- Research article
- Open Access
Validation of the kidney failure risk equation for end-stage kidney disease in Southeast Asia
BMC Nephrology volume 20, Article number: 451 (2019)
Patients with chronic kidney disease (CKD) are at high risk of end-stage kidney disease (ESKD). The Kidney Failure Risk Equation (KFRE), which predicts ESKD risk among patients with CKD, has not been validated in primary care clinics in Southeast Asia (SEA). Therefore, we aimed to (1) evaluate the performance of existing KFRE equations, (2) recalibrate KFRE for better predictive precision, and (3) identify optimally feasible KFRE thresholds for nephrologist referral and dialysis planning in SEA.
All patients with CKD visiting nine primary care clinics from 2010 to 2013 in Singapore were included and applied 4-variable KFRE equations incorporating age, sex, estimated glomerular filtration rate (eGFR), and albumin-to-creatinine ratio (ACR). ESKD onset within two and five years were acquired via linkage to the Singapore Renal Registry. A weighted Brier score (the squared difference between observed vs predicted ESKD risks), bias (the median difference between observed vs predicted ESKD risks) and precision (the interquartile range of the bias) were used to select the best-calibrated KFRE equation.
The recalibrated KFRE (named Recalibrated Pooled KFRE SEA) performed better than existing and other recalibrated KFRE equations in terms of having a smaller Brier score (square root: 2.8% vs. 4.0–9.3% at 5 years; 2.0% vs. 6.1–9.1% at 2 years), less bias (2.5% vs. 3.3–5.2% at 5 years; 1.8% vs. 3.2–3.6% at 2 years), and improved precision (0.5% vs. 1.7–5.2% at 5 years; 0.5% vs. 3.8–4.2% at 2 years). Area under ROC curve for the Recalibrated Pooled KFRE SEA equations were 0.94 (95% confidence interval [CI]: 0.93 to 0.95) at 5 years and 0.96 (95% CI: 0.95 to 0.97) at 2 years. The optimally feasible KFRE thresholds were > 10–16% for 5-year nephrologist referral and > 45% for 2-year dialysis planning. Using the Recalibrated Pooled KFRE SEA, an estimated 82 and 89% ESKD events were included among 10% of subjects at highest estimated risk of ESKD at 5-year and 2-year, respectively.
The Recalibrated Pooled KFRE SEA performs better than existing KFREs and warrants implementation in primary care settings in SEA.
According to the Global Burden of Disease Study 2015, total mortality for chronic kidney disease (CKD) rose by 31.7% from 2005 to 2015 worldwide . CKD stage 3 or worse (estimated glomerular filtration rate [eGFR] < 60 ml/min/1.73m2) is associated with increased risk of cardiovascular disease (CVD) and end-stage kidney disease (ESKD) that requires costly therapy including dialysis or kidney transplantation [2, 3].
Timely referral to nephrologists has shown to improve survival on dialysis  and reduce medical costs among patients who begin renal replacement therapy . Although a variety of factors may influence a decision for nephrologist referral, typical eGFR thresholds in clinical guidelines have varied from < 30, < 45 to < 60 mL/min/1.73 m2 [6,7,8]. Based on experts’ opinions, a systematic review suggested that referral at eGFR < 60 ml/min/1.73m2 is likely to be more cost-effective than at eGFR < 40 ml/min/1.73m2 . However, automated referrals for non-dialysis CKD have not been instituted in clinical practice and would likely overwhelm the health system as nephrologists are in short supply globally , with relative numbers of nephrologists ranging from 1 per million population in Southeast Asia to 31 per million in Western Europe . Heavy nephrologist caseload has been associated with mortality of dialysis patients , and less timely access to treatment for patients at higher risk of ESKD [12,13,14,15]. Therefore, accurate prediction scores to identify high-risk patients for ESKD are vital for efficient patient triage, decreasing waiting time and allocating limited resources to patients at highest risk. In 2011, a predictive model called the Kidney Failure Risk Equation (KFRE) incorporating four variables (age, sex, eGFR, urine albumin-to-creatinine ratio [ACR]) or eight variables (age, sex, eGFR, ACR, serum calcium, phosphate, bicarbonate, and albumin)  was developed with excellent predictive performance for ESKD risk in a Canadian population . Subsequently, the Original KFRE equation has been validated in more than 30 countries [17,18,19,20,21,22], recalibrated for non-North Americans using primarily European populations, and a Pooled KFRE equation has also been developed . However, these KFRE equations were developed and evaluated primarily in patients visiting the nephrology clinics [16,17,18]. CKD is largely asympotmatic, and the vast majority of patients (up to 90%), especially with earlier stages of CKD, are unaware of their conditions . Therefore, a well-performing KFRE would be highly relevant to the primary care settings to identify the fast progressors to ESKD. The Southeast Asian (SEA) population has been shown to have a heavy burden of ESKD  and may experience faster progression of CKD to ESKD compared to Caucasians [25,26,27]. However, the existing KFRE equations have not been evaluated in the SEA population.
Thus, we aimed to (1) compare performances of existing KFRE equations in a multi-ethnic population visiting primary care clinics in Singapore, (2) recalibrate KFRE to improve predictive precision for use in the SEA population, and (3) determine the optimally feasible KFRE thresholds to guide nephrologist referral and dialysis planning in SEA.
Singapore is a multi-ethnic country with major ethnic groups of Chinese, Malays and Indians. In 2017, there were 18 polyclinics (primary care clinics) located throughout Singapore, where about 60% Singaporeans with major risk factors for CKD (hypertension and diabetes) sought care . The 18 polyclinics were managed by two major healthcare groups (the SingHealth and the National Healthcare groups) before 2017, and the current study was derived from the electronic health records (EHR) at 9 SingHealth polyclinics at the time of study with the follow-up durations of two and five years.
For the 2-year follow-up, we included all 357,627 patients who visited the nine primary care clinics from January 1, 2010 to December 31, 2013 for eligibility screening. Eligibility criteria were 1) age ≥ 40 years, 2) not pregnant, 3) visited any primary care clinic at least twice with two visits at least 1 year apart, and 4) had ≥2 serum creatinine measurements taken at least 3 months apart to calculate eGFR by CKD-EPI equation  to screen for CKD. During the baseline screening window, if a patient met the eligibility criteria, he or she was immediately followed up on the date of recruitment. All other exposures including age, sex and urine albumin assessment were collected at the same time. During the follow-up, a patient may have died, or develop ESKD within the follow-up period, or remain ESKD-free at the end of the follow-up. Among the 150,344 eligible patients, 20,238 (14%) had persistent reductions in eGFR of < 60 mL/min/1.73m2. For the 5-year follow-up, we screened all 303,777 patients who visited the nine primary care clinics from January 1, 2010 to December 31, 2012, and 131,718 patients met the screening criteria. Among them, 19,857 (15%) had eGFR < 60 mL/min/1.73m2. Noteworthy, compared to the 5-year follow up, the 2-year follow-up had a shorter duration; thus, we extended the screening period to 3 years to capture more patients with ESKD. The flowcharts detailing the study design are shown in Fig. 1 and Fig. 2. The SingHealth Centralized Institutional Review Board granted ethics approval and consent waiver.
Incident ESKD cases and deaths were determined via linkage with the population-based Singapore Renal Registry and the Singapore Registry of Births and Deaths. Linkage was accomplished by matching the National Registration Identity Card number assigned to each citizen or permanent resident in Singapore and then verified by name. The Singapore Renal Registry identified ESKD as meeting one of the following criteria: 1) serum creatinine ≥880 μmol/L (10 mg/dL), 2) eGFR (based on either the Modification of Diet in Renal Disease Study equation , the Cockcroft Gault equation , or 24-h creatinine clearance) < 15 mL/min/1.73 m2, 3) undergoing peritoneal dialysis or haemodialysis, or 4) kidney transplantation has been undertaken . A diagnosis of ESKD required any of the criteria 1) - 3) above to be persistent for at least 3 months [6, 31].
Exposures assessment and KFRE validation
Information on age, gender, ethnicity, lifestyle factors, and co-morbidities was obtained from EHR data. Patients with hypertension or diabetes received blood and urine laboratory tests annually at primary care clinics. Creatinine measurements were calibrated to be traceable to isotope dilution mass spectrometry (IDMS) standardization . Urine albumin was measured as dipstick  in all patients and expressed as log-transformed ACR to be used in the KFRE (negative as 9, trace as 43, “+” as 81, “++” as 315, “>++” as 1073) [17, 34, 35].
We excluded patients without ACR (5-year, n = 2403; 2-year, n = 2516) and those developed ESKD before study baseline (5-year, n = 183; 2-year, n = 278), leaving 17,271 and 17,444 patients in the 5-year and 2-year cohorts for analysis. No missing data existed for other exposure variables (eGFR, age and sex). In calculating ESKD risks, we applied the existing 4-variable KFRE equations developed by Tangri et al.  (Additional file 1 and Additional file 2) based on the first eGFR and urine dipstick measurements. Three KFREs (Original KFRE, Original KFRE Calibrated for North American, Original KFRE Calibrated for non-north American) had the same regression coefficients but different baseline hazards, and the Pooled KFRE equation had both different regression coefficients and baseline hazards from the other three KFRE equations  (Additional file 1 and Additional file 2).
Recalibration of KFRE equations for SEA
We fit a Cox proportional hazards model using the same variables included in the Original KFRE (age, 70 years; 56% men, eGFR, 36 mL/min/1.73 m2, ACR 170 mg/g) [16, 17] and explored the baseline hazard and regression coefficients for the recalibration of KFRE equations for SEA population. We formed one recalibrated KFRE equation (Recalibrated Original KFRE SEA 1) by changing the baseline hazard in the Original KFRE, and formed one recalibrated KFRE (Recalibrated Original KFRE SEA 2) by changing both the baseline hazard and regression coefficients in the Original KFRE (Additional file 1 and Additional file 2). We also recalibrated the Pooled KFRE by changing the baseline hazard in a stepwise manner to look for an equation with the best calibration (Recalibrated Pooled KFRE SEA) (Additional file 1, Additional file 2 and Additional file 3).
Metrics for equation performance
The metrics used to compare the calibration (how closely the predicted risks agree with the observed risks) among different KFRE equations were the Brier score , bias , and precision . The Brier score was calculated as the squared difference of mean observed minus predicted risks  weighted according to the sample sizes in five risk categories (for 5 years, 0 to < 5%, 5 to < 15%, 15 to < 25%, 25 to < 50%, and ≥ 50%; for 2 years, 0 to < 2%, 2 to < 6%, 6 to < 10%, 10 to < 20%, and ≥ 20%) adopted from prior studies [16, 17]. Bias was expressed as the median difference between observed minus predicted risks  and precision was the interquartile range of bias . A KFRE equation with the lowest score of all three metrics would be chosen as the best-calibrated equation.
Discrimination and risk reclassification
We used area under receiver operating characteristic curve (AUC) of the best-calibrated KFRE equation as a continuous variable to that of continuous eGFR. We also used category-free net reclassification improvement (NRI) to compare the KFRE with eGFR: we assessed that how many more patients with ESKD were correctly assigned to higher predicted risks, as well as patients without ESKD to lower risks by using KFRE versus eGFR . We also applied AUC and NRI to compare between different KFRE equations.
Proportion of cases followed [PCF(p)] and proportion of the population needed to be follow [PNF(q)]
PCF(p) and PNF(q) were two recently developed measures that are highly relevant to the decision making in public health. PCF(p) represents the proportion of individuals who will develop disease who are included in the proportion p of individuals in the population of the highest risk, and PNF(q) is the proportion of the general population at highest risk that one needs to follow in order that a proportion q of those destined to become cases will be followed [39, 40]. For the equation with best calibration and predictive performance, we further calculated the PCF(p) and PNF(q) for the public health implication.
Explore the optimally feasible KFRE threshold in southeast Asians
For the best-calibrated KFRE equation, we applied the Youden Index to determine a statistically dichotomous risk threshold (‘low’ versus ‘high’) . The statistical threshold had the highest summation of the sensitivity and specificity. We compared the statistical KFRE thresholds identified in the current population to the 3, 5, and 10% at 5 years [17, 22] and 20 and 40% at 2 years , and also compared KFRE-based criteria to eGFR 30–60 ml/min/1.73m2 at 5 years and 20 ml/min/1.73m2 at 2 years suggested by Tangri et al. . We also compared KFRE thresholds with eGFRs that captured the same proportions of patients in this population to test the robustness of the results. The selection of the optimally feasible thresholds was based on sensitivity , specificity , positive predictive value (PPV) , negative predictive value (NPV) , positive likelihood ratio (LR+) , negative likelihood ratio (LR-) , and balanced by service-related considerations of (i) total number of patients with CKD referred to nephrologists, and (ii) number of patients with CKD needed to be evaluated by nephrologists to identify one patient who will progress to ESKD.
Stratified and sensitivity analyses
We conducted stratified analyses by age (40–75 vs. > 75 years), gender, ethnicity, type 2 diabetes mellitus status, and CKD stages (stage 3 and 4) for both 5-year and 2-year ESKD risks. To be consistent with Tangri et al., we also defined ESKD by limiting to those who started dialysis and received kidney transplantation only, and calculated the AUC of the best-calibrated KFRE equation. Since some patients died before the onset of ESKD of other causes, we evaluated the impact of competing risks of death on the KFRE risk prediction using the proportional hazards model proposed by Fine and Gray  compared to the non-competing risk model using the conventional Cox regression model. We used STATA software version 13.0 (Stata Corp, College Station, Texas) for all data analyses. Statistical significance was set at P ≤ 0.05.
Baseline characteristics of patients at 5- and 2-year ESKD risks are shown in Table 1 and Additional file 4. In both cohorts, the mean age was about 75 years, and approximately 50% were men, 80% were Chinese, 4% were Indians and 13% were Malays. At 5 years, 491 (2.8%) patients developed ESKD, among whom 147 (0.9%) with stage 3 CKD progressed to ESKD with a median time of 2.42 (range: 0.04–3.99) years, and 261 (1.5%) with stage 4 CKD developed ESKD in 1.78 (range: 0.09–3.93) years (Table 1). At 2 years, 330 (1.9%) developed ESKD; the median progression time from CKD to ESKD was 2.00 (range: 0.07–2.00) years among 48 (0.3%) patients with stage 3 CKD, and it was 1.13 (range: 0.01–2.00) years among 188 (1.1%) patients with stage 4 CKD (Additional file 4). In both cohorts, compared with non-ESKD patients, those with ESKD tended to be younger and have higher BMI, and were more likely to have type 2 diabetes mellitus, live in government housing and have a history of smoking (Table 1 and Additional file 4). For the 5-year cohort, compared to patients excluded for lack of established care or missing a second measurement of serum creatinine (n = 102,258), those included in the analysis (n = 131,718) were more likely to have hypertension (87.5% vs 32.5%, P < 0.001) or type 2 diabetes mellitus (44.6% vs 14.2%, P < 0.001).
Recalibrated KFRE equations
The recalibrated KFRE equations for 5- and 2-year risks are listed in Additional file 1 and Additional file 2. The baseline hazard from the Cox proportional hazard model for the current population was 0.9595 for 5-year and 0.9822 for 2-year risks (used for Recalibrated Original KFRE SEA 1). The regression coefficients for age, gender, and ACR are listed in Additional file 1 and Additional file 2, respectively (used for Recalibrated Original KFRE SEA 2). For the recalibrated Pooled KFRE equation, the best-calibrated baseline hazard was 0.8362 at 5-year and 0.8976 at 2-year risks (used for Recalibrated Pooled KFRE SEA) (Additional file 1, Additional file 2 and Additional file 3).
Comparison of performance
The Recalibrated Pooled KFRE SEA had the best calibration among all KFREs in terms of having the smallest Brier score (squared difference of mean observed minus predicted risks ; square root: 2.8% vs. 4.0–9.3% at 5 years; 2.0% vs. 6.1–9.1% at 2 years), the least bias (median difference between observed minus predicted risks ; 2.5% [− 2.0–4.5%] vs. 3.3–5.2% at 5 years; 1.8% [− 1.7–3.5%] vs. 3.2–3.6% at 2 years), and the best precision (interquartile range of bias ; 0.5% vs.1.7–5.2% at 5 years; 0.5% vs. 3.5–4.2% at 2 years) (Table 2). Compared to the observed risks, the Recalibrated Pooled KFRE SEA slightly underestimated the predicted ESKD risks at lower KFRE risk categories (< 25% at 5 years; < 20% at 2 years), and slightly overestimated the predicted risks at higher risk categories (≥25% at 5 years; ≥20% at 2 years) (Additional file 5).
Discrimination, thresholds and NRI
The AUCs and 95% CIs for the Recalibrated Pooled KFRE SEA were 0.94 (0.93 to 0.95) at 5-year ESKD risks and 0.96 (0.95 to 0.97) at 2-year, which were statistically significantly higher than eGFR alone (0.89 [0.88 to 0.91] at 5 years; 0.93 [0.92 to 0.95] at 2 years) (Tables 3,4 and 5 & Additional file 6). In addition, the AUCs for other KFRE equations were the same with the Recalibrated Pooled KFRE SEA (Table 3). Consistent with the AUC results, the NRIs comparing Recalibrated Pooled KFRE SEA to other KFRE equations showed similar performances (NRI ranging from − 0.23-1.72%) (Table 3).
At 5 years, we used eGFR 40 mL/min/1.73m2 (sensitivity 0.88, specificity 0.71) as reference to compare with thresholds of Recalibrated Pooled KFRE SEA because the eGFR 30 mL/min/1.73m2 sensitivity was suboptimal (0.70 [0.68–0.72]), and eGFR 45 and 60 mL/min/1.73m2 specificities were low (0.56 [0.55–0.57] and 0.01 [0.00–0.02]). Using eGFR < 40 mL/min/1.73m2 would identify 5283 (30%) patients requiring referral to a nephrologist, and this number was substantially higher than the 3506 (20%) of KFRE > 10%, and 2308 (13%) of the Youden Index-determined KFRE > 16%. Moreover, using KFRE thresholds ranging 10–16%, nephrologists need to evaluate 5.6 (5.0–6.0) to 7.8 (7.2–8.5) patients with CKD to find one ESKD case, and this resulted in higher referral efficiency than the 12.2 (11.1–13.3) patients using eGFR 40 mL/min/1.73m2. In addition to the higher referral efficiency, KFRE thresholds 10–16% also had similar sensitivity (0.86 [0.85–0.87] to 0.91 [0.90–0.92] vs. 0.88 [0.87–0.89]), higher specificity (0.82 [0.81–0.83] to 0.89 [0.88–0.90] vs. 0.71 [0.70–0.72]), higher PPV (0.10 [0.09–0.11] to 0.16 [0.14–0.18] vs. 0.08 [0.07–0.09]), similar NPV (0.995 [0.980–1.010] to 0.997 [0.990–1.004] vs. 0.995 [0.986–1.004]), and higher LR+ (5.01 [5.00–5.26] to 7.67 [7.14–8.33] vs. 3.03 [2.94–3.13]) compared to eGFR 40 mL/min/1.73m2 (Table 4).
In the 2-year cohort, using threshold of Recalibrated Pooled KFRE SEA > 45% had higher clinical efficiency than eGFR < 20 mL/min/1.73m2 in terms of fewer referral number to nephrologist (430 vs. 546), and similar number of patients to be evaluated by nephrologists to find one ESKD case (2.7 [2.4–3.1] vs. 3.2 [2.8–3.6]). In addition, KFRE threshold 45% also had similar sensitivity (0.54 [0.49–0.59] vs. 0.52 [0.48–0.56]), similar specificty (0.99 [0.98–1.00] vs. 0.98 [0.97–0.99]), similar NPV (0.992 [0.982–1.002] vs. 0.991 [0.989–1.001]), similar LR+ (36.8 [25.0–50.0] vs. 26.0 [20.0–33.3]), and higher PPV (0.42 [0.37–0.47] vs. 0.31 [0.27–0.35]) compared to those of eGFR 20 mL/min/1.73m2 (Table 5), suggesting a marginal superiority.
The Recalibrated Pooled KFRE SEA resulted in statistically significant improvement in NRI over eGFR alone in predicting ESKD. At 5 years, NRIs were ≥ 12.6% (10.6–15.3%) for KFRE 10–16% compared to eGFR 30, 40 and 45 mL/min/1.73m2 (Additional file 7). At 2 years, the NRI was 3.14% (2.86, 3.43%) for KFRE 45% compared to eGFR 20 mL/min/1.73m2 (Additional file 8).
In addition to the traditional eGFR cut-off values, we also compared the abovementioned KFRE thresholds to the eGFR cut-offs that captured the same proportions of patients in this population. Thus, at 5 years, the KFRE > 10% corresponded to eGFR < 35.4 mL/min/1.73m2, and the KFRE > 16% corresponded to eGFR < 30 mL/min/1.73m2 approximately (Table 4); at 2 years, the KFRE> 45% corresponded to eGFR< 18.8 mL/min/1.73m2 (Table 5). As a result, at 5 years, KFRE 10 and 16% had higher sensitivity and lower negative likelihood ratios compared to respective eGFR cut-off values, while other statistics (specificity, PPV, NPV, positive likelihood ratio and referral efficiency) remained the same (Table 4). At 2 years, KFRE > 45% had similar sensitivity, specificity, PPV, NPV, positive likelihood ratio, negative likelihood ratio, and referral efficiency compared to eGFR < 18.8 mL/min/1.73m2 (Table 5). However, all of the KFRE thresholds resulted in a positive NRI compared to the corresponding eGFR cut-off points (≥7.06% [6.77–7.34%]) (Additional file 7 and Additional file 8), indicating the robustness of the superiority of KFRE in clinical utility.
Using the Recalibrated Pooled KFRE SEA at 5 years, an estimated 82% ESKD events were included among 10% of subjects at highest estimated risk of ESKD (Fig. 3), and an estimated 92 and 96% cases were included among 20 and 30% of subjects at highest ESKD risks (Fig. 3). At 2 years, an estimated 89, 94 and 96% events were captured in 10, 20 and 30% of subjects at the highest estimated risk of ESKD (Fig. 3).
We also presented detailed statistics of a wide range of KFRE thresholds (3–21% at 5 year; 5–45% at 2 years) and observed that although using a higher KFRE threshold would refer fewer patients to a nephrologist to find one ESKD case, the sensitivity associated with KFRE also became less optimal (Fig. 4).
Stratified and sensitivity analyses
KFRE discrimination remained excellent across all subgroups defined by 1) age (40–75 vs. > 75 years), 2) gender, 3) ethnicity (Chinese, Malays, and Indians), 4) type 2 diabetes mellitus status, and 5) CKD stages at both 5- and 2-year risks. At 5 years, the AUC ranged from 0.82 to 0.96 in the subgroups and at 2 years, the AUC ranged from 0.83 to 0.98 in the subgroups (Additional file 9). Of note, at 5 years, the 95% CI of Malays and Chinese were comparable; and that of Indians, maybe due to smaller sample sizes, was much wider than Chinese and Malays (Additional file 9).
A total of 408 and 236 ESKD events started dialysis and received kidney transplantation at 5-year and 2-year, respectively. Limiting to these events as outcomes, the discrimination of the Recalibrated Pooled KFRE SEA was largely the same as the main analyses: the AUC was 0.93 (95% CI: 0.92–0.94) at 5-year and 0.95 (95% CI: 0.93–0.96) at 2-year risk of ESKD onset.
A total of 3241 and 1927 deaths occurred during the 5-year and 2-year follow-ups, respectively. The competing risk and non-competing risk models at 5 years had similar HRs and considerate overlaping 95% CIs (242 [60.4–972] vs. 227 [56.5–909]), suggesting the effect of competing mortality risks is unlikely to affect the ESKD prediction.
Statement of principal findings
Using electronic health records linked with national renal registry, we found that the recalibrated KFRE (Recalibrated Pooled KFRE SEA equation) had better performance than existing KFRE equations in terms of having a lower Brier score, less bias and improved precision for predicting ESKD in multi-ethnic patients visiting the primary care clinics. The overall predictive capability of the Recalibrated Pooled KFRE SEA for ESKD was significantly higher than using eGFR alone. In addition, 5-year KFRE thresholds ranging 10–16% for nephrologist referral and 2-year KFRE risk threshold at 45% for dialysis planning resulted in high referral efficiency, and substantially improved reclassification of ESKD risks relative to eGFR thresholds of 20, 30, 40 and 45 mL/min/1.73m2. Thus, automated referrals using KFRE thresholds warrant consideration in clinical practice for patients with CKD.
In relation to previous studies
Previous studies showed excellent predictive utility of the Original KFRE equation or the Pooled KFRE equation primarily among patients with European origins or those in nephrology clinics [16,17,18,19,20,21,22]. Our study expands on those findings in the primary care clinics in SEA. We observed high AUCs of the Recalibrated Pooled KFRE SEA at both 5-year (0.94; 95% CI: 0.93–0.95) and 2-year (0.96; 95% CI: 0.95–0.97) risks of ESKD. Moreover, evidence for KFRE thresholds based on empirical data is limited and previous suggestions on KFRE thresholds have been based on physicians’ opinions . Our study fills that gap by using comprehensive statistical metrics coupled with clinical consideration of nephrologist workload. Of note, our results suggested that the Recalibrated Pooled KFRE SEA thresholds ranging 10–16% for nephrology referral criterion over 5 years had high sensitivity, high specificity and high referral efficiency, and substantially improved reclassification of ESKD risk on top of eGFR thresholds of 30, 40 and 45 mL/min/1.73m2. In addition, the 2-year threshold of Recalibrated Pooled KFRE SEA for dialysis planning in the current study (45%) was close to the KFRE > 40% suggested previously  and was marginally better than eGFR 20 mL/min/1.73m2 . Moreover, as the availabilities of the healthcare resources and the balance among sensitivity, specificity and referral efficiency may vary from country to country, a universal optimal KFRE threshold may not be possible. Our study provides a wide range of Recalibrated Pooled KFRE SEA thresholds with useful statistics (sensitivity and referral efficiency) for clinicians and health planners to choose from based on local resources, which greatly enhanced the clinical application to the primary care settings with different availability of nephrology resources globally.
Meaning of the study
The Recalibrated Pooled KFRE SEA equation includes four routinely measured variables, which were available in > 86% of our study population with stage 3–5 CKD (although ACR was converted from urine albumin for all patients). Thus, our findings imply that the Recalibrated Pooled KFRE SEA equation is likely to aid referral decisions and dialysis planning across all general practitioner settings if integrated into EHR. Furthermore, the improved triage efficiency would enable patients at high-risk of ESKD to receive timely referrals to a nephrologist, which has been shown to shorten waiting time for nephrology care  and substantially reduce medical costs for initiating renal replacement therapy and dialysis compared to late referral . Since the nephrologist shortage is global , the implications of shortening patient wait time and reducing costs would have significant impact on health systems and patient well-being in resource-limited settings where CKD burden is rising, and accessibility to renal replacement therapy is limited at a global level .
Strengths and limitations
Our study has several strengths. First, this is likely to be the first report to determine the best-calibrated KFRE equation and potentially useful thresholds for nephrologist referral and dialysis planning in primary care population in SEA. When looking for the clinically useful thresholds, we applied rigorous statistical criteria, and combined service-related considerations for health planners, and the methodology provided as a yardstick for future studies. However, whehter the suggested thresholds are optimal for Singapore will need to be further tested taking into consideration of the workforce and work capacticy of Singapore nephrologists, and simulation studies are warranted to predict the performance of such thresholds over time. Second, we included all eligible patients visiting primary care clinics over the study duration and thus had a large sample size. Third, the multiple major ethnic groups in our sample (Chinese, Indians, and Malays) are a diaspora of populations from countries (China, India and Malaysia) that are homes to one-third of the world’s population. The excellent predictive utility of the Recalibrated Pooled KFRE SEA in all three ethnic groups shown in stratified analyses greatly enhanced the utilization of KFRE to many people globally. Fourth, serum creatinine measurements were calibrated to be traceable to an IDMS standard, thus increasing the validity of both eGFR and KFRE assessments. Fifth, we objectively assessed all ESKD cases with virtual follow-up completion via linkage to the nationwide Singapore Renal Registry.
However, our study also had some limitations. First, we deleted those without established care from the study (~ 30%), thus introducing the possibility of selection bias. Nevertheless, the prevalence of hypertension and diabetes was lower in the excluded population than that expected in the age-matched general population in Singapore [28, 48]. Second, the definition of ESKD in the current study was slightly different from that of Tangri et al. ; however, we conducted sensitivity analysis using the definition of ESKD from that of Tangri and found similar results. Third, the ACR value used in the KFRE score was converted from urine dipstick that was measured in all patients, and thus may be less precise compared to direct measurements. Specifically, the conversion between urine albumin to ACR for people with “+”, “++” and “+++” were based on limited data [17, 34, 35]. However, previous studies using the same conversion were included in the meta-analysis of KFRE validation and showed similar results of ESKD prediction , and thus suggested that the dipstick-converted ACR value is unlikely to have a large impact on the predictive performance of KFRE. Fourth, the current study did not have a validation dataset to examine the superior performance of Recalibrated KFRE SEA over other KFREs, and future studies among SEA populations are warranted to validate our results. Fifth, the sample size of Indian patients were small in the current population, and the 95% CI of AUC was wider compared to Chinese and Malays; thus, our results may not be generalizable to Indians. Future studies with bigger sample sizes of Indians are warranted to validate our results. Thus, the optimal threshold may be different in other countries. In addition, the current study did not have data on serum calcium, phosphate, bicarbonate, and albumin to validate the 8-variable KFRE equation. However, the 4- and 8-variable equations showed similar discrimination in the original development cohort  and subsequent meta-analysis . Therefore, the less complicated 4-variable KFRE may be a more convenient tool for clinical usage.
In conclusion, our results showed that the Recalibrated Pooled KFRE SEA equation is an excellent predictive tool and performed better in terms of having a lower Brier score, less bias and improved precision than existing KFRE for identifying patients with CKD at risk for progression to ESKD in a primary care setting in SEA. Our findings suggest that implementation of the equation using 5-year thresholds > 10–16% to guide dialysis planning and 2-year threshold > 45% to guide nephrologist referral would facilitate more efficient and accurate risk stratification of patients at high risk of ESKD. Future studies are warranted to validate our findings, evaluate the clinical and cost effectiveness of a CKD model of care that integrates EHR and the KFRE in primary care settings serving Asians as well as globally.
Availability of data and materials
Data are available on reasonable request from the corresponding author subject to approval by the IRB.
Area under receiver operating characteristic curve
Chronic kidney disease
Estimated glomerular filtration rate
Electronic health records
End-stage kidney disease
Isotope dilution mass spectrometry
Kidney Failure Risk Equation
Negative likelihood ratio
- LR + :
Positive likelihood ratio.
Negative predictive value
Net reclassification improvement
Positive predictive value
GBD 2015 Mortality and Causes of Death Collaborators. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980–2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet. 2016;388:1459–544.
Coresh J, Turin TC, Matsushita K, et al. Decline in estimated glomerular filtration rate and subsequent risk of end-stage renal disease and mortality. JAMA. 2014;311:2518–31.
Inker LA, Lambers Heerspink HJ, Mondal H, et al. GFR decline as an alternative end point to kidney failure in clinical trials: a meta-analysis of treatment effects from 37 randomized trials. Am J Kidney Dis. 2014;64:848–59.
Kim DH, Kim M, Kim H, et al. Early referral to a nephrologist improved patient survival: prospective cohort study for end-stage renal disease in Korea. PLoS One. 2013;8:e55323.
Lee J, Lee JP, Park JI, et al. Early nephrology referral reduces the economic costs among patients who start renal replacement therapy: a prospective cohort study in Korea. PLoS One. 2014;9:e99460.
Kidney disease: Improving global outcomes (KDIGO) CKD work group. KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2013;3:1–150.
National Collaborating Centre for Chronic Conditions. National Institute for Health and Clinical Excellence: Guidance. In: Chronic Kidney Disease: National Clinical Guideline for Early Identification and Management in Adults in Primary and Secondary Care. London: Royal College of Physicians (UK). Royal College of Physicians of London; 2008.
Section I. Measurement of renal function, when to refer and when to start dialysis. Nephrol Dial Transplant. 2002;17(Suppl 7):7–15.
Black C, Sharma P, Scotland G, et al. Early referral strategies for management of people with markers of renal disease: a systematic review of the evidence of clinical effectiveness, cost-effectiveness and economic analysis. Health Technol Assess. 2010;14:1–184.
Sharif MU, Elsayed ME, Stack AG. The global nephrology workforce: emerging threats and potential solutions! Clin Kidney J. 2016;9:11–22.
Harley KT, Streja E, Rhee CM, Molnar MZ, Kovesdy CP, Amin AN, Kalantar-Zadeh K. Nephrologist caseload and hemodialysis patient survival in an urban cohort. J Am Soc Nephrol. 2013;24:1678–87.
Naimark DM, Harel Z, Moineddin R, Bergman A. The impact of estimated glomerular filtration rate reporting on nephrology referral pattern, patient characteristics and outcome. Nephron Clin Pract. 2012;121:c10–5.
Noble E, Johnson DW, Gray N, et al. The impact of automated eGFR reporting and education on nephrology service referrals. Nephrol Dial Transplant. 2008;23:3845–50.
Hemmelgarn BR, Zhang J, Manns BJ, et al. Nephrology visits and health care resource use before and after reporting estimated glomerular filtration rate. JAMA. 2010;303:1151–8.
Jain A, Hemmelgarn BR. Impact of estimated glomerular filtration rate reporting on nephrology referrals: a review of the literature. Curr Opin Nephrol Hypertens. 2011;20:218–23.
Tangri N, Stevens LA, Griffith J, et al. A predictive model for progression of chronic kidney disease to kidney failure. JAMA. 2011;305:1553–9.
Tangri N, Grams ME, Levey AS, et al. Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis. JAMA. 2016;315:164–74.
Peeters MJ, van Zuilen AD, van den Brand JA, Bots ML, Blankestijn PJ, Wetzels JF. Validation of the kidney failure risk equation in European CKD patients. Nephrol Dial Transplant. 2013;28:1773–9.
Elley CR, Robinson T, Moyes SA, et al. Derivation and validation of a renal risk score for people with type 2 diabetes. Diabetes Care. 2013;36:3113–20.
Marks A, Fluck N, Prescott GJ, Robertson L, Simpson WG, Cairns Smith W, Black C. Looking to the future: predicting renal replacement outcomes in a large community cohort with chronic kidney disease. Nephrol Dial Transplant. 2015;30:1507–17.
Lennartz CS, Pickering JW, Seiler-Mussler S, et al. External validation of the kidney failure risk equation and re-calibration with addition of ultrasound parameters. Clin J Am Soc Nephrol. 2016;11:609–15.
Whitlock RH, Chartier M, Komenda P, et al. Validation of the kidney failure risk equation in Manitoba. Can J Kidney Health Dis. 2017;4:2054358117705372.
Ene-Iordache B, Perico N, Bikbov B, et al. Chronic kidney disease and cardiovascular risk in six regions of the world (ISN-KDDC): a cross-sectional study. Lancet Glob Health. 2016;4:e307–19.
United States Renal Data System 2018 USRDS Annual Data Report. Epidemiology of Kidney disease in the United States. Bethesda: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2018.
Barbour SJ, Cattran DC, Kim SJ, Levin A, Wald R, Hladunewich MA, Reich HN. Individuals of Pacific Asian origin with IgA nephropathy have an increased risk of progression to end-stage renal disease. Kidney Int. 2013;84:1017–24.
Barbour SJ, Er L, Djurdjev O, Karim M, Levin A. Differences in progression of CKD and mortality amongst Caucasian, oriental Asian and south Asian CKD patients. Nephrol Dial Transplant. 2010;25:3663–72.
Misra A, Tandon N, Ebrahim S, et al. Diabetes, cardiovascular disease, and chronic kidney disease in South Asia: current status and future directions. BMJ. 2017;357:j1420.
Ministry of Health Singapore. National Health Survey 2010. 2010.
Levey AS, Stevens LA. Estimating GFR using the CKD epidemiology collaboration (CKD-EPI) creatinine equation: more accurate GFR estimates, lower CKD prevalence estimates, and better risk predictions. Am J Kidney Dis. 2010;55:622–7.
Levey AS, Coresh J, Greene T, et al. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med. 2006;145:247–54.
National Registry of Diseases Office. Singapore Renal Registry Annual Registry Report 1999-2004. p. 2015.
National Institute of Diabetes and Digestive and Kidney Diseases. Creatinine Standardization.
Lew QLJ, Allen JC, Nguyen F, Tan NC, Jafar TH. Factors associated with chronic kidney disease and their clinical utility in primary care clinics in a multi-ethnic southeast Asian population. Nephron. 2018;138:202–13.
Konta T, Hao Z, Takasaki S, et al. Clinical utility of trace proteinuria for microalbuminuria screening in the general population. Clin Exp Nephrol. 2007;11:51–5.
Lamb EJ, MacKenzie F, Stevens PE. How should proteinuria be detected and measured? Ann Clin Biochem. 2009;46:205–17.
BRIER GW. Verification Of Forecasts Expressed In Terms Of Probability. Mon Weather Rev. 1950;78:1–3.
Jessani S, Levey AS, Bux R, et al. Estimation of GFR in south Asians: a study from the general population in Pakistan. Am J Kidney Dis. 2014;63:49–58.
Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157–72.
Pfeiffer RM, Gail MH. Two criteria for evaluating risk prediction models. Biometrics. 2011;67:1057–65.
Pfeiffer RM. Extensions of criteria for evaluating risk prediction models for public health applications. Biostatistics. 2013;14:366–81.
Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–5.
Tangri N, Ferguson T, Komenda P. Pro: risk scores for chronic kidney disease progression are robust, powerful and ready for implementation. Nephrol Dial Transplant. 2017;32:748–51.
Trevethan R. Sensitivity, specificity, and predictive values: foundations, Pliabilities, and pitfalls in research and practice. Front Public Health. 2017;5:307.
Hayden SR, Brown MD. Likelihood ratio: a powerful tool for incorporating the results of a diagnostic test into clinical Decisionmaking. Ann Emerg Med. 1999;33:575–80.
Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc. 1999;94:496–509.
Hingwala J, Wojciechowski P, Hiebert B, Bueti J, Rigatto C, Komenda P, Tangri N. Risk-based triage for nephrology referrals using the kidney failure risk equation. Can J Kidney Health Dis. 2017;4:2054358117722782.
Coresh J, Jafar TH. Disparities in worldwide treatment of kidney failure. Lancet. 2015;385:1926–8.
Seow LSE, Subramaniam M, Abdin E, Vaingankar JA, Chong SA. Hypertension and its associated risks among Singapore elderly residential population. J Clin Gerontol Geriatr. 2015;6:125–32.
The authors would like to thank Ms. Mingshi Cai and Ms. Sock Teng Chin from the Singapore Chronic Kidney Failure Registry for their assistance, as well as Ms. Patricia T. Kin and Ms. Caris Yang Thong Tan from the Department of Research in SingHealth Polyclinics.
The study obtained in-kind support from a SingHealth Analytics and Research Technologies grant. The research was also supported by research funding received from SingHealth Duke-NUS Health Services Research Institute (HSRI). THJ is supported by the National Medical Research Council, Singapore. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing manuscript.
Ethics approval and consent to participate
The SingHealth Centralized Institutional Review Board granted ethics approval and consent waiver.
Consent for publication
The corresponding author Professor Tazeen H. Jafar is the Editorial Board Member of BMC Nephrology. No other competing interests were declared.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Figure S1. Existing and recalibrated Kidney Failure Risk Equation (KFRE) for predicting 5-year risk of end-stage kidney disease among patients with chronic kidney disease stage 3–5. The figure shows the existing and recalibrated KFREs for predicting 5-year risk of end-stage kidney disease among patients with chronic kidney disease stage 3–5.
Figure S2. Existing and recalibrated Kidney Failure Risk Equation (KFRE) for predicting 2-year risk of end-stage kidney disease among patients with chronic kidney disease stage 3–5. The figure shows the existing and recalibrated KFREs for predicting 2-year risk of end-stage kidney disease among patients with chronic kidney disease stage 3–5.
Calibration (Brier score, bias and precision) plots of Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) with different constants for 5-year and 2-year risks of end-stage kidney disease. The figure shows the Brier score, bias and precision associated with different Recalibrated Pooled KFRE SEA constants at A) 5-year and B) 2-year risks of end-stage kidney disease to evaluate how closely the predicted risks agree with the observed risks.
Baseline characteristics of patients with chronic kidney disease stage 3–5 with 2-year follow-up. The table shows baseline characteristics of patients with chronic kidney disease stage 3–5 included in the cohort with 2-year follow-up.
Observed risk versus predicted probability of end-stage kidney disease using the Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) at five and 2 years. The predicted and observed end-stage kidney disease probability estimates represent the mean values of predicted risk and observed probabilities in the risk categories according to the Recalibrated Pooled KFRE SEA risks at A) 5-year and B) 2-year risks of end-stage kidney disease.
Area under receiver operating characteristic curves of the Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) for predicting the 5- and 2-year risks of onset of end-stage kidney disease. The figure shows the Recalibrated Pooled KFRE SEA equations and eGFR receiver operating characteristic curves for predicting the A) 5-year and B) 2-year risks of end-stage kidney disease among patients with chronic kidney disease.
Reclassification of 5-year risk of end-stage kidney disease onset among chronic kidney disease patients using the Recalibrated Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) equation thresholds at 10 and 16% compared to estimated glomerular filtration rate 30, 40 and 45 mL/min/1.73m2. The table shows the net reclassification improvement of the Recalibrated Pooled KFRE SEA equation threshold at 10 and 16% compared to estimated glomerular filtration rate 30, 40 and 45 mL/min/1.73m2 for predicting the 5-year risk of end-stage kidney disease among patients with chronic kidney disease.
Reclassification of 2-year risk of end-stage kidney disease onset among chronic kidney disease patients using the Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) equation threshold at 45% compared to estimated glomerular filtration rate 20 and 18.8 mL/min/1.73m2. The table shows the net reclassification improvement of the Recalibrated Pooled KFRE SEA equation threshold at 45% compared to estimated glomerular filtration rate 20 and 18.8 mL/min/1.73m2 for predicting the 2-year risk of end-stage kidney disease among patients with chronic kidney disease.
Stratified analyses of area under receiver operating characteristic curves of the Pooled Kidney Failure Risk Equation Southeast Asia (KFRE SEA) for predicting the 5-year and 2-year risks of end-stage kidney disease. The figure shows stratified analyses of area under receiver operating characteristic curves and 95% confidence interval of the Recalibrated Pooled KFRE SEA equations for predicting the A) 5-year and B) 2-year risk of onset of end-stage kidney disease.
About this article
Cite this article
Wang, Y., Nguyen, F.N.H.L., Allen, J.C. et al. Validation of the kidney failure risk equation for end-stage kidney disease in Southeast Asia. BMC Nephrol 20, 451 (2019). https://doi.org/10.1186/s12882-019-1643-0
- Chronic kidney disease
- End-stage kidney disease
- Kidney failure risk equation
- Southeast Asia