Validation of the kidney failure risk equation for end-stage kidney disease in Southeast Asia

Background Patients with chronic kidney disease (CKD) are at high risk of end-stage kidney disease (ESKD). The Kidney Failure Risk Equation (KFRE), which predicts ESKD risk among patients with CKD, has not been validated in primary care clinics in Southeast Asia (SEA). Therefore, we aimed to (1) evaluate the performance of existing KFRE equations, (2) recalibrate KFRE for better predictive precision, and (3) identify optimally feasible KFRE thresholds for nephrologist referral and dialysis planning in SEA. Methods All patients with CKD visiting nine primary care clinics from 2010 to 2013 in Singapore were included and applied 4-variable KFRE equations incorporating age, sex, estimated glomerular filtration rate (eGFR), and albumin-to-creatinine ratio (ACR). ESKD onset within two and five years were acquired via linkage to the Singapore Renal Registry. A weighted Brier score (the squared difference between observed vs predicted ESKD risks), bias (the median difference between observed vs predicted ESKD risks) and precision (the interquartile range of the bias) were used to select the best-calibrated KFRE equation. Results The recalibrated KFRE (named Recalibrated Pooled KFRE SEA) performed better than existing and other recalibrated KFRE equations in terms of having a smaller Brier score (square root: 2.8% vs. 4.0–9.3% at 5 years; 2.0% vs. 6.1–9.1% at 2 years), less bias (2.5% vs. 3.3–5.2% at 5 years; 1.8% vs. 3.2–3.6% at 2 years), and improved precision (0.5% vs. 1.7–5.2% at 5 years; 0.5% vs. 3.8–4.2% at 2 years). Area under ROC curve for the Recalibrated Pooled KFRE SEA equations were 0.94 (95% confidence interval [CI]: 0.93 to 0.95) at 5 years and 0.96 (95% CI: 0.95 to 0.97) at 2 years. The optimally feasible KFRE thresholds were > 10–16% for 5-year nephrologist referral and > 45% for 2-year dialysis planning. Using the Recalibrated Pooled KFRE SEA, an estimated 82 and 89% ESKD events were included among 10% of subjects at highest estimated risk of ESKD at 5-year and 2-year, respectively. Conclusions The Recalibrated Pooled KFRE SEA performs better than existing KFREs and warrants implementation in primary care settings in SEA.


Background
According to the Global Burden of Disease Study 2015, total mortality for chronic kidney disease (CKD) rose by 31.7% from 2005 to 2015 worldwide [1]. CKD stage 3 or worse (estimated glomerular filtration rate [eGFR] < 60 ml/min/1.73m 2 ) is associated with increased risk of cardiovascular disease (CVD) and end-stage kidney disease (ESKD) that requires costly therapy including dialysis or kidney transplantation [2,3].
Timely referral to nephrologists has shown to improve survival on dialysis [4] and reduce medical costs among patients who begin renal replacement therapy [5]. Although a variety of factors may influence a decision for nephrologist referral, typical eGFR thresholds in clinical guidelines have varied from < 30, < 45 to < 60 mL/min/ 1.73 m 2 [6][7][8]. Based on experts' opinions, a systematic review suggested that referral at eGFR < 60 ml/min/ 1.73m 2 is likely to be more cost-effective than at eGFR < 40 ml/min/1.73m 2 [9]. However, automated referrals for non-dialysis CKD have not been instituted in clinical practice and would likely overwhelm the health system as nephrologists are in short supply globally [10], with relative numbers of nephrologists ranging from 1 per million population in Southeast Asia to 31 per million in Western Europe [10]. Heavy nephrologist caseload has been associated with mortality of dialysis patients [11], and less timely access to treatment for patients at higher risk of ESKD [12][13][14][15]. Therefore, accurate prediction scores to identify high-risk patients for ESKD are vital for efficient patient triage, decreasing waiting time and allocating limited resources to patients at highest risk. In 2011, a predictive model called the Kidney Failure Risk Equation (KFRE) incorporating four variables (age, sex, eGFR, urine albumin-to-creatinine ratio [ACR]) or eight variables (age, sex, eGFR, ACR, serum calcium, phosphate, bicarbonate, and albumin) [16] was developed with excellent predictive performance for ESKD risk in a Canadian population [16]. Subsequently, the Original KFRE equation has been validated in more than 30 countries [17][18][19][20][21][22], recalibrated for non-North Americans using primarily European populations, and a Pooled KFRE equation has also been developed [17]. However, these KFRE equations were developed and evaluated primarily in patients visiting the nephrology clinics [16][17][18]. CKD is largely asympotmatic, and the vast majority of patients (up to 90%), especially with earlier stages of CKD, are unaware of their conditions [23]. Therefore, a well-performing KFRE would be highly relevant to the primary care settings to identify the fast progressors to ESKD. The Southeast Asian (SEA) population has been shown to have a heavy burden of ESKD [24] and may experience faster progression of CKD to ESKD compared to Caucasians [25][26][27]. However, the existing KFRE equations have not been evaluated in the SEA population.
Thus, we aimed to (1) compare performances of existing KFRE equations in a multi-ethnic population visiting primary care clinics in Singapore, (2) recalibrate KFRE to improve predictive precision for use in the SEA population, and (3) determine the optimally feasible KFRE thresholds to guide nephrologist referral and dialysis planning in SEA.

Study population
Singapore is a multi-ethnic country with major ethnic groups of Chinese, Malays and Indians. In 2017, there were 18 polyclinics (primary care clinics) located throughout Singapore, where about 60% Singaporeans with major risk factors for CKD (hypertension and diabetes) sought care [28]. The 18 polyclinics were managed by two major healthcare groups (the SingHealth and the National Healthcare groups) before 2017, and the current study was derived from the electronic health records (EHR) at 9 SingHealth polyclinics at the time of study with the follow-up durations of two and five years.
For the 2-year follow-up, we included all 357,627 patients who visited the nine primary care clinics from January 1, 2010 to December 31, 2013 for eligibility screening. Eligibility criteria were 1) age ≥ 40 years, 2) not pregnant, 3) visited any primary care clinic at least twice with two visits at least 1 year apart, and 4) had ≥2 serum creatinine measurements taken at least 3 months apart to calculate eGFR by CKD-EPI equation [29] to screen for CKD. During the baseline screening window, if a patient met the eligibility criteria, he or she was immediately followed up on the date of recruitment. All other exposures including age, sex and urine albumin assessment were collected at the same time. During the follow-up, a patient may have died, or develop ESKD within the follow-up period, or remain ESKD-free at the end of the follow-up. Among the 150,344 eligible patients, 20,238 (14%) had persistent reductions in eGFR of < 60 mL/min/1.73m 2 . For the 5-year follow-up, we screened all 303,777 patients who visited the nine primary care clinics from January 1, 2010 to December 31, 2012, and 131,718 patients met the screening criteria. Among them, 19,857 (15%) had eGFR < 60 mL/min/ 1.73m 2 . Noteworthy, compared to the 5-year follow up, the 2-year follow-up had a shorter duration; thus, we extended the screening period to 3 years to capture more patients with ESKD. The flowcharts detailing the study design are shown in Fig. 1 and Fig. 2. The SingHealth Centralized Institutional Review Board granted ethics approval and consent waiver.

Outcomes assessment
Incident ESKD cases and deaths were determined via linkage with the population-based Singapore Renal Registry and the Singapore Registry of Births and Deaths. Linkage was accomplished by matching the National Registration Identity Card number assigned to each citizen or permanent resident in Singapore and then verified by name. The Singapore Renal Registry identified ESKD as meeting one of the following criteria: 1) serum creatinine ≥880 μmol/L (10 mg/dL), 2) eGFR (based on either the Modification of Diet in Renal Disease Study equation [30], the Cockcroft Gault equation [30], or 24-h creatinine clearance) < 15 mL/min/1.73 m 2 , 3) undergoing peritoneal dialysis or haemodialysis, or 4) kidney transplantation has been undertaken [31]. A diagnosis of ESKD required any of the criteria 1) -3) above to be persistent for at least 3 months [6,31].

Exposures assessment and KFRE validation
Information on age, gender, ethnicity, lifestyle factors, and co-morbidities was obtained from EHR data. Patients with hypertension or diabetes received blood and urine laboratory tests annually at primary care clinics. Creatinine measurements were calibrated to be traceable to isotope dilution mass spectrometry (IDMS) standardization [32]. Urine albumin was measured as dipstick [33] in all patients and expressed as log-transformed ACR to be used in the KFRE (negative as 9, trace as 43, "+" as 81, "++" as 315, ">++" as 1073) [17,34,35].

Metrics for equation performance
The metrics used to compare the calibration (how closely the predicted risks agree with the observed risks) among different KFRE equations were the Brier score [36], bias [37], and precision [37]. The Brier score was calculated as the squared difference of mean observed minus predicted risks [36] weighted according to the sample sizes in five risk categories (for 5 years, 0 to < 5%, 5 to < 15%, 15 to < 25%, 25 to < 50%, and ≥ 50%; for 2 years, 0 to < 2%, 2 to < 6%, 6 to < 10%, 10 to < 20%, and ≥ 20%) adopted from prior studies [16,17]. Bias was expressed as the median difference between observed minus predicted risks [37] and precision was the interquartile range of bias [37]. A KFRE equation with the lowest score of all three metrics would be chosen as the best-calibrated equation.

Discrimination and risk reclassification
We used area under receiver operating characteristic curve (AUC) of the best-calibrated KFRE equation as a continuous variable to that of continuous eGFR. We also used category-free net reclassification improvement (NRI) to compare the KFRE with eGFR: we assessed that how many more patients with ESKD were correctly assigned to higher predicted risks, as well as patients without ESKD to lower risks by using KFRE versus eGFR [38]. We also applied AUC and NRI to compare between different KFRE equations.
Proportion of cases followed [PCF(p)] and proportion of the population needed to be follow [PNF(q)] PCF(p) and PNF(q) were two recently developed measures that are highly relevant to the decision making in public health. PCF(p) represents the proportion of individuals who will develop disease who are included in the proportion p of individuals in the population of the highest risk, and PNF(q) is the proportion of the general population at highest risk that one needs to follow in order that a proportion q of those destined to become cases will be followed [39,40]. For the equation with best calibration and predictive performance, we further calculated the PCF(p) and PNF(q) for the public health implication.

Explore the optimally feasible KFRE threshold in southeast Asians
For the best-calibrated KFRE equation, we applied the Youden Index to determine a statistically dichotomous risk threshold ('low' versus 'high') [41]. The statistical threshold had the highest summation of the sensitivity and specificity. We compared the statistical KFRE thresholds identified in the current population to the 3, 5, and 10% at 5 years [17,22] and 20 and 40% at 2 years [17], and also compared KFRE-based criteria to eGFR 30-60 ml/ min/1.73m 2 at 5 years and 20 ml/min/1.73m 2 at 2 years suggested by Tangri et al. [42]. We also compared KFRE thresholds with eGFRs that captured the same proportions of patients in this population to test the robustness of the results. The selection of the optimally feasible thresholds was based on sensitivity [43], specificity [43], positive predictive value (PPV) [43], negative predictive value (NPV) [43], positive likelihood ratio (LR+) [44], negative likelihood ratio (LR-) [44], and balanced by service-related considerations of (i) total number of patients with CKD referred to nephrologists, and (ii) number of patients with CKD needed to be evaluated by nephrologists to identify one patient who will progress to ESKD.

Stratified and sensitivity analyses
We conducted stratified analyses by age (40-75 vs. > 75 years), gender, ethnicity, type 2 diabetes mellitus status, and CKD stages (stage 3 and 4) for both 5-year and 2year ESKD risks. To be consistent with Tangri et al., we also defined ESKD by limiting to those who started dialysis and received kidney transplantation only, and calculated the AUC of the best-calibrated KFRE equation. Since some patients died before the onset of ESKD of other causes, we evaluated the impact of competing risks of death on the KFRE risk prediction using the proportional hazards model proposed by Fine and Gray [45] compared to the non-competing risk model using the conventional Cox regression model. We used STATA software version 13.0 (Stata Corp, College Station, Texas) for all data analyses. Statistical significance was set at P ≤ 0.05.

Baseline characteristics
Baseline characteristics of patients at 5-and 2-year ESKD risks are shown in Table 1 and Additional file 4. In both cohorts, the mean age was about 75 years, and approximately 50% were men, 80% were Chinese, 4% were Indians and 13% were Malays. At 5 years, 491 (2.8%) patients developed ESKD, among whom 147 (0.9%) with stage 3 CKD progressed to ESKD with a median time of 2.42 (range: 0.04-3.99) years, and 261 (1.5%) with stage 4 CKD developed ESKD in 1.78 (range: 0.09-3.93) years (Table 1). At 2 years, 330 (1.9%) developed ESKD; the median progression time from CKD to ESKD was 2.00 (range: 0.07-2.00) years among 48 (0.3%) patients with stage 3 CKD, and it was 1.13 (range: 0.01-2.00) years among 188 (1.1%) patients with stage 4 CKD (Additional file 4). In both cohorts, compared with non-ESKD patients, those with ESKD tended to be younger and have higher BMI, and were more likely to have type 2 diabetes mellitus, live in government housing and have a history of smoking (Table 1 and Additional file 4). For the 5-year cohort, compared to patients excluded for lack of established care or missing a second measurement of serum creatinine (n = 102,258), those included in the analysis (n = 131,718) were more likely to have hypertension (87.5% vs 32.5%, P < 0.001) or type 2 diabetes mellitus (44.6% vs 14.2%, P < 0.001).

Discrimination, thresholds and NRI
In addition to the traditional eGFR cut-off values, we also compared the abovementioned KFRE thresholds to the eGFR cut-offs that captured the same proportions of patients in this population. Thus, at 5 years, the KFRE > 10% corresponded to eGFR < 35.4 mL/min/1.73m 2 , and the KFRE > 16% corresponded to eGFR < 30 mL/min/1.73m 2 approximately (Table 4); at 2 years, the KFRE> 45% corresponded to eGFR< 18.8 mL/min/1.73m 2 ( Table 5). As a result, at 5 years, KFRE 10 and 16% had higher sensitivity and lower negative likelihood ratios compared to respective eGFR cut-off values, while other statistics (specificity, PPV, NPV, positive likelihood ratio and referral efficiency) remained the same (Table 4). At 2 years, KFRE > 45% had similar sensitivity, specificity, PPV, NPV, positive likelihood ratio, negative likelihood ratio, and referral efficiency compared to eGFR < 18.8 mL/min/1.73m 2 (Table 5). However, all of the KFRE thresholds resulted in a positive NRI compared to the corresponding eGFR cut-off points (≥7.06% [6.77-7.34%]) (Additional file 7 and Additional file 8), indicating the robustness of the superiority of KFRE in clinical utility.
Using the Recalibrated Pooled KFRE SEA at 5 years, an estimated 82% ESKD events were included among 10% of subjects at highest estimated risk of ESKD (Fig. 3), and an estimated 92 and 96% cases were included among 20 and 30% of subjects at highest ESKD risks (Fig. 3). At 2 years, an estimated 89, 94 and 96% events were captured in 10, 20 and 30% of subjects at the highest estimated risk of ESKD (Fig. 3).
We also presented detailed statistics of a wide range of KFRE thresholds (3-21% at 5 year; 5-45% at 2 years)      and observed that although using a higher KFRE threshold would refer fewer patients to a nephrologist to find one ESKD case, the sensitivity associated with KFRE also became less optimal (Fig. 4).

Stratified and sensitivity analyses
KFRE discrimination remained excellent across all subgroups defined by 1) age (40-75 vs. > 75 years), 2) gender, 3) ethnicity (Chinese, Malays, and Indians), 4) type 2-year of end-stage kidney disease. The proportion of cases followed represents the proportion of individuals who will develop disease who are included in the proportion p of individuals in the population of the highest risk, and the proportion of the population needed to be followed is the proportion of the general population at highest risk that one needs to follow in order that a proportion q of those destined to become cases will be followed.

Statement of principal findings
Using electronic health records linked with national renal registry, we found that the recalibrated KFRE (Recalibrated Pooled KFRE SEA equation) had better performance than existing KFRE equations in terms of having a lower Brier score, less bias and improved precision for predicting ESKD in multi-ethnic patients visiting the primary care clinics. The overall predictive capability of the Recalibrated Pooled KFRE SEA for ESKD was significantly higher than using eGFR alone. In addition, 5-year KFRE thresholds ranging 10-16% for nephrologist referral and 2-year KFRE risk threshold at 45% for dialysis planning resulted in high referral efficiency, and substantially improved reclassification of ESKD risks relative to eGFR thresholds of 20, 30, 40 and 45 mL/min/1.73m 2 . Thus, automated referrals using KFRE thresholds warrant consideration in clinical practice for patients with CKD.

In relation to previous studies
Previous studies showed excellent predictive utility of the Original KFRE equation or the Pooled KFRE equation primarily among patients with European origins or those in nephrology clinics [16][17][18][19][20][21][22]. Our study expands on those findings in the primary care clinics in SEA. We observed high AUCs of the Recalibrated Pooled KFRE SEA at both 5-year (0.94; 95% CI: 0.93-0.95) and 2-year (0.96; 95% CI: 0.95-0.97) risks of ESKD. Moreover, evidence for KFRE thresholds based on empirical data is limited and previous suggestions on KFRE thresholds have been based on physicians' opinions [22]. Our study fills that gap by using comprehensive statistical metrics coupled with clinical consideration of nephrologist workload. Of note, our results suggested that the Recalibrated Pooled KFRE SEA thresholds ranging 10-16% for nephrology referral criterion over 5 years had high sensitivity, high specificity and high referral efficiency, and substantially improved reclassification of ESKD risk on top of eGFR thresholds of 30, 40 and 45 mL/min/ 1.73m 2 . In addition, the 2-year threshold of Recalibrated Pooled KFRE SEA for dialysis planning in the current study (45%) was close to the KFRE > 40% suggested previously [17] and was marginally better than eGFR 20 mL/min/1.73m 2 [42]. Moreover, as the availabilities of the healthcare resources and the balance among sensitivity, specificity and referral efficiency may vary from country to country, a universal optimal KFRE threshold may not be possible. Our study provides a wide range of Recalibrated Pooled KFRE SEA thresholds with useful statistics (sensitivity and referral efficiency) for clinicians and health planners to choose from based on local resources, which greatly enhanced the clinical application to the primary care settings with different availability of nephrology resources globally.

Meaning of the study
The Recalibrated Pooled KFRE SEA equation includes four routinely measured variables, which were available in > 86% of our study population with stage 3-5 CKD (although ACR was converted from urine albumin for all patients). Thus, our findings imply that the Recalibrated Pooled KFRE SEA equation is likely to aid referral decisions and dialysis planning across all general practitioner settings if integrated into EHR. Furthermore, the improved triage efficiency would enable patients at high-risk of ESKD to receive timely referrals to a nephrologist, which has been shown to shorten waiting time for nephrology care [46] and substantially reduce medical costs for initiating renal replacement therapy and dialysis compared to late referral [5]. Since the nephrologist shortage is global [10], the implications of shortening patient wait time and reducing costs would have significant impact on health systems and patient well-being in resource-limited settings where CKD burden is rising, and accessibility to renal replacement therapy is limited at a global level [47].

Strengths and limitations
Our study has several strengths. First, this is likely to be the first report to determine the best-calibrated KFRE equation and potentially useful thresholds for nephrologist referral and dialysis planning in primary care population in SEA. When looking for the clinically useful thresholds, we applied rigorous statistical criteria, and combined service-related considerations for health planners, and the methodology provided as a yardstick for future studies. However, whehter the suggested thresholds are optimal for Singapore will need to be further tested taking into consideration of the workforce and work capacticy of Singapore nephrologists, and simulation studies are warranted to predict the performance of such thresholds over time. Second, we included all eligible patients visiting primary care clinics over the study duration and thus had a large sample size. Third, the multiple major ethnic groups in our sample (Chinese, Indians, and Malays) are a diaspora of populations from countries (China, India and Malaysia) that are homes to one-third of the world's population. The excellent predictive utility of the Recalibrated Pooled KFRE SEA in all three ethnic groups shown in stratified analyses greatly enhanced the utilization of KFRE to many people globally. Fourth, serum creatinine measurements were calibrated to be traceable to an IDMS standard, thus increasing the validity of both eGFR and KFRE assessments. Fifth, we objectively assessed all ESKD cases with virtual follow-up completion via linkage to the nationwide Singapore Renal Registry.
However, our study also had some limitations. First, we deleted those without established care from the study (~30%), thus introducing the possibility of selection bias. Nevertheless, the prevalence of hypertension and diabetes was lower in the excluded population than that expected in the agematched general population in Singapore [28,48]. Second, the definition of ESKD in the current study was slightly different from that of Tangri et al. [16]; however, we conducted sensitivity analysis using the definition of ESKD from that of Tangri and found similar results. Third, the ACR value used in the KFRE score was converted from urine dipstick that was measured in all patients, and thus may be less precise compared to direct measurements. Specifically, the conversion between urine albumin to ACR for people with "+", "++" and "+++" were based on limited data [17,34,35]. However, previous studies using the same conversion were included in the meta-analysis of KFRE validation and showed similar results of ESKD prediction [17], and thus suggested that the dipstick-converted ACR value is unlikely to have a large impact on the predictive performance of KFRE. Fourth, the current study did not have a validation dataset to examine the superior performance of Recalibrated KFRE SEA over other KFREs, and future studies among SEA populations are warranted to validate our results. Fifth, the sample size of Indian patients were small in the current population, and the 95% CI of AUC was wider compared to Chinese and Malays; thus, our results may not be generalizable to Indians. Future studies with bigger sample sizes of Indians are warranted to validate our results. Thus, the optimal threshold may be different in other countries. In addition, the current study did not have data on serum calcium, phosphate, bicarbonate, and albumin to validate the 8-variable KFRE equation. However, the 4-and 8-variable equations showed similar discrimination in the original development cohort [16] and subsequent meta-analysis [17]. Therefore, the less complicated 4-variable KFRE may be a more convenient tool for clinical usage.

Conclusions
In conclusion, our results showed that the Recalibrated Pooled KFRE SEA equation is an excellent predictive tool and performed better in terms of having a lower Brier score, less bias and improved precision than existing KFRE for identifying patients with CKD at risk for progression to ESKD in a primary care setting in SEA. Our findings suggest that implementation of the equation using 5-year thresholds > 10-16% to guide dialysis planning and 2-year threshold > 45% to guide nephrologist referral would facilitate more efficient and accurate risk stratification of patients at high risk of ESKD. Future studies are warranted to validate our findings, evaluate the clinical and cost effectiveness of a CKD model of care that integrates EHR and the KFRE in primary care settings serving Asians as well as globally.