Derivation and Validation of a Renal Risk Score for People With Type 2 Diabetes

Study populations

Outcomes

The primary composite outcome was fatal or nonfatal ESRD event (peritoneal dialysis or hemodialysis for ESRD, renal transplantation, or death with chronic renal failure [CRF] or ESRD coded as a contributing cause). Prospective as well as past events were identified from national hospital and mortality records between 1988 and December 2010 using ICD-9 and ICD-10 codes (Supplementary Table 1). Dialysis and renal transplantation events were confirmed against the information recorded in the Australia and NZ registry of renal replacement therapy (ANZDATA). Any inconsistencies in events or event dates between the two sources of outcome data were investigated further using hospital coded records to adjudicate. A summary of the outcome categories used from the ANZDATA registry can be found in Supplementary Table 2. All patients in the study were identified by an encrypted national health index number that maintained their anonymity but allowed linkage of their records to national hospital, mortality, and ANZDATA databases (21).

Measures

Risk variables involved in building the renal models included age, sex, ethnicity, smoking status, duration of diabetes, BMI, BP, sCr, serum lipids (HDL/total cholesterol ratio), urine albumin/creatinine ratio (UACR), HbA_1c, previous cardiovascular disease, and medication use (ACE inhibitors, ARBs, and other BP-lowering medications). Most of these variables were required data fields in the annual reviews from each primary care practice. However, sCr and medication fields were only required by some primary care organizations. Ethnic groups included European, East Asian, South Asian (Indian subcontinent), Maori, Pacific Island, and other. Socioeconomic status was derived from a deprivation score based on national small-area geographic classifications (22).

Analyses

Descriptive statistics including means, medians, and proportions were used to describe the characteristics of the derivation and validation cohorts. Clinically relevant categories were used for UACR because a high proportion of the skewed data were truncated to zero, making transformation difficult, and the predictive performance of the models was better using these categories than when using log-transformation. Categories included no albuminuria, microalbuminuria (≥2.5 mg/mmol in men or ≥3.5 mg/mmol in women and <30 mg/mmol in both), macroalbuminuria (≥30 and <100 mg/mmol), and advanced albuminuria (≥100 mg/mmol). The CKD-EPI equation was used to calculate eGFR for analyses (19).

A succession of predictive renal models was developed including variables routinely collected in primary care and hypothesized to be predictive of first end-stage renal events. Groups of variables were added sequentially using the approach previously used by Tangri et al. (14). Performance of the models to inform the inclusion of variables was assessed using C-statistics, Akaike, and Bayesian information criterion statistics. Cox proportional hazards models were used to estimate the hazard ratios and to calculate the coefficients for risk factors in the equations (23). In all of these models, the assumptions of proportional hazards were checked using log-log plots of survival and examining Schoenfield residuals, in particular using a test of the nonzero slope of the regression (23). Several methods were used to evaluate the performance of the predictive equations in the validation dataset. Calibration and discrimination were assessed using receiver operating curves, C-statistics, Hosmer-Lemeshow χ² tests, and observed versus predicted event rates in the validation cohort. Net reclassification improvement was assessed to quantify improvement in several models offered by adding different variables (24). Sensitivity, specificity, positive predictive values, and the number identified as high risk per subsequent renal event in 5 years, were calculated using different risk cutoff scores. The performance of the new risk stratification models was compared with using eGFR, UACR, the ADVANCE kidney outcomes model (16), and the renal risk models of Tangri et al. (14) in both of our cohorts.

Ethics approval

The DCS has ongoing approval from the NZ Multiregional Ethics committee (WGT/04/09/077).

Description of the cohorts

Model derivation

Table 2 presents the hazard ratios of five risk models with different combinations of predictive variables, using the 25,736 patients with complete data and no exclusion criteria. Models 2, 3, and 4 performed the best with the highest C-statistic (0.89) and lowest Akaike statistics in the full derivation sample. After the addition of systolic BP and HbA_1c, models 3 and 4 were only marginally better than model 2 (Table 2). Adding medication status to model 4 did not improve the performance in the derivation cohort and, furthermore, reduced the sample size to 15,856 (62%) due to missing information on its use (Supplementary Table 4). Models stratified by sex were tested but these did not improve predictive performance. BMI and serum lipid concentrations were not statistically significant and thus were not included.

Albuminuria, sCr, ethnicity, and previous cardiovascular disease were all important predictors after controlling for other risk factors. While glycemic and BP control was also statistically significantly related to future renal events, model 2 without HbA_1c and systolic BP performed almost as well as models 3 and 4. Women were at higher risk at any given level of sCr; however, women are known to have a lower sCr as normal. This sex risk association reversed when eGFR, rather than sCr, was included in the model due to sex adjustments in the eGFR formula. Ethnicity remained a significant predictor in all models. Full details of risk models 1, 2, 3, and 4 are included in Supplementary Table 5. Models 2, 3, and 4 were chosen as potential clinical risk score equations, and their validity was compared.

Table 3 demonstrates how the risk of developing an ESRD event varies considerably in a 55-year-old man with an eGFR of 58 mL/min depending on other demographic and clinical factors, which are incorporated in the renal risk model.

Model validation

Models 2, 3, and 4 also performed very well on calibration and discrimination in the validation cohort. When the validation cohort was divided into deciles of risk, predicted versus observed rates of renal events were very similar for models 2, 3, and 4 (Fig. 1). Models 2, 3, and 4 had substantial net reclassification improvements compared with model 1 (13.6, 12.5, and 14.1%, respectively) (Supplementary Table 6). There were small net reclassification improvements when using model 3 compared with model 2 (1.1%) or using model 4 compared with model 2 (0.5%). Sensitivity and specificity at different cutoff risks are presented in Supplementary Table 7. Using a cutoff of 2.5% 5-year renal risk, models 2, 3, and 4 had good sensitivity (76 to 77%) and specificity (88 to 89%). Using this cutoff, seven or eight individuals would be identified as high risk for every renal event occurring in the subsequent 5 years.

The DCS 5-year risk models 2, 3, and 4 discriminated future risk of ESRD accurately in this primary care–based validation cohort with C-statistics of 0.92, 0.92, and 0.91, respectively (Supplementary Fig. 3 and Supplementary Table 6). In comparison, C-statistics for eGFR (CKD-EPI equation) and UACR for predicting first ESRD event over 5 years were 0.77 and 0.80, respectively, in the DCS derivation cohort and 0.81 and 0.89, respectively, in the validation cohort. C-statistics using the 3-year models 2, 3, 4, and 5 of Tangri et al. (14) were 0.81, 0.86, 0.86, and 0.81, respectively, when applied to the DCS derivation cohort and 0.78, 0.84, 0.84, and 0.88, respectively, when applied to our validation cohort. The 5-year kidney outcomes model of Jardine et al. (16) from the ADVANCE trial also performed well in both our derivation and validation cohorts (C statistics 0.86 and 0.89, respectively).

Main findings

The global burden of renal disease in diabetes is growing, with huge associated costs in dialysis and early mortality. Few interventions are known to reduce progression of renal disease in diabetes apart from tight BP control, use of ACE inhibitors and ARBs, and good glycemic control (9,10). Early detection and treatment are recommended based on the rationale that this could slow progression of disease and that maximal benefit requires some years of treatment (26). Improved risk stratification at baseline may thus help to identify earlier those at increased risk of ESRD as well as aid future research into new interventions to reduce progression. Renal risk models 2, 3, and 4 demonstrated excellent discrimination and calibration in the validation cohort. These models were able to discriminate risk of ESRD events more accurately than using eGFR or albuminuria alone. The choice of model used clinically would depend on the variables available, although where all are available, DCS model 4 is recommended. The models also performed rather better than internationally available renal risk models. This may have been due to the inclusion of ethnic groups relevant to NZ, as ethnicity contributed significantly to the models.

Strengths and limitations

We used data from a large national cohort of unselected primary care patients under routine care of their type 2 diabetes, with long follow-up (median over 7 years). However, only one-third of those originally assessed had full data available for the analysis, primarily due to several primary care organizations not requiring the reporting of sCr during data collection. There were, however, few clinically significant differences between those with complete data and those with missing data. We also did not have access to data on dietary and physical activity levels, which probably contribute to risk.

Only baseline data were used to predict risk. Changes in treatment and progression of disease were not taken into account. This lack of account of improvements in treatment over time may have caused underestimation of risk. Despite these limitations, the risk models performed very well in both the derivation and validation cohorts and allow accurate risk delineation across a wide range of risks based on initial data alone. Importantly, it thus appears that clinical renal risk can be determined quite early in the course of the disease.

The ESRD diagnosis is based on data from national and international databases that are used clinically. Hospital coding using ICD-9 and ICD-10 codes for death from CRF was used as the outcome and represented the most common renal event. While it was not possible to independently adjudicate the death certificate coding of each cause of death, all dialysis and renal transplantation for CRF events were cross-checked between the national hospital coding database and the independent Australasian renal registry (ANZDATA), and so are likely to be reliable. Therefore, independent validity testing has not been carried out on these diagnoses.

Many of those with high renal risk died of other causes, particularly from cardiovascular disease, prior to an ESRD event. Such competition of diseases makes accurate prediction of renal events problematic. However, the models still performed very well. Cardiovascular and renal risk assessment could be undertaken together, as the risk factors are similar (27). Even so, there are some differences between risk profiles and the documentation of renal as well as cardiovascular risk is likely to be particularly clinically useful.

Compared with existing models

Although the recent renal risk models of Tangri et al. (14) have performed very well in secondary care populations with more advanced disease, some require variables that are not routinely measured in primary care (e.g., serum albumin, calcium, phosphate, and bicarbonate), while our models use variables routinely available in primary care. However, models 3, 4, and 5 of Tangri et al. (14), which included variables similar to our own, but without measures of glycemia, duration of diabetes, or history of CVD, performed well in our primary care cohorts (C-statistics 0.81–0.88). Even so, our DCS models 2, 3, and 4 demonstrated even better discrimination in the separate validation cohort (C-statistics 0.91–0.92).

Jardine et al. (16) recently published an equation predicting risk of major kidney events, including doubling of sCr to ≥2.26 mg/dL (≥200 µmol/L), renal replacement therapy, or renal death among people with type 2 diabetes from the ADVANCE trial. Their equation performed very well in both our derivation and validation cohorts (C-statistics 0.86–0.89). It is likely that the improvement in performance of our models 2–4 in our validation cohort was due to the inclusion of regionally relevant ethnic groups. Whether ethnicity is acting as a surrogate for other yet unmeasured clinical risk factors or whether there are genetic predispositions is not understood, but its inclusion in the model currently helps to identify those at high risk. Other countries might benefit by amending and validating the models to incorporate locally relevant high-risk ethnic or other groups. The difference may also reflect our slightly different definition of the composite outcome and the fact that we did not have retinopathy or educational variables so assumed no retinopathy at baseline and attendance at education until at least 16 years of age.

Other studies have found that serum hemoglobin (15), renal artery stenosis (28), and inflammatory biomarkers (29) are also predictive of ESRD. Decreasing serum hemoglobin is usually a relatively late sign of renal impairment, while the other two variables are often not available in primary care populations. Furthermore, risk prediction was very good without their inclusion. We could not compare the performance of the QKidney risk model with the DCS models, as we did not collect several of their risk variables (1). Even so, the performance of our models was better in our independent validation cohort than QKidney was in their general population validation cohort (1).

While this study found a significant increase in hazard ratio with increasing glycemia and systolic BP, their influence was small, producing only a marginal improvement in prediction. Other studies have similar findings, with only extremes of glycemia associated with increased risk of mortality in ESRD (30). This may be influenced by the fact that many of these studies, including ours, only took baseline data into account. Other studies have shown the importance of high HbA_1c, particularly sustained elevations, in the rate of progression of renal disease, independent of tight BP control (9). Interestingly, BMI was not associated with ESRD events, and smoking status as a whole did not reach statistical significance. A previous meta-analysis of cohort studies examining the associations between risk factors and renal death also found that neither excess weight nor smoking was related to renal death (31).

Implications for clinical practice and future research

This study enables the incorporation of multiple commonly measured parameters to derive risk and thus better define those who require intensification of standard therapy, closer monitoring of both adherence and therapeutic efficacy, or early referral for secondary renal and diabetes services. The paper presents the science behind a very clinical application. In practical use, we would expect the equations to be hidden behind clinical software or clinical decision support, which is now used quite extensively in primary care cardiovascular risk assessment and prompting of management decision making (e.g., http://www.nzssd.org.nz/cvd/ [27]).

Ability to prioritize and assess risk accurately can be improved over our current clinical approach. The use of model 4 was much more discriminating than using eGFR, or UACR, to assess the likelihood of progressing to ESRD event and urgency of need for intervention. Furthermore, it is not realistic or appropriate to refer all patients with diabetes with deteriorating eGFR to nephrologists, so more accurate clinical prioritization is helpful. Being able to enumerate the risk will also be important in treatment pathways, and the use of (externally validated) risk models will be able to be incorporated to ensure that escalation of care (e.g., chronic care programs, secondary care) can be based on risk rather than clinician factors, (in)equities, or less precise clinical descriptions.

DCS models 2, 3, and 4 could also be run across whole primary care populations to identify those at high risk or those not already on optimized preventative therapy or to find those not reaching recommended targets. A 5-year renal risk cutoff of 2.5% appears to give good sensitivity and specificity; it is, however, outside the scope of this study to specify therapy approaches, as the effect of altering therapy on outcomes as a response to a level of risk has not been studied.

Renal risk assessment may additionally help raise awareness of CKD or act as a motivational or educational tool to achieve better patient self-management, improved BP control (particularly with ACE inhibitors and ARB medications), tight glycemic control, avoidance of renotoxic medications such as nonsteroidal anti-inflammatory agents, and possibly proactive lifestyle changes including weight loss, physical activity, and smoking cessation (32,33).