Skip to main content
Download PDF

Developing quality indicators for Chronic Kidney Disease in primary care, extractable from the Electronic Medical Record. A Rand-modified Delphi method

BMC Nephrology volume 21, Article number: 161 (2020) Cite this article

Abstract

Background

Chronic kidney disease (CKD) is a common chronic condition and a rising public health issue with increased morbidity and mortality, even at an early stage. Primary care has a pivotal role in the early detection and in the integrated management of CKD which should be of high quality. The quality of care for CKD can be assessed using quality indicators (QIs) and if these QIs are extractable from the electronic medical record (EMR) of the general physician, the number of patients whose quality of care can be evaluated, could increase vastly. Therefore the aim of this study is to develop QIs which are evidence based, EMR extractable and which can be used as a framework to automate quality assessment.

Methods

We used a Rand-modified Delphi method to develop QIs for CKD in primary care. A questionnaire was designed by extracting recommendations from international guidelines based on the SMART principle and the EMR extractability. A multidisciplinary expert panel, including patients, individually scored the recommendations for measuring high quality care on a 9-point Likert scale. The results were analyzed based on the median Likert score, prioritization and agreement. Subsequently, the recommendations were discussed in a consensus meeting for their in- or exclusion. After a final appraisal by the panel members this resulted in a core set of recommendations, which were then transformed into QIs.

Results

A questionnaire composed of 99 recommendations was extracted from 10 international guidelines. The consensus meeting resulted in a core set of 36 recommendations that were translated into 36 QIs. This final set consists of QIs concerning definition & classification, screening, diagnosis, management consisting of follow up, treatment & vaccination, medication & patient safety and referral to a specialist. It were mostly the patients participating in the panel who stressed the importance of the QIs concerning medication & patient safety and a timely referral to a specialist.

Conclusion

This study provides a set of 36 EMR extractable QIs for measuring the quality of primary care for CKD. These QIs can be used as a framework to automate quality assessment for CKD in primary care.

Peer Review reports

Background

Chronic kidney disease (CKD) is a common chronic condition and a rising public health issue with increased morbidity and mortality, even at an early stage [1, 2]. CKD, defined as kidney damage or a glomerular filtration rate (GFR) <60ml/min for three months or more, has an estimated prevalence of around 11% [3, 4]. In people aged between 65 and 74 worldwide, it is estimated that one in five men and one in four women have CKD [5]. However, the concept of CKD is relatively new to patients and early disease is almost always asymptomatic [6]. The Australian Health Survey showed that only one in ten patients with evidence of kidney disease was actually aware of it [7], which illustrates how silent and under-recognized CKD is [8]. Furthermore, CKD is associated with reduced quality of life, early cardio-vascular disease and events, hospitalizations, progression to kidney failure and high healthcare cost [2, 9, 10].

The above findings emphasize the importance of identifying people with CKD at an early stage of the disease to take appropriate preventive measures as described in various evidence-based guidelines [11,12,13]. Primary care has a pivotal role in the early identification of CKD and the integrated management between primary and secondary CKD care, in collaboration with the patient, should be of high quality [14]. The implementation of chronic-care models have shown to improve renal and cardiovascular outcomes [15,16,17,18,19,20]. However, adherence to CKD guidelines is often low and CKD management in primary care could be improved [21,22,23]. The challenge for primary care is to screen the population at risk for CKD and to manage the disease appropriately [14, 24]. The electronic medical record (EMR) and more precisely, data extracted from the EMR, could be used for these purposes in an automated quality assessment [25,26,27,28,29]. However, in order to automate quality assessment for CKD, evidence-based and EMR-extractable quality indicators (QIs) covering all aspects of primary care for CKD are necessary.

In the past QIs for evaluating processes and outcomes of CKD care have been developed and although a systematic review identified 273 QIs, few QIs are available that focus on primary care, integrated management between primary and secondary care and EMR extractability to allow automatic audit and feedback [30,31,32,33]. Patients are also rarely involved in the design of the QIs even though a study which investigated the conservative management of kidney failure indicated that there is a discordance between QIs important for patients and for caregivers [34, 35].

The aim of our study is therefore to develop a set of EMR-extractable and evidence-based quality indicators using a multidisciplinary expert panel, including patients, which can be used as a framework to evaluate and improve the quality of primary care for patients with CKD.

Methods

Our study was performed between November 2017 and June 2018.

Study design

A RAND-modified Delphi method was used for the development of the QIs for CKD [35,36,37,38,39]. This consensus method consists of 5 steps: (1) Extraction of recommendations from international guidelines and inclusion in a written questionnaire (2) Individual assessment of the recommendations by an expert panel, including an analysis of these assessments and a feedback report (questionnaire round) (3) A consensus round with a face-to-face discussion with the panel based on the feedback report and assessment of the recommendations for their eligibility. (4) Final evaluation of the set of recommendations by the panelists (5) Transformation of the recommendations into the final set of QIs.

Study population

The expert panel consisted of 11 members: 3 nephrologists, 3 general practitioners (GP), 1 dietician, 1 nurse and 3 patients with CKD. One of the GPs was an expert in the use of EMRs. All of the professionals worked in Belgium and were selected on the basis of their expertise with CKD. The patients had the diagnosis of CKD for at least 5 years.

Data collection

Extraction of recommendations

We searched the Turning Research Into Practice (TRIP) database, which is a clinical search engine designed to allow users to quickly find high-quality research evidence [40], and MEDLINE for CKD guidelines. The search terms: “chronic kidney disease” OR “CKD” OR “chronic kidney failure” OR “renal insufficiency” AND “guideline” OR “quality indicator” OR “quality measure” OR “quality of health care” OR “quality of care” OR “recommendations” were used. The search was also extended to the World Wide Web. Both Dutch and English language guidelines were included. The quality of the guidelines was assessed using the AGREE II criteria [41]. A list of high-quality guidelines was obtained and from this list only guidelines or their updates published after 2008 were selected. Primary care guidelines were preferred since the goal of this study is the creation of quality indicators for primary care.

A list of all recommendations applicable to CKD was made. To select recommendations for inclusion in a written questionnaire, the SMART principle was used independently by two researchers (JS and KP). Disagreement was solved by discussion and if no consensus could be reached, a third researcher was consulted (SVdB). The SMART principle is known to guide the development of goals where each objective should be Specific, Measurable, Acceptable, Realistic and Time-bound [39, 42, 43]. For example, recommendations that were not specific or acceptable for primary care, were excluded because no useful indicators could be derived (e.g. ‘Patients with an estimated GFR < 20 mL/min/m2 may require initiation of renal replacement therapy if there are symptoms of uremia’). Since our goal was to develop EMR-extractable QIs, the SMART principle was also used to assess EMR-extractability.

Furthermore, if the questionnaire would remain too long for assessment by the expert panel in approximately 30-40 minutes, recommendations were selected by taking into account geography (country of origin closest to Belgium), demography (western European population or ancestry) and year of publication (most recent). The recommendations were also categorized according to subject/domain such as definition, classification, screening, diagnosis, management, treatment, medication & patient safety and referral to a specialist.

Because our expert panel included patients with CKD, a second questionnaire was prepared without the recommendations specifically aimed towards professionals. This resulted in a substantially shorter questionnaire for patients.

Questionnaire round

Completion

The panel members received the questionnaire by mail and they were asked to evaluate each recommendation with regards to its measurements of the quality of CKD care on a Likert scale ranging from 1 to 9, with 1 being the lowest score (poor recommendation to measure quality of care) and 9 being the highest score (excellent recommendation to measure the quality of care). For each recommendation, panel members were asked to confirm its relevance and its EMR-extractability. If a panel member could not assess the recommendation, there was an option to denote the recommendation as "not assessable". After each (sub) category, panel members were asked to assess the relevance of the recommendations in a top 5 (prioritization), which is explained in further detail below. The patients received assistance from one of the authors (KP or JS) to complete the questionnaire by going through each question and providing clarification when necessary.

Finally, all panel members were able to comment or suggest other recommendations.

Analysis

Before the consensus meeting the recommendations were divided into 3 categories: having a high, a low or an uncertain potential to measure the quality of primary care for CKD. Recommendations were allocated to these categories based on the median Likert scale scores, prioritization and the extent of consensus among the expert panel members [39].

Median Likert scale scores

The median of all panelists' scores for each recommendation, ranging from 1 to 9.

Prioritization

Prioritization was a percentage based on the score of the recommendation in a top-5. The first ranked recommendation received 5 points, the second 4 points, etc. If a recommendations was not included in the top-5 listing it received 0 points. Individual prioritization points were then added up and divided by the maximal possible points of the recommendation. For example, if 6 panelists ranked a recommendation first and 5 did not mention it in their top-5 score, the prioritization was 30/55 (11 x 5 = 55) or 54.5%.

Consensus

Consensus was determined as ≥70% of the panelists assigning a median Likert score of ≥7. Discussion was determined as ≥ 30% of the panelists assigning ≥7 AND ≥ 30% assigning ≤3. The other scores were determined as absence of consensus. (Table 1)

Table 1 Preselection and consensus criteria
Full size table

Classification of recommendations

The median Likert score and the prioritization determined the preselection of the recommendations. Based on the preselection and the consensus, the recommendations were categorized as having a high, low or uncertain potential to measure the quality of primary CKD care. (Table 1 and 2)

Table 2 Final classification of the recommendations into 3 categories
Full size table

Consensus round

A consensus round was organized to discuss the in- or exclusion of the recommendations based on the data in the feedback report. Each recommendation was awarded a color, rendering its potential for quality of care measurement: recommendations with a high, low or uncertain potential were awarded a green, red and orange color, respectively. Furthermore, the scores on the 9-point Likert scale awarded by all participants, the percentage of prioritization and the extent of consensus, were noted in the feedback report for each individual recommendation. The recommendations and comments that were added by the panelists finished the report. The high potential recommendations (green color) were briefly reviewed for inclusion, those with a low potential (red color) for exclusion. Uncertain recommendations (orange color) were discussed more extensively for in- or exclusion. Finally, the selected recommendations were discussed using the SMART principle to evaluate their EMR extractability and usefulness in primary care.

Final evaluation

The list of all selected recommendations was mailed to the panelists for their final evaluation.

Formulation of the final set QIs

The final set of recommendations was transformed into QIs. For example, the recommendation: “If a patient has CKD and the eGFR is ≥ 60 ml/min/1.73 m2 (stadium 1 and 2), test eGFR annually” was transformed into “the percentage of patients with CKD and a GFR ≥ 60 ml/min/1.73 m2 in whom the eGFR was tested annually”. The final set of QIs was approved by all authors.

Ethical approval

This study was evaluated by the ethical committee of KU Leuven (SCONE) with number MP003409, which decided ethical approval was unnecessary. All panel members gave their written informed consent to participate in this study.

Results

Extraction of recommendations

Ten guidelines met the eligibility criteria and were screened with the AGREEII method [41]. [11, 12, 44,45,46,47,48,49,50,51,52,53,54,55,56,57] These ten guidelines were selected, based on their relevance, publication date and applicability to primary care. A total of 563 recommendations were obtained out of these guidelines. Thereafter, recommendations were classified into eight main categories (definition, classification, screening, diagnosis, etiology, management, treatment and referral to a specialist). During this process, identical recommendations and recommendations that involved specialized laboratory findings were removed. This resulted in a combined list of 390 recommendations (see Addendum 1). These 390 recommendations were evaluated and this (see Figure 1) resulted in a list of 183 recommendations, which were reduced to 99 by merging resembling recommendations. Discrepancies were solved by consensus.

Fig. 1
figure 1

overview of recommendation extraction. *Unrepeated: repetitive recommendations were removed; ** Publication date: primary sources after 2008 were included; NICE: National Institute for health and Care Excellence; AAFP: American Association of Family Physicians; ACP: American College of Physicians; KDIGO: Kidney Disease Improving Global Outcomes; SIGN: Scottish Intercollegiate Guidelines Network; EBM: Evidence Based Medicine Practice Net; CMAJ: Canadian Medical Association Journal; NFN: Nederlandse Federatie van Nefrologen; MSN: Malaysian Society of Nephrology; DM: Domus Medica

Full size image

Questionnaire round

These 99 recommendations were included in the questionnaire and submitted for individual rating to the professionals. For patients with CKD, 47 recommendations were presented. These did not include the recommendations that were specifically aimed at professionals (see Addendum 2 and 3 for the questionnaires for professionals and patients, respectively).

The response rate of the questionnaire was 100%. Based on preselection and consensus results of both the professional and the patients’ questionnaire, 26 out of 99 recommendations were selected, 43 needed further discussion and 30 did not meet the selection criteria.

Consensus round

A consensus meeting was organized with 7 panel members participating (1 general practitioner, 2 nephrologists, 1 dietician, 1 nurse and 2 patients). In this round, 24 out of 26 pre-panel selected recommendations remained selected; 23 out of 43 pre-panel recommendations, categorized as “discussion”, were also selected and 6 out of 30 non-selected pre-panel recommendations were selected (see Addendum 4). These 53 recommendations were then further discussed during the consensus meeting by applying the SMART principle to evaluate EMR-extractability or to improve measurability. Some recommendations were rejected, some needed minimal modification or were merged, which resulted in a final set of 36 recommendations.

Final evaluation

After the consensus round, 36 recommendations were mailed to the panelists for final approval. All panelists approved the final set of recommendations in 2 rounds. After the first round, 6 recommendations were slightly adapted based on the response of the panelists who were unable to participate in the consensus meeting.

Formulation of the final set QIs

The 36 recommendations were all transformed into QIs, resulting in 1 QI concerning definition/classification, 1 QI concerning screening/diagnosis, 8 QIs concerning management, 8 QIs concerning treatment, 6 QIs concerning medication and patient safety and 12 QIs concerning referral to a specialist. (see Table 3).

Table 3 Quality indicators for CKD
Full size table

Discussion

Principal findings

We used a RAND-modified Delphi method to develop a set of 36 quality indicators which are EMR- extractable and which can be used to evaluate primary care for patients with CKD. This set of QIs is based on 10 (inter) national guidelines and includes a wide range of aspects of care for patients with CKD, namely definition & classification, screening, management: follow-up & vaccination, treatment of CKD, medication & safety of the patient and referral to a specialist. These EMR- extractable QIs can be used as a framework to improve the quality of primary care for patients with CKD.

The first QI with regard to definition & classifications is important because a correct diagnosis in the problem list of the EMR will be the basis for the utilization of many other QIs. After all, if the diagnosis is not correctly recorded in the EMR, a quality measurement evaluating the care of CKD will produce unreliable results, since patients who do not have a correct diagnosis in their problem list will not be recognized. Applying this QI in an EMR system is thus important and can be a part of the implementation of the so-called electronic CKD phenotype, as described by other authors [13, 58]. Furthermore, the quality indicators we developed are mainly process indicators, rather than outcome indicators. This is because we based our QIs on recommendations extracted from guidelines. For the purpose of quality assessment, process indicators are also more useful than outcome indicators [59]. Outcome indicators require long-term follow-up and are influenced by many other factors that are beyond the control of professionals (i.e. patient’s age, lifestyle choices, risk factors, compliance, and health status) [60]. In addition, the QIs we developed are EMR-extractable, which makes it possible to develop an automated quality assessment. This in turn could lead to an increase in the number of patients with CKD whose quality of care can be evaluated.

The inclusion of 12 QIs concerning referral to a specialist is not surprising, given the importance of integrated care pathways [15]. At the consensus meeting, the panel’s nephrologists expressed an opinion regarding the minimal acceptable level of information and investigations required for a referral to secondary care. Despite diabetes mellitus being an important cause of CKD, many recommendations about diabetes mellitus did not make it to the final set of quality indicators. The panel members agreed that most of them were more specific for the quality of care for patients with diabetes mellitus. NSAID use was only part of the QI about screening and not a QI on its own although it was the topic of 7 recommendations in the primary list we based our questionnaire on. The recommendations concerning the use of NSAIDs were not applicable to the SMART principle and were therefore not included in the questionnaire. However, during the consensus meeting the panel stressed the importance of mentioning NSAID use in our final list of recommendations. We do not provide any QI for the target value of LDL-cholesterol, which was the topic of 5 recommendations in the primary list, but which was not included in the questionnaire. These recommendations did not reach the final QI list because they were either not considered applicable to the SMART principle, or were not included at the consensus meeting. Nutritional guidance was not very prominent among the primary list of recommendations we based our questionnaire on. This can explain why previous work indicated a lack of QIs concerning lifestyle management [32]. Our final list of QIs included 1 indicator concerning lifestyle management. However, different aspects of lifestyle management are part of this QI, namely physical activity, smoking cessation, weight control and limiting alcohol intake.

QIs regarding the care of patients with CKD have been identified in previous work [30,31,32, 61, 62]. The differences we noticed between our work and that of other authors are mainly EMR- extractability of our list of QIs and the focus on referral to secondary care, emphasizing the importance of a timely referral to a specialist in the case of progressive CKD. We also included patients in the development of this set of QIs to avoid the discordant priorities that might exist in QIs between those receiving the care and those providing the care for CKD [34]. Patients were in agreement with clinical staff regarding the importance of a timely referral to a specialist, which also explains the higher number of QIs on this topic. Finally, the patients also stressed the importance of the QIs on medication and patient safety.

In addition, this study provides a set of QIs that cover all aspects of primary care for CKD. To this end, we used many different (inter) national guidelines to be as complete as possible. Use of the SMART principle for the development of QIs as in earlier work, has enabled us to incorporate our goals in the development of this set of QIs, namely EMR-extractability and the possibility to use this set as a framework to evaluate the primary care for CKD [39, 43]. We thus created the opportunity to automate quality assessment for CKD in primary care, which may help improve the care for CKD [26].

Strengths and limitations

One of the strengths of our study is that we developed our QIs based on 10 (inter) national guidelines, which provides our set of QIs with an extensive basis and a high validity. These QIs may therefore be used to evaluate care in other countries [35, 37, 39]. Furthermore, our panel included health professionals from the different disciplines involved in CKD care. Besides general physicians, nephrologists, a nurse and a dietician, 3 patients were also engaged in the design of this set of QIs, which has been lacking in many studies to date [31, 61]. After all, to provide good quality of care, it is important that the needs and preferences of the patients are taken into account [63]. Applying the SMART principle to develop this set of QIs made it possible to incorporate our main goals, namely extractability out of the EMR of the general physician and the creation of a framework to evaluate primary care for CKD. Taken together, this may pave the way for automated quality assessment.

Our study also has several limitations. We have not yet conducted practice testing of this set of QIs to confirm operational validity. Using a practice test prior to usage of QIs is an integral part of the implementation strategy and an essential step of the quality loop [35]. In a future study, these tests should be conducted in Belgium, using a clinical practice test. However, major difficulties are not expected because of the EMR-extractability of this set of QIs. We also did not test the EMR- extractability of these QI’s in practices outside Belgium since it is challenging to overlook international disparities between EMR systems. Thus, if these QIs are used outside of Belgium, EMR-extractability needs to be confirmed and a practice test needs to be performed. In addition, since these QIs are based on different recommendations with varying levels of evidence, it is not clear that an external group of experts would come to the same conclusions. Furthermore, there is a lack of a consistent relationship between process and outcome measures, where there is not always evidence that implementation of certain processes leads to improved outcomes. Although, in view of quality improvements, process indicators should remain central [64]. Finally, an important aspect of using data stored in the EMR is the completeness. Incomplete data could hamper the implementation of these EMR- extractable QIs. Potential sources of bias when using EMR data also need to be considered [65].

Conclusion

This study provides a set of 36 EMR-extractable quality indicators for CKD primary care, based on international guidelines and approved by medical professionals and patients. These quality indicators can be used as a framework to measure and improve the quality of primary care for CKD.

Availability of data and materials

The materials are available as additional files and the datasets (individual feedback reports) are available from the corresponding author on request.

Abbreviations

CKD:

Chronic Kidney Disease

EMR:

Electronic Medical Record

QIs:

Quality Indicators

References

  1. Eckardt K-U, Coresh J, Devuyst O, Johnson RJ, Köttgen A, Levey AS, Levin A. Evolving importance of kidney disease: from subspecialty to global health burden. Lancet. 2013;382(9887):158–69.

    Article  PubMed  Google Scholar 

  2. Gansevoort RT, Matsushita K, van der Velde M, Astor BC, Woodward M, Levey AS, de Jong PE, Coresh J. Lower estimated GFR and higher albuminuria are associated with adverse kidney outcomes. A collaborative meta-analysis of general and high-risk population cohorts. Kidney Int. 2011;80(1):93–104.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Hill NR, Fatoba ST, Oke JL, Hirst JA, O’Callaghan CA, Lasserson DS, Hobbs FDR. Global Prevalence of Chronic Kidney Disease – A Systematic Review and Meta-Analysis. PLoS One. 2016;11(7):e0158765.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  4. Bruck K, Stel VS, Gambaro G, Hallan S, Volzke H, Arnlov J, Kastarinen M, Guessous I, Vinhas J, Stengel B, et al. CKD Prevalence Varies across the European General Population. J Am Soc Nephrol. 2016;27(7):2135–47.

    Article  PubMed  Google Scholar 

  5. What is Chronic Kidney Disease? [https://www.worldkidneyday.org/faqs/chronic-kidney-disease/].

  6. Webster AC, Nagler EV, Morton RL, Masson P. Chronic Kidney Disease. Lancet (London, England). 2017;389(10075):1238–52.

    Article  Google Scholar 

  7. Biomedical Results for Chronic Diseases, 2011-12. [https://www.ausstats.abs.gov.au/ausstats/subscriber.nsf/0/01ECE269AAE6E736CA257C0700114DBA/$File/AHS%20-%20Biomedical%20Results%20for%20Chronic%20Diseases.pdf].

  8. Chronic Kidney Disease (CKD) Management in General Practice (3rd edition) [https://kidney.org.au/cms_uploads/docs/ckd-management-in-gp-handbook-3rd-edition.pdf].

  9. Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. New Engl J Med. 2004;351(13):1296–305.

    Article  CAS  PubMed  Google Scholar 

  10. Arora P, Vasa P, Brenner D, Iglar K, McFarlane P, Morrison H, Badawi A. Prevalence estimates of chronic kidney disease in Canada: results of a nationally representative survey. CMAJ. 2013;185(9):E417–23.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Diagnosis and management of chronic kidney disease - A national clinical guideline [https://www.sign.ac.uk/].

  12. Chronic kidney disease in adults: assessment and management [https://www.nice.org.uk/guidance/cg182].

  13. Tummalapalli SL, Peralta CA. An Electronic CKD Phenotype: A Step Forward in Improving Kidney Care. Clin J Am Soc Nephrol. 2019;14(9):1277.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Brady M, O'Donoghue D. The role of primary care in managing chronic kidney disease. Br J Gen Pract. 2010;60(575):396–7.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Wouters OJ, O'Donoghue DJ, Ritchie J, Kanavos PG, Narva AS. Early chronic kidney disease: diagnosis, management and models of care. Nat Rev Nephrol. 2015;11(8):491–502.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Richards N, Harris K, Whitfield M, O'Donoghue D, Lewis R, Mansell M, Thomas S, Townend J, Eames M, Marcelli D. Primary care-based disease management of chronic kidney disease (CKD), based on estimated glomerular filtration rate (eGFR) reporting, improves patient outcomes. Nephrol Dialysis Transplant. 2008;23(2):549–55.

    Article  Google Scholar 

  17. Ronksley PE, Hemmelgarn BR. Optimizing care for patients with CKD. Am J Kidney Dis. 2012;60(1):133–8.

    Article  PubMed  Google Scholar 

  18. Jolly SE, Navaneethan SD, Schold JD, Arrigain S, Konig V, Burrucker YK, Hyland J, Dann P, Tucky BH, Sharp JW, et al. Development of a chronic kidney disease patient navigator program. BMC Nephrol. 2015;16:69.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Lee B, Turley M, Meng D, Zhou Y, Garrido T, Lau A, Radler L. Effects of proactive population-based nephrologist oversight on progression of chronic kidney disease: a retrospective control analysis. BMC Health Serv Res. 2012;12:252.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Tuttle KR, Bakris GL, Bilous RW, Chiang JL, de Boer IH, Goldstein-Fuchs J, Hirsch IB, Kalantar-Zadeh K, Narva AS, Navaneethan SD, et al. Diabetic kidney disease: a report from an ADA Consensus Conference. Diab Care. 2014;37(10):2864–83.

    Article  Google Scholar 

  21. Stevens PE, O'Donoghue DJ, de Lusignan S, Van Vlymen J, Klebe B, Middleton R, Hague N, New J, Farmer CKT. Chronic kidney disease management in the United Kingdom: NEOERICA project results. Kidney Int. 2007;72(1):92–9.

    Article  CAS  PubMed  Google Scholar 

  22. Philipneri MD, Rocca Rey LA, Schnitzler MA, Abbott KC, Brennan DC, Takemoto SK, Buchanan PM, Burroughs TE, Willoughby LM, Lentine KL. Delivery patterns of recommended chronic kidney disease care in clinical practice: administrative claims-based analysis and systematic literature review. Clin Experimental Nephrol. 2008;12(1):41–52.

    Article  Google Scholar 

  23. Van Gelder VA, Scherpbier-De Haan ND, De Grauw WJC, Vervoort GMM, Van Weel C, Biermans MCJ, Braspenning JCC, Wetzels JFM. Quality of chronic kidney disease management in primary care: a retrospective study. Scandinavian J Prim Health Care. 2016;34(1):73–80.

    Article  Google Scholar 

  24. Multidisciplinaire richtlijn Chronisch nierlijden (CNI) Aanvulling op de richtlijn 'Chronische nierinsufficiëntie' 2012 van Domus Medica [https://www.researchgate.net/publication/320840197_Multidisciplinaire_richtlijn_Chronisch_nierlijden_CNI_Aanvulling_op_de_richtlijn_%27Chronische_nierinsufficientie%27_van_Domus_Medica_2012].

  25. Samal L, Linder JA, Bates DW, Wright A. Electronic problem list documentation of chronic kidney disease and quality of care. BMC Nephrol. 2014;15:70.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Drawz PE, Archdeacon P, McDonald CJ, Powe NR, Smith KA, Norton J, Williams DE, Patel UD, Narva A. CKD as a Model for Improving Chronic Disease Care through Electronic Health Records. Clin J Am Soc Nephrol. 2015;10(8):1488–99.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Sequist TD, Holliday AM, Orav EJ, Bates DW, Denker BM. Physician and patient tools to improve chronic kidney disease care. Am J Managed Care. 2018;24(4):e107–14.

    Google Scholar 

  28. Peralta CA, Frigaard M, Rubinsky AD, Rolon L, Lo L, Voora S, Seal K, Tuot D, Chao S, Lui K, et al. Implementation of a pragmatic randomized trial of screening for chronic kidney disease to improve care among non-diabetic hypertensive veterans. BMC Nephrol. 2017;18(1):132.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Baker DW, Persell SD, Thompson JA, Soman NS, Burgner KM, Liss D, Kmetik KS. Automated review of electronic health records to assess quality of care for outpatients with heart failure. Ann Internal Med. 2007;146(4):270–7.

    Article  Google Scholar 

  30. Tu K, Bevan L, Hunter K, Rogers J, Young J, Nesrallah G. Quality indicators for the detection and management of chronic kidney disease in primary care in Canada derived from a modified Delphi panel approach. CMAJ Open. 2017;5(1):E74–e81.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Fukuma S, Shimizu S, Niihata K, Sada KE, Yanagita M, Hatta T, Nangaku M, Katafuchi R, Fujita Y, Koizumi J, et al. Development of quality indicators for care of chronic kidney disease in the primary care setting using electronic health data: a RAND-modified Delphi method. Clin Experimental Nephrol. 2016.

  32. Smits KP, Sidorenkov G, Bilo HJ, Bouma M, Navis GJ, Denig P. Process quality indicators for chronic kidney disease risk management: a systematic literature review. Int J Clin Pract. 2016;70(10):861–9.

    Article  CAS  PubMed  Google Scholar 

  33. Verberne WR, Das-Gupta Z, Allegretti AS, Bart HAJ, van Biesen W, García-García G, Gibbons E, Parra E, Hemmelder MH, Jager KJ, et al. Development of an International Standard Set of Value-Based Outcome Measures for Patients With Chronic Kidney Disease: A Report of the International Consortium for Health Outcomes Measurement (ICHOM) CKD Working Group. Am J Kidney Dis. 2019;73(3):372–84.

    Article  PubMed  Google Scholar 

  34. Harrison TG, Tam-Tham H, Hemmelgarn BR, James MT, Sinnarajah A, Thomas CM. Identification and Prioritization of Quality Indicators for Conservative Kidney Management. Am J Kidney Dis. 2019;73(2):174–83.

    Article  PubMed  Google Scholar 

  35. Kotter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators--a systematic review. Implement Sci. 2012;7:21.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Dalkey N, Helmer O. An Experimental Application of the DELPHI Method to the Use of Experts. Manag Sci. 1963;9(3):458–67.

    Article  Google Scholar 

  37. Grypdonck L, Aertgeerts B, Luyten F, Wollersheim H, Bellemans J, Peers K, Verschueren S, Vankrunkelsven P, Hermens R. Development of quality indicators for an integrated approach of knee osteoarthritis. J Rheumatol. 2014;41(6):1155–62.

    Article  PubMed  Google Scholar 

  38. Wennekes L, Ottevanger PB, Raemaekers JM, Schouten HC, de Kok MW, Punt CJ, Grol RP, Wollersheim HC, Hermens RP. Development and measurement of guideline-based indicators for patients with non-Hodgkin's lymphoma. J Clin Oncol. 2011;29(11):1436–44.

    Article  PubMed  Google Scholar 

  39. Van den Bulck SA, Vankrunkelsven P, Goderis G, Broekx L, Dreesen K, Ruijten L, Mpoukouvalas D, Hermens R: Development of quality indicators for type 2 diabetes, extractable from the electronic health record of the general physician. A rand-modified Delphi method. Primary Care Diabetes 2019.

  40. Turning Research Into Practice (TRIP) database [https://www.tripdatabase.com/about].

  41. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–42.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Doran GT. There's a S.M.A.R.T. way to write management's goals and objectives. Manage Rev. 1981;70:35–6.

    Google Scholar 

  43. Smets M, Smeets M, Van den Bulck S, Janssens S, Aertgeerts B, Vaes B. Defining quality indicators for heart failure in general practice. Acta Cardiol. 2018:1–8.

  44. Cardiovascular disease : risk assessment and reduction, including lipid modification. [https:///guidance/cg181].

  45. Johnson CA, Levey AS, Coresh J, Levin A, Lau J, Eknoyan G. Clinical practice guidelines for chronic kidney disease in adults: Part I. Definition, disease stages, evaluation, treatment, and risk factors. Am Fam Physician. 2004;70(5):869–76.

    PubMed  Google Scholar 

  46. Johnson CA, Levey AS, Coresh J, Levin A, Lau J, Eknoyan G. Clinical practice guidelines for chronic kidney disease in adults: Part II. Glomerular filtration rate, proteinuria, and other markers. Am Fam Physician. 2004;70(6):1091–7.

    PubMed  Google Scholar 

  47. Snively CS, Gutierrez C. Chronic kidney disease: prevention and treatment of common complications. Am Fam Physician. 2004;70(10):1921–8.

    PubMed  Google Scholar 

  48. Rivera JA, O'Hare AM, Harper GM. Update on the management of chronic kidney disease. Am Fam Physician. 2012;86(8):749–54.

    PubMed  Google Scholar 

  49. Munar MY, Singh H. Drug dosing adjustments in patients with chronic kidney disease. Am Fam Physician. 2007;75(10):1487–96.

    PubMed  Google Scholar 

  50. Baumgarten M, Gehr T. Chronic kidney disease: detection and evaluation. Am Fam Physician. 2011;84(10):1138–48.

    PubMed  Google Scholar 

  51. Snyder S, Pendergraph B. Detection and evaluation of chronic kidney disease. Am Fam Physician. 2005;72(9):1723–32.

    PubMed  Google Scholar 

  52. Qaseem A, Hopkins RH Jr, Sweet DE, Starkey M, Shekelle P. Physicians ftCGCotACo: Screening, Monitoring, and Treatment of Stage 1 to 3 Chronic Kidney Disease: A Clinical Practice Guideline From the American College of Physicians. Ann Internal Med. 2013;159(12):835–47.

    Google Scholar 

  53. Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease (CKD) [https://kdigo.org/guidelines/ckd-evaluation-and-management/].

  54. Levin A, Hemmelgarn B, Culleton B, Tobe S, McFarlane P, Ruzicka M, Burns K, Manns B, White C, Madore F, et al. Guidelines for the management of chronic kidney disease. CMAJ. 2008;179(11):1154–62.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Chronische Nierschade (CNS) [https://www.nefro.nl/richtlijnen/chronische-nierschade-cns-2018].

  56. Management of Chronic Kidney Disease in Adults [https://www.msn.org.my/msn/Doc/PublicDoc_PB/CPGManagementOfChronicKidneyDisease.pdf].

  57. Richtlijn voor goede medische praktijkvoering: Chronische nierinsufficiëntie. [https://domusmedica.be/richtlijnen/chronische-nierinsufficientie].

  58. Norton JM, Ali K, Jurkovitz CT, Kiryluk K, Park M, Kawamoto K, Shang N, Navaneethan SD, Narva AS, Drawz P. Development and Validation of a Pragmatic Electronic Phenotype for CKD. Clin J Am Soc Nephrol. 2019;14(9):1306.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Eddy DM. Performance measurement: problems and solutions. Health affairs (Project Hope). 1998;17(4):7–25.

    Article  CAS  Google Scholar 

  60. Malin JL, Asch SM, Kerr EA, McGlynn EA. Evaluating the quality of cancer care: development of cancer quality indicators for a global quality assessment tool. Cancer. 2000;88(3):701–7.

    Article  CAS  PubMed  Google Scholar 

  61. Smits KP, Sidorenkov G, Bilo HJ, Bouma M, van Ittersum FJ, Voorham J, Navis G, Denig P: Development and initial validation of prescribing quality indicators for patients with chronic kidney disease. Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association 2016.

  62. Manns L, Scott-Douglas N, Tonelli M, Weaver R, Tam-Tham H, Chong C, Hemmelgarn B. A Population-Based Analysis of Quality Indicators in CKD. Clin J Am Soc Nephrol. 2017;12(5):727–33.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Hermens RP, Ouwens MM, Vonk-Okhuijsen SY, van der Wel Y, Tjan-Heijnen VC, van den Broek LD, Ho VK, Janssen-Heijnen ML, Groen HJ, Grol RP, et al. Development of quality indicators for diagnosis and treatment of patients with non-small cell lung cancer: a first step toward implementing a multidisciplinary, evidence-based guideline. Lung Cancer (Amsterdam, Netherlands). 2006;54(1):117–24.

    Article  CAS  Google Scholar 

  64. Bilimoria KY. Facilitating Quality Improvement: Pushing the Pendulum Back Toward Process Measures. JAMA. 2015;314(13):1333–4.

    Article  CAS  PubMed  Google Scholar 

  65. Verheij AR, Curcin V, Delaney CB, McGilchrist MM. Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse. J Med Internet Res. 2018;20(5):e185.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank and acknowledge the panelists for their collaboration in the development of quality indicators for CKD: Dr. Peeters, Dr. Stas, Prof. Dr. Claes, Dr. Adriaansen, Dr. Coursier, Dr. Van Pottelbergh, N. Bangels, E. Van Nuland and the CKD patients.

Funding

Not applicable

Author information

Authors and Affiliations

  1. Academic Center for General Practice, Department of Public Health and Primary Care, KU Leuven, Kapucijnenvoer 33 blok J, 3000, Leuven, Belgium

    Steve A. Van den Bulck, Patrik Vankrunkelsven, Geert Goderis, Gijs Van Pottelbergh, Jonathan Swerts, Karolien Panis & Rosella Hermens

  2. Radboud Institute for Health Sciences, Scientific Institute for Quality in Healthcare, Radboud University Medical Center, Radboud University Nijmegen, Nijmegen, Netherlands

    Rosella Hermens

Authors
  1. Steve A. Van den Bulck
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrik Vankrunkelsven
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Geert Goderis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gijs Van Pottelbergh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jonathan Swerts
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Karolien Panis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rosella Hermens
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SVDB, PV, GG and RH designed and conceptualized the study. SVDB, JS and KP performed the data analysis and drafted the manuscript. SVDB chaired the consensus meeting. SVDB, RH, PV, GG and GVP revised the manuscript. SVDB and RH are the guarantors of this work. All authors read and approved the final manuscript

Corresponding author

Correspondence to Steve A. Van den Bulck.

Ethics declarations

Ethics approval and consent to participate

This study was evaluated by the ethical committee of KU Leuven (SCONE) with number MP003409, which decided ethical approval was unnecessary. All panel members gave their written informed consent to participate in this study.

Consent for publication

Not applicable

Competing interests

Not applicable

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Addendum 1: Exhaustive list of recommendations

Additional file 2.

Addendum 2: Questionnaire for the professionals

Additional file 3.

Addendum 3: Questionnaire for the patients

Additional file 4.

Addendum 4: Table of recommendations switched from category

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Van den Bulck, S.A., Vankrunkelsven, P., Goderis, G. et al. Developing quality indicators for Chronic Kidney Disease in primary care, extractable from the Electronic Medical Record. A Rand-modified Delphi method. BMC Nephrol 21, 161 (2020). https://doi.org/10.1186/s12882-020-01788-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12882-020-01788-8

BMC Nephrology

ISSN: 1471-2369

Contact us