Setting an agenda for comparative effectiveness systematic reviews in CKD care
BMC Nephrology volume 13, Article number: 74 (2012)
Systematic reviews comparing the effectiveness of strategies to prevent, detect, and treat chronic kidney disease are needed to inform patient care. We engaged stakeholders in the chronic kidney disease community to prioritize topics for future comparative effectiveness research systematic reviews. We developed a preliminary list of suggested topics and stakeholders refined and ranked topics based on their importance. Among 46 topics identified, stakeholders nominated 18 as ‘high’ priority. Most pertained to strategies to slow disease progression, including: (a) treat proteinuria, (b) improve access to care, (c) treat hypertension, (d) use health information technology, and (e) implement dietary strategies. Most (15 of 18) topics had been previously studied with two or more randomized controlled trials, indicating feasibility of rigorous systematic reviews. Chronic kidney disease topics rated by stakeholders as ‘high priority’ are varied in scope and may lead to quality systematic reviews impacting practice and policy.
Evidence to inform the optimal care of patients at risk of chronic kidney disease (CKD) incidence or progression is greatly needed. Patients, clinicians, and policy makers [1–7] want evidence about effective strategies to: (1) prevent incident CKD, (2) improve the accuracy and timely detection of CKD, (3) treat CKD effectively to limit progression to end stage renal disease (ESRD), and (4) decrease morbidity from CKD-related comorbid illnesses. Comparative effectiveness research (CER) compares “the benefits and harms of various interventions and strategies for preventing, diagnosing, treating and monitoring health conditions in real-world settings.”  The American Recovery and Reinvestment Act (ARRA) of 2009 provided $1.1 billion for CER which was divided and disbursed to the National Institutes of Health, the Agency for Healthcare Research and Quality and the Office of the Secretary of Health and Human Services . The Institute of Medicine was tasked to recommend national priorities for research questions to be addressed by CER and supported by ARRA funds. Among the Institute’s list of 100 initial priority topics, two addressed kidney disease .
A core research method of CER is systematic review and synthesis of existing literature, but this has not been widely employed to study CKD care. Findings from systematic reviews can help decision-makers draw conclusions about effective care strategies employed among heterogeneous populations and clinical settings. Systematic reviews also help to identify gaps in the existing evidence that need to be addressed by future primary research studies .
CER studies may be distinguished from other types of clinical research by their explicit goal of being highly responsive to priorities of community stakeholders (persons or groups who have a vested interest in a clinical decision and the evidence that supports that decision) who will use study findings to support their decisions about care . Research funding agencies increasingly recommend formal solicitation of stakeholder input as an important initial step in designing CER studies. [11, 12]. To date, there has been little effort to identify priorities for CER among stakeholders in the CKD community and to share this with the CKD community at large. We engaged stakeholders within the CKD community to identify and prioritize topics for future CER systematic reviews and to help set an agenda for future primary CER studies of CKD care.
The Johns Hopkins University Evidence-Based Practice Center (EPC) assembled our team under a contract from the Agency for Healthcare Research and Quality (AHRQ) of the Department of Health and Human Services. Our primary goal was to use the input of community stakeholders to identify topics for systematic reviews of primary literature comparing the effectiveness of strategies to prevent, detect, and treat CKD or its complications. We also sought to identify topics for future CER studies of CKD care where the performance of systematic reviews might not be currently feasible due to lack of sufficient primary literature to review. Our team had clinical expertise in pediatric nephrology, adult nephrology, and general internal medicine, and research expertise in CER, clinical epidemiology and prevention of CKD, clinician practice patterns in the care of patients with CKD, health disparities, and systematic review methodology.
We established a preliminary protocol for identifying and prioritizing topics, which we later revised with input from the stakeholders. The protocol consisted of three main activities: (1) identifying and categorizing potential topics of interest to the CKD community for systematic review and primary CER studies, (2) asking stakeholders to rank topics that would be of greatest interest to the CKD community, and (3) exploring the feasibility of performing CER systematic reviews on these high priority topics (Figure 1).
Preliminary categorization and identification of topics for CER studies
From the outset, our team acknowledged the breadth of topics potentially relevant to the care of patients with CKD, ranging from studies of strategies to prevent CKD, to studies of strategies to prevent morbidity and mortality among patients with end stage renal disease (ESRD). We decided a priori to restrict our efforts to identifying priorities for CER reviews relevant to the care of patients with CKD stages I-IV because of time limitations (the process was funded to occur in less than 1 year) and differing considerations relevant to the care of patients with early or progressive CKD versus the care of patients with ESRD. To categorize the topics, we developed a preliminary ‘map’ reflecting broad domains of CKD care (prevention, detection, progression and complications) prior to engaging stakeholders. Within each broad domain, we developed a preliminary list of potential topics through an iterative process informed by our review of recent clinical practice guidelines [12–14] and perspectives pieces published by CKD thought leaders. We identified topics without regard to whether they were the subject of ongoing reviews. For many of the topics, we also identified numerous subtopics that could be of interest to the CKD community.
Selection of CKD community stakeholder representatives
We sought to identify individuals representing varied stakeholder perspectives relevant to the performance of CER in CKD care. Many of the individuals we identified were employed or had active memberships in organizations representing relevant decision-makers in the CKD community and/or had clinical expertise relevant to the care of patients with CKD. (Table 1) Because we hypothesized government agencies such as the National Institutes of Health (NIH) might use the results of systematic reviews to guide funding decisions, we also engaged program officers overseeing clinical studies for their input.
Community Stakeholder Engagement
We asked stakeholders to provide input on our approach to identifying potential CER research topics and to rank the importance of topics. We engaged them via two conference calls and several electronic and mailed communications.
We reviewed definitions of CER as presented by AHRQ, the Institute of Medicine and the Federal Coordinating Council for Comparative Effectiveness Research  with stakeholders. We then presented the preliminary map of domains for CER topics to stakeholders. The map reflected four areas of potential interest to the CKD community: (1) prevention of CKD, (2) detection of CKD, (3) progression of CKD, and (4) complications of CKD. We also presented stakeholders with a preliminary list of 30 potential topics for CER research. We solicited stakeholders’ feedback regarding: (1) the relevance and appropriateness of our preliminary conceptual map, and (2) the relevance and comprehensiveness of the preliminary list of topics we proposed. We asked stakeholders to add to our preliminary list of topics.
We also asked stakeholders to establish a set of criteria for ranking topics. We proposed four criteria which they might consider, including (1) the burden of the topic to patients with CKD or patients at risk of developing CKD (including consideration of whether that topic was of particular relevance to priority subpopulations such as patients with diabetes, pediatric patients, racial/ethnic minorities or other vulnerable populations); (2) the extent to which there is uncertainty regarding optimal approaches to the topic; (3) the extent to which a systematic review with definitive findings could impact decision-making by patients, clinicians, or policy makers, and (4) the feasibility of performing a systematic review (i.e., adequacy of literature facilitating a rigorous review and the absence of recent previous systematic reviews). However, stakeholders expressed concern about the availability of literature to perform systematic reviews in several areas, and therefore decided to prioritize topics without regard to feasibility. Our team subsequently carried out a feasibility assessment of the stakeholder-prioritized topics.
Stakeholders revised our preliminary list of CER topics by refining the proposed topics and adding additional topics. Stakeholders then each independently ranked their 10 highest priority topics for CER studies (10 for highest, 1 for lowest). We calculated a summary score reflecting each topic’s overall standing among stakeholders (Additional file 1: Appendix A). We allowed ties for overall rankings and shared the final ranked list of topics with stakeholders to learn whether the composite rankings reflected the consensus of the entire group. We then asked them to identify topics they thought should be recommended for systematic reviews by AHRQ’s Evidence Based Practice Centers.
Assessment of CER systematic review feasibility
We conducted preliminary literature searches to estimate the feasibility of performing CER systematic reviews on topics ranked highly by stakeholders. We searched publication titles in PubMed to identify relevant studies using terms sufficient to identify a reasonable core of the studies on a topic. We did not intend for these searches to be as exhaustive as would be required for formal systematic reviews. We assessed the availability of published systematic reviews/meta-analyses (within the past 5 years), randomized controlled trials (RCTs), and observational studies (if no systematic reviews/meta-analyses or RCTs were identified on a topic) for each top tier topic area. For RCTs and observational studies, we reviewed up to 300 of the most recent PubMed records for each topic. We noted whether studies pertained to specific priority populations and whether trial interventions were pharmacological or non-pharmacological. Search protocol and search terms are contained in Additional file 2: Appendix B.
Final categorization and list of potential topics for CER studies
Stakeholders agreed that the four proposed domains for categorizing topics for CER appropriately reflected the range of potential CER topics about CKD care, and added 16 additional topics to our preliminary list of 30 topics. Table 2 lists the 46 topics (grouped by domain, in no particular order) and example CER questions. The example questions are intended to illustrate ways in which CER topics could be framed, but they are not intended to limit the range of CER questions which could be asked regarding CER topic areas.
Final Stakeholder Prioritization of Topics for CER Studies
After stakeholders collectively reviewed the summary rankings, they agreed that topics with a global rank of 12 or better should be considered ‘high priority’ for future funded CER reviews. Because of ties, this cutoff resulted in 18 topics being considered ‘high priority.’ There was diversity in the opinions of the stakeholders regarding the appropriateness of topic areas for CER systematic reviews, with some stakeholders expressing concern regarding the likely paucity of evidence to compare existing strategies of care in certain topic areas. However, in general, stakeholders’ rankings were consistent, reflecting overall consensus regarding the appropriateness of topics for which CER systematic reviews would be of greatest interest to the CKD community. One anomaly was that one stakeholder’s top ranked topic, ‘inflammation in the progression of CKD’, was not ranked at all by any other stakeholder. Yet that top ranking by a single stakeholder was sufficient for it to be included among the top tier of topics. (Table 3) Half (n = 9) of top tier topics focused on strategies to slow or stop progression of CKD, while fewer top tier topics focused on prevention of CKD (n = 6), detection of CKD (n = 2), and complications of CKD (n = 1). Two topics emerged among the top tier priorities in more than one area of the conceptual map, patient safety (in prevention and progression areas) and hypertension (in prevention and progression areas).
Feasibility of performing CER systematic reviews on priority topics
Our preliminary assessment suggested some systematic reviews could be feasibly performed, with some topics having sufficient trial data and others having only observational data. (Table 3) Some of the priority topics identified, including ‘screening for CKD’ and ‘management of hypertension and lipid levels’, were previously reviewed in the AHRQ-funded evidence report Screening for and Management of Chronic Kidney Disease Stages 1-3 .
This project is one of the first attempts to systematically assess priorities for CER studies among stakeholders in the CKD community. Stakeholders’ high priority rankings reflect topics posing significant burden to patients with CKD or at risk of CKD, and aspects of CKD care for which there is significant uncertainty regarding effectiveness. High rankings also reflect stakeholders’ views that high-quality systematic reviews could help inform clinical practice or policy. While topics identified through this process may not include all topics that warrant CER systematic reviews or primary CER studies, this process may provide a preliminary road map for researchers seeking to perform CER studies relevant to the CKD community.
Many topics receiving a ‘high priority’ designation by stakeholders focused on studying the effectiveness of strategies to prevent CKD incidence and to slow CKD progression. Some highly ranked topic areas, such as ‘patient safety’, were reflected in multiple domains of the conceptual map. Because they are viewed as relevant to several aspects of CKD care and prevention, rigorous CER reviews on these topics could have high impact on patient care and clinical outcomes. Other topics, such as studies evaluating the comparative effectiveness of health information technology and the study of collaborative care strategies (both ranked as ‘top tier’ by our stakeholders), represent novel areas of inquiry. We identified few studies of these topics during our feasibility assessment, potentially reflecting a potential need for primary CER studies in these areas.
There are several caveats to our engagement of stakeholders for their input. First, the group of stakeholders was relatively small. Thus, individual stakeholders’ rankings could substantially affect topics’ overall final rankings and might not reflect fully the range of opinions that might be present among the entire CKD community. Indeed, some stakeholders did express concerns regarding the appropriateness of some topics for CER reviews, given a paucity of evidence to address these topics. Second, the composition of our stakeholder group and the manner in which we contacted stakeholders for participation in this exercise could also affect our findings. Third, while our stakeholder group included representatives from two patient advocacy organizations, we did not include actual patients with CKD. Methods for selecting patients with appropriate levels of expertise (i.e., medical knowledge, awareness of issues affecting patients with CKD relevant to comparative effectiveness research) for this type of scientific effort are not yet well defined. Fourth, stakeholders’ input may also have been further enriched by our inclusion of non-physician medical professionals such as nurses, social workers and dietitians. Future efforts to identify topics for CER reviews should consider broadening stakeholder representation to include patients and non-physician medical professionals. Fifth, some stakeholders did submit additional comments in with their rankings, which provided additional context to their rankings. However, because all stakeholders did not comment to the same extent, we did not formally analyze these comments. Finally, as this was one of the first funded efforts to identify priorities for CER systematic reviews in CKD, diversity in stakeholders’ opinions could also reflect their varying interpretations of the ultimate goal of activities such as this.
Other characteristics of our approach could also have influenced our findings. For instance, we developed the map and topic lists via iterative discussions prior to obtaining stakeholders’ input, and without regard to ongoing systematic reviews, which may have influenced the range of the final list of topics that the stakeholders ranked. Future projects of this kind may benefit from assessing and comparing independently determined priorities of different groups of stakeholders, including policy makers, funding agencies, clinicians and patients. Also, while our preliminary findings suggest several topics may have enough evidence to make CER systematic reviews feasible, several factors determine whether a systematic review will add value to inform medical decision-makers, including the number and quality of studies available to review, how definitively identified studies answer the question of interest, and how answers to questions will reduce uncertainty or add to the current knowledge and ultimately impact care. These factors should all be considered prior to undertaking CER systematic reviews.
In conclusion, we systematically identified priorities for CER research relevant to the care of patients with Stages I-IV CKD among stakeholders in the CKD community. Future efforts such as ours might benefit from identification of an even more broadly defined group of stakeholders (e.g., patients and non-physician CKD health professionals), further refinement of protocols for engaging stakeholders throughout the topic identification process, and development of strategies for identifying areas in need of both CER systematic reviews as well as primary CER studies. Findings from this project may be a useful guide for researchers and research funders seeking to address CER questions highly relevant to improving clinical care and outcomes of patients with CKD.
Mehrotra R: Bridging the care gap around dialysis initiation: Is CKD education part of the solution?. Am J Kidney Dis. 2011, 58 (2): 160-161. 10.1053/j.ajkd.2011.06.002.
Smith DH, Thorp ML: Nurse-coordinated care in CKD: time for translation into practice?. Clin J Am Soc Nephrol. 2011, 6 (6): 1229-1231. 10.2215/CJN.03620411.
Pauly RP, Tonelli M: Predicting development of CKD in the general population–early days in a rapidly evolving field. Am J Kidney Dis. 2011, 57 (6): 805-807. 10.1053/j.ajkd.2011.02.378.
Litvin CB, Nietert PJ, Wessell AM, Jenkins RG, Ornstein SM: Recognition and management of CKD in primary care. Am J Kidney Dis. 2011, 57 (4): 646-647. 10.1053/j.ajkd.2010.11.013.
Ikizler TA: CKD classification: time to move beyond KDOQI. J Am Soc Nephrol. 2009, 20 (5): 929-930. 10.1681/ASN.2009030309.
Fink JC, Chertow GM: Medication errors in chronic kidney disease: one piece in the patient safety puzzle. Kidney Int. 2009, 76 (11): 1123-1125. 10.1038/ki.2009.315.
Wasse H, McClellan WM: Increasing physician knowledge about the diagnosis and management of CKD: how can we help primary care providers?. Am J Kidney Dis. 2009, 54 (2): 187-190. 10.1053/j.ajkd.2009.05.002.
Federal Coordinating Council for Comparative Effectiveness Research: Report to the President and the Congress. June 2009, Department of Health and Human Services, Washington, DC, Accessed January 10, 2012, at http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf
Agency for Healthcare Research and Quality: Recovery Act Awards. September 2010, Rockville, MD, Accessed February 6, 2012, at http://www.ahrq.gov/fund/recoveryawards/
Brewer NT, Salz T, Lillie SE: Systematic review: the long-term effects of false-positive mammograms. Ann Intern Med. 2007, 146 (7): 502-510.
Agency for Healthcare Research and Quality: The Effective Health Care Program Stakeholder Guide 2011. July 2011, Rockville, MD, Accessed February 6, 2012, at http://www.ahrq.gov/clinic/epcpartner/stakeholderguide/stakeholdr.pdf
Patient-Centered Outcomes Research Institute (PCORI). Washington, DC, Accessed February 1, 2012, at http://www.pcori.org/
National Kidney Foundation: KDOQI Guidelines and Commentaries. 2009, Accessed January 20, 2012, at http://www.kidney.org/professionals/kdoqi/guidelines_commentaries.cfm
London G, Coyne D, Hruska K, Malluche HH, Martin KJ: The new kidney disease: improving global outcomes (KDIGO) guidelines - expert clinical focus on bone and vascular calcification. Clin Nephrol. 2010, 74 (6): 423-432.
Agency for Healthcare Research and Quality Effective Health Care Program: Screening for and Management of Chronic Kidney Disease Stages 1–3. July 2010, Accessed December 20, 2012, at http://www.effectivehealthcare.ahrq.gov/ehc/products/163/480/CKD%20Protocol%20with%20logos%207%2020%2010.pdf
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2369/13/74/prepub
We wish to thank the following individuals for participation among our CKD Community Stakeholders: Chester H. Fox, MD, Susan L. Furth, MD, PhD, Marva Moxey-Mims, MD, Andrew Narva, MD, Neil R. Powe, MD, Jennifer St. Clair Russell, MSEd, Nikolay Tchopev, MD, Paul M. Palevsky, MD, and Joseph A. Vassalotti, MD. Two of our stakeholders declined acknowledgement, however, we are grateful for their contributions to the project.
This manuscript and the work it is derived from was commissioned by the Agency of Healthcare Research and Quality (AHRQ), through a contract to the Johns Hopkins University Evidence-based Practice Center (contract no. HHSA 290-2007-100061).
Dr. Crews was supported by the Amos Medical Faculty Development Program of the Robert Wood Johnson Foundation, Princeton NJ.
Dr. Fadrowski was supported by K23ES016541 from the National Institute of Environmental Health Science, Bethesda MD.
Dr. Greer was supported by 5KL2RR025006 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and the NIH Roadmap for Medical Research.
There are no financial or non-financial competing interests amongst the authors.
DC participated in the design of the study, the preliminary identification of topics, and the feasibility assessment; and drafted the manuscript. RG, JF and MC each participated in the design of the study, the preliminary identification of topics, and the feasibility assessment; and helped to draft the manuscript. DD participated in the design and coordination of the study, the feasibility assessment and helped to draft the manuscript. JS participated in the design of the study and helped to draft the manuscript. KF and PC participated in the coordination of the study and the feasibility assessment. LEB oversaw all study activities, participated in the design of the study, participated in the preliminary identification of topics, conducted stakeholder engagement, and drafted the manuscript. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Crews, D.C., Greer, R.C., Fadrowski, J.J. et al. Setting an agenda for comparative effectiveness systematic reviews in CKD care. BMC Nephrol 13, 74 (2012). https://doi.org/10.1186/1471-2369-13-74