Skip to main content


We’d like to understand how you use our websites in order to improve them. Register your interest.

Patient-reported outcome measures: selection of a valid questionnaire for routine symptom assessment in patients with advanced chronic kidney disease – a four-phase mixed methods study



Patient-reported outcome measures (PROMs) are becoming increasingly important in healthcare. In nephrology, there is no agreement on which chronic kidney disease (CKD) symptom questionnaire to use. Therefore, the aim of this study is to select a valid symptom questionnaire for routine assessment in patients with advanced CKD.


A four-phase mixed methods approach, using qualitative and quantitative research methods, was applied. First, a systematic literature search was conducted to retrieve existing symptom questionnaires. Second, a symptom list was created including all symptoms in existing questionnaires and symptoms mentioned in interviews with patients with CKD, from which symptom clusters were identified. Next, questionnaires were selected based on predefined criteria regarding content validity. Last, two online feedback panels of patients with CKD (n = 151) and experts (n = 6) reviewed the most promising questionnaires.


The literature search identified 121 questionnaires, of which 28 were potentially suitable for symptom assessment in patients with advanced CKD. 101 unique symptoms and 10 symptom clusters were distinguished. Based on predefined criteria, the Dialysis Symptom Index (DSI) and Palliative Care Outcome Scale-Renal Version (IPOS-Renal) were selected and reviewed by feedback panels. Patients needed 5.4 and 7.5 min to complete the DSI and IPOS-Renal, respectively (p < 0.001). Patients experienced the DSI as more specific, complete and straightforward compared to the IPOS-Renal.


The DSI was found to be valid and reliable, the most relevant, complete, and comprehensible symptom questionnaire available for routine assessment in patients with advanced CKD. Routine PROMs collection could be of great value to healthcare, both at individual patient and national level. Feedback on scores and involvement of healthcare providers may promote adaptation and implementation in healthcare.

Peer Review reports


The last decade there has been a shift in focus towards a more patient-centred and value-based healthcare. As described by Michael E. Porter, value in healthcare depends on the outcomes achieved and should be defined around the patient [1]. With this change, patient-reported outcomes (PROs) are becoming increasingly important in healthcare [1,2,3,4]. PRO measures (PROMs) can be used to quantify a wide variety of concepts of health that are relevant to the patient, such as quality of life, functional status and symptom burden [2, 5].

Until recently, PROMs were mainly used in research settings. However, PROMs are increasingly being applied for clinical management in individual patients and evaluation of quality of care [4, 6]. PROMs may enhance understanding of patients’ symptoms and needs, and have the potential to improve patient outcomes and engagement in decision making [4, 7, 8]. The use of PROMs is nowadays recommended to be implemented and routinely used in clinical practice [4, 9, 10].

Broadly a PROM can be classified as a generic or disease specific instrument. Generic PROMs measure general aspects of patients’ health status, such as functional status or quality of life. Disease specific PROMs are tailored to a specific condition and address aspects of disease experience and symptoms, making these PROMs in general more sensitive and responsive to change in disease burden [2, 4, 5, 10]. Often, both generic and disease specific PROMs are used to enable comparisons across and within populations [4, 5, 10].

Also in nephrology, routine collection of PROMs can be of added value. [11] Patients with advanced chronic kidney disease (CKD) experience a poor health-related quality of life (HRQOL) and numerous physical and emotional disease related symptoms [12,13,14]. Moreover, in patients with advanced CKD, HRQOL levels generally decrease and symptom burden generally increases as the disease progresses [15]. Despite their relevance, many symptoms in patients with advanced CKD remain unnoticed. This may be partly explained by patients being reluctant to share their experienced symptoms, particularly due to feelings of guilt about wasting clinicians’ or other patients’ time [16]. Additionally, clinicians frequently are not able to identify the full spectrum of experienced symptoms and their severity, resulting in under-recognition and under-treatment of symptoms [14, 17,18,19]. Routine symptom assessment, using a questionnaire that fits patients’ needs, could provide insight and guidance for symptom management [16, 20]. Symptom management has been identified as top priority by patients with advanced CKD [21].

Although the relevance of patients’ perspective is recognized, PROMs have not yet been widely implemented in nephrology [2, 9, 11]. Currently, methods and instruments needed for implementation of PROMs in patients with advanced CKD, including patients with end-stage kidney disease (ESKD) with and without dialysis, are being explored in the Netherlands. Some generic health questionnaires are considered to be appropriate instruments for this purpose [9, 22]. However, there is no agreement on which questionnaire is most suitable to measure the broad spectrum of symptoms that patients with advanced CKD experience [9, 23]. Therefore, the aim of this study is to systematically select the most suitable CKD-specific symptom questionnaire for routine assessment in patients with advanced CKD and ESKD using a four-phase mixed methods approach.



This study is part of the development of a national registry of PROMs, which will be included in the Dutch Renal Registry (Renine) []. For now, the PROMs registry is primarily aimed at patients with advanced CKD, including patients with ESKD receiving dialysis or without renal replacement therapy (RRT). Patients will be followed over time across different stages and treatments (e.g. advanced CKD, ESKD, with and without RRT), and therefore, we have chosen not to restrict this study to subpopulations, but to focus on CKD in general, taking all existing CKD-specific symptom questionnaires into consideration.

In this study, the focus was on the content validity of the symptom questionnaire, defined as “the relevance, comprehensiveness, and comprehensibility of the PROM for the construct, target population, and context of use of interest” [24]. According to the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) standards, content validity is the most important and first to be considered measurement property in selecting a PROM [24, 25]. Furthermore, since numerous symptom questionnaires are already available [2, 26], it would be preferable to select an existing questionnaire instead of developing a new one. As an alternative for organizing focus groups and interviews with patients to identify domains of symptoms relevant to patients, we searched and used existing CKD symptom questionnaires, assuming that they all have attempted to include the most important domains and items. By combining all these questionnaires, we make use of a much wider variation in patients, methods, clinical settings and countries to gather content-wise relevant domains for CKD.

A four-phase approach, combining qualitative and quantitative research methods, was applied: 1) conduct a systematic literature search to retrieve all existing symptom questionnaires used in patients with CKD. 2) Create a complete list of unique symptoms from all symptom questionnaires and interviews with patients with advanced CKD. Cluster these symptoms into relevant symptom groups. 3) Select symptom questionnaires based on criteria to ensure content validity, including the completeness, relevance and comprehensibility for the advanced CKD population and context of routine care [24]. 4) Evaluate the most promising symptom questionnaires using a panel of patients with advanced CKD and experts (i.e. experienced questionnaire assessors). Below the four phases are described in detail.

Systematic literature search - phase 1

A systematic literature search was performed to identify all existing symptom questionnaires developed and/or used in patients with CKD. A query was constructed using numerous synonyms or identifiers for the keywords ‘chronic kidney disease’, ‘symptoms’ and ‘questionnaires’ (Additional file 1). The search was restricted to studies published in the English or Dutch language. Studies conducted in individuals < 18 years of age were excluded.

The search was executed in PubMed by two independent reviewers (EvdW and GvR). Titles were screened and found to be eligible when describing one or more symptoms or the use of a symptom questionnaire in patients with CKD. Next, the abstracts of articles included by at least one of the reviewers were screened to identify existing symptom questionnaires. Systematic reviews describing the use of questionnaires in patients with CKD were screened full text to make sure that all existing symptom questionnaires were included.

We aim to select a symptom questionnaire addressing the full range of symptoms experienced by the total CKD population. To distinguish such broad symptom questionnaires from in-depth questionnaires addressing only one or two specific symptoms (e.g. depression or fatigue questionnaires), we excluded symptom questionnaires addressing less than four physical or emotional symptoms [26]. Additionally, questionnaires focussing only on transplant-specific symptoms and generic health questionnaires (e.g. HRQOL or activities of daily living questionnaires) were excluded.

Symptom list and clustering - phase 2

Symptoms from questionnaires

A list of symptoms was created from all symptoms included in the questionnaires. To collect only unique symptoms, overlapping symptoms were combined (e.g. ‘Tingling in feet or hands’ as a combination of ‘Tingling in feet’ and ‘Tingling in hands’).

Analysis of videotaped interviews

To assure completeness of the symptom list, 18 videotaped interviews with patients with advanced CKD were analysed to check for missing symptoms. Patients received haemodialysis (n = 13), peritoneal dialysis (n = 3) or no RRT (n = 2), were 20–83 years old, and half of them was male. The interviews were conducted by two experienced male interviewers (HB and FvdZ), who were not involved in the patients’ treatment. The videos were obtained from the Dutch Kidney Patients Association (NVN) [27] and were developed to inform and support patients with CKD in making future choices regarding therapy. During the semi-structured interviews, different aspects of living with CKD were discussed, including aspects about disease, treatment, physical functioning, psychosocial aspects, relationships and quality of life. As a result of patient’s answers, additional themes were sometimes introduced including symptoms that patients experience and considered relevant. The NVN and the interviewed patients gave permission to use this material for this research purposes. The videotaped interviews were watched and analysed by two independent researchers (GvR and EvdW). All symptoms mentioned by patients were written down verbatim and subsequently compared to the symptom list derived from the questionnaires (phase 1). Symptoms that were not yet on the list were added.

Clustering of symptoms

The total list of unique symptoms was divided into clusters to identify themes that describe the broad spectrum of symptoms experienced by patients with CKD. Clustering was done by two independent healthcare professionals: a nephrologist (JR) and a nurse practitioner (NBB) specialized in pre-dialysis and dialysis care, both experienced in clinical practice and research. JR and NNB discussed the symptoms and identified clusters inductively by constant comparison and grouping of similar type of symptoms. Clusters and corresponding symptoms were discussed until consensus was reached.

Preliminary selection of symptom questionnaire - phase 3

A set of criteria (Table 1) was applied to make a preliminary selection of symptom questionnaires that are relevant, complete and comprehensible for patients with advanced CKD or ESKD in routine care setting.

Table 1 Criteria for symptom questionnaires suitable for routine assessment in patients with advanced chronic kidney disease (CKD)

Feedback panels - phase 4

Dutch versions of the most promising questionnaires were evaluated by two online feedback panels facilitated by the NVN. One panel consisted of 151 patients receiving different treatments: pre-dialysis (CKD stage 4-5), haemodialysis, peritoneal dialysis and transplantation. The patients in this panel were randomly assigned to one of the selected questionnaires. Patients assessed only one questionnaire so that their judgement on the assigned questionnaire was based on their personal opinion, experiences and needs, and not influenced by the content or structure of another questionnaire. The second panel consisted of six experienced questionnaire assessors, namely NVN patient representatives who advise on research (e.g. questionnaire development). Five of these experts were CKD patient and one person was a relative of a CKD patient. To enable a direct comparison of the questionnaires, this panel of experts compared all questionnaires from the previous phase.

To review the questionnaire, patients were asked to complete the questionnaire and to answer additional questions. Questions concerned the content and structure of the questionnaire, including: time needed for completion, burden of completing the questionnaire, desired frequency of questionnaire assessment, unclear questions, unnecessary questions, missing questions with room to report three additional symptoms, and other suggestions or comments. The time to complete the questionnaire was measured electronically (i.e. objective time). Patients also estimated the time to complete the questionnaire, hereafter referred to as subjective time. Differences between the questionnaires in objective and subjective time to complete were presented as geometric mean.

Statistical analyses were performed using SPSS V.23.0. P < 0.05 was considered statistically significant. To evaluate differences in patient, treatment and questionnaire characteristics, Student’s t-test and Chi-square tests of association were performed. To test the reliability of the symptom burden score, Cronbach’s alpha coefficients were calculated. A sensitivity analysis using one-way ANOVA and Chi-square tests was conducted to determine if the results from the patient panel are the same for transplant patients compared to patients on dialysis or without RRT.


Systematic literature search - phase 1

Figure 1 shows a flow diagram of the literature search and questionnaire selection. The search strategy identified 571 articles, of which 223 articles were included based on title and abstract. From these articles, including two full text reviews, and through snowballing, 121 unique symptom questionnaires were identified. Of these questionnaires, 93 were excluded (mainly because less than four symptoms were addressed, see Fig. 1), resulting in 28 symptom questionnaires for further investigation.

Fig. 1

Flow chart of the selection of a valid CKD-specific symptom questionnaire

Symptom list and clustering - phase 2

A complete symptom list was created from the 28 symptom questionnaires. One hundred unique symptoms were identified from these questionnaires. Analysis of the videotaped interviews with patients with advanced CKD resulted in one additional symptom (Additional file 2: Table S2). From this symptom list, two healthcare professionals distinguished the following ten clusters: general symptoms, night’s rest, gastroenterology, cardiopulmonary, central nervous system, musculoskeletal, skin, head/throat, psychosocial and sex. The total symptom list categorized into ten clusters is available in Additional file 2: Table S2.

Preliminary selection of symptom questionnaire - phase 3

In the third phase, the previous two steps were combined: the 28 symptom questionnaires were judged on their coverage of at least nine out of ten symptom clusters. Fifteen questionnaires were excluded based on this criterion (criterion A). An additional three questionnaires were excluded due to their extensive length (criterion B), leaving ten questionnaires for further examination.

Table 2 shows the characteristics of the ten questionnaires on which the questionnaires were compared and evaluated. Six out of ten questionnaires were both developed and validated in an advanced CKD population, meeting criterion C. Two of these six are derivatives of a third questionnaire; the two questionnaires include exactly the same symptoms but also distinguish how much a symptom bothers, the severity and the frequency of symptoms and hereby exceed the determined maximum length (criterion B). Another two questionnaires address a broader perspective than symptoms only. The questions regarding symptoms are spread across the questionnaire, which does not satisfy criterion D. Based on the criteria two questionnaires were selected for further consideration in the next phase.

Table 2 Characteristics of 10 suitable symptom questionnaires for routine assessment in patients with advanced CKD based on cluster coverage and suitable length

Feedback panels - phase 4

Feedback panels

The Dialysis Symptom Index (DSI) [34] and Palliative Care Outcome Scale – Renal Version (IPOS-Renal) [35] were judged by two online panels of patients and experts. Patients were randomly assigned to a questionnaire (Table 3). In total 127 patients (84.1%) received RRT, of which 27 dialysis (17.9%) and 100 transplantation (66.2%). The second panel of six experts evaluated and compared both questionnaires.

Table 3 Comparison of two CKD-specific symptom questionnaires based on feedback of the patient panel (n = 151)

Time to complete

The patient panel needed on average (standard deviation; SD) 5.4 (1.6) minutes to complete the DSI and 7.5 (1.8) minutes to complete the IPOS-Renal (p < 0.001). Also subjectively the DSI was less time consuming than the IPOS-Renal, with a difference in geometric mean of 1.6 min (p < 0.001). The time to complete estimated by experts ranged from 2 to 15 and 3 to 20 minutes for the DSI and IPOS-Renal, respectively.

Burden and frequency

Four and two patients of the online patient panel experienced, respectively, the DSI and IPOS-renal as burdensome. All experts indicated that both questionnaires were not burdensome. For both questionnaires, most patients prefer to complete the questionnaire two or four times per year. Most experts (4 out of 6) desired four times per year. In both panels, participants noted that the questionnaire should be filled in prior to each consultation with the nephrologist.


Both panels indicated that, overall, the questions in both questionnaires were clear. All experts fully agreed that both questionnaires were easy to interpret and one expert added that the questionnaires were also comprehensible for patients with low literacy. For the IPOS-Renal some patients and one expert noted that the questions might be too generally formulated, which can cause confusion or difficulties to interpret a question. Also, some patients indicated that some questions might not be applicable to all patients or treatment modalities. For the DSI some patients mentioned that questions about sexual problems may be not applicable to all patients. Two experts indicated that, in comparison to the DSI, some symptoms may be missed when using the IPOS-Renal, but these experts did not mention which symptoms were lacking. For both questionnaires patients reported additional symptoms, which were all covered by the defined clusters. Patients reported more symptoms using the DSI than using the IPOS-Renal, with 12.0 and 8.0 experienced symptoms, respectively (p < 0.001) (Table 3).


Patients made comments similar to the answers described above (see Questions). About the DSI several patients reported that they experienced the questionnaire as pleasant, clear and enlightening. For both questionnaires patients suggested to add questions on treatment and “how patients experience their lives”. Additionally, the patients pointed out that feedback on their results and involvement of the nephrologist are highly important.


The experts compared both questionnaires. Five out of six experts preferred the DSI. They qualified the DSI as more specific and complete, and believed that the questions were more clear and easier to fill in than the IPOS-Renal. Two experts, however, also mentioned that the lay-out of the IPOS-Renal was visually more attractive than the DSI.

Sensitivity analysis

Results of the patient panel stratified for transplant and non-transplant patients are available in Additional file 3: Table S3. Similar differences between the DSI and IPOS-Renal were found in transplant and non-transplant patients. Transplant patients completed both questionnaires faster (p = n.s.) and reported less symptoms (p = n.s.) compared to non-transplant patients. However, both transplant and non-transplant patients needed less time to complete the DSI (p < 0.05) and reported more symptoms using the DSI compared to the IPOS-Renal (p < 0.05). Also comments regarding the content and structure of the questionnaires were similar in transplant and non-transplant patients.


The aim of this study was to select a valid CKD specific symptom questionnaire for routine assessment in patients with advanced CKD or ESKD. The first two phases, the literature search and symptom clustering, resulted in 28 potentially suitable symptom questionnaires and ten symptom clusters. During the third phase, two questionnaires were selected based on their relevance, completeness and comprehensibility to routine assessment in patients with CKD: the DSI and IPOS-Renal. These two questionnaires were reviewed by panels of patients and experts in the fourth phase. The results of the panel reviews showed that the DSI was the most complete, specific and comprehensible symptom questionnaire. Therefore, the DSI was considered to be the most suitable symptom questionnaire currently available for routine assessment in patients with advanced CKD or ESKD.

Previous literature and current findings support the completeness and straightforwardness of the DSI. First, the patient panel reported 12 symptoms using the DSI, which is 1.5 times the number of symptoms reported when using the IPOS-Renal. We believe that this increased score is due to differences in completeness of the questionnaires rather than differences in characteristics between patients. Similar numbers of symptoms are also presented in previous literature [41]. Furthermore, a recent study showed that symptoms of insomnia, fatigue, cramping, anxiety, depression and frustration were considered top-priority by dialysis patients. Such physical and emotional symptoms are also included in the DSI [42]. Still, additional symptoms were mentioned by the patients assessing the DSI. Therefore, we propose to retain the possibility to report additional symptoms as this may favour the completeness and patient satisfaction [43]. Besides this, the time to complete the questionnaire reflects the straightforwardness of the DSI. Although the DSI contains more items than the IPOS-Renal, patients needed less time to complete the questionnaire. This might suggest that the DSI is more clear and easier to complete for patients.

We believe that routine symptom assessment can contribute to a more patient-centred healthcare system and improvement in quality of care. Routine assessment enables patients and healthcare professionals to track changes in symptom burden over time, which may result in a more complete and better understanding of patients’ symptoms and needs. Routine assessment may also yield valuable information for the evaluation of effectiveness of treatment and the progression of symptoms.

Herein, the provision of feedback on PROM score to patients and healthcare providers, both on individual and on aggregated level, may be of great importance [44]. At the individual patient level, feedback may enhance communication between patients and healthcare professionals, which is considered highly important by patients with advanced CKD [21]. Moreover, results of similar patients could provide insight in what to expect in the future and may promote patient engagement in decision making [3, 45]. Additional to the provision of feedback, the involvement of clinicians was considered very important by several patients and is expected to contribute to a successful implementation and a more patient-centred healthcare [46].

At centre or national level, performance and variation in outcomes between centres can be mapped out and may promote initiatives to improve quality of care. Besides, patient outcomes are of great value to the already available clinical performance measures, which mainly consider structure and process of care [47, 48]. So far, PROMs have been mainly used in scientific research and less often for nationwide assessment in clinical practice [4, 9]. Consequently, little is known about how PROMs can be best deployed to achieve quality improvement [49, 50].

Further research is needed to investigate how PROMs can be best used in clinical practice to improve symptom management, shared decision making and to address patients’ needs. We propose to assess and discuss symptoms using the DSI twice per year, in order to gain insight into symptom development with a minimal burden to patients and to healthcare professionals. In addition to a suitable questionnaire, successful implementation of PROMs into routine care requires planning, facilities (e.g. electronic system to collect and report PROM scores) and involvement of all stakeholders (e.g. patients, healthcare professionals and researchers) [51]. Furthermore, barriers may be encountered when implementing PROMs into routine care, including low response rates, organizational struggles or low commitment from patients or healthcare professionals [11]. To facilitate implementation and sustainability, it is vital to take these barriers into account, by, for example, providing information and communication systems to adequately collect data and discuss PROM-scores. We suggest to test the PROMs in collaboration with all stakeholders so that it fits the workflow and priorities in routine care.

A unique feature of this study is the four-phase mixed methods approach with both qualitative and quantitative research methods. Especially with the combination of these methods we believe to have selected a valid and reliable symptom questionnaire that is relevant, complete and appropriate for the population and context of interest. First, this method addressed all criteria for evaluation of content validity as established in the COSMIN standard [24]. Second, with the use of all existing symptom questionnaires, we believe to have reached completeness and to have identified the domains that are most relevant to the patient, more so than would be possible when conducting a single study. This conclusion is also supported by patients’ input: the analysis of the interviews with patients with CKD resulted in only one additional symptom and no new symptoms or domains were mentioned by the patient panel. Third, patients, healthcare professionals and experts were involved in this study. Particularly patient involvement was considered highly important, because patients’ perspective helps to select the questionnaire that is most complete, comprehensible and relevant to them. This might increase the probability of completing the questionnaire when implemented in daily practice [52]. By using this mixed methods design, a symptom questionnaire that was preferred by experts and very positively assessed by patients was selected.

On the downside, the patient panel might be not representative of the entire advanced CKD population. First, patients participating in an online panel may be more health conscious, familiar with online questionnaires and involved in healthcare compared to those who do not participate (i.e. healthy responder bias). Second, most participants in the patient panel received a kidney transplant. However, the results and comments on the questionnaires of the transplant patients did not differ from those of the patients on dialysis or without RRT. Besides this, within the context of interest, patients will be followed over time, through different stages and treatments. Many patients receive (pre-)dialysis care prior to their transplantation, and thus, we do not expect that the inclusion of patients who received a kidney transplant affected the evaluation of the questionnaires.

With the method used, the focus was on the most important PROM property, namely the content validity of the symptom questionnaire. Additionally, the DSI showed good reliability: excellent internal consistency of the symptom burden score in this current study and good test-retest reliability in the development-study [34]. However, more research is needed to further explore the reliability and validity of this questionnaire [34]. Additionally, further research is needed to investigate if the DSI detects (clinically relevant) changes in symptom burden (i.e. responsiveness). Moreover, the smallest detectable change and the minimal important change need to be investigated for the interpretation of changes in symptom burden over time [53].


In conclusion, the DSI was found to be valid and reliable, the most relevant, complete, and comprehensible symptom questionnaire currently available for routine assessment in patients with advanced CKD or ESKD. The use of PROMs could be of great added value to healthcare, both at the individual patient and national level. Feedback on results and involvement of healthcare providers may promote adaptation and implementation of PROMs into healthcare.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.



Chronic Kidney Disease


COnsensus-based Standards for the selection of health Measurement INstruments


Dialysis Symptom Index


End-Stage Kidney Disease


Health-Related Quality Of Life


Palliative Care Outcome Scale – Renal Version


Dutch Kidney Patients Association


Patient-Reported Outcome(s)


Patient-Reported Outcome Measure(s)


Renal Replacement Therapy


Standard Deviation


  1. 1.

    Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–81.

    CAS  PubMed  Article  Google Scholar 

  2. 2.

    Tang E, Bansal A, Novak M, Mucsi I. Patient-reported outcomes in patients with chronic kidney disease and kidney transplant-part 1. Front Med (Lausanne). 2017;4:254.

  3. 3.

    Rotenstein LS, Huckman RS, Wagle NW. Making patients and doctors happier - the potential of patient-reported outcomes. N Engl J Med. 2017;377(14):1309–12.

    PubMed  Article  Google Scholar 

  4. 4.

    Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167.

    PubMed  Article  Google Scholar 

  5. 5.

    Nilsson E, Orwelius L, Kristenson M. Patient-reported outcomes in the Swedish National Quality Registers. J Intern Med. 2016;279(2):141–53.

    CAS  PubMed  Article  Google Scholar 

  6. 6.

    Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. BMJ. 2010;340:c186.

    PubMed  Article  Google Scholar 

  7. 7.

    Etkind SN, Daveson BA, Kwok W, Witt J, Bausewein C, Higginson IJ, et al. Capture, transfer, and feedback of patient-centered outcomes data in palliative care populations: does it make a difference? A systematic review. J Pain Symptom Manag. 2015;49(3):611–24.

    Article  Google Scholar 

  8. 8.

    Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18(1):115–23.

    PubMed  Article  Google Scholar 

  9. 9.

    Breckenridge K, Bekker HL, Gibbons E, van der Veer SN, Abbott D, Briancon S, et al. How to routinely collect data on patient-reported outcome and experience measures in renal registries in Europe: an expert consensus meeting. Nephrol Dial Transplant. 2015;30(10):1605–14.

    PubMed  PubMed Central  Article  Google Scholar 

  10. 10.

    Devlin NJ, Appleby J. Getting the most out of PROMs: putting health outcomes at the heart of NHS decision-making 2010.

    Google Scholar 

  11. 11.

    Nimmo A, Bell S, Brunton C, Campbell J, Doyle A, MacKinnon B, et al. Collection and determinants of patient reported outcome measures in haemodialysis patients in Scotland. QJM. 2018;111(1):15–21.

    Article  Google Scholar 

  12. 12.

    Mittal SK, Ahern L, Flaster E, Maesaka JK, Fishbane S. Self-assessed physical and mental function of haemodialysis patients. Nephrol Dial Transplant. 2001;16(7):1387–94.

    CAS  PubMed  Article  Google Scholar 

  13. 13.

    Fukuhara S, Lopes AA, Bragg-Gresham JL, Kurokawa K, Mapes DL, Akizawa T, et al. Health-related quality of life among dialysis patients on three continents: the Dialysis outcomes and practice patterns study. Kidney Int. 2003;64(5):1903–10.

    PubMed  Article  Google Scholar 

  14. 14.

    Raj R, Ahuja KDK, Frandsen M, Jose M. Symptoms and their recognition in adult haemodialysis patients: interactions with quality of life. Nephrology (Carlton). 2017;22(3):228–33.

    Article  Google Scholar 

  15. 15.

    de Goeij MC, Ocak G, Rotmans JI, Eijgenraam JW, Dekker FW, Halbesma N. Course of symptoms and health-related quality of life during specialized pre-dialysis care. PLoS One. 2014;9(4):e93069.

    PubMed  PubMed Central  Article  Google Scholar 

  16. 16.

    Pugh-Clarke K, Read SC, Sim J. Symptom experience in non-dialysis-dependent chronic kidney disease: a qualitative descriptive study. J Ren Care. 2017;43(4):197–208.

    PubMed  Article  Google Scholar 

  17. 17.

    Stephens RJ, Hopwood P, Girling DJ, Machin D. Randomized trials with quality of life endpoints: are doctors' ratings of patients' physical symptoms interchangeable with patients' self-ratings? Qual Life Res. 1997;6(3):225–36.

    CAS  PubMed  Article  Google Scholar 

  18. 18.

    Weisbord SD, Fried LF, Mor MK, Resnick AL, Unruh ML, Palevsky PM, et al. Renal provider recognition of symptoms in patients on maintenance hemodialysis. Clin J Am Soc Nephrol. 2007;2(5):960–7.

    PubMed  Article  Google Scholar 

  19. 19.

    van der Veer SN, Aresi G, Gair R. Incorporating patient-reported symptom assessments into routine care for people with chronic kidney disease. Clin Kidney J. 2017;10(6):783–7.

    PubMed  PubMed Central  Article  Google Scholar 

  20. 20.

    Davison SN, Levin A, Moss AH, Jha V, Brown EA, Brennan F, et al. Executive summary of the KDIGO controversies conference on supportive care in chronic kidney disease: developing a roadmap to improving quality care. Kidney Int. 2015;88(3):447–59.

    PubMed  Article  Google Scholar 

  21. 21.

    Manns B, Hemmelgarn B, Lillie E, Dip SC, Cyr A, Gladish M, et al. Setting research priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol. 2014;9(10):1813–21.

    PubMed  PubMed Central  Article  Google Scholar 

  22. 22.

    Chronic kidney disease data collection reference guide. International Consortium for Health Outcomes Measurement (ICHOM); 2017.

  23. 23.

    Thong MS, van Dijk S, Noordzij M, Boeschoten EW, Krediet RT, Dekker FW, et al. Symptom clusters in incident dialysis patients: associations with clinical variables and quality of life. Nephrol Dial Transplant. 2009;24(1):225–30.

    PubMed  Article  Google Scholar 

  24. 24.

    Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–70.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  25. 25.

    Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–57.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  26. 26.

    Flythe JE, Powell JD, Poulton CJ, Westreich KD, Handler L, Reeve BB, et al. Patient-reported outcome instruments for physical symptoms among patients receiving maintenance dialysis: a systematic review. Am J Kidney Dis. 2015;66(6):1033–46.

    PubMed  PubMed Central  Article  Google Scholar 

  27. 27.

    Kidney Decision Aid: Dutch Kidney Patients Association [Nierwijzer, Nierpatienten Vereniging Nederland (NVN)]. 2017. Accessed 26 February 2018.

  28. 28.

    Basch E, Abernethy AP, Mullins CD, Reeve BB, Smith ML, Coons SJ, et al. Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology. J Clin Oncol. 2012;30(34):4249–55.

    PubMed  Article  Google Scholar 

  29. 29.

    Lloyd GG, Guthrie E. Handbook of Liaison Psychiatry. Cambridge University Press; 2007. p. 96–98.

  30. 30.

    Wu AW, Fink NE, Cagney KA, Bass EB, Rubin HR, Meyer KB, et al. Developing a health-related quality-of-life measure for end-stage renal disease: the CHOICE health experience questionnaire. Am J Kidney Dis. 2001;37(1):11–21.

    CAS  PubMed  Google Scholar 

  31. 31.

    Almutary H, Bonner A, Douglas C. Which patients with chronic kidney disease have the greatest symptom burden? A comparative study of advanced CKD stage and dialysis modality. J Ren Care. 2016;42(2):73–82.

    PubMed  Article  Google Scholar 

  32. 32.

    Curtin R. C Bultman D, Thomas-Hawkins C, a J Walters B, Schatell D. hemodialysis patients' symptom experiences: effects on physical and mental functioning. Nephrol Nurs J. 2003;29(6):562–74.

    Google Scholar 

  33. 33.

    Danquah FV, Zimmerman L, Diamond PM, Meininger J, Bergstrom N. Frequency, severity, and distress of dialysis-related symptoms reported by patients on hemodialysis. Nephrol Nurs J. 2010;37(6):627–38 quiz 39.

    PubMed  Google Scholar 

  34. 34.

    Weisbord SD, Fried LF, Arnold RM, Rotondi AJ, Fine MJ, Levenson DJ, et al. Development of a symptom assessment instrument for chronic hemodialysis patients: the Dialysis symptom index. J Pain Symptom Manag. 2004;27(3):226–40.

    Article  Google Scholar 

  35. 35.

    Murphy EL, Murtagh FE, Carey I, Sheerin NS. Understanding symptoms in patients with advanced chronic kidney disease managed without dialysis: use of a short patient-completed assessment tool. Nephron Clin Pract 2009;111(1):c74–c80.

    PubMed  Article  Google Scholar 

  36. 36.

    Hays RD, Kallich JD, Mapes DL, Coons SJ, Carter WB. Development of the kidney disease quality of life (KDQOL) instrument. Qual Life Res. 1994;3(5):329–38.

    CAS  PubMed  Article  Google Scholar 

  37. 37.

    Portenoy RK, Thaler HT, Kornblith AB, Lepore JM, Friedlander-Klar H, Kiyasu E, et al. The Memorial Symptom Assessment Scale: an instrument for the evaluation of symptom prevalence, characteristics and distress. Eur J Cancer. 1994;30a(9):1326–36.

    Article  Google Scholar 

  38. 38.

    Chang VT, Hwang SS, Feuerman M, Kasimis BS, Thaler HT. The memorial symptom assessment scale short form (MSAS-SF). Cancer. 2000;89(5):1162–71.

    CAS  PubMed  Article  Google Scholar 

  39. 39.

    Murtagh FE, Addington-Hall JM, Edmonds PM, Donohoe P, Carey I, Jenkins K, et al. Symptoms in advanced renal disease: a cross-sectional survey of symptom prevalence in stage 5 chronic kidney disease managed without dialysis. J Palliat Med. 2007;10(6):1266–76.

    PubMed  Article  Google Scholar 

  40. 40.

    de Haes JCJM, Olschewski M, Fayers P, Visser MRM, Cull A, Hopwood P, et al. Measuring the quality of life of cancer patients with the Rotterdam Symptom Checklist (RSCL): a manual. Groningen: Northern Centre for Healthcare Research (NCH); 2012.

  41. 41.

    Almutary H, Bonner A, Douglas C. Symptom burden in chronic kidney disease: a review of recent literature. J Ren Care. 2013;39(3):140–50.

    PubMed  Article  Google Scholar 

  42. 42.

    Flythe JE, Hilliard T, Castillo G, Ikeler K, Orazi J, Abdel-Rahman E, et al. Symptom prioritization among adults receiving in-center hemodialysis: a mixed methods study. Clin J Am Soc Nephrol. 2018;13(5):735–45.

    PubMed  PubMed Central  Article  Google Scholar 

  43. 43.

    Marcinowicz L, Chlabicz S, Grebowski R. Open-ended questions in surveys of patients' satisfaction with family doctors. J Health Serv Res Policy. 2007;12(2):86–9.

    PubMed  Article  Google Scholar 

  44. 44.

    Bingham CO 3rd, Noonan VK, Auger C, Feldman DE, Ahmed S, Bartlett SJ. Montreal accord on patient-reported outcomes (PROs) use series - paper 4: patient-reported outcomes can inform clinical decision making in chronic care. J Clin Epidemiol. 2017;89:136–41.

    PubMed  PubMed Central  Article  Google Scholar 

  45. 45.

    O'Hare AM, Armistead N, Schrag WL, Diamond L, Moss AH. Patient-centered care: an opportunity to accomplish the "three aims" of the National Quality Strategy in the Medicare ESRD program. Clin J Am Soc Nephrol. 2014;9(12):2189–94.

    PubMed  PubMed Central  Article  Google Scholar 

  46. 46.

    O'Hare AM. Patient-centered care in renal medicine: five strategies to meet the challenge. Am J Kidney Dis. 2018;71(5):732–6.

    PubMed  Article  Google Scholar 

  47. 47.

    Kliger AS. Quality measures for dialysis: time for a balanced scorecard. Clin J Am Soc Nephrol. 2016;11(2):363–8.

    PubMed  Article  Google Scholar 

  48. 48.

    Nissenson AR. Improving outcomes for ESRD patients: shifting the quality paradigm. Clin J Am Soc Nephrol. 2014;9(2):430–4.

    PubMed  Article  Google Scholar 

  49. 49.

    Howell D, Molloy S, Wilkinson K, Green E, Orchard K, Wang K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26(9):1846–58.

    CAS  PubMed  Article  Google Scholar 

  50. 50.

    Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12(5):559–68.

    PubMed  Article  Google Scholar 

  51. 51.

    Anderson NE, Calvert M, Cockwell P, Dutton M, Kyte D. The use of patient-reported outcomes in patients treated with maintenance hemodialysis: a perspective. Am J Kidney Dis. 2019.

  52. 52.

    Wiering B, de Boer D, Delnoij D. Patient involvement in the development of patient-reported outcome measures: the developers' perspective. BMC Health Serv Res. 2017;17(1):635.

    PubMed  PubMed Central  Article  Google Scholar 

  53. 53.

    De Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. United Kingdom: Cambridge University Press; 2011. 338 p.

    Google Scholar 

  54. 54.

    CCMO. Central Committee on Research Involving Human Subjects memorandum Behavioural research. 2002.

Download references


The authors thank J.I. Rotmans, N.C. Berkhout-Byrne and F. van der Zwaard for their contribution. The authors are grateful to the patients and experts for participating the panel. The authors thank the NVN for providing the videotaped interviews, and the interviewed patients for their participation and permission.


This study was supported by unrestricted grants from the Dutch Kidney Foundation (A1D1P04), the KIDZ program of the Dutch Patient Federation and the Dutch Health Insurers Association. The funding organizations were not involved in the study design, data collection, statistical analysis, interpretation of the data, writing the report and the decision to submit the report for publication.

Author information




Authors EvdW, HB, MH and FD designed the study. Data acquisition was done by EvdW, KP, GvR, HB and FD. Authors EvdW, GvR and YM conducted the data analysis and drafted the manuscript. All authors (EvdW, YM, KP, GvR, WJB, FvI, HB, MH and FD) supported the interpretation of results, provided important intellectual content, and revised and approved the final version of the manuscript.

Corresponding author

Correspondence to Esmee M. van der Willik.

Ethics declarations

Ethics approval and consent to participate

According to national guidelines, permission of the Medical Ethics Committee was not needed since participants were not subjected to procedures or required to follow rules or behaviour [54]. The NVN and interviewed patients gave written permission to use the videotaped interviews for this research purpose. Patients and experts who participated the feedback panel gave written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Search string for systematic literature search for symptom questionnaires used in patients with chronic kidney disease. (DOCX 30 kb)

Additional file 2:

Table S2. Unique symptoms identified from questionnaires and interviews with patients with chronic kidney disease, divided into ten symptom clusters. (DOCX 34 kb)

Additional file 3:

Table S3. Comparison of two CKD-specific symptom questionnaires based on feedback of transplant and non-transplant patients. (DOCX 32 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

van der Willik, E.M., Meuleman, Y., Prantl, K. et al. Patient-reported outcome measures: selection of a valid questionnaire for routine symptom assessment in patients with advanced chronic kidney disease – a four-phase mixed methods study. BMC Nephrol 20, 344 (2019).

Download citation


  • Chronic kidney disease (CKD)
  • End-stage kidney disease (ESKD)
  • Pre-dialysis
  • Dialysis
  • Symptom burden
  • Questionnaire
  • Patient-reported outcome measures (PROMs)
  • Value-based healthcare