Skip to main content
  • Research article
  • Open access
  • Published:

Accuracy and concordance of measurement methods to assess non-adherence after renal transplantation - a prospective study

Abstract

Background

Non-adherence (NA) to immunosuppressants (IS) among renal transplant recipients (RTRs) is associated with higher risk of allograft rejection, graft loss, and mortality. A precise measurement of NA is indispensable, although its prevalence differs greatly depending on the respective measurement methods. The objective of this study was to assess the accuracy and concordance of different measurement methods of NA in patients after renal transplantation.

Design and methods

This was a single-center prospective observational study. At baseline (T0), NA was measured via physicians’ estimates (PE), self-reports (SR), and tacrolimus trough level variability (CV%) in 78 RTRs. A Visual Analogue Scale (VAS, 0–100%) was applied both for SR and PE. In addition, we used BAASIS© for SR and a 5-point Likert scale for PE. NA was measured prospectively via electronic monitoring (EM, VAICA©) during a three month period. Meanwhile, all participants received phone calls in a two week interval (T1-T6) during which SRs were given.

Results

Seventy-eight RTRs participated in our study. At t0, NA rates of 6.4%, 28.6%, and 15.4% were found for PE, SR, and CV%, respectively. No correlation was found between these methods. During the study, the percentages of self-reported and electronically monitored adherence remained high, with a minimum mean of 91.2% for the strictest adherence measure (Timing Adherence ±30 min). Our results revealed a moderate to high association between SR and EM. In contrast to PE and CV%, SR significantly predicted electronically monitored adherence. Overall, a decreasing effect of electronically monitored adherence was found for both taking and timing adherence (±2 h, ±30 min) over the course of the study.

Discussion

The moderate to high concordance of SR and EM suggests that both methods measure NA equally accurately. SR seems to be a method that can adequately depict electronically monitored NA and may represent a good and economical instrument to assess NA in clinical practice. The increased adherence at the beginning of the study and its subsequent decrease suggests an intervention effect. Surveillance of IS intake via EM with intermittent phone calls could improve adherence on a short-term basis. To establish long-term effects, further research is necessary.

Peer Review reports

Background

Adherence is defined as “the process by which patients take their medications as prescribed” [1]. In case of Non-Adherence (NA) to the immunosuppressive (IS) regimen after renal transplantation, detrimental effects on graft survival and functioning can occur. NA has been identified as a major contributing factor for acute cellular and chronic allograft rejection, graft loss, and mortality [2,3,4,5]. Nonetheless, a substantial number of renal transplant recipients (RTRs) are non-adherent to their IS medication, on average 35.6% of patients per year [6]. This prevalence differs greatly depending on the respective measurement method, leading to NA rates ranging from 1.6% to 58.7% [7]. A wide array of different NA measures has emerged in the past years including direct and indirect measures with various advantages and disadvantages. Direct measures comprise direct observation of medication intake, biomarkers (tracers consumed with drug), and measurement of IS trough level variability in the blood [8,9,10]. Indirect measures include pill counts, self-reports (diaries, questionnaires, interviews), collateral reports from family members or clinicians, prescription refills, and electronic monitoring [8, 9]. For measures of NA in RTRs, self-reports, collateral reports, and IS trough level variability are most frequently applied, due to their inexpensiveness and practicability [8].

However, collateral reports such as physicians’ estimates are mostly considered unreliable and biased [11, 12]. Although self-reports yield the risk of underestimating NA due to social desirability or memory bias [8, 13], they are viewed as inexpensive and practical tools for clinical settings [14,15,16]. In particular the self-report questionnaire BAASIS©, which includes the assessment of taking, timing, and dosing adherence, is a widely used and recommended instrument [15]. The IS trough level variability has also become a standard measure of patients’ adherence in clinical practice, since it reflects the patient’s consumption of immunosuppressive drugs [8, 17]. Especially the coefficient of variation is increasingly used and has been associated with rejection and poor renal graft outcome [18,19,20]. However, results can be influenced by several biological parameters (metabolic rates, drug interactions, etc.) and distort interpretation [9, 11].

Electronic monitoring (EM) has recently gained great popularity. It is often declared as the best estimate of NA [13, 15, 21], since it provides insight into the dynamics of both taking and timing adherence over a period of time [8, 15]. Although the utilization of EM as a method for monitoring immunosuppressive NA is on the rise and used in a variety of studies [22,23,24,25,26,27], the accuracy of this method is still being debated. Electronic malfunction, incorrect use, negative change in routine, but also an induced intervention effect resulting in improved adherence, are considered potential causes for imprecise EM measures [21, 28,29,30,31]. Studies on the correlations of EM with other adherence measures show mixed results. In a review on adherence in a general population, a high to moderate concordance between EM and self-reports could be found [32], whereas another review found self-reports to be systematically underestimating NA compared to EM [33]. Studies examining the correlations of different adherence measures in RTRs are sparse. One study found self-reported NA to be a significant predictor of electronically monitored NA during a three months course. However, it only showed a 26% sensitivity compared to EM [14]. Similar results were found in a study by Butler et al. [12]. Another study of Schäfer-Keller et al. [34] monitored immunosuppressive NA over a three months period. IS trough levels, self-reported NA, and clinician’s collateral reports showed significant but low correlations with EM. The concordance of other adherence measures depicts equally heterogeneous results. Liu et al. [16] found self-reported NA to be associated with serum concentration fluctuation for tacrolimus but not cyclosporine. Similar results were found by Lalic et al. [35], while other studies found no correlation at all between self-reports and IS trough levels [10, 34]. Equally, no associations were detected between collateral reports and self-reports [11, 34, 36], nor between physicians’ estimates and the variability of trough levels [11, 34]. Some research also recommends combining several measurement methods to increase sensitivity [9, 15, 34, 37]. Overall, there still seems to be some lack of consensus on how exactly measures of NA correlate with each other and which approach is the best. Especially in RTRs, evidence seems scarce. With this study we aim to close this gap by prospectively examining the concordance of different adherence measures in RTRs. To our knowledge, no study has yet examined both self-reported and electronically monitored immunosuppressive NA in a longitudinal design.

Methods

Objectives of the study

The primary objective of this work was to investigate the correlation between different measurement methods of immunosuppressive NA during the phase of medication implementation [1, 38]. Besides IS trough levels and physicians’ estimates on NA, special attention was paid to electronically monitored and self-reported NA. The following issues were addressed: 1) To what degree do the relevant measurement methods interrelate at the beginning of the study? 2) Does electronically monitored NA coincide with patients’ self-reports during the course of the study? 3) Can self-reported NA, physicians’ estimates and/or trough levels predict the degree of electronically monitored NA during the study? 4) How does the measured NA change over time? Does the study design induce an intervention effect on NA?

Study population and recruitment methods

This work is part of the APT (Adherence and Psychological Health after Transplantation) research project of the Department of Psychosomatic Medicine and Psychotherapy and took place in cooperation with the Department of Nephrology and Hypertension and the Department of Cardiac Surgery in Erlangen. Our sample was consecutively derived from RTRs attending their routine follow-up examination at the renal transplant outpatient clinic from March 2018 to April 2019. Potential participants were contacted via telephone prior to their regular appointment. If interested, study information and questionnaires were sent to the respective patients. RTRs aged 18 years or more who were at least six months post-transplantation and received tacrolimus (Advagraf© or Prograf©) as their prescribed IS medication were eligible. We specifically focused on medication implementation and excluded cases of initiation [1, 38]. Exclusion criteria encompassed insufficient German language skills, severe mental disorders, and/or cognitive impairments. Institutional ethic board approval was obtained from the Clinical Ethics Committee of the University Hospital Erlangen (Friedrich-Alexander-University, Erlangen-Nürnberg, FAU). Written informed consent was given by all participants.

Data collection and measurement of non-adherence

Electronic monitoring

At enrollment, all participants received an electronic multicompartment pillbox (VAICA SimpleMed©) which allows the storage and organization of medication up to seven days and up to four doses a day. The patients’ medication plan for their main immunosuppressant (Advagraf© or Prograf©) was entered on the web portal of the corresponding device. We defined two time intervals of ±30 min and ± 2 h of the prescribed IS intake time. Pill extraction times were registered electronically and transferred to the respective patients’ web-based pillbox record. Reminder functions (visual and acoustic sounds) were disabled for all patients during the whole course of the study. To improve validity, participants were asked to keep log of incidents when the opening of the compartments did not correspond with medication intake (e.g. IS were extracted early and taken later) or when the medication was taken from another source (take-away box, pocket, purse etc.) [39]. To determine whether electronic monitoring might influence the patients’ normal adherence behavior [28, 29], we asked the participants to rate its subjective effect on a scale from − 10 (negatively influenced) to + 10 (positively influenced).

Self-report

For self-reports on IS intake, we used the Basel Assessment of Adherence to Immunosuppressive Medication Scale (BAASIS© [15]). It consists of four items on immunosuppressive NA (dose taking, drug holidays, timing adherence ±2 h, dose reduction) to be rated on a 6-point scale (0 = no/never, 5 = every day) referring to the last four weeks. Consent to at least one of the items is considered non-adherent (= dichotomous). Further, a total score of the four items results in a continuous adherence rating with scores ranging from 0 to 20 [40]. We used an altered version of this questionnaire as we added a further question on timing adherence, asking for a deviation of intake time of ±30 min. We further reduced the recall period to two weeks during the course of the study. In addition to the 6-point rating scale, we asked for the absolute frequency of deviations from the prescribed medical regimen. At the beginning of the study, we also asked each participant to rate his overall adherence on a 10 cm (0–100%) Visual Analogue Scale (VAS) (BAASIS© Interview).

Physician’s estimates

After the patients’ routine follow-up examination, the treating nephrologist was asked to assess the patients’ overall adherence on a 5-point scale (1 = very good, 5 = very bad). Patients who were rated less than good (> 2) where considered non-adherent. Similar measurement methods have been used by other researchers [11]. In addition, the patients’ general adherence was assessed on a 10 cm (0–100%) Visual Analogue Scale (VAS) to reach comparability with the patients’ self-reports.

IS trough levels

IS trough levels (Tacrolimus) are routinely measured during the patients’ follow-up examination at the outpatient clinic of the nephrologic department of the University Hospital in Erlangen. We used the IS trough level which was measured at the day of study enrollment as well as up to three available antecedent measures [10], irrespective of the individual interval each patient attended their follow-up visits. IS levels from several time points are considered more reliable than only one [11, 17]. We excluded successive measures that were taken during hospitalization. We then assessed the IS trough level variability (CV% [17]), which was previously found predictive of graft rejection, organ loss, and NA [18,19,20, 41]. If only one IS trough level was available and variability was not calculable, we excluded this case for this specific analysis. For the calculation of the CV% we first standardized IS levels by dividing each IS level by their respective target level, which is determined individually for each patient and can be changed by the physicians depending on its clinical course. For each standardized IS trough level, we calculated means (M) and standard deviations (SD). The CV% was then calculated by dividing SD through M and multiplied by 100. Higher CV%s reflect a higher degree of irregularity in IS trough levels [10, 17]. A CV% threshold of approximately > 30 was previously associated with poor renal allograft outcomes [20, 41,42,43] and non-adherence [19], therefore we categorized patients with CV% > 30 as non-adherent and CV% < 30 as adherent, respectively.

Study design and measurement points

This is a prospective single-center observational study. Electronic monitoring took place for three months. During this time, each participant was called every second week and asked to give a self-report on potential NA of the last two weeks (Fig. 1).

Fig. 1
figure 1

Timeline of study procedure

At the end of the study, patients were asked to return the electronic pillbox and received a detailed feedback on their electronically monitored adherence behavior.

Data management

Deficient data from the electronic pillbox were corrected via notes provided by the patients’ diaries. At the end of the three months EM period, the participants were asked to assess the consistency of diary and pillbox use on a 0–10 scale. If patients stated inconsistent use of the dairies and/or the pillboxes (ratings from 0 to 5) or if technical failure occurred over the whole study or a certain period of the study, the respective measurement points were eliminated from analysis [28, 29]. In cases of which patients were not using the pillbox for a certain period of time (e.g. travel, hospitalization, weekend etc.), we extended the individual study period for the respective time, if possible. If patients could not be reached via telephone in the two week interval, we extended this period to a maximum of three weeks. After four weeks, the respective measurement points were specified as missing.

EM-Imputations were used to replace single missing values. Missing values for EM, whole psychometric scales, and measurement points (due to drop-outs, eliminated measures etc.) were not replaced. The measurement points or number of patients included in the respective analyses are indicated. It has to be noted that Timing Adherence ±30 min (TM ± 30 min) includes all cases of Timing Adherence ±2 h (TM ± 2 h) while TM ± 2 h includes all cases of Taking Adherence (TA).

Statistical analysis

For descriptive statistics, we depicted frequencies, mean values, standard deviations, and ranges. For group comparisons (Non-Responder Analysis), independent T-tests, Mann-Whitney-U-tests, and Chi2-tests were used. Analyses on the associations of measurement methods comprised Cohen’s kappa and the Intraclass correlation coefficient (ICC). Three linear regression models were conducted with the percentage of electronically monitored adherence (TA, TM ± 2 h, TM ± 30 min) as the outcome variable and sex, age, physicians’ estimates (VAS), self-reports (VAS), and IS trough level variability (CV%) as independent variables. To assess changes in probability of adherence over time, we dichotomized the electronically monitored adherence scores (for TA, TM ± 2 h, and TM ± 30 min respectively; 0: 100% adherent, 1: ≤ 100% adherent). Two alternative linear mixed models (linear and piecewise linear) were conducted for each variable. Data were processed and analyzed using the software SPSS 21 for Microsoft Windows©. Linear mixed models were carried out using the nlme-package [44] and the lme4-package [45] for R version 3.5.1. All results were interpreted on a significance level of p < .05. We used Bootstrap Confidence intervals with 10,000 iterations in case of severe deviations from the normal distribution.

Results

Of 231 eligible RTRs, a total of 184 patients could be contacted by phone. 89 initially agreed to participate in our study. 11 dropped out before activating the pillbox or during the first week, one patient resigned during the first month, and another during the second month of pillbox usage (=NR). A total of 76 patients completed the study (=P). Reasons for not participating or drop-outs included sickness, lack of time, or impracticability of the pillbox. A Non-Responder Analysis (including drop-outs and refusals, n = 108) revealed no significant difference in age (P: mean = 55.64 years; NR: mean = 52.12 years; t = − 1.85, p = .065), year of transplantation (P: median = 2013; NR: median = 2015; U = 3452.00, p = .065) and sex (χ2 = 3.12, p = .077) between those who participated and those who did not. Sociodemographic and biomedical data of participating patients are displayed in Table 1.

Table 1 Sociodemographic and biomedical data

Prevalence of non-adherence at the beginning of the study (t0)

Physicians’ estimates ranged from 1 (very good) to 3 (moderate), with a total mean of 1.36 (SD: .60). 73 of the 77 patients (93.6%) were rated with good or very good (= adherent), whereas 5 patients (6.4%) were rated to have moderate adherence (= not adherent). On the Physicians’ Visual Analogue Scale (VAS), adherence ratings ranged from 73% to 100%, with a mean of 92.27% (SD: 5.47). 80.8% of all patients were rated to have an overall adherence of 90% or better. Self-reports (dichotomous assessment of BAASIS©) revealed NA in 22 patients (28.6%). 67 patients (85.9%) claimed not to have missed any dose during the past four weeks, whereas 10 (12.9%) said it happened at least once. No one claimed to have skipped several doses in a row. 59 patients (75.6%) stated that they always took their IS within a 2-h interval, whereas 18 patients (23.1%) said they deviated from it at least once. No one stated an autonomous reduction of dosage. 59 of our participants received the additional question on the 30 min intake interval. Of those, 18 (30.5%) patients never deviated from their intake time more than 30 min, whereas 22 (37.3%) delayed their intake once per months and 19 (32.2%) said it happened at least every other week or more often. On the patients’ VAS, self-reported adherence ranged from 10% to 100%, with a mean of 92.03% (SD: 13.51). 81.8% of all patients rated themselves to have an adherence of 90% or better.

The IS trough level variability (CV%) ranged from 4.4% to 64.5% with a mean of 21.1% (SD: 10.2). Of the 78 participants, 12 (15.4%) exhibited a IS trough level variability (CV%) > 30, 65 (84.3%) stayed below this threshold, while one patient (1.3%) was excluded from the analysis due to missing data. In conformity with the recommendation of triangulation in order to increase the sensitivity of measurement methods [9, 15, 34, 37], we combined the dichotomous scores of physicians’ estimates, self-report, and IS trough levels to attain a composite NA score. We found 31 (40.8%) patients who had at least one rating of NA in any of these three categories.

Association of Measurement Methods at the beginning of the study (t0)

There was no association between the physicians’ estimates (VAS) and self-reports (VAS) of adherence (ICC = .196, p = .174). Similar results appeared regarding the dichotomous classification of adherence, revealing no agreement of the respective assessments: Only 5 (6.6%) patients were rated non-adherent by their treating nephrologist, while a total of 22 patients (28.21%) rated themselves as non-adherent. Of those, only 3 (13.64%) were classified accordingly by their physician, whereas the remaining 2 were rated adherent (k = .130, p = .108). Equally, IS trough level variability (dichotomous) was not significantly associated with physician’s estimates (k = .158, p = .120): Only 2 of the 5 patients rated as non-adherent had an increased IS trough level variability. Neither was the continuous CV% associated with the physicians’ estimates (VAS, ICC = −.003, p = .701). The dichotomous IS trough level variability was also not associated with the dichotomous self-rating of BAASIS© (k = .039, p = .715): Only 4 of the 22 who rated themselves non-adherent showed a CV% above the critical threshold. The continuous scores of CV% and self-report (VAS) did also not show any correlation (ICC = .014, p = .134). An overview of the associations between the different measurement methods at the beginning of the study (t0) can be viewed in Table 2.

Table 2 Association of adherence measurement methods at t0

Prevalence of electronically monitored and self-reported non-adherence during the study (t1-t6)

During the course of the study (t1-t6), the NA rate measured by BAASIS© (patients who agreed to any of the four items, two weeks recall) varied between 9.3% (t2) and 20.5% (t6). 70.3% of all participants reported full adherence throughout the whole study. 29.7% claimed NA at least once during study course. Taking NA alone was stated by 1.3% (t3) to 5.6% (t5) of all patients. A deviation from the ±30 min interval, which included all incidences of taking and timing NA ±2 h, was confirmed by 38.2% (t1) – 51.4% (t5) participants and only 23.2% reported full timing adherence ±30 min during the whole study. When extending the recall period to 4 weeks, equal to the original BAASIS© (by adding t1 + t2, t3 + t4, and t5 + t6, respectively), the prevalence rates were as follows: 10.8% (t1 + t2), 17.1% (t3 + t4), and 26.8% (t5 + t6).

The percentages of self-reported adherence remained relatively high throughout the whole study. Self-reported TA showed a relatively stable mean with a minimum of 99.47% (±2.3) at t5 and a maximum of 99.8% (±1.2) at t2 with minimal ranges (85.7–100%) during the study (t1-t6). Drug holiday never occurred. The percentage of self-reported TM ± 2 h (including TA) ranged from 21.4% to 100% from t1-t6 while the mean percentage per measure stayed continuously high with a minimum mean of 97.7% (±9.4) at t6 and a maximum mean of 99.4% (±1.7) at t4. The percentage of self-reported TM ± 30 min (including TA and TM ± 2 h) displayed a wide range from 0% to 100% during study course, whilst the mean still only reached a minimum of 94.3% (±12.7) at t5 and a maximum 96.1% (±6.6) at t1. Dose reduction did not occur from t1-t5, but occurred in two cases (2.7%) at t6, whereas 97.3% complied with their prescribed dosage.

Electronically monitored adherence showed similarly high percentages. TA showed a minimum percentage of 85.7% (t6) with constantly high means ranging from 98.9% (±2.4) at t6 to 99.6% (±1.4) at t1. The percentage minimum of electronically monitored TM ± 2 h (including TA) was 61.9% (t2), whilst it displayed constantly high means ranging between 97.2% (±4.9) at t6 and 99.0% (±4.6) at t2. TM ± 30 min dropped to a minimum percentage of 0% (t4, t5). Means again remained high with a range between 91.2% (±14.9) at t6 and 94.0% (±19.1) at t2. Corresponding figures can be found in the Additional files (1, 2, 3, 4, 5 and 6).

Associations of self-reported and electronically monitored non-adherence during the study (t1-t6)

The associations between self-reported NA and electronically monitored NA are based on absolute frequencies of the respective deviations and are depicted in Table 3. The correlations between Taking NA ranged from a fair correlation at T3 (ICC = .493) to a good agreement at T2 (ICC = .739). The correlations of Timing NA ±2 h ranged from ICC = .50 (T3, T6) to ICC = .895, while those for Timing NA ±30 min correlated poorly at T3 (ICC = .394) and very high at T4 (ICC = .905). In total, the correlation of methods was highest for Timing NA ±30 min (ICC = .879) and lowest for Timing NA ±2 h (ICC = .634). It has to be noted that the absolute frequencies (Total ∑T1-T6) of electronically monitored NA had constantly higher means than self-reported NA (Taking NA: .83 vs. .26, Timing NA ±2 h: 2.19 vs. 0.94, Timing NA ± 30 min: 8.44 vs. 5.8).

Table 3 Correlation of self-reported and electronically monitored non-adherence

As depicted in Table 4, linear regression analyses revealed only self-reported adherence (VAS) as a significant predictor for the percentage of electronically monitored TA, TM ± 2 h, and TM ± 30 min, respectively.

Table 4 Linear regression analysis

Effect of measurement and change of non-adherence over time (t1-t6)

The linear mixed models confirmed a significant decrease in the probability of electronically monitored TA (p < .001), TM ± 2 h (p < .001) as well as TM ± 30 min (p < .001) during the course of the study (Table 5). Figs. 2 and 3 depict the steady decrease of electronically monitored TM ± 30 min. Although the piecewise linear model did not reach significance for any specific measurement points, there was a declining trend in the probability of electronically monitored TM ± 30 min from T3 to T4 (p = .07). A slight increase, however, could be witnessed from T5 to T6 (Fig. 2), indicating again a slight rise in adherence at the end of the study compared to T5. No significance could be reached, though (p = .31).

Table 5 Linear and piecewise linear mixed models on the change of adherence over time
Fig. 2
figure 2

Percentage Frequency of Timing Adherence ±30 min – Electronic Monitoring (raw scores)

Fig. 3
figure 3

Estimated probability of Timing Adherence ±30 min – Electronic Monitoring

For the self-assessed intervention effect, no significant change over time could be confirmed. Its mean ranged between 1.7 (±3.6) at t2 and 2.3 (±3.8) at t6. All corresponding figures are available in the Additional files (7, 8 and 9).

Discussion

Prevalence of non-adherence and Association of Measurement Methods at the beginning of the study (t0)

This study prospectively examined the concordance and accuracy of different measurement methods of immunosuppressive NA in RTRs with special focus on self-reported and electronically monitored NA. In our study, the prevalence of self-reported NA was 28.6% when measured with BAASIS© at t0. Similar prevalence rates of self-reported NA can be found in other studies using BAASIS©, displaying ranges between 27% and 36.6% [10, 11, 46]. For the self-reported NA via VAS, we found a mean of 92.03% skewing towards highly adherent individuals. This also corresponds to previous results that found a median of 99% with an interquartile range of 10 [46]. Physicians’ estimates resulted in a NA prevalence of 6.4% which is similar to results previously found in other studies, with rates of 3.8% [36] and 9.2% [11]. IS trough level variability displayed a mean of 21.1% in our study which is also comparable to other studies that found CVs% of 18.15% [41], 21.26% [10] and 17.0% [18], respectively. 15.4% of our cohort displayed a CV% > 30. In the study by Rodrigo et al. [20], up to 37.4% of patients were found to have a CV% above 30, though. It must be kept in mind that there is no standardized threshold of CV% that has reliably been associated with adverse outcomes, yet. It still varies considerably in different studies and makes further comparisons challenging [19, 20, 41,42,43]. Our composite NA score was 40.8%. Other studies that combined different measures reached NA rates up to 70.5% [37], however this specific study displayed high NA rates for each single measure, 46.6% for BAASIS, 38.6% for collateral reports, and 34.1% for sub-therapeutic IS trough levels, respectively. Another study found 26.4% non-adherent and 36.8% partially non-adherent patients when combining self-reports and collateral reports of NA [36]. However, different NA measures were used in both cases.

Our results suggest no agreement between the different NA measures applied at t0 which is well in line with previous studies exhibiting no association between various instruments whatsoever [10, 11, 34, 36]. Especially in case of high trough level variability, no equivalence to other measures is likely. Although NA might contribute to a high variability, many other factors influence this parameter, for instance metabolism, nutrition, drug interactions, and “white-coat adherence” (better adherence before clinic visits) [9, 11]. Our results underscore the assumption that each measurement method functions as a partial indicator for NA and captures different aspects thereof [10, 34].

Prevalence of electronically monitored and self-reported non-adherence during the study (t1-t6)

During our study, self-reported NA by BAASIS© (two weeks recall) ranged between 9.3% and 20.5% from t1 to t6. When extending the recall period to 4 weeks by adding the respective time points, prevalence rates were 10.8%, 17.1%, and 26.8%, respectively. This rate is lower than at t0 (28.6%) as well as in previously reported literature (27 to 36.6%) [10, 11, 46]. This might be attributable to a potential intervention effect by the study, resulting in a rapid decline of self-reported NA in the first month of the study and a subsequent gradual rise during the following weeks. A comparable effect has been observed in previous studies [28, 29].

Further, self-reported Taking NA remained relatively low from t1 to t6 (1.3–5.6%), indicating that timing deviations of ±2 h constituted the largest part of NA when measured by BAASIS©. When we included timing deviations of ±30 min, NA rose to 38.2–51.4% from t1 to t6. These results are in line with literature displaying much lower rates for Taking NA than Timing NA, as well as only rarely occurring events of drug holiday and dose reductions [36, 46].

Despite these rates of NA, the percentages of self-reported Taking and Timing adherence remained high throughout the whole study. The same occurred for electronically monitored adherence, with a minimum mean of 91.2% for the strictest adherence measure (TM ± 30 min). Previous research [14, 47] found similar high rates of adherence when using EM. NA rates seem to depend on the mode of measure (percentage, count of NA incidents) as well as the definition of the NA threshold (TA, TM ± 2 h, TM ± 30 min respectively) which seem crucial to distinguish when comparing NA rates across studies.

Associations of self-reported and electronically monitored non-adherence during the study (t1-t6)

The association between self-reported NA and electronically monitored NA in RTRs has only sparsely been examined. Studies undertaking this attempt only received modest results [14, 34]. We used the absolute frequencies of NA incidence for both self-reports and EM in order to receive the highest possible accuracy. Throughout the study, the association between self-reports and EM remained moderate to high. Overall, highest correlations could be found for TM ± 30 min. As suggested by previous literature [32], self-reports and EM seem to produce similar results on NA which leads to the conclusion that they measure the same construct. Since the highest accuracy was reached for the strictest measure of NA (TM ± 30 min), it can be assumed that small deviations from the dosing schedule are generally reported by patients. Lower correlations for TA or TM ± 2 h might occur due to social desirability [8, 13], as major deviations from the medication regimen might rather want to be disguised than smaller ones. However, it was suggested that disclosure of NA to a researcher could be more accurate than to clinical staff [12] and hence our results could also display genuine reports of NA. Low correlations could therefore also result from missed diary entries. In these cases, NA might have been registered by EM, even though the IS was taken from another source [28, 39]. Another possibility is electronic failure, since EM devices may fail to register openings due to technical malfunctions [28] or impaired cellular reception.

We further found self-reported NA (VAS) to be the only significant predictor of the percentage of electronically monitored adherence. This is in line with previous research that also found self-reports to be predictive of electronically monitored NA [12, 14]. IS trough level variability and physicians’ estimates on NA (VAS) seem to play a subordinate role.

Effect of measurement and change of non-adherence over time (t1-t6)

Our results revealed a steadily decreasing probability of electronically monitored adherence over time, both for TA, TM ± 2 h, and TM ± 30 min. As discussed earlier, this decrease could reflect the waning of an intervention effect induced by the study design. Applying EM as a NA measure was previously associated with an interference of normal drug intake in the first month of monitoring. It was shown that NA was very low in the beginning, but was shortly followed by an increase of NA until it stabilized after 35 to 40 days [28, 29]. In our study the probability of adherence continuously decreased until the end of the study displaying no stabilization of adherence behavior. However, we found a trend towards a significant change in intake behavior after t3 in TM ± 30 min, suggesting a shift in adherence after 4–6 weeks (28–42 days respectively). Our results are not fully comparable to previous studies, though, since we depicted dichotomized adherence behavior during a two week interval, while other research focused on dichotomized intake behavior for each single day [28, 29]. Further, there was no self-assessed intervention effect. This might be due to subtle changes in adherence behavior over a period of three months which were not consciously observed by the respective patients.

Summary, conclusions and limitations

This study examined the concordance of different adherence measurement methods in RTRs with special focus on prospective measures of self-reported and electronically monitored NA. We found no correlation between IS trough levels, physicians’ estimates, and self-reports. Self-reports however, were found to be moderately to highly concordant with EM which allows for the conclusion that they both measure NA equally. Methodological flaws cannot be excluded for both methods (memory bias, social desirability vs. electronic malfunction, misuse of device [8, 13, 28, 30, 31]), though. We conclude that self-reports are equally accurate as EM, but seem to be superior to EM in clinical practice due to its practicability and economy [8, 14, 15]. EM on the other side can be viewed as a valid tool in research for gaining greater insight in the dynamics of NA [8, 15]. The possibility of an intervention effect induced by EM must be kept in mind when measuring NA. Surveillance of IS intake via EM with intermittent phone calls could improve adherence on a short-term basis. To establish long-term effects, further research is necessary.

Certain limitations of our study need to be mentioned. Although we assessed the consistency of diary use, we did not eliminate cases in which diary keeping was partially incomplete [28]. There might have been incidents in which patients indeed took their medication but forgot to use the diary. For these patients, we rather measured adherence to pillbox and diary use than actual adherence to IS medication. Better diary keeping could be intrinsic to patients that are already more adherent [39] which could increase the gap between adherent and non-adherent patients. In further studies, it is recommendable to evaluate both the patients’ acceptance and intention of future pillbox and diary use before enrollment. If necessary, adequate adjustments to the patient’s routine could be made by replacing pillboxes by pill bottles [39, 48] or diaries by mobile apps. Generalization of results must also be treated with caution. A responder bias towards adherent patients cannot be excluded, since we had a highly adherent population with low variation in data. Replication of our results in a more non-adherent population is necessary. Future studies should also focus more strongly on the influence of biological aspects on trough level variability when using this parameter as a measure of adherence.

Availability of data and materials

The data supporting our findings can be requested from Dipl.-Psych. Marietta Lieb (marietta.lieb@uk-erlangen.de) and Prof. Yesim Erim (yesim.erim@uk-erlangen.de).

Abbreviations

TA:

Taking Adherence

TM ± 2 h:

Timing Adherence ±2 h

TM ± 30 min:

Timing Adherence ±30 min

VAS:

Visual Analogue Scale

NA:

Non-Adherence

IS:

Immunosuppressant/immunosuppressive

RTR:

Renal Transplant Recipient

LMM:

Linear Mixed Model

ICC:

Intraclass Correlation Coefficient

ns:

Not significant

EM:

Electronic Monitoring

References

  1. Vrijens B, De Geest S, Hughes DA, Przemyslaw K, Demonceau J, Ruppar T, et al. A new taxonomy for describing and defining adherence to medications. Br J Clin Pharmacol. 2012;73(5):691–705.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Vlaminck H, Maes B, Evers G, Verbeke G, Lerut E, Van Damme B, et al. Prospective study on late consequences of subclinical non-compliance with immunosuppressive therapy in renal transplant patients. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2004;4(9):1509–13.

    Article  Google Scholar 

  3. Russell CL, Ashbaugh C, Peace L, Cetingok M, Hamburger KQ, Owens S, et al. Time-in-a-bottle (TIAB): a longitudinal, correlational study of patterns, potential predictors, and outcomes of immunosuppressive medication adherence in adult kidney transplant recipients. Clin Transpl. 2013;27(5):E580–90.

    Article  Google Scholar 

  4. Prihodova L, Nagyova I, Rosenberger J, Majernikova M, Roland R, Groothoff JW, et al. Adherence in patients in the first year after kidney transplantation and its impact on graft loss and mortality: a cross-sectional and prospective study. J Adv Nurs. 2014;70(12):2871–83.

    Article  PubMed  Google Scholar 

  5. Denhaerynck K, Dobbels F, Cleemput I, Desmyttere A, Schafer-Keller P, Schaub S, et al. Prevalence, consequences, and determinants of nonadherence in adult renal transplant patients: a literature review. Transpl Int. 2005;18(10):1121–33.

    Article  PubMed  Google Scholar 

  6. Dew MA, DiMartini AF, De Vito DA, Myaskovsky L, Steel J, Unruh M, et al. Rates and risk factors for nonadherence to the medical regimen after adult solid organ transplantation. Transplantation. 2007;83(7):858–73.

    Article  PubMed  Google Scholar 

  7. Belaiche S, Decaudin B, Dharancy S, Noel C, Odou P, Hazzan M. Factors relevant to medication non-adherence in kidney transplant: a systematic review. Int J Clin Pharm. 2017;39(3):582–93.

    Article  PubMed  Google Scholar 

  8. Foster BJ, Pai ALH. Adherence in adolescent and Young adult kidney transplant recipients. Open Urology Nephrology J. 2014;7:133–43.

    Article  Google Scholar 

  9. Osterberg L, Blaschke T. Adherence to medication. N Engl J Med. 2005;353(5):487–97.

    Article  CAS  PubMed  Google Scholar 

  10. Scheel J, Reber S, Stoessel L, Waldmann E, Jank S, Eckardt KU, et al. Patient-reported non-adherence and immunosuppressant trough levels are associated with rejection after renal transplantation. BMC Nephrol. 2017;18(1):107.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Pabst S, Bertram A, Zimmermann T, Schiffer M, de Zwaan M. Physician reported adherence to immunosuppressants in renal transplant patients: prevalence, agreement, and correlates. J Psychosom Res. 2015;79(5):364–71.

    Article  PubMed  Google Scholar 

  12. Butler JA, Peveler RC, Roderick P, Horne R, Mason JC. Measuring compliance with drug regimens after renal transplantation: comparison of self-report and clinician rating with electronic monitoring. Transplantation. 2004;77(5):786–9.

    Article  PubMed  Google Scholar 

  13. Nevins TE, Nickerson PW, Dew MA. Understanding medication nonadherence after kidney transplant. J Am Soc Nephrol. 2017;28(8):2290–301.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Denhaerynck K, Steiger J, Bock A, Schafer-Keller P, Kofer S, Thannberger N, et al. Prevalence and risk factors of non-adherence with immunosuppressive medication in kidney transplant patients. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2007;7(1):108–16.

    Article  CAS  Google Scholar 

  15. Dobbels F, Berben L, De Geest S, Drent G, Lennerling A, Whittaker C, et al. The psychometric properties and practicability of self-report instruments to identify medication nonadherence in adult transplant patients: a systematic review. Transplantation. 2010;90(2):205–19.

    Article  PubMed  Google Scholar 

  16. Liu J, Liu S, Yan J, Yi Q, Huang H. Adherence to immunosuppressive medication in renal transplant recipients from follow-up outpatient in China: association of 2 different measurement methods. Clin Ther. 2015;37(11):2572–80.

    Article  PubMed  Google Scholar 

  17. Hsiau M, Fernandez HE, Gjertson D, Ettenger RB, Tsai EW. Monitoring nonadherence and acute rejection with variation in blood immunosuppressant levels in pediatric renal transplantation. Transplantation. 2011;92(8):918–22.

    Article  CAS  PubMed  Google Scholar 

  18. Huang CT, Shu KH, Ho HC, Wu MJ. Higher variability of Tacrolimus trough level increases risk of acute rejection in kidney transplant recipients. Transplant Proc. 2016;48(6):1978–80.

    Article  CAS  PubMed  Google Scholar 

  19. Pizzo HP, Ettenger RB, Gjertson DW, Reed EF, Zhang J, Gritsch HA, et al. Sirolimus and tacrolimus coefficient of variation is associated with rejection, donor-specific antibodies, and nonadherence. Pediatr Nephrol. 2016;31(12):2345–52.

    Article  PubMed  Google Scholar 

  20. Rodrigo E, Segundo DS, Fernandez-Fresnedo G, Lopez-Hoyos M, Benito A, Ruiz JC, et al. Within-patient variability in Tacrolimus blood levels predicts kidney graft loss and donor-specific antibody development. Transplantation. 2016;100(11):2479–85.

    Article  CAS  PubMed  Google Scholar 

  21. Williams A, Low JK, Manias E, Dooley M, Crawford K. Trials and tribulations with electronic medication adherence monitoring in kidney transplantation. Res Social Adm Pharm. 2016;12(5):794–800.

    Article  PubMed  Google Scholar 

  22. Foster BJ, Pai A, Zelikovsky N, Amaral S, Bell L, Dharnidharka VR, et al. A randomized trial of a multicomponent intervention to promote medication adherence: the teen adherence in kidney transplant effectiveness of intervention trial (TAKE-IT). Am J Kidney Dis. 2018;72(1):30–41.

  23. Panyi N, Erim Y. Manualisierte Psychotherapie zur Optimierung der Immunsuppressiva-Adhärenz bei Patienten nach Nierentransplantation: Ergebnisse einer Machbarkeitsstudie. Z Psychosom Med Psychother. 2017;63(2):189–201.

    PubMed  Google Scholar 

  24. Reese PP, Bloom RD, Trofe-Clark J, Mussell A, Leidy D, Levsky S, et al. Automated reminders and physician notification to promote immunosuppression adherence among kidney transplant recipients: a randomized trial. Am J kidney Dis. 2017;69(3):400–9.

    Article  PubMed  Google Scholar 

  25. Nevins TE, Robiner WN, Thomas W. Predictive patterns of early medication adherence in renal transplantation. Transplantation. 2014;98(8):878–84.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. Russell C, Conn V, Ashbaugh C, Madsen R, Wakefield M, Webb A, et al. Taking immunosuppressive medications effectively (TIMELink): a pilot randomized controlled trial in adult kidney transplant recipients. Clin Transpl. 2011;25(6):864–70.

    Article  Google Scholar 

  27. Russell CL, Moore S, Hathaway D, Cheng AL, Chen G, Goggin K. MAGIC study: aims, design and methods using SystemCHANGE to improve immunosuppressive medication adherence in adult kidney transplant recipients. BMC Nephrol. 2016;17(1):84.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Denhaerynck K, Schafer-Keller P, Young J, Steiger J, Bock A, De Geest S. Examining assumptions regarding valid electronic monitoring of medication therapy: development of a validation framework and its application on a European sample of kidney transplant patients. BMC Med Res Methodol. 2008;8:5.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Deschamps AE, Van Wijngaerden E, Denhaerynck K, De Geest S, Vandamme AM. Use of electronic monitoring induces a 40-day intervention effect in HIV patients. J Acquir Immune Defic Syndr. 2006;43(2):247–8.

    Article  PubMed  Google Scholar 

  30. De Bleser L, De Geest S, Vandenbroeck S, Vanhaecke J, Dobbels F. How accurate are electronic monitoring devices? A laboratory study testing two devices to measure medication adherence. Sensors. 2010;10:1652–60.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Riekert KA, Rand CS. Electronic monitoring of medication adherence: when is high-tech best? J Clin Psychol Med Settings. 2002;8(1):25–34.

    Article  Google Scholar 

  32. Monnette A, Zhang Y, Shao H, Shi L. Concordance of Adherence Measurement Using Self-Reported Adherence Questionnaires and Medication Monitoring Devices: An Updated Review. PharmacoEconomics; 2017.

  33. El Alili M, Vrijens B, Demonceau J, Evers SM, Hiligsmann M. A scoping review of studies comparing the medication event monitoring system (MEMS) with alternative methods for measuring medication adherence. Br J Clin Pharmacol. 2016;82(1):268–79.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Schäfer-Keller P, Steiger J, Bock A, Denhaerynck K, De Geest S. Diagnostic accuracy of measurement methods to assess non-adherence to immunosuppressive drugs in kidney transplant recipients. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2008;8(3):616–26.

    Article  Google Scholar 

  35. Lalic J, Velickovic-Radovanovic R, Mitic B, Paunovic G, Cvetkovic T. Immunosuppressive medication adherence in kidney transplant patients. Med Princ Pract. 2014;23(4):351–6.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Schmid-Mohler G, Thut MP, Wuthrich RP, Denhaerynck K, De Geest S. Non-adherence to immunosuppressive medication in renal transplant recipients within the scope of the integrative model of behavioral prediction: a cross-sectional study. Clin Transpl. 2010;24(2):213–22.

    Article  Google Scholar 

  37. Silva AN, Moratelli L, Tavares PL, Marsicano EO, Pinhati RR, Colugnati FA, et al. Self-efficacy beliefs, locus of control, religiosity and non-adherence to immunosuppressive medications in kidney transplant patients. Nephrology. 2016;21(11):938–43.

    Article  PubMed  Google Scholar 

  38. De Geest S, Zullig LL, Dunbar-Jacob J, Helmy R, Hughes DA, Wilson IB, et al. ESPACOMP medication adherence reporting guideline (EMERGE). Ann Intern Med. 2018;169(1):30–5.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Bova CA, Fennie KP, Knafl GJ, Dieckhaus KD, Watrous E, Williams AB. Use of electronic monitoring devices to measure antiretroviral adherence: practical considerations. AIDS Behav. 2005;9(1):103–10.

    Article  PubMed  Google Scholar 

  40. De Bleser L, Dobbels F, Berben L, Vanhaecke J, Verleden G, Nevens F, et al. The spectrum of nonadherence with medication in heart, liver, and lung transplant patients assessed in various ways. Transpl Int. 2011;24(9):882–91.

    Article  PubMed  Google Scholar 

  41. Goodall DL, Willicombe M, McLean AG, Taube D. High Intrapatient variability of Tacrolimus levels and outpatient clinic nonattendance are associated with inferior outcomes in renal transplant patients. Transplant Direct. 2017;3(8):e192.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. O'Regan JA, Canney M, Connaughton DM, O'Kelly P, Williams Y, Collier G, et al. Tacrolimus trough-level variability predicts long-term allograft survival following kidney transplantation. J Nephrol. 2016;29(2):269–76.

    Article  CAS  PubMed  Google Scholar 

  43. Abu Bakar K, Mohamad NA, Hodi Z, McCulloch T, Williams A, Christian M, et al. Defining a threshold for tacrolimus intra-patient variability associated with late acute cellular rejection in paediatric kidney transplant recipients. Pediatr Nephrol. 2019;34:2557–62.

  44. Pinheiro J, Bates D, DebRoy S, Sarkar D, Team RC. Nlme: linear and nonlinear mixed effect models. R Package Version. 2014;3:1–117.

    Google Scholar 

  45. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48.

    Article  Google Scholar 

  46. Massey EK, Tielen M, Laging M, Beck DK, Khemai R, van Gelder T, et al. The role of goal cognitions, illness perceptions and treatment beliefs in self-reported adherence after kidney transplantation: a cohort study. J Psychosom Res. 2013;75(3):229–34.

    Article  PubMed  Google Scholar 

  47. Leino AD, King EC, Jiang W, Vinks AA, Klawitter J, Christians U, et al. Assessment of tacrolimus intrapatient variability in stable adherent transplant recipients: establishing baseline values. Am J Transplant Off J Am Soc Transplant Am Soc Transplant Surg. 2019;19(5):1410–20.

    Article  CAS  Google Scholar 

  48. Kalichman SC, Cain D, Cherry C, Kalichman M, Pope H. Pillboxes and antiretroviral adherence: prevalence of use, perceived benefits, and implications for electronic medication monitoring devices. AIDS Patient Care STDs. 2005;19(12):833–9.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We wish to thank the staff of the nephrologic outpatient clinic at the University Hospital of Erlangen for enabling the organization and conduction of this project. We especially would like to thank all patients for taking part in our study.

The present work was performed in (partial) fulfillment of the requirements for obtaining the degree „Dr. rer. Biol. hum. “of Marietta Lieb.

Funding

No funding was obtained for the conduction of this project.

Author information

Authors and Affiliations

Authors

Contributions

ML designed and conducted the study, processed statistical data and drafted the manuscript. TH supported the processing and analysis of statistical data. YE supervised and supported the conceptualization and conduction of the study. MS and MO enabled and supported the realization of the study. All authors (ML, TH, YE, MS, MO) revised the manuscript critically for important intellectual content, have given final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Authors’ information

ML works as research assistant in the Department of Psychosomatic Medicine and Psychotherapy, Friedrich-Alexander-University Erlangen-Nürnberg (marietta.lieb@uk-erlangen.de). TH works as statistician in the Institute of Medical Informatics, Biometry and Epidemiology, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) (tobias.hepp@uk-erlangen.de). MS is head of the Department of Nephrology and Hypertension, University Hospital of Erlangen. MO is senior physician at the Department of Nephrology and Hypertension, University Hospital of Erlangen. YE is the head of the Department of Psychosomatic Medicine and Psychotherapy, Friedrich-Alexander-University Erlangen-Nürnberg (yesim.erim@uk-erlangen.de).

Corresponding author

Correspondence to Marietta Lieb.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the ethics committee of the University Hospital Erlangen. All patients gave written informed consent to participate. No organs/tissues were obtained from participants.

Consent for publication

Not applicable (Our manuscript contains no individual person’s data in any form, consent to publish therefore does not have to be obtained).

Competing interests

All authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Percentages of Taking Adherence t1-t6 – Self-Reports

Additional file 2.

Percentages of Timing Adherence ±2 h t1-t6 – Self-Reports

Additional file 3.

Percentages of Timing Adherence ±30 min t1-t6 – Self-Reports

Additional file 4.

Percentages of Taking Adherence t1-t6 – Electronic Monitoring

Additional file 5.

Percentages of Timing Adherence ±2 h t1-t6 – Electronic Monitoring

Additional file 6.

Percentages of Timing Adherence ±30 min - Electronic Monitoring

Additional file 7.

a: Percentage Frequency of Taking Adherence - Electronic Monitoring (raw scores). 8 b: Estimated probability of Taking Adherence - Electronic Monitoring.

Additional file 8.

a: Percentage Frequency of Timing Adherence ±2 h – Electronic Monitoring (raw scores). 8 b: Estimated probability of Timing Adherence ±2 h – Electronic Monitoring.

Additional file 9.

a: Course of unadjusted means of the self-assessed intervention effect. 9 b: Boxplots for the self-assessed intervention effect.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lieb, M., Hepp, T., Schiffer, M. et al. Accuracy and concordance of measurement methods to assess non-adherence after renal transplantation - a prospective study. BMC Nephrol 21, 114 (2020). https://doi.org/10.1186/s12882-020-01781-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12882-020-01781-1

Keywords