Psychometric properties of instruments for assessing adherence to oral antineoplastic agents: a COSMIN systematic review

Search strategy

Four English databases (PubMed, EMBASE, Web of Science, and Cumulative Index to Nursing and Allied Health Literature (CINAHL)) and four Chinese databases (China National Knowledge Infrastructure (CNKI), Wanfang, Weipu, and Sinomed) were searched for relevant studies published from the database inception to December 31, 2023. A search strategy (Table S2) consisting of medication adherence, cancers, oral administration, PROMs, and measurement properties was used. The search filters (Terwee et al., 2009) were also used to enhance the sensitivity of searches, where available. Additionally, all retrieved papers’ reference lists were manually searched for potential studies.

Inclusion and exclusion criteria

The inclusion criteria were as follows: (1) studies that focused on the adherence to OAAs, including endocrine, cytotoxic, targeted therapies and immunomodulators across cancer types; (2) the target population consisted of adult patients (age > 18 years) diagnosed with cancer; (3) studies aiming to develop or validate the PROMs; (4) studies describing at least one psychometric property listed in the COSMIN; (5) full-text articles are available.

The exclusion criteria included the following: (1) studies that included mainly oral agents other than OAAs (such as analgesics and antiemetics); (2) studies in which a PROM was solely used to measure an outcome (e.g., randomized clinical trials); (3) repeated or overlapping publication.

Study selection

We imported all the references searched and removed duplicates in EndNote X9. The formal screening was conducted in April 2024. Two reviewers (MMS, KHH), trained in evidence-based methodologies, independently screened the titles and abstracts of the references according to the predefined eligibility criteria and then retrieved the full text of articles that may have wholly or potentially met the eligibility criteria. A third reviewer (SXL) resolved disagreements.

Data extraction

Two reviewers (MMS, KHH) performed the data extraction independently in June 2024. A third reviewer (SXL) resolved disagreements. A standardized extraction table was utilized, which included the details of the studies (first author, published year, country or region, the PROM used, participants’ characteristics, study design, response rate, and article’s language), characteristics of the included PROMs (item generation, dimensions and items, target population, range of scores, mode of administration, response options, original language/translation, and measurement properties). Original authors were contacted for missing or unreported data.

Appraisal of methodological quality and measurement properties

Two reviewers (MMS, KHH) independently appraised the methodological quality and measurement properties of the PROMs. A third reviewer (SXL) resolved disagreements.

The methodological quality of the eligible studies was rated using the COSMIN Risk of Bias checklist (Mokkink et al., 2018). The checklist consists of ten boxes: PROM development, content validity, structural validity, internal consistency, cross-cultural validity/measurement invariance, reliability (test–retest), measurement error, criterion validity, hypothesis testing for construct validity, and responsiveness. The methodological quality of items in each box was rated as “very good,” “adequate,” “doubtful,” or “inadequate.” The quality of the criteria for good measurement properties of each PROM was rated as sufficient (+), insufficient (-), inconsistent (±), or indeterminate (?), according to the proposed criteria (Mokkink et al., 2018; Prinsen et al., 2018).

Since the COSMIN does not provide evaluation criteria for assessing structural validity using exploratory factor analysis (EFA), we considered a sufficient result when the total variance explained was at least 60% (Wu, 2010). In addition, we suggested that the statistical results from assessing the relationship between the PROM and the direct objective measures should be viewed as a criterion validity result when evaluating criterion validity (Oliveira et al., 2023).

Grading the quality of evidence and formulate recommendations

Two reviewers (MMS, KHH) independently graded the quality of the evidence of all studies according to the adapted version of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE), proposed by the COSMIN initiative (Guyatt et al., 2011; Prinsen et al., 2018). A third reviewer (SXL) resolved disagreements. The quality of evidence for each measurement property was graded as “high”, “moderate”, “low”, or “very low”. COSMIN recommends categorizing the included PROMs into three categories (Prinsen et al., 2018): PROMs with evidence for sufficient content validity (at any level) and at least low-quality evidence for sufficient internal consistency can be categorized as “strongly recommended”; PROMs with high-quality evidence for an insufficient psychometric property should be categorized as “not recommended”; and other situations are categorized as “weakly recommended.”

Literature search

A total of 1,985 records were initially identified from the databases. After removing duplicates, title/abstract screening, full-text reviews, and additional searches, eight studies and eight instruments were eventually included in the systematic review (see Fig. 1).

Characteristics of included study and PROMs

Table 1 presents detailed information on the studies that were included. Eight studies were published between 2013 and 2023. There were five studies reported in English (Bagcivan & Akbayrak, 2015; Daouphars et al., 2013; Gambalunga et al., 2022; Silveira et al., 2021; Talens et al., 2023), two in Chinese (Li, Sun & Dong, 2018; Qin et al., 2020), and one in French (Baudot et al., 2016). France (Baudot et al., 2016; Daouphars et al., 2013) and China (Li, Sun & Dong, 2018; Qin et al., 2020) each contributed two papers; Turkey (Bagcivan & Akbayrak, 2015), Brazil (Silveira et al., 2021), Italy (Gambalunga et al., 2022) and Spain (Talens et al., 2023) each published one paper. Except for two cohort studies (Baudot et al., 2016; Daouphars et al., 2013), the remaining studies were cross-sectional (Bagcivan & Akbayrak, 2015; Gambalunga et al., 2022; Li, Sun & Dong, 2018; Qin et al., 2020; Silveira et al., 2021; Talens et al., 2023). The sample sizes ranged from 46 to 306. The mean/median age ranged from 54.6 to 68.6 years. Most of the cancer diagnoses of the participants were breast cancer, gastrointestinal tract cancer, hematologic cancer, lung cancer, and genitourinary system cancer. The OAAs taken by the participants mainly included cytotoxic drugs (e.g., capecitabine, tegafur, etoposide, hydroxyurea, and vinorelbine), targeted drugs (e.g., imatinib, sunitinib, and ibrutinib), hormone drugs (e.g., tamoxifen, anastrozole, exemestane, letrozole, and abiraterone) and immunomodulator (e.g., lenalidomide, thalidomide, and pomalidomide). Three studies (Daouphars et al., 2013; Li, Sun & Dong, 2018; Talens et al., 2023) reported a PROM response rate ranging from 93.9% to 98.0%.

Table 2 describes detailed information on the included PROMs. Eight PROMs were identified, four of which were universal to cancer patients treated with any OAA: the Turkish-Version Oral Chemotherapy Adherence Scale (T-OCAS) (Bagcivan & Akbayrak, 2015), Chinese-Version Oral Chemotherapy Adherence Scale (C-OCAS) (Li, Sun & Dong, 2018), Morisky Medication Adherence Scale-8 (MMAS-8) (Qin et al., 2020), and Experience with and Adherence to Oral Antineoplastic Agents Scale (EXPAD-ANEO) (Talens et al., 2023). The other four instruments were only suitable for patients with specific types of cancer or patients using a particular kind of OAA: the Self-assessment Adherence Questionnaire (Daouphars et al., 2013) for patients with chronic myeloid leukemia treated with imatinib; Measuring Adherence and Management of Side Effects in Patients Treated with Capecitabine (Baudot et al., 2016) for patients treated with capecitabine; Treatment Adherence Measure (TAM) (Silveira et al., 2021) for patients with multiple myeloma treated with immunomodulators; Adherence-Breast Endocrine Therapy Questionnaire (A-BET) (Gambalunga et al., 2022) for patients with breast cancer treated with oral endocrine therapy. Although all identified instruments were validated among cancer patients taking OAAs, three PROMs (Daouphars et al., 2013; Qin et al., 2020; Silveira et al., 2021) were not initially developed for cancer patients taking OAAs: MMAS-8 was developed for patients with essential hypertension (Morisky, Green & Levine, 1986); TAM was designed for patients with chronic diseases (Delgado & Lima, 2001); and the Self-assessment Adherence Questionnaire was modeled after previously published questionnaires related to other disease states (Chisholm et al., 2005; Girerd et al., 2001). One PROM has Turkish (Bagcivan & Akbayrak, 2015) and Chinese versions (Li, Sun & Dong, 2018), which we treated as independent instruments because their items and subscale compositions varied. All instruments were either self-reported by patients through self-completed questionnaires or interview-based approaches. The summary of measurement properties for each PROM is presented in Table S3.

Methodological quality of included studies

Table 3 demonstrates the methodological quality results of the included studies. Five studies presented the PROM development process (Bagcivan & Akbayrak, 2015; Baudot et al., 2016; Daouphars et al., 2013; Gambalunga et al., 2022; Talens et al., 2023). The quality of development studies for two PROMs (Bagcivan & Akbayrak, 2015; Daouphars et al., 2013) was rated inadequate as no appropriate qualitative methods were used to evaluate relevant items for the instruments. The quality of three PROM development studies (Baudot et al., 2016; Gambalunga et al., 2022; Talens et al., 2023) was rated doubtful because each item was tested in an inappropriate number of patients or it was unclear if all items were tested in their final form. Five studies (Bagcivan & Akbayrak, 2015; Baudot et al., 2016; Gambalunga et al., 2022; Li, Sun & Dong, 2018; Talens et al., 2023) reported content validity, and all of them were rated adequate or doubtful because it was unclear if professionals from all required disciplines were included or each item was tested in an inappropriate number of patients or professionals.

For structural validity, one study (Talens et al., 2023) was rated very good owing to using confirmatory factor analysis (CFA), three studies (Bagcivan & Akbayrak, 2015; Li, Sun & Dong, 2018; Qin et al., 2020) were rated adequate with EFA conducted, and one study (Baudot et al., 2016) was rated inadequate with correlation coefficient reported. All included studies assessed internal consistency. Two studies (Li, Sun & Dong, 2018; Talens et al., 2023) were rated very good due to calculating Cronbach’s alpha or omega for each subscale. Three studies (Daouphars et al., 2013; Gambalunga et al., 2022; Silveira et al., 2021) were rated doubtful due to uncertainty if the scale was unidimensional. Three studies (Bagcivan & Akbayrak, 2015; Baudot et al., 2016; Qin et al., 2020) were rated inadequate because no Cronbach’s alpha was calculated for each internal subscale, or only the correlation coefficient between entries was calculated.

Four studies (Bagcivan & Akbayrak, 2015; Gambalunga et al., 2022; Li, Sun & Dong, 2018; Qin et al., 2020) reported reliability, but none calculated the intraclass correlation coefficient (ICC). Furthermore, none of the four studies specified whether the patients remained stable, if the conditions were comparable, or if the time interval was suitable, making the reliability doubtful. Criterion validity evaluated by three studies (Bagcivan & Akbayrak, 2015; Daouphars et al., 2013; Talens et al., 2023) was rated inadequate due to lack of the area under the receiver operating curve (AUC) or choosing another medication adherence tool for comparison. Hypothesis testing for construct validity assessed by three studies (Baudot et al., 2016; Silveira et al., 2021; Talens et al., 2023) was classified as very good or adequate. None of the PROMs explored measurement error, cross-cultural validity/measurement invariance, and responsiveness.

Quality of psychometric properties and evidence

Table 4 displays the quality of psychometric properties and evidence for each PROM. Two PROMs (Baudot et al., 2016; Talens et al., 2023) demonstrated sufficient quality for content validity. The quality of content validity of three PROMs (Bagcivan & Akbayrak, 2015; Gambalunga et al., 2022; Li, Sun & Dong, 2018) was rated indeterminate due to a lack of asking the patients about the relevance of the items or the patients and professionals about the comprehensiveness of the items. Two PROMs (Bagcivan & Akbayrak, 2015; Gambalunga et al., 2022) had high-quality evidence of content validity, three PROMs (Baudot et al., 2016; Li, Sun & Dong, 2018; Talens et al., 2023) had moderate-quality evidence, and one PROM (Daouphars et al., 2013) had very low-quality evidence owing to no content validity studies and the inadequate quality of the PROM development study.

Three PROMs (Li, Sun & Dong, 2018; Qin et al., 2020; Talens et al., 2023) had sufficient quality for structural validity, one PROM (Baudot et al., 2016) had indeterminate quality, and one PROM (Bagcivan & Akbayrak, 2015) had insufficient quality because the total variance explained was merely 43%. The quality of evidence for structural validity of one PROM (Talens et al., 2023) was high, whereas three PROMs (Bagcivan & Akbayrak, 2015; Li, Sun & Dong, 2018; Qin et al., 2020) were downgraded to moderate quality of evidence due to a serious risk of bias, and one PROM (Baudot et al., 2016) was downgraded to very low due to an extremely serious risk of bias.

One PROM (Talens et al., 2023) had insufficient internal consistency because the Omega coefficient was less than 0.7 for one of the factors. One PROM’s (Li, Sun & Dong, 2018) internal consistency was rated sufficient; the other six PROMS (Bagcivan & Akbayrak, 2015; Baudot et al., 2016; Daouphars et al., 2013; Gambalunga et al., 2022; Qin et al., 2020; Silveira et al., 2021) were rated indeterminate because the criteria for “at least low evidence for sufficient structural validity” were not met, or Cronbach’s alpha for each subscale was not reported. The quality of evidence for internal consistency of two PROMs (Li, Sun & Dong, 2018; Talens et al., 2023) was high, one PROM (Bagcivan & Akbayrak, 2015) was downgraded to low due to a very serious risk of bias, and five PROMs (Baudot et al., 2016; Daouphars et al., 2013; Gambalunga et al., 2022; Qin et al., 2020; Silveira et al., 2021) were downgraded to very low due to a very serious risk of bias and imprecision.

Four PROMs (Bagcivan & Akbayrak, 2015; Gambalunga et al., 2022; Li, Sun & Dong, 2018; Qin et al., 2020) failed to report ICC, leaving indeterminate quality for reliability. The quality of evidence for the reliability of one PROM (Bagcivan & Akbayrak, 2015) was downgraded to low due to a very serious risk of bias, and three PROMs (Gambalunga et al., 2022; Li, Sun & Dong, 2018; Qin et al., 2020) were downgraded to very low due to a very serious risk of bias and imprecision. Three PROMs (Bagcivan & Akbayrak, 2015; Daouphars et al., 2013; Talens et al., 2023) reported indeterminate quality for criterion validity owing to inappropriate selection of the gold standard or the failure to calculate AUC. The quality of evidence for the criterion validity of one PROM was downgraded to low due to a very serious risk of bias (Talens et al., 2023), and two PROMs (Bagcivan & Akbayrak, 2015; Daouphars et al., 2013) were downgraded to very low due to a very serious risk of bias and imprecision. The quality of hypothesis testing for construct validity for two PROMs (Baudot et al., 2016; Talens et al., 2023) was sufficient; one PROM (Silveira et al., 2021) was insufficient because only 50% of the hypothesis was confirmed. The quality of evidence for hypothesis testing for construct validity for three PROMs (Baudot et al., 2016; Silveira et al., 2021; Talens et al., 2023) was downgraded to moderate due to a serious risk of bias or a sample size of less than 100.

Recommended grade of PROMs

The results of the recommended grades for eight PROMs are shown in Table 4. Seven PROMs (Bagcivan & Akbayrak, 2015; Baudot et al., 2016; Daouphars et al., 2013; Gambalunga et al., 2022; Li, Sun & Dong, 2018; Qin et al., 2020; Silveira et al., 2021) were weakly recommended for use to assess OAA adherence, whereas one PROM (Talens et al., 2023) was not recommended for use to assess OAA adherence due to high-quality evidence for insufficient internal consistency.