Development and psychometric evaluation of a patient-reported symptom index for patients with non-muscle invasive bladder cancer: the NMIBC-SI

Abstract

Background and objective

Non-muscle invasive bladder cancer (NMIBC) is a chronic condition requiring frequent follow-up with endoscopic examinations, tumour resections and intravesical treatments. In this clinical context, patient-reported outcomes (PROs) have enormous potential to inform treatment assessment and recommendations for NMIBC. We aimed to develop and evaluate a patient-reported NMIBC Symptom Index (NMIBC-SI) to facilitate clinical research and enhance care.

Methods

NMIBC-SI items were developed based on existing literature and qualitative interviews with patients and clinicians, and evaluated in two field tests: item reduction, using NMIBC-SI data from 220 patients on active treatment from nine Australian centres; reliability and validity evaluation of item-reduced version using NMIBC-SI data from 232 patients from five countries.

Results

NMIBC-SI assesses disease and treatment-related symptom burden and two treatment-specific side-effects (cystoscopy, intravesical BCG/Chemotherapy). Composite analysis supported a single composite model including core symptom and cystoscopy index items (Intravesical index items were not tested due to small sample). Test-retest reliability was strong (range 0.894–0.91). As expected, the NMIBC-SI was able to discriminate between no treatment and any treatment groups, and no treatment and chemo/BCG groups, providing evidence towards validity.

Conclusions and clinical implications

NMIBC-SI assesses patients’ self-reported symptom burden and can be used to evaluate NMIBC treatments from the perspective of patients. The NMIBC-SI is acceptable to patients and has evidence for reliability and validity. Future validation work with patients with greater symptom burden is warranted.

Plain English summary

Patients with NMIBC require long-term monitoring with regular endoscopic examinations and various treatments. This research describes the development of the NMIBC-SI, a patient-completed questionnaire that assesses the impact of those treatments on health outcomes important to patients. The NMIBC-SI can be used in research and in the clinic to improve communication and treatment decisions.

Introduction

Bladder cancer (BC) is the ninth most common cancer worldwide with an estimated 614,298 people diagnosed in 2022 [1]. Most cases (70–80%) diagnosed are non-muscle invasive bladder cancer (NMIBC) [2], with 60–70% recurring, and 5-year progression to muscle invasion ranging from 1% for low risk to 21% for high risk [3], with a 40% probability of progression in a recently identified very high risk group [4]. For NMIBC, five year survival is > 80% [5], yielding prevalence 10 times greater than incidence rates [6]. Treatment for NMIBC involves endoscopic resection of bladder tumours, usually followed by intravesical chemotherapy or immunotherapy in recurrent and/or high-risk cases. Intensive follow-up is mandatory, with repeated endoscopic examinations ± bladder biopsy/tumour resection initially 3-monthly, decreasing according to risk and radiological imaging. Intravesical treatments are used when appropriate. Adjuvant therapies may reduce recurrence, but have sequelae such as flu-like symptoms, urinary frequency, urgency, fatigue, and dysuria. Follow-up examinations are recommended to detect and treat recurrences. In this context, patient-reported outcomes (PROs) including symptom burden, physical function, and health-related quality of life (HRQL) are important issues for patients and managing clinicians.

A PRO is a report coming directly from patients, without interpretation by another, about how they feel in relation to a health condition and its therapy [7]. The role of PRO measures (PROMs) is important in clinical trials that evaluate the effectiveness of treatment because they incorporate the patient’s perspective. In oncology, PROs are useful for improving service provision and informing treatment decisions [8]. PROs are particularly important where survival outcomes may not be the most relevant treatment outcome, particularly in chronic conditions such as NMIBC. They also offer a patient-centred measure of treatment impact, burden on patients and presence of symptoms or side-effects not assessable with clinical tests [9]. Due to the combination of these factors, there is enormous potential for PROs to inform treatment assessment, patient information and recommendations for NMIBC management.

Despite a large number of RCTs conducted (~ 200 since 1970) evaluating effectiveness of NMIBC treatments, many focused on assessing adverse events, with few assessing PROs in the short and long-term [10]. Where PROs were assessed, the evidence is limited by small sample sizes, lack of comparison groups, poor adjustment for baseline function, and failure to distinguish between patients with varying degrees of risk and between treatments in the analysis [11]. Consequently, key evidence on the impact of contemporary therapies for NMIBC on patient’s HRQL is lacking [10].

NMIBC generally has good clinical outcomes with available treatment options. Hence choice of treatment should consider additional aspects such as trade-offs in certain PROs. Our earlier work developed an empirically derived conceptual framework of PROs important to people with NMIBC, including proximal effects that can occur directly as a consequence of the NMIBC and/or treatment for the disease (e.g. blood in urine, urinary frequency) and their consequential effects on a person’s ability to function and their overall well-being (i.e. distal effects) [12]. This framework includes six domains covering symptoms and treatment side-effects, five functioning domains, and seven experience of care domains, guiding the design of a comprehensive PRO assessment plan for clinical practice in NMIBC and future clinical trials of treatments for NMIBC.

We conducted a systematic review of PROMs available to assess PROs identified in our conceptual framework, evaluating optimality (i.e. efficient and focused measurement) of existing PROMs as measures of symptom benefit for use as clinical trial endpoints in contemporary treatment for NMIBC [11]. We found that some PROs important in NMIBC were inadequately covered by generic and cancer-specific PROMs (e.g. skin problems, severe systemic side-effects of treatment such as joint pain and stiffness, physical function impairment due to restricted movement, psychological distress) and several PROMs used in NMIBC studies contained some content irrelevant to people undergoing contemporary treatment for NMIBC (e.g. appetite, nausea, vomiting) [11]. The European Organisation for Research and Treatment of Cancer (EORTC) have developed a suite of PROMs designed to assess HRQL in cancer clinical trials. The Quality of Life Questionnaire (QLQ-C30) is a core 30-item PROM designed to measure HRQL in all cancer patients [13]. Disease and treatment-specific modules complement the core measure, improving the sensitivity and specificity of HRQL assessments in specific groups of patients (for example, the QLQ-NMIBC24 – a disease-specific module assessing HRQL issues affecting patients with NMIBC) [14]. The EORTC QLQ-C30 plus the EORTC QLQ-NMIBC24 module was best aligned with our NMIBC-specific conceptual model [12], but failed to represent some PROs important to NMIBC patients (e.g. skin problems, joint pain/stiffness).

The EORTC PROMs were developed to enable a comprehensive HRQL assessment, assessing a range of issues including aspects of functioning, side-effects of cancer treatment, and symptoms of specific cancers. Symptom indexes, on the other hand, are PROMs which provide a more focused approach, typically including only relevant symptoms/side-effects summed into a single index. This offers an alternative measurement approach to the EORTC’s more comprehensive and modular approach to the assessment of HRQL. A brief focussed symptom measure intended to comprehensively assess direct symptoms and side-effects of contemporary treatments for NMIBC would enable accurate, robust, and clinically relevant assessment of differences in PROs among current and emerging therapies for NMIBC.

Extensive work with patients was conducted previously to develop and pre-test a draft NMIBC-specific symptom index (NMIBC-SI) [12, 15, 16]. First, an exhaustive list of clinically relevant issues was generated from three sources: (1) a systemic review and narrative analysis of the NMIBC-PRO literature [12]; (2) in-depth qualitative interviews with 26 NMIBC patients that explored patients’ experience of receiving treatment; and (3) in-depth qualitative interviews with 20 treating clinicians (specialist nurses and urologists) [12]. The list of issues was framed into questions (items) and constructed into the draft NMIBC-SI. This version was pre-tested with 15 patients and clinicians for content clarity and comprehensiveness, and appropriateness of the NMIBC-SI’s time frame, question stem, and response options [16].

This study aimed to:

• 1. Undertake a preliminary evaluation of the draft NMIBC-SI by examining the legitimacy of summing items into an index and identify items with poor psychometric performance for possible elimination (field test 1).

• 2. Psychometrically evaluate reliability, validity, and clinical utility of the final NMIBC-SI (field test 2).

Methods

This multi-centre study included two field tests. Field test 1 was a national cross-sectional study involving NMIBC patients aged ≥ 18, recruited during active treatment or within one week after final treatment. Eligible patients were recruited from nine Australian sites by treating urologists and urology nurses between February 2017 and March 2018. Data collected included patient demographics, clinical features of the tumour, risk category [17], treatment type, comorbidity, and the draft NMIBC-SI.

Field test 2 was a prospective longitudinal study involving patients aged ≥ 18, newly diagnosed with NMIBC after imaging or flexible cystoscopy but before endoscopic resection. Eligible patients were recruited from nine Australian, two New Zealand, three United States, one United Kingdom, and one Canadian, sites by treating urologists and urology nurses between July 2018 and July 2020. Data collected included patient demographics, clinical features of the tumour, risk category [17], treatment type, comorbidity, the NMIBC-SI, and HRQL assessed with EORTC QLQ-C30 [13] and QLQ-NMIBC24 [14] self-completed in English at baseline and four follow-up times (Table 1). Detailed methods are reported in the study protocol [15].

Table 1 PRO assessment schedule

Analysis

Field test 1

Our conceptual framework comprised physical symptoms caused by NMIBC or treatment, physical movement, psychological symptoms, sexual function, treatment burden, cystoscopy side-effects, and Bacillus Calmette-Guerin (BCG)/chemotherapy side-effects [12]. The symptom index included only the physical symptoms (Core index) and side-effects of cystoscopy and intravesical therapy (BCG/chemotherapy), assessed across 88-items.

We adopted a clinimetric rather than psychometric approach to analysis [18], as our goal was to identify key symptoms for inclusion, regardless of their co-occurrence and/or correlations with other items. Items were selected for their clinical relevance and combined into clinically-meaningful groups [19], rather than positing the existence of a latent variable and selecting a sample of interchangeable items to represent such a variable, as in the reflective approach [20,21,22]. Items were composite (formative) indicators [20], which need not be strongly correlated with each other to form a ‘composite’ [20] or ‘index’ [23], thus rendering factor analysis, item response theory, and measures of internal consistency inappropriate [22, 23]. Instead, we flagged items for possible exclusion based on the following:

• Low prevalence - frequency of each item response option (floor/ceiling effects), mean, and standard deviation was calculated. Items with marked floor/ceiling effects were considered for exclusion;

• High missing data - Items with > 20% missing data were considered for exclusion. The denominator was those patients to whom the items were applicable (e.g. having BCG);

• Redundancy – items conceptually and/or functionally similar and highly correlated (Pearson correlation ≥ 0.50) with an already included item (to avoid double-counting of essentially the same symptom). We calculated correlation coefficients for each pair of items and examined all correlations in descending order of magnitude. Pairs that correlated > 0.5 were flagged for further scrutiny and decisions about retaining/removing items were made by an expert group of nine urologists and the PRO methodologists on the team. The item considered to be phrased most clearly was retained, the other considered redundant and removed;

• Patient-reported experiences (PREs) of care (e.g. “treatment inconvenience”) rather than symptoms of disease or treatment. Questionnaires assessing PREs of care are available [24, 25] and could be used in studies with PRE endpoints;

• Too general (could have a range of causes; e.g. bodily pain) or not NMIBC-specific (e.g. cough).

Nine urologists reviewed all items and results from field test 1 analysis and provided input to item inclusion/exclusion decisions. Clinicians selected items to retain from items that were highly correlated and perceived to be assessing the same concept (e.g. blood in urine retained, passing blood clots removed) and ensured clinically important items were not excluded due to low prevalence (e.g. fever), until consensus was reached by all nine urologists.

Field test 2

Descriptive statistics included participant demographics, clinical characteristics, and loss to follow-up at each time-point.

Composite analysis

We used Confirmatory Composite Analysis (CCA) [26, 27] to examine the NMIBC-SI. To maximise data available for this analysis, we combined data from different time-points. For all analyses we used T3 if available. If T3 was missing for a participant we used their T4 data, if T4 was also missing we used their T5 data, and if T5 was also missing we used their T2 data. We first examined response frequencies for each item in the data set to identify skew and/or floor/ceiling effects.

CCA is described in detail elsewhere [26,27,28]. Briefly, whereas factor analysis evaluates items based on inter-item correlations (assuming a latent variable as a common underlying cause), composites are evaluated on the basis of their covariance with external criteria; thus, the focus in on the utility of a collection of items, not how strongly they are associated with one another. A composite variable is an exact linear combination of its indicators, such that the construct of interest is defined by its indicators. This approach is arguably better suited to the validation of symptom indexes than traditional approaches [28].

We tested three models (Table 2). Because CCA analyses uses complete cases, and only a small subset (N = 65) of the sample completed the Intravesical index, this index was excluded from CCA. CCA was conducted using the cSEM package in R [29].

To determine acceptability of each index score we examined score distributions and floor/ceiling effects (Table 2).

Test-retest reliability of index scores was assessed with single measurement, random effects intra-class correlations (ICCs) in psych() package using data from T3 and 3–7 days after T3 (T4; Table 1); we anticipated scores to be relatively stable at these times (Table 2).

Clinical known-groups validity was compared for NMIBC-SI scores for patients by treatment groups. We hypothesised: at T3, patients experiencing any treatment (transurethral removal of bladder tumour (TURBT), chemotherapy, BCG) would have greater symptom burden than those experiencing no treatment; patients experiencing chemotherapy/BCG would have greater symptom burden than those experiencing no treatment; and those undergoing BCG would have greater symptom burden than those undergoing chemotherapy (Table 2).

Table 2 Psychometric tests and criteria to assess reliability and validity

Results

Field test 1

Field test 1 included 220 participants: 178 male, mean age 69.3, representing all risk groups (Table 3 and Table S1). Of 88 items, most participants (> 80%) did not report experiencing 21 items. Seven highly correlated items were excluded, seven excluded as > 50% of urologists rated them as not directly related to NMIBC treatment or considered too general (e.g. “have you had a cough?”), 30 physical, psychological and sexual function items were excluded because distal impacts of treatment on physical, psychological and sexual function are covered in existing PROMs developed for assessing these domains. This produced a 23-item NMIBC-SI assessing disease or treatment-related symptoms (NMIBC-Core, 23-items) and two treatment-specific optional indexes assessing cystoscopy-specific (NMIBC-Cyst, 6-items) or intravesical BCG/Chemotherapy-related side-effects (NMIBC-Intra, 10-items).

Table 3 Field test 1 participant demographic and clinical information (n = 220)

Field test 2

Field test 2 included 232 participants (Fig. 1): 176 males, mean age 68yrs, representing Low 53.5%, Intermediate 11.2% and High 25.4% risk groups [17] (Tables 4 and S2). Average NMIBC-SI completion time was 3 min, 41 sec.

Fig. 1

Flow of participants through field test 2

Table 4 Field test 2 participant baseline demographic and clinical information (n = 232)

Composite analysis

Item response option frequencies for the three indexes are presented in Figs. 2, 3 and 4. Few participants responded 4 ‘Very much’ to any of the NMIBC-SI items (< 10% of participants responded very much to any of the items), and many participants responded 1 ‘Not at all’ to numerous items. For example, ≥ 50% of participants responded 1 ‘Not at all’ for 16/23 (70%) items in the NMIBC-Core index (Figs. 2) and 8/10 (80%) items in the NMIBC-Intra (Fig. 3). There was greater spread of participants across the response options for the NMIBC-Cyst index (Fig. 4).

Fig. 2

NMIBC-Core 23-item response frequency

Fig. 3

NMIBC-Intra 10-item response frequency

Fig. 4

NMIBC-Cyst 6-item response frequency

All three models tested exhibited good fit according to most fit indices (Table 5). The AIC and BIC both suggested that the one-composite model including both core and cystoscopy index items was best fitting, although the two-composite (core and cystoscopy) model had the highest R². Based on these results, we conclude that the core and cystoscopy index items can be represented as separate indexes. A single 29-item index is defensible, although the 6 cystoscopy items are applicable only to patients who have received a recent cystoscopy. For subsequent analyses, we analysed the proximal, cystoscopy, and Intravesical composites separately.

Table 5 Fit indices for the three composite models tested

Index-level performance

The three index summary scores had measures of central tendency close to the low end of the scale range (i.e. predominance of low scores), indicating most participants were not exhibiting many serious difficulties (Table 6).

Table 6 Index-level performance

Reliability

The intraclass correlations for test-retest reliability are shown in Table 7. For the three index scores, test-retest reliability was good (range 0.89–0.91).

Table 7 Test-retest reliability for NMIBC-SI indexes

Clinical validity

As expected, we found significant differences between no treatment and any treatment groups and between no treatment and chemo/BCG groups for the NMIBC-Core index (Table 8a, b). We also found significant differences between BCG and chemotherapy groups, with significantly higher scores for BCG patients. No other differences between treatment groups were statistically significant, although several others, including all of the NMIBC-Intra comparisons, were moderate to large.

Table 8a Known groups: no treatment vs. treatment (TURBT, chemotherapy, BCG) mean scores

Table 8b Known groups: no treatment vs. treatment (TURBT, chemotherapy, BCG) between group differences

Discussion

This multicentre international study followed the iterative process recommended by the FDA [7] and EORTC [31] for developing PROMs, and tested the conceptual framework of the NMIBC-SI, confirmed it with scoring rules, and assessed reliability and validity of the resultant index. NMIBC-SI was developed using a rigorous, theoretically, and empirically driven process supporting its content and face validity [10, 12, 16]. The CCA results supported all models tested. We conclude that the core and cystoscopy index items can be represented as separate indexes (23 and 6 items respectively), and although our results also support a single 29-item index, the cystoscopy items are applicable only to patients who have received a recent cystoscopy, so it makes more sense to aggregate those items separately.

Item response frequencies illustrate that participants reported few or mild symptoms, perhaps a reflection of > 50% of our field test 2 sample being graded low risk [17], resulting in less variability in responses than expected. The most commonly reported symptoms (daytime/nighttime urinary frequency and urinary urgency) are consistent with what would be expected in this population, with a likely multifactorial etiology, including from the cancer itself, from repetitive procedures on the bladder, from intravesical treatment (e.g. BCG and chemotherapy both elicit bladder irritation, which can manifest as urgency, frequency, bladder irritation, incomplete emptying and small volume voids), and for some, from age-related changes. Although frequent, these symptoms are less serious effects of NMIBC treatment. The next most common group of symptoms (lack of energy/disrupted sleep/small volume voids) could be a direct consequence of frequency/nighttime urinary disruptions. Rash, smell, fever and haematuria, although less commonly reported, are more serious symptoms possibly indicative of infection, and concerning and troublesome to patients, therefore important to measure. In addition, many patients treated for NMIBC can develop recurrent tumors needing future administrations of the same type of therapy. Having a comprehensive assessment of the symptom burden associated with therapy for NMIBC can serve to guide future decisions regarding choice of agent to treat recurrent tumors.

The NMIBC-SI indexes also satisfied psychometric criteria for reliability and validity. Test-retest reliability was strong indicating stability in the construct they were designed to assess, and they were able to discriminate between no treatment and treatment groups providing evidence of clinical utility. Patients who had no treatment experienced lower symptom burden scores compared to treatment group patients. The NMIBC-Intra index was able to discriminate between BCG and chemotherapy groups. BCG patients experienced worse intravesical-specific symptoms compared to chemotherapy patients as expected [32].

The NMIBC-SI provides a brief method for clinically relevant assessment of differences in PROs among contemporary treatments for NMIBC, which can provide an evidence base for the ongoing improvement of future therapies for NMIBC. It is intended for self-completion and patients rate the amount of “bother” attributed during the past week on a 4-point response scale. A key requirement for using the NMIBC-SI in clinical trials is having scoring rules yielding index scores sensitive to both improvement (symptom benefit) and deterioration (burden of treatment). The NMIBC-SI indexes are scored by taking the average of each item included in an index with linear rescaling to an observable range of 0-100, with higher scores representing worse symptoms or functional impairment. It is a flexible method where indexes can be selected depending on research aims and included as PRO endpoints in future NMIBC trials to:

• describe baseline burden of disease-defining symptoms (NMIBC-Core);

• assess the extent of symptom benefit or harm with contemporary treatments for NMIBC by prospective assessment of change from baseline in the NMIBC-Core;

• assess specific burden of cystoscopy and/or intravesical therapy (BCG or chemotherapy) by prospective assessment of change from baseline in NMIBC-Cyst and NMIBC-Intra respectively.

NMIBC-SI can also be used in observational studies of patterns and outcomes of care to document common and persistent problems associated with NMIBC treatment, providing information about treatment sequelae for use by both patients and clinicians. Current gaps in evidence limit our understanding of PRO trajectories from diagnosis through to long-term survivorship and treatment effects [10].

NMIBC-SI provides an alternative measurement system to existing PROMs, focusing exclusively on proximal disease and treatment-related symptoms, overcoming some of the limitations of existing PROMs previously described [10, 11] Researchers wishing to assess other aspects of HRQL should use the NMIBC-SI together with PROMs developed specifically to assess those HRQL aspects, although care must be taken not to over-burden patients, particularly with questionnaires that overlap in content. The NMIBC-SI is a relatively short symptom-focussed measure that can help clinical researchers meet international directives to “measure what matters” in trials of treatments for NMIBC where symptoms are a key focus [33,34,35].

Our evaluation component had some limitations. Item response frequencies demonstrated lower symptom burden in our study sample than expected, and although the psychometric results were strong, it is uncertain how the instrument would perform in a sample with greater variability. Our inability to test the Intravesical index using CCA due to unexpectedly small numbers of patients experiencing BCG/chemo should be addressed in future research with more targeted recruitment of these patients. Further, clinical validity analysis was also affected by unexpectedly low numbers of respondents in certain clinical groups (patients receiving BCG or chemotherapy at relevant PRO assessment time-points). Further research is needed with a diverse patient sample (e.g. patients experiencing more severe symptoms) to provide additional support for the NMIBC-SI’s measurement properties. A longitudinal observational study is underway that will explore change in PROs over time (acute to 5-year survivorship), the effect of maintenance BCG in addition to induction, the effect of recurrent disease and disease progression over time, and by patient risk category groups [15].

In future, the NMIBC-SI could also be used in clinical practice. Although developed to be a PROM for clinical research, its application in clinical practice was also considered during development [36, 37]. The layout, question content, and uniform response format were designed with ease of completion and rapid interpretation in mind. The 23-item NMIBC-SI can be completed within 4-min, and the responses reviewed by clinicians and compared with previous patient assessments within 1-minute if data are captured electronically and auto-generated comparative reports enabled. Implementing this would require resources and infrastructure. Envisaged applications of the NMIBC-SI in clinical practice would be to identify and appreciate the nature, severity, and time course of troublesome symptoms. Ready access to this information should improve shared understanding, discussions, and treatment decisions [33, 38].

Conclusion

Comparative effectiveness research in NMIBC requires a strong evidence-base that incorporates assessment of PROs. Such research is becoming more important in NMIBC due to the rapidly evolving treatment landscape with multiple new therapies being developed and approved. To fully capture and quantify patients’ perspectives, appropriately constructed and validated PROMs are required. The NMIBC-SI provides a brief method for clinically relevant assessment of differences in PROs among contemporary treatments for NMIBC for use in research and potentially clinical practice. The NMIBC-SI would benefit from further examination of its measurement properties in a sample of patients experiencing more severe symptoms.