|
|
||||||||
Guest Access | Sign In via User Name/Password |
|||||||||
* From the Health Services Research Unit (Drs. Lamping and Schroter), Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK; MAPI Values (Dr. Marquis and Ms. Marrel), Lyon, France; and Bayer plc and Drs. Duprat-Lomon and Sagnier), Stoke Court, UK.
Correspondence to: Donna L. Lamping, PhD, Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, Keppel St, London WC1E 7HT, United Kingdom; e-mail: donna.lamping{at}lshtm.ac.uk
| Abstract |
|---|
|
|
|---|
Design: A psychometric study within an international, prospective, randomized, double-blind study. The CAP-symptom questionnaire (CAP-Sym) is a new, 18-item, patient-reported outcome measure that evaluates the bothersomeness of CAP-related symptoms during the past 24 h using a 6-point Likert scale. We used "gold standard" psychometric methods to comprehensively evaluate the acceptability, reliability, validity, and responsiveness of the CAP-Sym.
Setting: Sixty-four centers in 13 countries (France, Germany, Hungary, Israel, Italy, Norway, Poland, Portugal, South Africa, Spain, Sweden, Switzerland, United Kingdom).
Patients: Five hundred fifty-six patients with CAP, recruited from outpatient clinics, general practice, and hospital centers.
Interventions: Randomization 1:1 to moxifloxacin (400 mg once daily), oral or standard oral treatment (amoxicillin, 1 g tid, or clarithromycin, 500 mg bid), alone or in combination, for up to 14 days.
Results: Standard psychometric tests confirmed the acceptability (item nonresponse, item-endorsement frequencies, item/scale floor and ceiling effects), reliability (internal consistency, item-total and inter-item correlations, test-retest reliability), validity (content, construct, convergent, discriminant, known groups), and responsiveness of the CAP-Sym.
Conclusions: The CAP-Sym is a practical and scientifically sound patient-based outcome measure of CAP-related symptoms that has been developed using "gold standard" methods. As the only fully validated measure of symptoms in patients with CAP, which is quick and easy to administer and is more responsive than the generic Medical Outcomes Study 36-Item Short-Form Health Survey, the CAP-Sym provides a practical and rigorous method for improving the evaluation of outcomes in clinical trials and audit.
Key Words: community-acquired pneumonia outcomes patient-based assessment questionnaire symptoms
| Introduction |
|---|
|
|
|---|
Recently introduced guidelines for the management of CAP5 6 7 8 9 provide algorithms to guide clinical decision making about the choice of antimicrobials. As an alternative to aminopenicillins and/or macrolides, current guidelines also recommend a new "respiratory" fluoroquinolone as a potential first-line option. In addition to their known efficacy, fluoroquinolones offer the potential advantage of quicker symptom resolution and improved quality of life due to their rapid bactericidal activity. It is therefore important that new antimicrobial treatments in CAP be evaluated on the basis of rigorous assessment of patient-based outcomes such as symptoms and quality of life in addition to clinical outcomes.
Most studies evaluate treatment efficacy on the basis of clinical outcomes such as mortality,10 11 12 bacteriologic response,10 13 14 15 temperature,16 respiratory and heart rate,17 clinical cure/response,13 15 nature and severity of adverse events/safety,10 13 15 17 and hospitalization.10 11 12 18 More recent studies have also evaluated outcomes on the basis of the health-care costs associated with outpatient visits,19 inpatient hospitalizations,17 18 the use of antibiotics/antimicrobials,10 11 and time to return to work or usual activities.10
Despite clear recognition that patient-based outcomes are a key component in evaluating health outcomes,20 21 22 only three studies have evaluated treatment efficacy in CAP using rigorous, validated measures of quality of life,10 17 19 and none have assessed symptoms using scientifically robust measures. Two studies10 13 that have evaluated symptoms used unvalidated, clinician-reported ratings of patients symptoms, and two other studies16 19 assessed patient-reported symptoms using scales that have not been fully evaluated for reliability, validity, and responsiveness. Our extensive review of the literature and expert opinion from pneumologists, clinical researchers, and outcome researchers pointed to the need for a practical and scientifically rigorous patient-based outcome measure to evaluate symptoms in CAP in clinical trials and audit.
We describe the development and validation of the CAP symptom questionnaire (CAP-Sym), a new, patient-based measure of symptoms in CAP. We used rigorous psychometric methods23 to guide the development and evaluation of the CAP-Sym. These "gold standard" scientific methods, borrowed from the social sciences for application in health care,24 25 allow regulatory bodies, clinicians, researchers, and patient advocacy groups to determine whether an instrument is a "good" measure that provides scientifically credible information. Psychometrics provide well-established scientific methods for measuring subjective judgements using numeric scales and evaluating the quality of measurement scales (ie, reliability, validity, responsiveness). Rigorous criteria are now available for evaluating the scientific robustness of health-outcome measures.26 27 We28 29 30 and others31 32 33 have used these methods extensively to develop and validate outcome measures in several areas of clinical medicine.
We undertook a comprehensive evaluation of the acceptability, reliability, validity, and responsiveness of the CAP-Sym questionnaire as part of an international, prospective, randomized, double-blind study to compare the effectiveness of treatment with moxifloxacin oral tablets to standard oral regimes in patients with CAP.
| Materials and Methods |
|---|
|
|
|---|
Trained interviewers conducted telephone or face-to-face interviews using an interview guide. Interviews included open and closed questions asking patients about their daily life with CAP, their symptoms, the circumstances in which they were most bothered/limited because of CAP, and the consequences of CAP and its treatment. Patients verbatim reports were used to develop questions for the CAP-Sym based on a predefined format to evaluate patients views about the bothersomeness of their symptoms.
The questionnaire was developed in English (for use in the United Kingdom and South Africa) and then translated into 12 other languages: French, German (for use in Germany and Switzerland), Spanish, Italian, Portuguese, Swedish, Norwegian, Polish, Hungarian, Hebrew (for use in Israel), Russian (for use in Israel), and Afrikaans (for use in South Africa). Linguistic validation was performed according to the standard forward/backward methodology34 : (1) a single forward translation from English to the target language; (2) review of the translation by linguistic experts; (3) backtranslation into English; and (4) amendments to the forward translation based on the backtranslation. Each language version of the questionnaire was pretested by interviewing two pulmonary specialists in each country to check for completeness, relevance, and the appropriateness of the wording used by patients to describe their condition. Modifications to the questionnaire were then made and final translations agreed.
CAP-Sym Questionnaire
The CAP-Sym (see Appendix) measures 18 CAP-related symptoms: coughing, chest pains, shortness of breath, coughing up phlegm/sputum (secretion from the chest), coughing up blood, sweating, chills, headache, nausea, vomiting, diarrhea, stomach pain, muscle pain, lack of appetite, trouble concentrating, trouble thinking, trouble sleeping, and fatigue. It is a patient reported questionnaire that is administered by interview. Patients are asked to rate each symptom for bothersomeness during the past 24 h using a 6-point Likert scale. In this study, the researcher asked the patient "In the past 24 h, how much have you been bothered by ... ", then read each symptom aloud and recorded how much the patient rated the bothersomeness of the symptom on the 6-point response scale (0 = patient did not have the symptom/problem; or patient had the symptom/problem and reported that it bothered him/her: 1 = not at all, 2 = a little, 3 = moderately, 4 = quite a bit, 5 = extremely). All 18 items are summed to produce a CAP-Sym score (range, 0 to 90). High values indicate poorer outcomes (ie, higher symptom bothersomeness).
CAP 2000 Study
The CAP 2000 study is a multicenter, prospective, randomized, double-blind trial to compare the effectiveness of moxifloxacin and standard recommended therapy in CAP. Full details of the study are described elsewhere.35
Briefly, patients from 13 countries, recruited from outpatient clinics, general practice, and hospital centers, were randomized to one of two treatment arms: moxifloxacin (400 mg once daily) oral or standard treatment (amoxicillin, 1 g tid, or clarithromycin, 500 mg bid, or both amoxicillin and clarithromycin) for at least 5 days and up to 14 days of treatment. The choice of standard treatment was made by study clinicians prior to randomization and represented the most appropriate local first-line therapy based on clinical presentation, potential pathogens, and local susceptibility data. All randomized patients were followed up to study termination on an intention-to-treat basis.
Outcomes were assessed at baseline (study entry), days 3 to 5 during treatment, days 7 to 10 (test of cure), and days 28 to 35 after treatment. Clinical outcomes and patient-reported symptoms and energy were assessed at all four assessment points, and quality of life was assessed at baseline and days 28 to 35. Clinical outcome measures included temperature and the pneumonia severity index (PSI).36 We used the CAP-Sym to assess symptoms, the acute version of the Medical Outcomes Study 36-Item Short-Form Health Survey (SF-36)37 to assess quality of life, and the vitality subscale of the SF-36 to assess energy. Use of health-care resources (ie, medications, including alternative antibacterial therapy, diagnostic and therapeutic procedures, hospitalization, and other health-care visits) was recorded throughout the study period. Ethics approval was obtained from the relevant committees in each country, and written informed consent was obtained from all patients prior to study entry.
Psychometric Evaluation of the CAP-Sym
We used "gold standard" psychometric tests and criteria23
25
for item reduction and to evaluate the acceptability, reliability, validity, and responsiveness of the CAP-Sym (Table 1
, 1A
). The first step in the psychometric field testing of the CAP-Sym was to perform standard item-reduction analyses to determine whether the initial 18-item CAP-Sym (CAP-Sym 18) could be reduced to a smaller number of items. This was done by selecting items that performed best on psychometric tests. The second step was to perform extensive psychometric analyses to evaluate the acceptability, reliability, validity, and responsiveness of the CAP-Sym.
|
|
| Results |
|---|
|
|
|---|
|
Results from cross-validation analyses of the item-reduction strategy performed separately on the two random split-half subsamples confirmed the robustness of the item-reduction strategy. Item-reduction analyses in both subsamples resulted in very similar items being eliminated in these two subsamples as in the main pooled data set. There were only minor differences in the actual items that were eliminated across all three samples.
We evaluated the psychometric properties of both the CAP-Sym 18 and the CAP-Sym 12. This was done to enable a direct comparison of the measurement properties of the full-length and item-reduced versions of the CAP-Sym to determine whether item reduction resulted in a more robust measure.
Acceptability
The full-length and item-reduced versions of the CAP-Sym show good acceptability. Table 3
shows a low proportion of missing data, low floor/ceiling effects, and skewness values within the recommended range for the CAP-Sym 18 and the CAP-Sym 12.
|
coefficients for the CAP-Sym 18 and the CAP-Sym 12 indicate high internal consistency. Values exceed the standard criterion of 0.70. Item-total correlations ranged from 0.22 to 0.56 (mean, 0.40) for the CAP-Sym 18, and from 0.28 to 0.60 (mean, 0.42) for the CAP-Sym 12. One item in the CAP-Sym 18 (coughing up blood; 0.22) failed the criterion of > 0.25, whereas all items in the CAP-Sym 12 passed this criterion. Mean inter-item correlations were 0.20 and 0.23 for the CAP-Sym 18 and CAP-Sym 12 scales, respectively.
Test-Retest Reliability:
As shown in Table 3
, both the CAP-Sym 18 and the CAP-Sym 12 show good test-retest reliability. Intraclass correlation coefficients (ICCs) were > 0.95.
Validity
Content Validity:
The content validity of the CAP-Sym was evaluated during the development of the questionnaire. Evidence from qualitative interviews with patients, opinion from experts in CAP, and a review of literature supports the content validity of the CAP-Sym.
Construct Validity (Within-Scale Analyses):
Evidence of high internal consistency (Table 3)
and findings from the principal component factor analysis support the construct validity of the CAP-Sym 18 and the CAP-Sym 12. Moderately high item-total correlations, high
coefficients, and the results of the factor analysis indicate that a single construct is being measured, and that the items can be combined to form summary scores.
Construct Validity (Known Group Differences/Hypothesis Testing):
Table 4
presents mean CAP-Sym scores for patients defined as clinically cured or clinical failures. As hypothesized, CAP-Sym 18 and CAP-Sym 12 scores are significantly lower (indicating lower symptom bothersomeness) in patients defined as clinically cured than in the small number of patients (n = 7) who did not demonstrate clinical improvement (p = 0.03 and 0.02, respectively). Similarly, SF-36 Vitality scores are significantly lower (indicating lower energy) in patients defined as clinically cured than in those who did not demonstrate clinical improvement (p = 0.03).
|
|
Responsiveness
Table 6
shows effect sizes for change in CAP-Sym scores between baseline and days 3 to 5, days 7 to 10, and days 28 to 35. As hypothesized, CAP-Sym 18 and CAP-Sym 12 scores show improvement from baseline to follow-up at all three assessment points and increase in magnitude across time, indicating good responsiveness. Effect sizes are large at all three assessments. Similar results for the responsiveness of the CAP-Sym were found using standardized response means (results not shown).
|
| Discussion |
|---|
|
|
|---|
Both the CAP-Sym 18 and the CAP-Sym 12 meet standard criteria for acceptability, reliability, validity, and responsiveness. As the two versions show very similar psychometric properties, it is difficult to make a strong case for a psychometric advantage of either the CAP-Sym 18 or the CAP-Sym 12. The CAP-Sym 12 shows a slight psychometric advantage (ie, all item-total correlations meet the criterion, whereas one item fails the criterion in the CAP-Sym 18), as well as the practical advantage of being shorter, and therefore has a slightly lower respondent burden. However, the CAP-Sym 18 includes all symptoms and could therefore be argued to have optimal content and face validity. We recommend both measures for use in clinical trials and audit.
The CAP-Sym is designed to be administered by interview. This method of questionnaire administration typically results in little missing data. If the CAP-Sym is administered by self-completion or postal survey rather than by interview, it is likely that there will be more missing data. In this case, an alternate scoring method would be more appropriate. We suggest that if the CAP-Sym is to be administered by self-completion, missing data should be imputed using the same algorithm recommended for scoring the SF-36.39 Using this method, a person-specific estimate is imputed for missing questions in cases in which the patient has answered at least 50% of the items on the CAP-Sym.
It is important to consider possible methodologic limitations in the development of the CAP-Sym. First, were the patients who participated in the CAP 2000 and the field testing of the CAP-Sym a representative sample? Comparison with previous studies shows that patients in our study were similar in age and gender to patients with CAP in another large international trial.15 Second, does the lack of correlation between the CAP-Sym and clinical outcome measures limit its use or interpretation as an outcome measure in clinical trials? Discrepancies between measures of outcome assessed from the patients point of view and clinical assessments have been demonstrated in many areas of health care.40 41 42 This does not mean that patient-based outcome measures of symptoms or quality of life are not valid. Rather, this common finding confirms that clinical and patient-based measures are evaluating different and not necessarily related aspects of outcome, both of which must be included in any comprehensive evaluation of outcomes. Moreover, we have shown that none of the clinician-rated signs and symptoms scales used in the CAP 2000 study were adequately reliable to be used to evaluate treatment outcome or to validate the CAP-Sym.43 Third, as expected in a study of this design and the population studied, the number of clinical failures was very small. It could be argued, therefore, that the test of validity based on differences in mean CAP-Sym scores between patients defined as clinically cured and those who did not demonstrate clinical improvement is less robust than if applied to a sample with a higher number of clinical failures. However, as other pieces of evidence from different psychometric tests provide further confirmation of the validity of the CAP-Sym, we do not believe that this arguably weaker piece of evidence compromises the overall validity of the CAP-Sym. It would, nevertheless, be important to further evaluate the ability of the CAP-Sym to discriminate groups in trials that produce more mixed results in terms of treatment efficacy.
Our conclusions about the acceptability, reliability, validity, and responsiveness of the CAP-Sym are based on the results of field testing of 556 patients with CAP in 13 countries. Subsequent studies are needed to evaluate the psychometric properties of the 13 individual language versions of the CAP-Sym. As data begin to accumulate from the use of the CAP-Sym, we will be able to establish normative population values for CAP-Sym scores in different countries. It is also possible that the CAP-Sym may prove to be useful as a measure of outcomes in patients with hospital-acquired pneumonia or those with CAP requiring hospitalization, although the measure should be revalidated for use in more severe conditions. Findings from this study suggest a possible advantage of using the disease-specific CAP-Sym in clinical trials as it may be more responsive in detecting treatment effects than the generic SF-36. Further evaluation of the comparative responsiveness of the CAP-Sym and SF-36 should be carried out in order to confirm possible differences in sensitivity to treatment effects.
Evaluating the scientific adequacy of patient-based health-outcome measures is a key task in developing recommendations about which measures should be used in clinical trials and to routinely monitor health care. Use of the CAP-Sym as an outcome measure in clinical trials ensures the availability of scientifically credible information about the effect of new treatments on patient-reported symptoms. For example, the CAP-Sym could be used to evaluate new pharmaceutical treatments or to compare the efficacy of oral vs IV administration. Routine use of the CAP-Sym in clinical practice will enable clinicians and managers to assess outcomes in CAP on a continuing basis. Such information can be used to identify strengths and deficiencies in quality of care, and thus can be used to improve medical practice.
| APPENDIX |
|---|
|
|
|---|
|
| Acknowledgements |
|---|
| Footnotes |
|---|
This work was funded by Bayer plc. Drs. Lamping and Schroter have received research funding and support for attending con- ferences from Bayer. MAPI Values (Dr. Marquis and Ms. Marrel) has received funding from Bayer for consulting, questionnaire development, and linguistic validation.
| References |
|---|
|
|
|---|
and the internal structure of tests. Psychometrika 1951;16,297-334[CrossRef][ISI]
This article has been cited by other articles:
![]() |
S K Gupta, K Viswanath, R D Thulasiraj, G V S Murthy, D L Lamping, S C Smith, M Donoghue, and A E Fletcher The development of the Indian vision function questionnaire: field testing and psychometric evaluation Br. J. Ophthalmol., May 1, 2005; 89(5): 621 - 627. [Abstract] [Full Text] [PDF] |
||||
![]() |
S Schroter and D L Lamping Coronary revascularisation outcome questionnaire (CROQ): development and validation of a new, patient based measure of outcome in coronary bypass surgery and angioplasty Heart, December 1, 2004; 90(12): 1460 - 1466. [Abstract] [Full Text] [PDF] |
||||
![]() |
R El Moussaoui, B C Opmeer, P M M Bossuyt, P Speelman, C A J M de Borgie, and J M Prins Development and validation of a short questionnaire in community acquired pneumonia Thorax, July 1, 2004; 59(7): 591 - 595. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Wilson, L. Allegra, G. Huchon, J.-L. Izquierdo, P. Jones, T. Schaberg, and P.-P. Sagnier Short-term and Long-term Outcomes of Moxifloxacin Compared to Standard Antibiotic Treatment in Acute Exacerbations of Chronic Bronchitis Chest, March 1, 2004; 125(3): 953 - 964. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. Hilari, S. Byng, D. L. Lamping, and S. C. Smith Stroke and Aphasia Quality of Life Scale-39 (SAQOL-39): Evaluation of Acceptability, Reliability, and Validity Stroke, August 1, 2003; 34(8): 1944 - 1950. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |