|
|
||||||||
Guest Access | Sign In via User Name/Password |
|||||||||
* From Childrens Medical Group, Hamden, CT; Department of Pediatrics and Center for Primary Care Education and Research, New York Medical College Valhalla, NY; and Department of Pediatrics, Bridgeport Hospital-Yale New Haven Health, Bridgeport, CT.
Correspondence to: Kirsten M. Baker, MD, Childrens Medical Group, 299 Washington Ave, Hamden, CT 06518; e-mail: kirsten{at}pol.net
| Abstract |
|---|
|
|
|---|
Design: A survey containing eight case summaries was mailed to 24 board-certified pediatric allergists and pulmonologists, who were asked to classify each case according to the national guidelines. The case summaries included the patients medical history, physical examination, and chest radiograph and pulmonary function test results. Physicians were also asked to interpret the pulmonary function tests, to indicate the main factors used to classify each case (daytime symptoms, nighttime symptoms, pulmonary function testing, or various combinations), and to make treatment recommendations.
statistics were used to measure agreement.
Results: Fourteen of 24 surveys mailed (58%) were completed and returned. Agreement was poor for classifying asthma (
= 0.29; 95% confidence interval [CI], 0.25 to 0.33) and for the main factors used to make the classifications (
= 0.19; 95% CI, 0.14 to 0.23). Specialists exhibited higher agreement in their interpretation of pulmonary function tests (no asthma,
= 0.66; asthma on baseline,
= 0.53; exercise-induced asthma,
= 0.65). While physicians treatment recommendations were consistent with their severity classifications, the low level of agreement in those classifications led to substantial variability in the treatments recommended.
Conclusions: The low level of agreement among pediatric asthma specialists in classifying asthma severity suggests the need to refine the classification system used in the national guidelines to help ensure the consistent application of those guidelines.
Key Words: asthma classification guidelines
| Introduction |
|---|
|
|
|---|
For a classification system to be useful, different raters must be able to apply it in a consistent manner. A high level of interrater agreement could bolster acceptance of the guidelines among primary care physicians, who have been slow to adopt them.5 6 7 A low level of agreement would suggest the need for refining the classification system.
Although researchers have documented poor agreement among specialists in the interpretation of pulmonary function tests,8 9 we were unable to identify previous observer variation studies concerning the classification of asthma. The goal of the present study was therefore to assess the interrater reliability of the asthma classification system by measuring agreement among pediatric asthma specialists.
| Materials and Methods |
|---|
|
|
|---|
|
Survey Instrument
Each case summarized a patients first visit to the Pediatric Asthma Center, including medical, family, and environmental histories; physical examination; report of chest radiograph results; and pulmonary function test results if they were performed for the initial evaluation (the Appendix presents case 2 in its entirety). Cases were chosen to represent a variety of age groups and the four classification categories. One patient was included who had respiratory symptoms but did not have asthma in the judgment of the patients physician.
In addition to the asthma severity classification question, the questionnaire asked, for each case, whether symptoms vs pulmonary function determined the classification (Appendix). It also asked the physician to note any comorbid conditions and make treatment recommendations.
Since the classification system is based in part on pulmonary function, the physician was asked to interpret pulmonary function test results included with five of the eight case summaries. Three case summaries did not include test results because children < 5 years old are not usually able to perform the test. Respondents were therefore asked to interpret three additional pulmonary function tests to yield a total of eight tests, an adequate number for analyzing interobserver agreement. All of the pulmonary function tests were performed according to the American Thoracic Society standards.10 11 The questionnaire asked the physician to indicate his or her specialty, years in practice, and practice characteristics.
Statistical Analysis
Statistics were used to measure agreement among the respondents.12
Measures the extent of agreement among different raters by calculating the observed agreement corrected for the agreement that would be expected by chance.
Can range from 1 to 1, with the following interpretation: <0.40 represents poor agreement, 0.40 to 0.75 represents fair to good agreement, and > 0.75 represents excellent agreement. The Stata software package (Intercooled Stata 7.0 for Windows 98/95/NT; Stata Corporation; College Station, TX) was used to perform statistical analysis.
| Results |
|---|
|
|
|---|
Table 2 shows how the 14 physicians classified each of the eight cases. For example, case 1 was classified no asthma by 11 physicians, mild intermittent by 1 physician, and moderate persistent by 2 physicians.
|
statistics. The overall
score of 0.29 for question 1 (95% confidence interval [CI], 0.25 to 0.33) indicates that the 14 asthma specialists exhibited poor agreement in classifying these cases. They demonstrated the greatest agreement for the mild intermittent category (
= 0.44). Even when mild persistent, moderate persistent, and severe persistent were combined into a single "persistent" category (data not shown), the
value for this combined category was 0.45 (95% CI, 0.41 to 0.49). While reducing the number of categories increased the overall
value to 0.43 (95% CI, 0.38 to 0.48), this value still represents only fair agreement.
|
= 0.20), ranging from 0.03 to 0.31 for the different individual answers.
Since comorbid conditions can mimic or exacerbate asthma symptoms, respondents were asked to identify additional diagnoses that would apply to each case (question 3). Allergic rhinitis, sinusitis, vocal cord dysfunction, and gastroesophageal reflux were listed explicitly, and space was provided for writing in other diagnoses.
Scores ranged from 0.30 for vocal cord dysfunction to 0.80 for gastroesophageal reflux. It was not possible to compute an overall
score for question 3, 4, or 5 because the response categories were not mutually exclusive (ie, the physician was able to choose more than one category).
The specialists exhibited a higher level of agreement in their interpretation of the pulmonary function tests (question 4) than in their classification of asthma (question 1). Only one of the five responses to question 4 listed in Table 3 asthma revealed by bronchodilatorfell into the "poor agreement" range.
One pulmonary function test came from a patient with a known fixed extrathoracic obstruction. Although the questionnaire did not include this abnormality as one of the choices, the physician was provided space to write in additional comments. If the physician noted obstruction of the upper airways or large airways, or described blunted inspiratory and expiratory loops in this space, that response was coded as positive for an extrathoracic obstruction. Otherwise, negative was assumed.
The specialists were asked to choose one or more treatments from a list of options (question 5). To assess agreement level for use of any anti-inflammatory medication, responses for daily inhaled steroid and other anti-inflammatory were combined into a single anti-inflammatory category (data not shown). The
value for the combined category was 0.41 (95% CI, 0.34 to 0.48).
Table 4 reports the respondents treatment recommendations according to their asthma classifications. Altogether, there were 112 classifications (eight cases x 14 respondents). Since the respondent could select more than one treatment option for a given patient, there are > 112 treatment recommendations. No treatment options were selected for five of the classifications, which are indicated as "question left blank" in Table 4 .
|
| Discussion |
|---|
|
|
|---|
Use of a simplified classification that combined mild, moderate, and severe persistent asthma into a single persistent asthma category still yielded only fair agreement among these physicians. This finding suggests that simplifying the existing classification in this manner would not, itself, result in a satisfactory level of agreement.
While it was surprising to discover the extent of disagreement among pediatric asthma specialists in how they classified patients, perhaps the variability should have been expected. Studies have documented observer variability within a number of subspecialties: in radiologists readings of sonograms of breast masses,13 in pathologists grading of dysplasia in ulcerative colitis,14 and in orthopedic surgeons classification of scoliosis,15 16 to name a few. Clearly, some disagreement is inevitable in the application of clinical judgment.
Reasons for Disagreement
What might account for the variability in the application of the asthma classification system by this group of asthma specialists? There are at least three potential explanations: variability in interpretation of symptoms, variability in interpretation of pulmonary function test results, and variability in the importance placed on symptoms vs pulmonary function.
Overlap in the symptoms of asthma and other comorbid conditions can result in differences of interpretation that may affect physicians classifications. A patient with frequent cough may be considered to have more severe asthma by a physician who interprets that cough as a symptom of asthma. This same patient may be considered to have mild asthma by a physician who interprets that cough as a symptom of allergic rhinitis. Since respondents exhibited a higher level of agreement in their interpretations of pulmonary function tests than in their asthma classifications, those interpretations probably do not account for a large portion of the variability in the classifications.
In contrast, respondents exhibited very low agreement on which factorssymptoms or pulmonary functionmost influenced a patients classification. The perceived importance of symptoms vs pulmonary function, not merely how physicians interpreted each of these, affected their classification choices.
The handling of case 2 (Appendix) by different respondents illustrates how variation in the processing of clinical information can lead to different classifications and treatment choices. One respondent interpreted the pulmonary function test as negative for asthma, classified the patient accordingly, and recommended no asthma medications. This respondent indicated that the classification was based primarily on pulmonary function. Another respondent also interpreted the pulmonary function test as negative for asthma, but classified the patient as moderate persistent asthma and recommended daily and as-needed bronchodilator, daily inhaled steroid, and other anti-inflammatory asthma medications. This physician indicated that the classification was based primarily on daytime symptoms. While the treatments selected by these two physicians matched their classifications, their disagreement about the classification would have consequences for the patient: either the first physician recommended inadequate treatment or the second physician recommended excessive treatment.
Implications for Clinical Practice
The asthma guidelines developed by the National Institutes of Health were intended to improve patient outcomes by facilitating the diagnosis and treatment of this chronic condition. While asthma specialists in the present study recommended treatments that usually matched their severity classifications, they disagreed among themselves about those classifications. Since proper therapy depends on proper classification, one is forced to question the effectiveness of the official guidelines in serving their intended purpose.
In 2002, the National Institutes of Health released an update on selected topics of its national guidelines.4 Since the classification system in the updated guidelines remains essentially unchanged except for the omission of pulmonary function testing in patients < 5 years old, the findings reported here continue to be pertinent.
The clinical importance of the present findings rests on a chain of assumptions: (1) that our sample of specialists does not differ substantially from the general population of pediatric asthma specialists in ways that would markedly affect the findings, (2) that physicians would classify and treat actual patients in a manner similar to the way they handled the cases presented in the survey, (3) that disagreements could lead to nontrivial differences in treatment strategies, and (4) that those differences could potentially result in harm to some patients who would receive inadequate or inappropriate therapies.
The first two assumptions cannot be proven from our data, but they seem plausible and we have no reason to believe that they are not true. The remaining assumptions are supported by our findings. At minimum, therefore, our results suggest the need for confirmatory studies.
Study Limitations and Future Research
A survey cannot duplicate the experience of examining a patient and gathering a medical history. While this is a real limitation, it is also an advantage: physicians were all given the same information on which to base their decisions. It follows that disagreements must have occurred because physicians differed in their processing of the information rather than in their gathering of the information.
While a response rate close to 100% would be desirable, it has been shown that response rates to mail surveys are lower for physicians (54%) compared with nonphysicians (68%).17 Our 58% response rate is respectable for such a demanding survey given to busy physicians.
Since our survey was administered to a limited sample of specialists, a larger study could be useful for confirming the results reported here. Nevertheless, it should be noted that observer variation studies typically rely on a relatively small number of experts. For example, in the studies cited earlier13
14
15
16
the number of physicians ranged from 5 to 13. It is also important to note that data from the present sample yielded quite narrow CIs for most of the
statistics reported in Table 3
. It is therefore unlikely that a larger sample will produce markedly different results.
Future studies should also examine intraobserver reliability, that is, the extent to which an individual physician classifies the same asthma patient the same way when presented with identical information on different occasions. Additional studies should also evaluate these issues using adult asthma patients. Finally, studies should include a more detailed examination of the reliability of pulmonary function test interpretations.
| Conclusions |
|---|
|
|
|---|
| Appendix |
|---|
|
|
|---|
Case 2
John is a 13-year-old boy presenting to the specialist for initial evaluation in October 2000 (certain clinically insignificant details have been altered to ensure patient anonymity).
Pulmonary History: Johns breathing problems began when he was 2 years old, when he received a diagnosis of bronchitis, then pneumonia. Two years later, asthma was diagnosed. He did well until 2 years ago, when his symptoms increased. He did not do well until he was put on his present regimen 1 month ago. Now he has inspiratory wheezing when he is stressed, playing sports, or upset. Respiratory symptoms include shortness of breath, wheeze, dry cough, and tightness in the chest. Inspiratory wheeze can occur daily; nighttime symptoms occur once or twice a month.
Hospitalizations and Emergency Department Visits: John had one emergency department visit 1 year ago for asthma.
Warning Signs: Warning signs include cough, stuffed nose, runny nose, colds, behavior or personality changes, sneezing, dark circles under eyes, and raised shoulders.
Triggers: Triggers include colds, runny nose, changes in weather, spring and fall seasons, cats, air pollutants, cold air, emotions (laughing/upset), smoke, strong smells, dust/dust mites, molds, or mildew.
Other Medical Consultants: When John was 5 years old, he was seen by another pulmonologist, who recommended medications as needed.
Sports/Activities: John plays baseball and basketball. Problems during these activities include respiratory wheeze in the throat.
Allergies: Cat.
Medical History: Birth weight was 8 lb, 7 oz. Pregnancy, labor, and delivery were normal. Growth and development were normal. John as a history of GI reflux, and currently has food come up once a week. He has a history of hay fever. There is no history of sinusitis, eczema, nasal polyps, or ear infections.
Family History: Johns mother is 36 years old and has gastroesophageal reflux disease and allergies. Johns father is 44 years old and has hay fever. Johns brother is healthy, and his sister has seafood allergies. Johns grandmother has asthma, bronchitis, and gastroesophageal reflux disease.
Irritants and Controls in the Home Environment: Irritants include oil heat, damp basement, dog (never in bedroom), smoker at fathers house (visits every other weekend), real Christmas tree, curtains seldom washed, no dust mite covers on pillows/mattress/box spring, dust. Controls include frequently used room air conditioner and no carpets.
Vital Signs: Johns weight is 93 lb, heart rate is 69 beats/min, BP is 94/52 mm Hg, height is 63 inches, respiratory rate is 22 breaths/min, and oxygen saturation on room air is 98%.
Physical Examination: John is well developed, well nourished, and has no respiratory distress. Head examination is within normal limits, except for allergic shiners and mild nasal congestion. Pupils were equal, round, and reactive to light. Tympanic membranes were normal. Nose was clear for rhinorrhea, with gray, pale, and swollen turbinates. Throat showed no erythema or exudates, positive for postnasal drip, and 2+ tonsils. Neck was supple, with no masses; trachea was midline. Chest showed normal anteroposterior diameter, and no retractions. Cardiovascular was benign, and abdomen showed no hepatosplenomegaly. Extremities showed no clubbing. Neurologic evaluation was grossly intact. Skin showed no eczema, and lymph nodes showed no adenopathy.
Chest Radiography: A chest radiograph revealed mild hyperinflation at the time of evaluation.
Pulmonary Function: A pulmonary function test was performed on the day of evaluation (Appendix Table 1 ; Fig 1 ).
|
|
| Acknowledgements |
|---|
| Footnotes |
|---|
This work was performed at New York Medical College.
This study was supported in part by Health Resources and Services Administration award #1D12HP00022.
Received for publication February 25, 2003. Accepted for publication June 24, 2003.
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
J. W. Stout, C. M. Visness, P. Enright, C. Lamm, G. Shapiro, V. N. Gan, G. K. Adams III, and H. E. Mitchell Classification of Asthma Severity in Children: The Contribution of Pulmonary Function Testing Arch Pediatr Adolesc Med, August 1, 2006; 160(8): 844 - 850. [Abstract] [Full Text] [PDF] |
||||
![]() |
B. P. Yawn, S. K. Brenneman, F. C. Allen-Ramey, M. D. Cabana, and L. E. Markson Assessment of Asthma Severity and Asthma Control in Children Pediatrics, July 1, 2006; 118(1): 322 - 329. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. M. Graham Classifying Asthma Chest, July 1, 2006; 130(1_suppl): 13S - 20S. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Van Sickle Perceptions of asthma among physicians: an exploratory study with the ISAAC video Eur. Respir. J., November 1, 2005; 26(5): 829 - 834. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. P. Galant, L. J.R. Crawford, T. Morphew, C. A. Jones, and S. Bassin Predictive Value of a Cross-Cultural Asthma Case-Detection Tool in an Elementary School Population Pediatrics, September 1, 2004; 114(3): e307 - e316. [Abstract] [Full Text] [PDF] |
||||
![]() |
G. L. Colice Categorizing Asthma Severity: An Overview of National Guidelines Clin. Med. Res., August 1, 2004; 2(3): 155 - 163. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |