|
|
||||||||
Guest Access | Sign In via User Name/Password |
|||||||||
* From the Department of Pediatrics (Drs. Jones, Clement, Kimia, Dyer, Li, and Hanley-Lopez), Keck School of Medicine at the University of Southern California, Los Angeles, CA; and the Southern California Chapter of the Asthma and Allergy Foundation of America (Ms. Morphew), Los Angeles, CA.
Correspondence to: Craig A. Jones, MD, FCCP, 1801 E Marengo, Rm 1G1, General Laboratories Bldg, Los Angeles, CA 90033; e-mail: craigjones{at}adelphia.net
| Abstract |
|---|
|
|
|---|
Objective: To establish and evaluate a large-scale, school-based case-detection process designed to efficiently and reliably identify inner city children with asthma.
Methods: A bilingual, seven-question, self-administered, parental asthma screening survey was developed. Clinical validation was achieved in a sample of 675 consecutive parents bringing a child to the school-based Breathmobile Program for initial evaluation, using a comprehensive evaluation by a physician specialist (ie, allergist) as the standard. Survey response patterns were used to construct a novel seven-model, tiered scoring algorithm and an abbreviated algorithm that predict the probability of a child being clinically classified as "yes asthma" or "no asthma." A systematic survey distribution process administered by a single coordinator was developed, and the impact of a classroom-oriented incentive offering a $25 school supply gift certificate for survey return rates of
80% was evaluated.
Results: A total of 636 parents provided one or more survey responses and information sufficient for clinical classification. The scoring algorithm correctly identified children with asthma (
80% probability) with a sensitivity of 86.5%, a specificity of 83.6%, and a misclassification rate of 14.3% (91 of 636 children). The sensitivity for identifying persistent asthma was 91.3%. Asthma prevalence estimates derived using survey results from a larger sampling of the general population were similar to rates previously reported for comparable populations. The inclusion of an inexpensive incentive increased the median survey return rates from 35.3 to 65% (z= -11.9; p < .001). The screening process has been used to conduct 27,526 surveys at inner city schools.
Conclusions: The Breathmobile case-detection process offers a validated, comprehensive, large-scale method with which to identify children with asthma at their school sites.
Key Words: asthma children inner city screening urban validation
| Introduction |
|---|
|
|
|---|
An important component of any intervention model designed to mitigate the impact of asthma on inner city children is the ability to efficiently and reliably identify children who are likely to have poorly controlled asthma. Ideally, this process would identify children with previously diagnosed asthma who are not receiving adequate therapy as well as children with undiagnosed asthma. Schools have received increasing attention as strategic sites for this process. School-based case identification methods that have been tested include parental surveys and exercise challenge procedures.14 15 16 17 Although surveys can be used more widely than other more labor-intensive approaches, an effective survey should demonstrate useful sensitivity and specificity when applied in large heterogeneous populations. It must be simple in content (to facilitate completion), yet sufficiently detailed to collect the information needed to construct prediction models. An ideal survey should accommodate the "real-world" variable response patterns that occur with self-administered surveys. School-based approaches also must incorporate methods to efficiently distribute surveys and achieve high return rates without disrupting the work environment for school personnel.
In this article, we describe the clinical validation as well as the complete and abbreviated scoring algorithms for a one-page, bilingual, self-administered, case identification survey. In addition, we describe an efficient, scalable school-based distribution process that can be used in a heterogeneous urban population to reliably identify children who may have either undiagnosed or poorly controlled asthma. Our model has been developed and utilized in the context of identifying and subsequently delivering specialized care to large numbers of inner city children with asthma in a comprehensive school-based treatment programthe Breathmobile Program.
| Materials and Methods |
|---|
|
|
|---|
Study Population for Clinical Validation of the Survey
Clinical validation of our asthma case identification survey was completed in a sequential sampling of 675 parents of children seen at 71 school sites during their initial evaluation in the Breathmobile Program between January 1, 2001, and January tenth, 2002. Participants were recruited by two methods that have been used routinely in the Breathmobile Program since its inception in 1995. First, fliers were sent to all parents at a school site. The fliers provided general information about asthma and offered a free asthma evaluation (as well as ongoing care for asthma and allergic disease in the program if indicated) to all children at the school site. In addition, school nurses at each site contacted the parents of children they suspected to have asthma, and offered the opportunity for evaluation and care in the program. Parents responding to either recruitment approach were given an appointment to be seen when the Breathmobile visited their childs school site. Our previous experiences indicated that these complementary recruitment efforts would provide a heterogeneous population including children with no asthma as well as children with mild, moderate, and severe asthma that would be appropriate for developing prediction models that would discriminate between no asthma and all asthma severity levels.
Validation of the Survey and Development of the Scoring Algorithms
The parental survey was validated by comparing survey responses to a comprehensive clinical assessment that was conducted by the physician specialist in the Breathmobile clinic. At the patients initial visit, the parent or guardian was asked to complete the survey prior to the childs clinical evaluation. Parents were not provided any specific instructions about completing the survey or any interpretation of the survey contents. Surveys were collected by program staff and were not viewed by the Breathmobile physician, who immediately completed a structured clinical assessment that included the following: vital signs; history; targeted physical examination; spirometry; and, when appropriate, allergy skin testing. Breathmobile physicians then classified the child as "yes asthma" or "no asthma." An asthma severity rating (scale, 0 to 4 [in accordance with the 1997 National Heart, Lung, and Blood Institute guidelines18
]) also was assigned if sufficient information was available.
Parental survey answers were evaluated to determine which response combinations provided the best probability estimates of a child being classified as yes-asthma or no-asthma. Seven different scoring models (models A through G) were developed to account for all of the parental response patterns that were observed, and a tiered scoring algorithm based on the presence of parental answers to different question elements was designed. This complete algorithm, which included all seven models, ensures that the most predictive model (in descending order of predictive validity) will be applied to assess parental survey responses. In our setting, the application of the complete scoring algorithm to a large number of surveys is most readily accomplished after the survey response data have been scanned into a database.
A second, abbreviated, scoring model was developed to facilitate a simpler survey and case identification process. In this model, a simple visual scoring method can be used. The best predictive value for the abbreviated model was achieved by assessing responses to five survey questions and two possible response patterns.
Field Application and Prevalence Estimates Derived From the Scoring Algorithm
Since the case identification survey was validated in a population with a high prevalence of asthma, prevalence estimates derived from the survey process and scoring algorithm were evaluated in a large representative school-aged population. The scoring algorithm was applied to surveys returned by parents of 4,110 children from 210 classrooms that had survey return rates of
80% during routine screening (see below). The
80% return rate was selected to establish prevalence estimates that were likely to represent the general school-aged population in urban Los Angeles. Asthma prevalence estimates based on survey answers indicating either "asthma" (a "yes" answer to question element 1A) or "probable asthma" (a
80% probability of being classified as yes-asthma using the complete scoring algorithm) were determined.
Classroom-Based Survey Process
For routine case identification, a classroom-based survey process was developed that facilitated the distribution and collection of surveys by mimicking the distribution process routinely used in the school environment. All survey procedures were designed to allow a single individual to conduct surveys at multiple sites at the same time, thereby minimizing costs.
The following stepwise approach was used to survey classrooms at participating schools. (1) The school administrative office provides the survey project coordinator with a list of the classroom numbers, the teachers names, and the number of children in each classroom. (2) The coordinator prepares a survey packet for each child in every classroom. Each packet contains a survey form with a preprinted numerical identifier. A bilingual (ie, English and Spanish) cover sheet with the same identifier is stapled to the survey form. This cover sheet explains the nature of the survey and informs parents that they will receive an individualized letter of the findings that may be given to their childs physician if the survey is returned. Parents are asked to write the childs name on the cover letter, and to return both the anonymous survey form and the paired cover letter to their childs teacher. (3) All survey packets for each classroom are placed in an envelope. A letter to the teacher explaining the survey process and the potential benefits for children found to have asthma is also included in each classroom envelope. The coordinator keeps a list of the survey identifier numbers distributed to each classroom. (4) The coordinator takes the classroom envelopes to the school office and places them in the appropriate teachers mailbox. (5) The teacher distributes a survey packet to each child with instructions to take the packet home, have a parent complete both forms, and return the packet to the classroom. (6) Returned surveys are collected by the teacher and placed in the original envelope, which is then taken to the school office, to be picked up by the project coordinator. (7) The coordinator scans the surveys into a database, using specially developed scanning software that includes quality-assurance checks for partially filled bubbles. The software automatically applies the scoring algorithm and produces an individualized letter with the appropriate identifier number (in the same language used for survey completion) summarizing the parents answers and suggesting whether their child may benefit from further evaluation for asthma. Three different letters, based on the probability estimate of the child having asthma, can be generated. (8) The cover page (which has the childs name and the survey identifier number) is stapled to the matched response letter, and the letters are returned to the appropriate classroom for distribution to the students.
Because the efficiency of a self-administered screening process depends heavily on survey return rates, the effect of a teacher-oriented incentive on survey return rates was evaluated during routine screening. Surveys were distributed to 27,526 children in 1,212 classrooms at 24 school sites (19 elementary schools and 5 middle schools). Survey response rates were evaluated in two phases. In phase I, surveys were distributed to classrooms in 11 schools without the use of any incentives. In phase II, classroom teachers at 13 schools were offered a $25 gift certificate from a commercial school supply entity if the survey return rate for their classroom was
80%.
Statistical Analysis
The parent response to each survey item was compared to the physicians assessment of their childs asthma status by calculating sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Univariate discriminant analysis procedures were used to estimate the probability of asthma based on the parent response to each item. Multivariate discriminant analysis assisted in identifying the combination of survey items that contribute most in classifying individuals as asthmatic or nonasthmatic. Discriminant functions were selected using stepwise procedures and prior probabilities of asthma based on physician assessment. F- to enter values19
in the stepwise analyses corresponded to a p value of 0.15, as recommended by Constanza and Afifi.19
The degree of success of the classification procedure was assessed through the examination of error rates computed using cross-validation by the jackknife method.20
The analysis required that a cut point be determined for the graded answer options offered in survey questions 2 to 7. Receiver operating characteristic curve estimates were evaluated to determine the cut point for each question that optimized sensitivity and specificity.21 To further validate the cut point selection, logistic regression analyses were conducted to assess where differences in the likelihood of asthma between response option levels were significant. For each survey item, any positive response compared to "never" resulted in a substantial increase in the likelihood of asthma. With the exception of question 4, the likelihood of being classified clinically as having asthma was not significantly influenced by gradations within the positive response options. In question 4, children who experienced exercise-related symptoms "rarely," compared to "sometimes," were less likely to be clinically classified as asthmatic (odds ratio, 0.40; p = 0.004). However, the reduction in relative risk was more pronounced for children who "never" compared to "rarely" experienced asthma-related symptoms during play or exercise (odds ratio, 0.15; p < 0.001). These results led to combining positive responses into an "ever" category and treating never as the null category for survey questions 2 to 7.
The impact of incentives on average return rates was evaluated by means of the Mann-Whitney U test. The effect of incentives on the percentage of classrooms with
80% return rates was evaluated using a
2 test. All statistical analyses were performed using a statistical software package (SPSS; SPSS Inc; Chicago IL).
| Results |
|---|
|
|
|---|
80% survey return rate. The survey validation population (self-selected) had a higher prevalence of asthma, a higher percentage of men, more African Americans, and a broader representation of all age groups than did the general population sampled during the general screening process.
|
|
These seven prediction models then were used in a tiered "best-fit" cascading scoring algorithm to evaluate survey responses. Thus, model A was applied if the survey response pattern was consistent with model A. If this was not the case, then model B was evaluated and applied if appropriate, and so on. The scoring algorithm (models A through G), the question elements included in each model, and the sensitivity and specificity for each model are shown in Table 3
. For each model, response patterns that predict a
80% probability of a child being classified as having asthma are shown.
|
The sensitivity and specificity following the application of the complete algorithm for a child to be classified as yes-asthma or no-asthma (with a probability of
80%) are shown in Table 4
. Also shown is the validity of the algorithm for intermittent vs persistent asthma, and for different demographic groups that were part of the clinical validation population. The survey demonstrated a greater sensitivity for detecting persistent vs intermittent asthma and showed greater specificity for women, younger children, and African Americans.
|
The sensitivity and specificity following the application of the abbreviated algorithm for a child to be classified as yes-asthma or no-asthma (with a probability of
80%) is shown in Table 5
. The abbreviated algorithm demonstrated patterns of sensitivity and specificity that were quite similar to the complete algorithm for intermittent vs persistent asthma and different demographic groups. Overall, the abbreviated version yielded a lower sensitivity (83.4% vs 86.5%, respectively), a higher specificity (85.4% vs 83.6%, respectively), a similar PPV (93.9% vs 93.5%, respectively), and a lower NPV (65.5% vs 69.4%, respectively) than the complete model. The abbreviated algorithm was able to estimate asthma probability for 635 of the 636 children in the validation population who were classified using the complete scoring algorithm.
|
80% (and, thus, were likely to be representative of the general population). In this largely elementary school population, the complete algorithm estimated the prevalence of known asthma to be 6.1%, with an overall prevalence of 14.1% including all children with "probable" asthma (ie, algorithm indicating
80% probability of being classified as yes-asthma). Our prevalence estimate for known asthma is in close agreement with the rate previously reported19
by the Los Angeles County Department of Health Services (Table 6
). Although prevalence estimates for probable asthma in urban Los Angeles have not been previously reported, our current estimate (14.1%) is similar to that derived from a similar asthma-screening effort using a bilingual take-home survey in six elementary schools in New York (Table 6)
. Our abbreviated algorithm (which could be used for 4,108 of the 4,110 children classified with the complete algorithm) estimated overall asthma prevalence at 12.2%.
|
80%). Since response rates appeared to be classroom-dependent (and, thus, possibly teacher-dependent), an incentive for classroom teachers was offered in phase II of the survey. Of the 15,449 surveys at 11 elementary schools (586 classrooms) and 2 middle schools (101 classrooms), a total of 8,896 were returned. Thus, the incentives increased survey return rates (median, 25th percentile, and 75th percentile) from 35.3% (0% and 60%, respectively) to 65% (40% and 80%, respectively) (zstatistic, -11.9; p < .001). The percentage of classrooms with
80% return rates increased from 9.3 to 27.5% (p < 0.001 [
2 test]). Return rates were highest in elementary schools, where incentives increased the median return rates from 45 to 65% (zstatistic, -8.4; p < 0.001). In middle schools, return rates improved from 21.7 to 59.4% (zstatistic, -6.1; p < 0.001). | Discussion |
|---|
|
|
|---|
The survey-based case identification process we have developed for use at inner city schools differs in several respects from the approaches described by others. First, our survey and case identification process are intended to identify all children who are likely to have asthma-related respiratory difficulties. This would include children with previously diagnosed asthma who are not receiving adequate controller therapy as well as children with undiagnosed asthma. Numerous studies (and our unpublished experiences during the past 7 years in the Breathmobile Program) have found that asthma is both underdiagnosed and undertreated in inner city children. Although efforts to selectively identify children with undiagnosed asthma have merit, it is likely that the impact of efforts to mitigate the effects of this disease on inner city children will be greatest if all children with asthma-related problems can be identified. We also have developed novel multimodeled, tiered scoring algorithms that accommodate the real-world variability in the completion of self-administered surveys. This approach allows children to be classified even when parents only partially complete the survey, a predictable event in large-scale case identification programs. Finally, our classroom-based survey process can be administered by a single coordinator with minimal assistance from busy school personnel.
The survey scoring algorithms that we have developed were also suitable for estimating the prevalence of asthma in a large representative sample of school-aged children in urban Los Angeles. The prevalence estimates determined for both known and probable cases of asthma are similar to previously reported values. The abbreviated algorithm estimated a prevalence of probable asthma in the general population that was 1.9% lower than that for the complete algorithm. This may reflect the fact that the abbreviated algorithm relies on only five question elements and has a sensitivity that is 3.1% lower than that of the complete algorithm.
Sample bias can significantly influence the results obtained during any survey validation. Our validation effort included the sampling of a sequential population of 675 parents who either responded to a widely distributed flier offering an asthma evaluation or who had children observed to have recurrent respiratory symptoms. It is likely that self-selection in response to the flier, and the inclusion of children with recurrent respiratory symptoms accounts for the high percentage of children in the validation population (465 of 636 children, 73.1%) who were found to have asthma. Prediction models for this and other inner city projects would likely be more precise or accurate if clinical validation was based on a true random sampling of the population. However, a number of factors, such as participation rates, subject age, school attendance policies, and socioeconomic factors combine to make studies of this nature problematic among urban children.
Because PPVs and NPVs are directly affected by prevalence, we have emphasized sensitivities and specificities in our analyses because these parameters are less likely to be influenced by the prevalence of asthma in the validation population. The values determined for our survey compare favorably to those reported for other instruments. Using the overall sensitivity and specificity for our complete scoring algorithm, an asthma prevalence of 15% would predict a PPV of 48% and a NPV of 97%. These calculated values suggest that a negative survey is a reliable predictor that a child would not be classified with asthma. On the other hand, a positive response to our survey will include some children who eventually will not be classified as having asthma. Overall, the false-positive rate for our survey would be estimated as 16.4% for the complete algorithm, and 14.6% for the abbreviated algorithm (1 - specificity). We view this as a favorable characteristic. Because asthma is both underdiagnosed and undertreated in inner city children, it is both reasonable and prudent that interventions intended to effectively change the health of inner city populations include case identification methods that emphasize not missing children with poorly controlled asthma.
It is important to note that clinical validation methods for any asthma case detection survey may not be generally applicable to all populations or in all settings. Our validation scheme, which utilized clinical assessment by an asthma specialist as the standard, was targeted to identify children with asthma at inner city schools. Thus, prediction estimates derived from use of the Breathmobile survey could vary if this survey were used to screen a different population, under different screening conditions, and/or in another setting.
A number of case identification methods using surveys to identify childhood asthma have been reported previously. Although the scope and setting of the survey process, as well as the validation methods used, have varied widely, these studies have demonstrated that parental answers to self-administered survey questions offer a useful and perhaps superior method for identifying children with asthma. Our findings support this notion. Moreover, the use of a simple, self-administered survey to identify children with asthma appears to have several advantages over more labor-intensive screening methods, such as the free running asthma screening test or spirometry. For the 636 children in our validation population, abnormal spirometry (ie, FEV1, < 80% predicted) would have detected asthma with a sensitivity of 20.1%, a specificity of 80.7%, a PPV of 86.7%, and a NPV of 20.9% (data not shown). Thus, spirometry alone, which is labor-intensive and requires face-to-face contact with each individual child and parent, would have missed most of the children with asthma. Comparable limitations were observed during validation of the International Study of Asthma and Allergy in Childhood survey.24 25 26
We also have examined some of the factors that influence survey response rates and/or affect screening program acceptance at inner city schools. Our experiences during the distribution of surveys to 27,526 children in 24 Los Angeles schools suggest that it may be difficult to implement a routine survey process that requires substantial time or commitment by school personnel at large, often overcrowded, urban schools. This may adversely affect the implementation of an exercise challenge as a routine asthma case detection procedure in inner city schools. We also have found that a single individual can coordinate a large-scale, routine survey program for asthma in urban schools when a short, take-home, self-administered parental survey is used. Survey distribution and collection can be done at two inner city school sites at a time, whenever schools are in session, and without any linkage to other school events. Finally, we have found that survey return rates of
80% can be achieved in many classrooms using inexpensive, teacher-oriented incentives that help classrooms obtain needed school supplies.
The Breathmobile case identification process and clinically validated survey offer an efficient, scalable, school-based approach for identifying children with asthma in a large urban population. Our findings, and those reported in other urban settings,14 indicate that large numbers of children can be reliably evaluated with a short parental survey administered at urban school sites. The prediction models derived from our clinical validation scheme accommodate the variable response patterns that are typically encountered in a heterogeneous urban setting. An efficient, scalable case detection process is an important step toward meeting the greater challenge of establishing interventions that lead to a sustainable improvement in the health of inner city children with asthma.
| Appendix |
|---|
|
|
|---|
| Acknowledgements |
|---|
| Footnotes |
|---|
The Los Angeles Breathmobile Program and survey screening project are funded in part by the Southern California Chapter of the Asthma and Allergy Foundation of America. Ongoing operations for the Los Angeles Breathmobile Program are funded by the Los Angeles County Department of Health Services. Development of the scanning software for the screening survey project was supported by Schering Inc.
Received for publication April 25, 2003. Accepted for publication October 15, 2003.
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
S. P. Galant, T. Morphew, S. Amaro, and O. Liao Current Asthma Guidelines May Not Identify Young Children Who Have Experienced Significant Morbidity Pediatrics, April 1, 2006; 117(4): 1038 - 1045. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. P. Galant, L. J.R. Crawford, T. Morphew, C. A. Jones, and S. Bassin Predictive Value of a Cross-Cultural Asthma Case-Detection Tool in an Elementary School Population Pediatrics, September 1, 2004; 114(3): e307 - e316. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |