Chest ACCP Education Calendar
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

Guest Access | Sign In via User Name/Password
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF) Free
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Article Archive
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (13)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Carson, S. S.
Right arrow Articles by Bach, P. B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Carson, S. S.
Right arrow Articles by Bach, P. B.
(Chest. 2001;120:928-933.)
© 2001 American College of Chest Physicians

Predicting Mortality in Patients Suffering From Prolonged Critical Illness*

An Assessment of Four Severity-of-Illness Measures

Shannon S. Carson, MD and Peter B. Bach, MD

* From the Department of Medicine, Division of Pulmonary and Critical Care Medicine (Dr. Carson), University of North Carolina, Chapel Hill, NC; and the Health Outcomes Research Group, Department of Epidemiology and Biostatistics and Department of Medicine (Dr. Bach), Memorial Sloan-Kettering Cancer Center, New York, NY.

Correspondence to: Shannon S. Carson, MD, Division of Pulmonary and Critical Care Medicine, University of North Carolina, 420 Burnette Womack Bldg, CB# 7020, Chapel Hill, NC 27599; e-mail: scarson{at}med.unc.edu


   Abstract
TOP
 Abstract
Introduction
 Materials and Methods
 Results
 Discussion
 References
 
Study objectives: Investigators have been using severity-of-illness indexes such as APACHE II (acute physiology and chronic health evaluation score II) to describe patients with prolonged critical illness. However, little is known about the utility of these indexes for this patient population. We evaluated the ability of four severity-of-illness indexes to predict mortality rates in 182 patients with prolonged critical illness.

Design: Retrospective inception cohort study.

Setting: A single, urban, long-term, acute-care hospital in Chicago.

Patients: One hundred eighty-two patients transferred from 37 acute-care hospital ICUs.

Measurements and results: We assessed four indexes: the acute physiology and chronic health evaluation II, the simplified acute physiology score II, the mortality prediction model II, and the logistic organ dysfunction system using variables measured on admission to the long-term acute-care hospital ICU. We found that none of these indexes distinguished well between the patients who lived and the patients who died (area under ROC [receiver operating characteristics] curve < 0.70 for all), nor did they assign correct probabilities of death to individual patients (Hosmer-Lemeshow goodness-of-fit statistics, p < 0.01 for all).

Conclusions: Investigators and clinicians should use caution in using severity-of-illness measures developed for acutely ill patients to describe critically ill patients admitted to long-term care units. As clinical practice and research focus more on these latter patients, development of adequately performing severity-of-illness measures appropriate to this patient population will be needed.

Key Words: chronically critically ill • epidemiology (methods) • hospital mortality • intensive care • long-term care • outcome assessment (health care) • prolonged critical illness • severity-of-illness index


   Introduction
TOP
 Abstract
 Introduction
Materials and Methods
 Results
 Discussion
 References
 
Patients experiencing prolonged critical illness demand high levels of resource intensive care. As a result of the unique medical needs of these patients and of the fiscal and bed-space burdens that they place on acute-care hospital ICUs, care for many of these patients has shifted to free-standing facilities such as skilled nursing facilities, regional weaning centers, long-term acute-care hospitals (LTACs), or specialized units within acute-care hospitals.1 2 3 4 5 Asthe number of patients experiencing prolonged critical illness continues to increase, understanding variations in care and costs, improving outcomes, and clarifying prognoses will all become increasingly important goals. To understand these patients and these facilities better, a reliable standard for measuring and comparing illness severity will be required.

Several investigators have used severity-of-illness measures developed in the acute-care hospital ICU setting to describe, for research purposes, illness severity in patients with prolonged critical illness.6 7 8 9 10 Others have described their use of these measures in triage and hospital admission decisions.11 12 While it is a fact that many patients admitted to LTAC hospitals are critically ill and require intensive care services, existing acute-care ICU scoring systems were not developed in the LTAC ICU setting and, therefore, may not be appropriate to this population. Before the use of acute-care ICU scoring systems in this patient population becomes more common, a detailed analysis of the performance of acute-care ICU severity-of-illness measures in critically ill patients admitted to LTAC hospitals is warranted.

Severity-of-illness measures widely used in acute- care hospital ICUs include the APACHE (acute physiology and chronic health evaluation) scores,13 the simplified acute physiology score (SAPS),14 the mortality prediction model (MPM),15 and the logistic organ dysfunction system (LODS).16 These measures combine physiologic variables and sometimes measures of chronic health problems to calculate a score that corresponds to relative illness severity. In each case, the score can be transformed into a probability of death through the use of a logistic regression equation. In this study, we assessed the ability of APACHE II, SAPS II, MPM II, and LODS to predict mortality rates in a group of critically ill patients admitted to an LTAC hospital ICU.


   Materials and Methods
TOP
 Abstract
 Introduction
 Materials and Methods
Results
 Discussion
 References
 
Setting
The patient data we used for this study have been previously described, as have the characteristics of the facility, the methods for gathering patient information, and the time period of observation.6 7 In brief, the facility is a 75-bed LTAC hospital in an urban location. The hospital has a seven-bed ICU, with full cardiovascular monitoring capabilities. Patients are accepted regardless of illness severity provided their conditions are stable for transfer. Transfer of patients receiving positive end-expiratory pressure of > 10 cm H2O or fraction of inspired oxygen of > 0.60 requires approval by the medical director. Patients receiving mechanical ventilation are required to have a tracheostomy. The facility is staffed by specialists in pulmonary and critical care medicine. Consultation services are available for most medical and surgical subspecialties, psychiatry, and clinical psychology. Patients are discharged from the LTAC after complex medical issues are resolved and they are safely liberated from treatment with mechanical ventilation or if multiple attempts at liberation from mechanical ventilation are unsuccessful.

Study Patients
Study patients include all patients transferred to the LTAC hospital between August 1, 1995 and July 31, 1996, from ICUs, respiratory care units, or step-down units at 37 acute-care hospitals. Only the initial hospital admission was considered for analysis. Patients who received mechanical ventilation long-term before their index hospitalizations were excluded.

Patient Data
Acute physiology variables measured during the first 24 h of LTAC admission were obtained by review of medical records. Records from the transferring facility were reviewed for comorbidities, the primary diagnosis leading to initial ICU admission, and the diagnosis responsible for continued ICU admission. Patients were considered to have survived to discharge from the LTAC if they were discharged to any other facility or to home. Any patient who died at the LTAC was considered a death, regardless of the level of care he or she was receiving at the time of death. All patients were observed until one of these events occurred.

Severity-of-Illness Calculations
The techniques and scoring algorithms for each of these scores have been published previously.13 14 15 16 Approaches and adaptations specific to this patient population are mentioned below:

• APACHE II: All patients received five chronic health points because of their ventilator dependence or other organ system dysfunction that was characteristic of the patient population. The diagnosis leading to continued ICU admission, as determined by the investigators, was used as the input for the diagnostic category weight.

• MPM II: Complete data were not available on prior GI bleeding. All patients were considered to be physician-recommended ICU admissions, as none had undergone surgery within 1 week of admission. For the cardiovascular support score, the requirement of vasoactive drug therapy was recorded, but specific dosages of vasopressor drugs were not available.

• LODS: With the exception of serum urea measurement within 24 h of ICU admission, data were complete for calculation of this score.

• SAPS II: Data were complete for calculation of this score.

Analytic Methods
Severity-of-illness indexes were evaluated for two separate performance characteristics: discrimination and calibration.17 Discrimination refers to the ability of the index to correctly rank a group of patients in order of the likelihood that each individual patient dies. To assess each instrument’s ability to discriminate, we examined three types of outcomes. For the binary outcome (dead or alive at hospital discharge), we examined the mean difference in scores between survivors and nonsurvivors using the Student’s t test and the empiric area under the receiver operating characteristic (ROC) curve (AUC) generated from a logistic regression model.18 AUC values > 0.5 reflect discriminative power, with higher values reflecting better discrimination. For the time-dependent outcome death, we examined the Harrell’s concordance probability (or C statistic).19 20 Similar to ROC analysis, the Harrell statistic is a measure of the likelihood that, of two randomly chosen patients, the one with the lower score will live longer than the one with the higher score. As such, better scores on the concordance probability reflect the ability of the index to predict survival time. For this latter analysis, patients were censored at the time of hospital discharge.

Calibration refers to the ability of the index to accurately determine the probability of death. We calculated an individual patient’s probability of death as proposed by the developers of each index. We evaluated both general calibration and specific calibration of each index. To determine the general level of calibration of each index, we compared the calculated probability of death for the cohort to the observed death rate. We then determined the patient-specific calibration of each index by calculating the Hosmer-Lemeshow goodness of fit statistic for eight equally sized groups and 8 degrees of freedom (df), ordered based on their predicted probability of death.18 Each analysis was performed on all available data. All p values are two-sided. Analyses were performed with STATA 6.0 software (Statacorp; College Station, TX).

The project was reviewed and approved by the Institutional Review Board of the University of Chicago, Chicago, IL.


   Results
TOP
 Abstract
 Introduction
 Materials and Methods
 Results
Discussion
 References
 
Patient Characteristics
During the study period, there were 187 patients admitted to the LTAC from outside hospitals, of whom 182 had complete information necessary to calculate all four severity-of-illness indexes. The average patient had been in an acute-care hospital ICU or step-down unit for 27 days (range 10 days to 101 days) before transfer. The average age was 70 years. Roughly half of the patients were women. In general, patients were awake and following commands. Fourteen percent were receiving vasoactive drugs, and 73% were receiving mechanical ventilation for an average of 25 days before transfer. The predominant diagnoses leading to initial ICU admission at the acute-care hospital included acute lung injury (26%), primary cardiac disease (17%), exacerbation of chronic lung disease (14%), and CNS disease (including seizures and cerebrovascular events, 13%). Diagnoses leading to continued ICU admission are included in Table 1 . Seventy-seven patients (42%) died before discharge from the long-term care facility; 105 patients (58%) survived to discharge.


View this table:
[in this window]
[in a new window]

  		Table 1. Diagnoses Leading to Continued ICU Admission*

 
Discrimination of Severity-of-Illness Measures
Each of the four severity-of-illness measures demonstrated some ability to discriminate between patients who died and those who survived, based on three assessments (Table 2 ). In aggregate, survivors had lower SAPS II scores, lower LODS scores, lower APACHE II scores, and lower MPM II scores than did patients who died (all p values < 0.01). By the ROC, all four scales had statistically significant discriminative ability. SAPS II had the most discriminative power, with an AUC of 0.69. LODS had an AUC of 0.65, APACHE II had an AUC of 0.63, and MPM II24 had an AUC of 0.59.


View this table:
[in this window]
[in a new window]

  		Table 2. Ability of Four Severity-of-Illness Indexes to Discriminate Among In-hospital Deaths of Patients Admitted to an LTAC Hospital (n = 182), as Assessed by Mean Difference in Score, ROC Analysis, and Concordance Probability

 
To gain some insight into these AUCs, we compared them to the published AUCs of these indexes when used to predict mortality rates in acutely ill patients. In contrast to the AUCs for this LTAC population, which ranged from 0.59 to 0.69, SAPS II had an AUC of 0.87, APACHE II had an AUC of 0.81, SAPS II had an AUC of 0.87, and MPM II24 had an AUC of 0.84. Therefore, each index demonstrated substantially poorer discrimination in the LTAC population than in the acutely ill ICU populations.

Concordance Probability
In Table 2 , we present the values for the Harrell’s concordance probability for each statistic. Similar to ROC analysis, a value of 0.5 would denote that the scale provided no discrimination. The values in the table reflect a slight improvement over chance but in no case suggest a high degree of discriminatory power.

Calibration of Severity-of-Illness Measures
Overall, 42% of the patients at the LTAC died. Generally, only the APACHE II scoring system was calibrated correctly, producing an estimated mortality rate of 42% as well. All other indexes, which relied more heavily on acute physiology and less on chronic health impairments, underpredicted mortality. SAPS II predicted a 23% mortality rate, MPM II predicted a 21% mortality rate, and LODS predicted a 17% mortality rate.

Because the APACHE II system was calibrated correctly to the cohort overall, we display the results of the Hosmer-Lemeshow test on our cohort for this severity-of-illness measure (Table 3 ). The Hosmer-Lemeshow goodness-of-fit test provides a further assessment of calibration, in that it assesses the extent to which each scale predicts an individual patient’s risk of death. In this test, the cohort is broken into a number of groups based on the predicted probability of death for each individual. For the example in Table 3 , the entire cohort is broken up into eight groups. Group 1 has 24 patients in it, and, based on their APACHE II scores, this group has a predicted probability of death of 15.6%. In other words, based on the APACHE II scores of these 24 patients, we would expect 0.156 x 24 patients (or 3.74 patients) to die. We actually observed five deaths in this group. Group 2 contains the patients with the next highest probabilities of death, averaging 23.6%. For the 22 patients in this group, we anticipated 0.236 x 22, or 5.19 deaths. We observed nine deaths. At each level of probability, the predicted and the actual mortalities are compared. Greater deviations between predicted deaths and actual deaths increase the C statistic, where higher values reflect poorer calibration. In the case of the APACHE II, the patient-specific calibration was poor (C = 20.5, df = 8, p = 0.0086).


View this table:
[in this window]
[in a new window]

  		Table 3. Hosmer-Lemeshow Goodness-of-Fit Test for a Group of Patients at an LTAC, Using the APACHE II Severity Index*

 
Given that our primary analysis demonstrated that the other three indexes vastly underpredicted mortality rates in the cohort, we would not expect these indexes to predict individual probabilities of death accurately. In fact, the Hosmer-Lemeshow goodness-of-fit statistic demonstrates very poor calibration for each of these scales: MPM II (C = 85.1, df = 8, p < 0.0001); SAPS II (C = 57.9, df = 8, p < 0.0001); LODS (C = 49.5, df = 8, p < 0.0001).


   Discussion
TOP
 Abstract
 Introduction
 Materials and Methods
 Results
 Discussion
References
 
This study was undertaken because investigators have been using acute severity-of-illness measures to describe illness severity, to conduct research, and to make triage decisions in the care of long-term critically ill patients.6 7 8 9 10 11 12 Prior to this study, there were no data regarding the suitability of these measures to this population. In this study, we evaluated four common severity-of-illness indexes to determine which, if any, could potentially be used in this setting. We found that SAPS II, LODS, APACHE II, and MPM II all discriminated between patients who would live and patients who would die, and that SAPS II demonstrated the best discrimination in the cohort, with an AUC of 0.69. In reviewing the published performance of these indexes on acute-care ICU cohorts, we found that this level of discrimination was well below that achieved in acute-care hospital ICU settings.

We also evaluated the calibration of these indexes and found that the indexes were poorly calibrated to the patient population. Three of the four indexes underpredicted mortality in our cohort, whereas APACHE II predicted overall mortality correctly but was poor at discriminating between individual patients in terms of risk of death. Whereas all four of the indexes are based on measures of acute physiologic derangements, APACHE II takes into account chronic organ dysfunction and diagnosis leading to ICU admission. If outcomes for chronically critically ill patients are more dependent on underlying disease, long-term organ dysfunction, and physical functioning than on physiologic derangements measured during the first day of hospital admission, this result would make sense. It is also possible that the similarity between observed mortality rates and mortality rates predicted by APACHE II was a chance occurrence. Therefore, the findings of this study should be confirmed in other groups of LTAC patients.

We must emphasize that, at present, none of the indexes, including APACHE II, are calibrated well for individual patients. When patients were stratified based on risk of death, the differences in the number of observed vs predicted deaths were large, as evidenced by a large Hosmer-Lemeshow test statistic (Table 3) . Therefore, we can conclude that none of these scoring systems were well calibrated to individuals in this LTAC hospital, despite the fact that one of the systems was well calibrated to the group overall. There are other examples in the literature where severity-of-illness indexes performed very well when applied to large groups of patients, but mortality rate predictions for individual patients were less reliable.21 22

This study has several limitations. Data were obtained by review of medical records from the LTAC and referring hospitals; therefore, data for all variables in the MPM II model were not complete. Additionally, this study was performed at a single institution, so the data may not be applicable to all LTACs or long-term ventilator units.

Severity-of-illness scoring systems such as the ones in this study were designed and validated in acutely ill patients using variables gathered during the first 24 h of admission to acute-care hospital ICUs. The patients, facilities, and approaches to care are different in LTAC hospitals, so it is not surprising that these indexes do not perform at the same level in LTAC patients as they do in the populations for whom they were intended. These instruments should not be used for clinical decision making in this patient population, and their use for description of research cohorts should be interpreted with caution.

To optimally perform the tasks expected of severity-of-illness measures, new scales that are developed and validated in a large population of patients with prolonged critical illnesses are necessary. Development of a severity-of-illness measure for chronically critically ill patients may require a novel approach. Survival to discharge may not be an appropriate outcome, as patients whose conditions clinically worsen can be transferred to acute-care hospitals causing the most severely ill patients to be classified as hospital survivors. Other outcomes, such as long-term survival, return to independence, or other qualitative measures may be more appropriate. In a model that we described, but which has not yet been validated, poor functional status prior to acute illness and advanced age were stronger predictors of 1-year mortality in patients requiring prolonged treatment with mechanical ventilation at an LTAC than were any measures of organ dysfunction.7


   Footnotes
 
Abbreviations: APACHE II = acute physiology and chronic health evaluation score II; AUC = area under the ROC curve; df = degrees of freedom; LTAC = long-term acute-care hospital; LODS = logistic organ dysfunction score; MPM II = mortality prediction model II; ROC = receiver operating characteristic; SAPS II = simplified acute physiology score II

This study was supported by an unrestricted grant from the Transitional Hospitals Corporation and by the Robert Wood Johnson Foundation.

Received for publication August 21, 2000. Accepted for publication February 21, 2001.


   References
TOP
 Abstract
 Introduction
 Materials and Methods
 Results
 Discussion
 References
 

  1. Scheinhorn, DJ, Artinian, BM, Catlin, JL (1994) Weaning from prolonged mechanical ventilation: the experience at a regional weaning center. Chest 105,534-539[Abstract/Free Full Text]
  2. Bagley, PH, Cooney, EA (1997) Community-based regional ventilator weaning unit: development and outcomes. Chest 111,1024-1029[Abstract/Free Full Text]
  3. Daly, RF, Rudy, EB, Thompson, KS, et al (1991) Development of a special care unit for chronically critically ill patients. Heart Lung 20,45-52[ISI][Medline]
  4. Elpern, EH, Silver, MR, Rosen, RL, et al (1991) The noninvasive respiratory care unit: patterns of use and financial implications. Chest 99,205-208[Abstract/Free Full Text]
  5. Gracey, DR, Naessens, JM, Viggiano, RW, et al (1995) Outcome of patients cared for in a ventilator-dependent unit in a general hospital. Chest 107,494-499[Abstract/Free Full Text]
  6. Bach, PB, Carson, SS, Leff, A (1998) Outcomes and resource utilization for patients with prolonged critical illness managed by university-based or community-based subspecialists. Am J Respir Crit Care Med 158,1410-1415[Abstract/Free Full Text]
  7. Carson, SS, Bach, PB, Brzozowski, L, et al (1999) Outcomes after long-term acute care: an analysis of 133 mechanically ventilated patients. Am J Respir Crit Care Med 159,1568-1573[Abstract/Free Full Text]
  8. Scheinhorn, DJ, Chao, DC, Stearn-Hassenpflug, M, et al (1997) Post-ICU mechanical ventilation: treatment of 1,123 patients at a regional weaning center. Chest 111,1654-1659[Abstract/Free Full Text]
  9. Latriano, B, McCauley, P, Astiz, ME, et al (1996) Non-ICU care of hemodynamically stable mechanically ventilated patients. Chest 109,1591-1596[Abstract/Free Full Text]
  10. Nasraway, SA, Button, GJ, Rand, WM, et al (2000) Survivors of catastrophic illness: outcome after direct transfer from intensive care to extended care facilities. Crit Care Med 28,19-25[CrossRef][ISI][Medline]
  11. Daly, BJ, Gorecki, J, Sadowski, A, et al (1996) Do-not-resuscitate practices in the chronically critically ill. Heart Lung 25,310-317[CrossRef][ISI][Medline]
  12. Rudy, EB, Daly, BJ, Douglas, S, et al (1995) Patient outcomes for the chronically critically ill: special care unit versus ICUs. Nurs Res 44,324-331[ISI][Medline]
  13. Knaus, WA, Draper, EA, Wagner, DP, et al (1985) APACHE II: a severity of disease classification system. Crit Care Med 13,818-829[ISI][Medline]
  14. Le Gall, J-R, Lemeshow, S, Saulnier, F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270,2957-2963[Abstract]
  15. Lemeshow, S, Teres, D, Klar, J, et al (1993) Mortality probability models (MPM II) based on an international cohort of intensive care patients. JAMA 270,2478-2486[Abstract]
  16. LeGall, J-R, Klar, J, Lemeshow, S, et al (1996) The logistic organ dysfunction system: a new way to assess organ dysfunction in the ICUs. JAMA 276,802-810[Abstract]
  17. Lemeshow, S, Klar, J, Teres, D (1995) Outcome prediction for individual intensive care patients: useful, misused, or abused? Intensive Care Med 21,770-776[CrossRef][ISI][Medline]
  18. Hosmer, DW, Lemeshow, S (1989) Applied logistic regression. Wiley New York, NY.
  19. Harrell, FEJ, Lee, KL, Mark, DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15,361-387[CrossRef][ISI][Medline]
  20. May, S, Hosmer, DW (1998) A simplified method of calculating an overall goodness-of-fit test for the Cox proportional hazards model. Lifetime Data Anal 4,109-120[CrossRef][ISI][Medline]
  21. Knaus, WA, Harrell, FE, Lynn, J, et al (1995) The SUPPORT prognostic model: objective estimates of survival for seriously ill hospitalized adults. Ann Intern Med 122,191-203[Abstract/Free Full Text]
  22. Fox, E, Landrum-McNiff, K, Zhong, Z, et al (1999) Evaluation of prognostic criteria for determining hospice eligibility in patients with advanced lung, heart, or liver disease. JAMA 282,1638-1645[Abstract/Free Full Text]



This article has been cited by other articles:


Home page
 ChestHome page
N. R. MacIntyre, S. K. Epstein, S. Carson, D. Scheinhorn, K. Christopher, and S. Muldoon
Management of Patients Requiring Prolonged Mechanical Ventilation: Report of a NAMDRC Consensus Conference
Chest, December 1, 2005; 128(6): 3937 - 3954.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF) Free
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Article Archive
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via ISI Web of Science (13)
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Carson, S. S.
Right arrow Articles by Bach, P. B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Carson, S. S.
Right arrow Articles by Bach, P. B.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS