| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
Guest Access | Sign In via User Name/Password
|
|
|
|||||||
Guest Access | Sign In via User Name/Password |
|||||||||
* From the Case Western Reserve University School of Medicine at MetroHealth Medical Center, Cleveland, OH.
Correspondence to: Allan Garland, MD, MA, Division of Pulmonary and Critical Care Medicine, MetroHealth Medical Center, 2500 MetroHealth Dr, Cleveland, OH 44109; e-mail: agarland{at}metrohealth.org
 |
Abstract |
|---|
 TOP Abstract IntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
Key Words: health services research • ICU • organization and administration • quality of health care • total quality management
|
Introduction |
|---|
|
TOPAbstractIntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
The importance of ICUs is evidenced by a number of observations. One third to one half of Americans spend time in an ICU during their final year of life,12 and one fifth die there.345 Beyond death rates, suffering is common among ICU patients.6789 Substantial dissatisfaction among relatives and friends of ICU patients101112 indicates that suffering is not limited to the patients. In addition, the economic costs of ICU care are staggering. One day in an ICU costs $2,000 to $3,000, which is sixfold higher than those for non-ICU care.1314 Although they contain just 8% of acute care hospital beds,1516 ICUs consume 20% of the total in-patient expenditures.161718 This equals 0.9% of all economic activity in the United States, or $91 billion in 2001,19 and ICU utilization is increasing rapidly.2 ICUs are a smaller portion of the health-care systems in other industrialized countries, but they still represent an important and disproportionate segment of medical care and costs.17202122232425
A necessary starting point for efforts to improve health care is to recognize that problems of quality are common and serious throughout our health-care system. The US health-care system is by far the most expensive in the world,19 while ranking 37th in overall performance and 72nd in population health.26 This poor performance is not explained by the violence, health habits, or cholesterol levels of society.27 The Canadian health-care system is the 10th most expensive, ranking 30th and 35th on these measures, respectively.
There are many contributors to this poor performance. A recent Institute of Medicine Roundtable28 stated that:
Serious and widespread quality problems exist throughout American medicine. These... occur in small and large communities alike, in all parts of the country and with approximately equal frequency in managed care and fee-for-service systems of care. Millions of Americans are not reached by proven effective interventions that can save lives and prevent disability. Perhaps an equal number suffer needlessly because they are exposed to the harms of unnecessary health services. Large numbers are injured because preventable complications are not averted... Quality of care is the problem.
With nonadherence to established standards of care being related to poor outcomes,2930 only 50 to 70% of Americans receive the care that is recommended for their situations31 and 20 to 30% receive inappropriate medical interventions.27313233 Medical errors and hospital-acquired complications are prevalent, often leading to disability, resulting in large costs and 27,000 to 98,000 deaths per year.2734353637
The data that exist demonstrate that ICUs share these problems. Certain subsets of iatrogenic complications in ICUs occur in 31% of patients and are severe in 13% of patients.38 Errors were observed to occur in 1% of all the activities performed each day in patients in an Israeli ICU, with a higher rate for physicians than nurses.39 These problems are rendered even more disturbing by the finding that one third of ICU nurses and physicians denied having ever made an error in the ICU, whereas at the same time they said that many errors are neither acknowledged nor discussed.40
Poor communication, teamwork, and problem solving among ICU staff are common, and are perceived as being more prevalent and important by ICU nurses than physicians.404142 These lead to a poor understanding of shared goals and worse patient outcomes.4344
Another problem is the variation in practice and outcomes that is not explained by patient or illness characteristics. Because wide and widespread variation could not exist if most practitioners practiced optimally, such variation is evidence that suboptimal care is common. Variation has been found for many outcomes in a broad range of settings, with important differences noted by geographic region,14546 hospital,474849 physician,4850515253545556 and insurance status or payer system.505758596061 Such variation occurs in ICUs as well.535759626364 For example, the odds ratio across 34 ICUs for using pulmonary artery flotation catheters was found to vary by 38% according to the patient’s race and by 33% according to their insurance status, but by 200 to 400% according to how the ICU was organized and staffed.64 Lest one think otherwise, such variation is not just an American phenomenon.56656667686970
Given the frequency of death in ICUs, it is particularly troubling that ICUs suffer from major deficiencies regarding palliative and end-of-life care. ICU patients or their surrogates are often dissatisfied with the amount, nature, and clarity of communications with caregivers.117172 These contacts, which are often delayed73747576 and too brief,71 lead to confusion,71 conflict,1172 and uncertainty about the goals of therapy.73 Thus, in addition to the high prevalence of suffering among the ICU patients referenced above, ICU patients frequently receive care the philosophy and intent of which are inconsistent with their wishes.673 In addition, the variation in practice present in other aspects of ICU care includes end-of-life and palliative care.1707475777879
These data demonstrate that ICU care is important, expensive, and problematic. Therefore, vigorous efforts are needed to critically examine and improve ICUs. Because quality is a vague term that is used to subtend a variety of other vague concepts, I will instead refer to the more operational concept of ICU performance and will use the term performance improvement (PI) in place of the various alternatives.
Because it would require us to believe that ICU care is different than every other area of human endeavor, it is not plausible that all ICUs perform equally well or that any given ICU is performing optimally. To identify how well a given ICU performs requires the quantitation of relevant indexes of performance. However, ICUs are complex organizations, and it is challenging to clearly define or measure such indexes. In fact, the meaning, scope, and measurement of performance in health care have evolved and broadened over the past 2 decades. Some of the most widely used approaches to improving performance in health care have proven inadequate. This article will discuss both the conceptual basis and the practical aspects of a superior method of evaluating and improving ICU performance.
|
Defining ICU Performance |
|---|
|
TOPAbstractIntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
There are a number of domains within which an ICU should be judged (Table 1 ). Although ultimately an ICU exists to serve the medical needs of critically ill patients, it also provides important services for families and friends of patients, health-care workers in the ICU, the hospital, and society. Judging ICUs based only on patients’ health outcomes fails to recognize the larger social value associated with expert care of these patients. In addition, no single metric is adequate to address any of the categories of outcomes listed in Table 1.
|
The ICU readmission rate is not listed in Table 1 because it is, at best, a questionable indicator of ICU performance. Its potential value derives from observations that readmitted patients have higher mortality rates and longer lengths of stay.82 However, for it to be a meaningful surrogate requires the following: (1) a detrimental outcome occurred after the patient left that was due to a problem present prior to the original ICU discharge; and (2) it would not have occurred if the patient had remained longer in the ICU. There are no data8283 that have demonstrated this. The optimal ICU readmission rate is unknown, and a low rate might actually indicate that, on average, patients are inappropriately remaining in the ICU longer than necessary, increasing the costs of care and their exposure to virulent pathogens.
All of the performance parameters listed in Table 1 have limitations. ICU or hospital mortality rates are commonly used measures that are relatively simple to collect. Whereas some data indicate that hospitals with higher short-term death rates have more preventable deaths,29 it is long-term survival and quality of life (QOL) that are most important to people.848586 Also, certain attitudes about death and dying can mean that higher short-term mortality rates represents superior care by virtue of being more concordant with patients’ end-of-life wishes.8788 Similarly, ICU and hospital lengths of stay are problematic outcome measures. Although daily costs are reduced by transferring patients out of the ICU to ward beds,14 premature transfers lead to worse outcomes.899091 Reductions in the lengths of stay and short-term mortality rates may merely reflect a shifting of the place of death from one location to another, with no real net improvements.92 Thus, using short-term mortality rates or lengths of stay as outcomes can lead to erroneous conclusions. The best way to avoid this pitfall would be to interpret the short-term outcomes in the context of longer term outcomes.
The laborious nature of collecting data on posthospital survival and QOL undoubtedly contributes to low usage of these important measures.93 It is even more work to combine these into a measure of long-term survival that is adjusted for the QOL, such as quality-adjusted life years.94 A variety of questionnaire-based tools have been developed for quantifying patients’ QOL.95969798 The most commonly used is the Medical Outcomes Study 36-item short form.99 These questionnaires generally gauge health-related QOL in multiple domains, with heavy emphasis on function.95100 Indeed, ICU survivors often have demonstrated poor QOL.101102103104 The assessment of cognitive function, depression, and posttraumatic stress disorder also shed light on the long-term consequences of ICU care.105106
Complication and error rates are often used as measures of ICU performance. These are relevant because of potential causal relationships of such adverse events with increased mortality, morbidity, or costs.34 However, such adverse events do not necessarily lead to clinically relevant consequences, and some studies3738 have found little effect. Therefore, care must be taken to ensure that an established relationship to relevant outcomes exists for any medical error or complication the rate of which is used as a surrogate of ICU performance. Because only a fraction of medical practices have been rigorously proven to be efficacious,107 deviations from recommended practice may have no such relationship.
Symptom control and end-of-life decision making are important aspects of ICU care. As discussed above, there is much room for improvement in this area. Use of these outcomes as measures of ICU performance has been limited by lack of training and orientation among physicians,108109 a paucity of preexisting tools to measure them,110 and other factors.77
Because ICU care is expensive, resource consumption should be part of the assessment of ICU performance at every institution. The best measure that balances simplicity and information content is ICU length of stay, although as discussed above, this has limitations. Other measures that require much effort to acquire include total monetary charges or costs,111 the usage of various diagnostic and/or therapeutic procedures, and therapeutic intervention scoring system (TISS) score.112 The TISS is a measure of ICU resource utilization that works well for cohorts.113114 However, because spending a lot of money is justifiable if the benefits are commensurately large, whereas even small expenditures that generate no benefits are wasted, resource use is most relevant in combination with the noneconomic outcomes in Table 1. Although the formalism associated with this concept, cost-effectiveness, is well beyond the scope of most local attempts to improve ICU performance,115 it would be a powerful tool to assist society in clarifying the value of ICUs, as well as to assess the performance of individual ICUs. A somewhat simpler approach to assessing cost-effectiveness that can be adapted to use within a single ICU has been described by Rapoport et al.116
The effective use of ICU beds is important, because they are an expensive and limited resource. Arguing that the rationing of critical care is common but often inequitable, Kalb and Miller117 developed a thoughtful framework for ICU triage in which they proposed that the use of critical care be limited to clinical settings in which it has been demonstrated, or at least is presumed, to be cost-effective. They and others118119 have provided arguments for using medical suitability as the primary determinant of such triage decisions. They propose that a low priority for ICU beds be assigned to those patients who are unlikely to benefit either because they are not very ill or because they are hopelessly ill. In practice, ICU triage decisions are often inefficient but can be made more effective without adverse medical consequences.120 The dynamic between need and availability makes this a complex topic, as evidenced by studies8991 showing that the transferring of patients out of the ICU because of limited bed availability has serious negative consequences. Possible measures of utility include the following: (1) the percentage of ICU patients who receive care that is achievable elsewhere in the hospital; (2) the fraction of patients for whom ICU care represents an exercise in medical futility;121122123 and (3) patients who remain in an ICU longer than their need for its special capabilities. Although the rates of adherence to written or published ICU admission and discharge standards is a common means of measuring the quality of ICU bed utilization, such standards have not been subjected to the scientific validation of whether they affect relevant outcomes and, thus, cannot yet be endorsed for this purpose.124 On the other hand, for some conditions data exist that permit an evidence-based approach to assessing the appropriateness of ICU triage decisions.125126127 There is even a small amount of literature128 addressing structural approaches to improving clinically relevant outcomes by improving ICU triage decisions.
The importance of satisfaction among patients and their families as measures of ICU performance is highlighted by data11127172 documenting that poor communication and dissatisfaction are common. However, because acquiring satisfaction data requires questionnaires or interviews that are unfamiliar and time-consuming to administer and analyze, little is usually done in this area. There are numerous potential dimensions to such surveys in ICU care, including satisfaction with the following: (1) level of care from physicians, nurses, and other ancillary health-care personnel; (2) involvement in decisions regarding care; (3) amount and quality of communications with health-care and administrative personnel; (4) outcomes of care; (5) administrative hospital functions, such as admissions, discharges, and billing; (6) food; and (7) housekeeping. Although some have disputed the validity of currently used methods,129130 a variety of tools exist to measure these aspects of satisfaction, including some created specially for use in ICUs.101272131132133134
The satisfaction of all those who work in an ICU is another key component of ICU performance. Job dissatisfaction contributes to higher rates of staff turnover,135 which has the following effects: (1) training time and money is wasted136137; (2) staff morale is further degraded while stress on managers and remaining staff is increased; and (3) the ability of the ICU to perform as an experienced, highly functioning team is diminished,138 possibly leading to worse patient outcomes.139 The nursing shortage is already substantial,140141142 especially in ICUs.143 A respiratory therapist shortage is not far behind,144 and a shortage of ICU physicians is looming.145 Problems related to dissatisfaction and turnover are obviously worse if those who quit cannot be replaced. In fact, dissatisfaction and burnout is common among nurses,146147148149 respiratory therapists,150 and ICU physicians.151152153154155 Whereas staff retention rates are easily obtained from personnel records, data about job satisfaction are collected from questionnaires or interviews. Many survey tools exist to assess job satisfaction, burnout, and related constructs.146151154156157158159160161162163 Because of the disparities in the importance assigned to these issues among different groups of ICU personnel,42 in these efforts it is necessary to separately address the job satisfaction of each group.
A large class of performance measures quantify the processes, procedures, and functions occurring within the ICU or linking the ICU to the rest of the hospital. Although the importance of such measures derives mainly from their coupling to the more directly relevant measures of performance listed in Table 1, they possess independent importance because of their relationship to the quality of interpersonal interactions and the staff’s perceptions of their workplace. An example is inefficiency or conflict in the course of interactions between the ICU and the pharmacy. Even in the absence of identified errors in drug administration resulting from such incidents, they have the potential to lead to errors, waste, or rework, and, in any case, generate negative feelings between the staffs.
|
Measuring and Interpreting ICU Performance |
|---|
|
TOPAbstractIntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
Although many performance measures can be phrased in terms of an AE that occurs in relation to individuals, it is fundamental that the detection and analysis of individual AEs cannot be used to measure performance. This does not mean that the analysis of individual AEs has no role in PI, but knowing that any given number of deaths or even potentially avoidable errors occurred cannot inform us about performance without determining the rate of AEs. The number of individual events must be pooled (numerator) and expressed as the rate of events by dividing by the aggregate number of at-risk patients, patient-days, or other appropriate denominator. This rate then is compared with some standard. The benchmark for comparison could be some accepted norm (ie, the rate in other ICUs, the rate derived from databases, or the prior rate in that same ICU). It is this form of cumulative information (eg, rates of death, errors, complications, staff turnover, or family dissatisfaction) that truly represents ICU performance.
There are multiple reasons why simply accumulating information about AEs, without their rates, is inadequate and potentially misleading. First, not all AEs are attributable to poor care. Even with the implementation of every known beneficial practice, it is not plausible that AEs will disappear; rather, the goal is to reduce their rate to an acceptable level or below what it was previously. Second, an individual AE does not necessarily cause harm or increase costs (eg, administering the incorrect dose of a drug often does no harm). Third, even an AE that is apparently caused by human error that led to harm is not proof that the overall performance is poor. For example, a surgeon who once amputated the wrong leg could have a superior overall complication rate. Fourth, the usual methods for identifying AEs, incident reports, or chart review are inefficient, inaccurate, and subject to significant disagreement.3739166167168169 This last point highlights the necessity of creating reliable methods of systematically collecting accurate data on the performance parameter being surveyed. Both the denominator and the numerator must be accurate. This topic is beyond our scope, but there are a few guidelines, as follows: (1) prospectively collected data are superior to retrospective data; (2) information collected by computerized information systems is superior to that collected by humans, especially if the systems are specifically programmed to acquire the desired information; (3) data collected by personnel dedicated to that job are superior to that collected by health-care workers tasked with collecting data in addition to their usual clinical responsibilities; and (4) identification of AEs should use objective, predefined criteria rather than human judgment.
Measuring ICU performance by variables such as those in Table 1 is effected by calculating the rate for binary variables, such as survival or complications, and the mean, median, or interquartile range for continuous variables, such as a 0-to-100 patient satisfaction scale. Whatever the data type, care must be taken to collect a sample size that is large enough to allow for reliable statistical comparisons.88170171 This point is illustrated by the example of a complication that occurs in approximately 3% of patients, with monthly tracking showing rates that fluctuate between 0% and 6%. If the rate jumps from 1 to 3% from January to February, this cannot not be taken as indicative of anything other than statistical variation, unless the confidence intervals are narrower than these fluctuations. In this example, there would have to be 1,120 patients at risk per month to reduce the 95% confidence interval to ± 2%.172 Fewer patients are needed if the AE rate under consideration is higher and if wider confidence intervals will suffice to convince oneself that the observed changes are "real." A related issue in the interpretation of comparative statistical performance data is the phenomenon of regression to the mean.173 This principle tells us that if a performance indicator is either high or low at one time, because of statistical fluctuation it will tend on the next measurement to be more intermediate. Thus, it is important not to overinterpret short-term changes in performance measurements.
A very important issue is that performance parameters are strongly influenced by patient demographics, comorbidities, and type and severity of acute illness. These factors are cumulatively referred to as case mix. This may be true even for institutional outcomes, such as staff satisfaction.174 Although raw data are "correct," they are problematic for comparing ICU performance between different time periods (a primary element of PI) if there have been important changes in case mix. For example, an increase in ICU mortality over time may not represent worse care if sicker patients were admitted to the ICU in the second epoch. Even the seasonal differences that occur in some critical illnesses175176177 can create this problem. Case-mix differences are even more important in comparing performance between different ICUs.87178179 Thus, it is important for PI efforts to collect at least some case-mix data.
Within a single ICU, the simplest way to handle this potential problem is to collect and compare case-mix data for the separate time periods under consideration. When appropriate to the performance issue being surveyed, comparisons might be made in the same season of successive years. If the case-mix variables are similar in the different intervals, then it is reasonable to believe that the performance comparisons are valid. The argument against this simple approach is that finding similarity among any given number of case-mix variables cannot exclude the possibility that there are other differences between the cohorts that remain undetected. Although the point is valid, this simple approach is reasonable, and it is the only practical one in institutions having limited resources for PI efforts.
The case-mix variables chosen for assessment should follow the purpose of the PI project. For general purposes, it is straightforward to record patient age, gender, race, the presence or absence of important comorbid states, and the organ system most responsible for ICU admission. With more data collection resources, one should consider collecting socioeconomic variables, such as insurance category (eg, private, Medicare, Medicaid, or none64180181) and measures of severity of illness, such as an the acute physiology and chronic health evaluation (APACHE) score, TISS score, or multiple organ dysfunction score.112171182 Special case-mix variables should be chosen for special purposes. For example, measuring the outcomes of patients with hypoxemic respiratory failure leads one to record each patient’s initial PaO2/fraction of inspired oxygen ratio as a measure of the severity of this state.
When the case mix is not similar across time periods, it is best to make adjustments for the observed differences. A common method is to use one of the available ICU risk prediction systems.116183184 Some of these, such as the APACHE, the mortality probability model (MPM), the simplified acute physiology score (SAPS), and Project IMPACT, are for general adult ICU patients,182185 Pediatric Risk of Mortality (PRISM) is for general pediatric ICUs,186 and others have been developed for specialized subsets of critically ill patients, such as trauma or cardiac surgery patients.187188 These systems are equations that are derived from multivariable linear modeling of demographic and clinical data that predict outcome variables for each patient by comparison with the large inception cohort of ICU patients used to create the equations. Nonlinear models, such as neural networks, achieve similar predictive power.189 This approach is equivalent to comparing outcomes of patients in the ICU to those of the inception cohort from which the system was derived. Most of these systems explicitly generate scores used within the equations that should not be confused with the predictions, but are themselves convenient tools to use in evaluating the case mix.
Unfortunately, these "prefabricated" systems are very limited; most of them only predict short-term mortality. This can be used to measure performance using the standardized mortality ratio (SMR). The SMR is calculated as the observed mortality rate of an ICU cohort divided by the average mortality rate predicted by the equations. The SMR is, thus, a death rate that is case mix-adjusted by comparison with the original inception group.88183184 Although unable to predict individual outcomes with sufficient accuracy to evaluate care for individual patients,190191 the SMR performs reasonably well for patient cohorts.183185192193194195 Other than short-term mortality, the list of performance measures for which validated prediction equations allow case-mix adjustments is brief. The proprietary APACHE III system also predicts ICU length of stay, days spent receiving mechanical ventilation, and the likelihood of receiving "active" intervention; however, there has been published validation62 only for the length of stay. Similarly, in pediatrics the PRISM investigators have published a prediction of ICU length of stay.63 Validated, ready-made tools to case-mix adjust the other important ICU performance measures do not yet exist.
Even for the few performance parameters included in these systems, there are problems related to calibration,178179190196 accuracy within some diagnostic subsets,190197 generalizability to other countries,198199 and aging of the inception data set.184200 In addition, none of these systems adjust for nonphysiologic parameters, which may be important determinants of outcomes, such as socioeconomic factors. Whereas the older versions of these systems are in the public domain, some of the most current and usable ones, such as APACHE III and Project IMPACT, are proprietary and expensive.201202 It should be noted, however, that most data indicate that the latest versions produce only modest increases in predictive power over earlier, free versions.195203204
All of the prognostic systems mentioned are prospective in that predictions are generated from data available at the start of the ICU admission, representing the clinical situation before ICU care was provided. However, the use of commercially available, retrospective prognostic systems based on administrative or claims data is growing, in some cases mandated by regulatory agencies. Many of these methodologies factor information about events that occurred during the course of care into the predictions. This represents a fundamentally flawed approach to evaluating performance and should not be used for this purpose. Not only is the accuracy of administrative databases problematic,205206207 but such an approach is not capable of distinguishing whether poor outcomes are attributable to bad care or to the severity of illness.87205208 For example, an ICU where many patients die because of avoidable complications could fail to be identified as a poor performer by a system that incorporated those complications into its prediction of the risk of death.
A more flexible approach that allows customized case-mix adjustment for any numerical performance parameter is to create a personalized multivariable regression analysis.171209 This avoids many of the disadvantages of the ready-made systems. Instead of comparing the outcomes of an individual ICU with those of the development cohort of a prefabricated system, this approach compares the performance of one ICU with itself over time by adjusting for changes in case mix that could invalidate meaningful comparison of the unadjusted outcomes. The method involves putting all of the data into spreadsheet format with each ICU admission as a separate row. The spreadsheet columns comprise each one of the desired outcome variables, each of the case-mix variables, and one or more indicator variables that identify the desired epochs of interest (eg, this year vs last year). Using any of the standard, commercially available statistical computer programs, a regression equation is generated for each performance measure of interest. This measure is the dependent variable, the case-mix variables are the independent covariates, and the different time periods being compared represent the indicator variable. The magnitude and statistical significance of the coefficient for the time period indicator variable tells whether there has been a change in performance after adjusting for case mix. Continuous outcome variables are analyzed with linear regression, and binary variables (eg, ICU mortality) are analyzed with logistic regression. A relatively simple implementation might include covariates of age, gender, race, important comorbid states, the organ system most responsible for ICU admission, and a measure of the severity of illness. One can even use the basic elements of a ready-made system as the case-mix covariates. For example, a custom equation using the APACHE III acute physiology score, the source of ICU admission, and ICU admission diagnosis provided superior predictive power to the APACHE III predictions themselves.210 Furthermore, there is much redundancy within clinical ICU data, and a custom equation that excluded vital signs resulted in predictive power similar to that of APACHE III.211 However, any custom modeling scheme requires nontrivial expertise in computer applications and statistical modeling but can be implemented within individual ICUs.212
Special comments must be made discouraging the use of diagnosis-related groups (DRGs) in PI efforts. In most hospitals, the DRG choices are not made by clinicians and contain significant inaccuracies.213 In addition, the DRG on hospital discharge is generally chosen to maximize reimbursement and need not bear any relationship to the reason for ICU admission. Indeed, the DRG may reflect a potentially avoidable complication of care.214 Likewise, the admission DRG may not represent the reason for transfer to ICU.
The discussion above highlights the difficulties involved in using the measurement of outcomes to improve ICU performance. However, a balanced treatment of PI efforts must address the concerns of those who go further. Lilfor et al215 wrote that comparisons based on outcomes cannot be used to measure the quality of hospital care because of problems, including the following: (1) changes over time in the definitions of outcomes and risk factors; (2) changes over time in the source and quality of data; (3) imperfect methods of predicting the outcomes of interest; and (4) physicians "gaming" any system of assessment that is put into place. Indeed, when comparing performance among different ICUs using prefabricated systems for predicting outcomes, some data show194216 that relative performance depends on which system is used. In addition, there is evidence of gaming in the form of changes in the coding of risk factors when a system for measuring risk-adjusted outcomes is first implemented.217 However, this nihilistic view fails to recognize a number of important factors. First and foremost is that data with even limited predictive power cannot possibly be more inaccurate than having no data at all. Because limitations, such as unmeasured variables, can never be totally eliminated, it will never be possible to make any set of data perfectly predictive. Thus, the view of Lilfor et al215 is akin to saying that these efforts are forever hopeless. Second, claiming that there is no connection between outcomes and quality is incorrect, simply because quality and performance are fundamentally defined by outcomes. The proposal of Lilfor et al215 that we use surrogate measures, such as structures and processes, to define performance, overlooks the basic reason for health care, to improve health outcomes. Observing that outcomes often correlate poorly with expert opinions of what constitutes the "best" structures or processes of care is not proof that outcomes do not define performance; rather, it demonstrates that such surrogate measures cannot substitute for the outcomes that are the true measures of performance. Third, whereas the gaming of a system that compares different ICUs is a real concern, this should only be a major confounder when first implementing a new system for case-mix adjustment or risk prediction. The clearest data showing its existence, the case of the initial experience of New York state with the public release of risk-adjusted mortality rates for coronary bypass surgery, found that gaming accounted for a minority of the observed improvement in performance.217218 And last, the criticisms of Lilfor et al215 mainly concern the use of prefabricated systems to make relative judgments comparing the performance of one ICU against another. I propose instead that the primary thrust of local PI efforts be aimed at comparisons over time within a single ICU. This approach renders irrelevant the limitations inherent in the use of the prefabricated prediction systems.
Mention must be made of external benchmarking in ICU PI efforts. This is the process of comparing performance from one ICU against that of other ICUs.219 Contemporaneously benchmarking is usually affected by membership in a consortium, such as the University HealthSystem Consortium220 or the Institute for Healthcare Improvement.221 Although this strategy is valuable,222 it is often expensive and impractical. Retrospective benchmarking is easier. As mentioned, calculating the SMR in one ICU from a system such as APACHE III is equivalent to retrospective benchmarking against the patient cohort on which the APACHE equations were created. An even simpler method of retrospective benchmarking is to compare the rates of the outcome variable of interest in one ICU to the rates reported in the literature from other ICUs. Of course, due care must be taken to ensure that the comparisons are valid ones.171219
|
Summary |
|---|
|
TOPAbstractIntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
The second part of this review224 establishes a practical framework for PI and examines specific strategies that can be used to improve ICU performance, including the use of information systems.
|
Footnotes |
|---|
Learning Objectives: 1. Improving ICU performance requires a shift from a paradigm that focuses on individual performance to one that emphasizes improvement in ICU systems and processes. 2. Defining and measuring performance in the ICU requires cumulative data to calculate relevant summary measures of performance. Detecting individual adverse events (errors) is not sufficient.
Dr. Garland has indicated to the ACCP that he has not received anything of value, either directly or indirectly, from a commercial or other party related directly or indirectly to the subject of this article submission.
Received for publication September 3, 2004. Accepted for publication January 20, 2005.
|
References |
|---|
|
TOPAbstractIntroductionDefining ICU PerformanceMeasuring and Interpreting ICU...SummaryReferences |
|---|
