(Chest. 2000;118:53S-58S.)
© 2000
American College of Chest Physicians
Performance Measurement Through Audit, Feedback, and Profiling as Tools for Improving Clinical Care*
Kevin B. Weiss, MD and
Robin Wagner, RN, MHSA
*
From the Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, Chicago, IL.
Correspondence to: Kevin B. Weiss, MD, Center for Health Services Research, Rush Primary Care Institute, Rush-Presbyterian-St. Luke’s Medical Center, 1653 West Congress Parkway, Chicago, IL 60612
 |
Abstract
|
|---|
Clinical audits and practice profiling have become popular tools in
the attempt to change physician behavior to improve quality of care.
Unfortunately, the growing need for information on quality of care has
often outpaced the development of standard, valid, and reliable
approaches to using these tools. The studies of performance measurement
published in the literature to date demonstrate varying impact on
ability to improve clinical care; few are randomized controlled trials.
While performance measurement has become a common practice, the science
surrounding this field is still in its early stages of development;
while it seems promising, it should be viewed as largely
experimental.
Key Words: Health Plan Employer Data Information Set • quality of care
|
Introduction
|
|---|
Since
the seminal work of Wennberg and Gittelsohn1
in the 1970s,
numerous studies have sought to understand the reasons for large
geographic variations in the use of clinical services, as well as
variations in patient outcomes.2
For much of the 1980s and
early 1990s, these studies have served to indicate that there might be
important differences in the quality of medical care as a result of
overuse, underuse, and/or misuse of medical interventions.3
4
In response, the health-care system has begun to explore ways
to use this information to influence changes in provider behavior to
improve care.
Use of clinical data to improve outcomes is not a new concept. For
decades, hospitals have routinely held surgical morbidity and mortality
reviews as a means by which to learn from experience. More recently,
the focus has shifted to clinical data review across multiple settings,
and among differing types of provider groups. This report presents an
overview of the use of clinical audits/practice feedback, practice
profiling/benchmarking, and regulatory oversight as tools for changing
physician behavior to improve outcomes. Before exploring these
different tools, it may be useful to review a couple of the basic
concepts of performance measurement.
|
Individual vs Population Performance Measurement
|
|---|
Individual case review provides different data from
population-based reviews, and both types of review have their strengths
and weaknesses. Individual case review is principally used to explore
concerns that are associated with rare, but sentinel, events. An
example would be the review of an incident in which a patient received
mismatched blood products. This type of review is best suited for
events that are infrequent but important enough to warrant the use of
resources so as to minimize the chances of a similar error in the
future. Yet, one may or may not be able to generalize from the
knowledge gained from such a review, and the infrequent nature of such
events makes it infeasible to examine multiple similar events. There is
also little opportunity to use epidemiologic and statistical tools to
assist in judging the degree of certainty of the findings from
individual case reporting.
The alternative to individual case review is the population-based
approach. Aggregated experience from multiple cases can provide
insights to patterns of clinical behavior for more common conditions
that affect many more patients. An example would be a measure of the
proportion of patients within a particular health plan who received the
flu vaccine. With population-based assessment, it is possible to use
standard epidemiologic and statistical techniques to help assess the
degree of certainty of the conclusions drawn from the observed clinical
experiences.
|
Quality as Measured by Structure, Process, and Outcomes
|
|---|
Donabedian5
offered the concept that quality could be
measured based on structure, process, and outcomes. Structure
encompasses physical factors, such as buildings, as well as
professional and institutional factors, such as the regulatory and
financing environments in which health care is delivered. Process
refers to the actions that health-care providers take in delivering
medical care, such as performing examinations, ordering tests, and
prescribing medications. Outcomes are the end result of the process
interventions: the effects on the patient’s health and well being.
While early attempts at measuring the quality of health care focused on
the structure, much of the current focus relates to exploring clinical
processes and outcomes.
Although patient outcomes are the ultimate judge of the quality of
health care, there are several advantages to using process measures
instead of outcome measures for purposes of performance evaluation.
Most notably, it is much easier for physicians or other health-care
providers to accept responsibility for their actions in providing care
than to accept responsibility for their patients’ outcomes, because
there are many factors affecting patient outcomes that are not directly
under the control of providers. For example, while a provider might
make a concerted effort to ensure that a patient has been offered the
flu vaccine, the patient may choose not to take the vaccine and may
subsequently develop influenza. In this situation, performance
evaluation will produce very different results depending on whether it
is the process (providing access to the vaccine) or the outcome
(influenza) that is measured.
Process measures are also useful in evaluating the quality of care for
common chronic conditions for which the ultimate outcomes may take
years to determine, such as hypertension and stroke, or glycemic
control and complications from diabetes. For these reasons, it is
attractive to focus on using process measures rather than outcomes
measures for performance measurement. However, it would seem that the
best measure of health-care performance rests with patient outcomes,
including physiologic status, health-related quality of life, and
satisfaction with the health-care system.
|
Formative vs Evaluative Information
|
|---|
A third central concept for performance measurement relates to how
the data are used. Formative data are gathered for immediate use, to
guide clinical decisions affecting ongoing patient care.6
As such, this type of information is different from the kind used for
evaluation. Although evaluative data may be collected at any time in
the process of care, they are generally examined retrospectively in an
attempt to evaluate good vs bad quality health care, overuse vs
underuse of services, or perhaps to compare one type of service to
another. Currently, there are several major types of performance
measurement in use. These include clinical audits/practice feedback,
practice profiling/benchmarking, and regulatory oversight of
performance indicator systems.
|
Clinical Audits and Practice Feedback
|
|---|
For decades, health-care systems have used clinical audits as a
tool for quality assessment. Audits of this type usually seek to
characterize care through the systematic review of a series of patient
experiences. Most often, the information is obtained by examining
charts or medical records for documentation of specific clinical
practices/procedures. Since the 1970s, the British have used audits to
examine issues of quality surrounding clinical management of minor
acute problems or preventive health practices,7
8
chronic
disease management (eg, diabetes,9
and
asthma10
), and the use of specialty
consultations.11
While clinical audits are widely used to assess performance, there is
conflicting evidence regarding whether or not they are effective in
changing provider behavior. For example, a study at one hospital
demonstrated significant improvements in preventive health processes
that were audited vs other health-care processes that were not
monitored.12
Two small studies, examining the quality of
Papanicolaou smears, demonstrated that performance of both residents
and faculty physicians substantively improved after they received
feedback from clinical audits.13
14
By contrast, a study
by Reilly and Patten15
demonstrated little change in
targeted prescribing patterns for various clinical conditions as a
result of audit and feedback.
The Ambulatory Care Medical Audit Demonstration Project16
is the largest formal study of the use of audit information in the
United States. The project was designed as a randomized controlled
clinical trial of the use of quality-improvement techniques to improve
clinical performance in areas of primary care. Although audit
information was only one element in a multidimensional intervention,
this study demonstrated that it is possible to improve the quality of
care with feedback of audit information.
Unfortunately, there has been no formal synthesis of studies on the use
of audits to affect clinical performance. Many of the studies conducted
to date were not well controlled and did not include a strategy for
randomizing the physicians who were given feedback. Rather, most were
preevaluation/postevaluation designs, based on interventions conducted
at a single site or with a small number of practices. Therefore, while
clinical audit with feedback is an attractive approach to changing
physician behavior, its efficacy is unclear.
|
Practice Profiling/Benchmarking
|
|---|
Another approach to performance measurement compares the
performance of a single provider against that of a panel of similar
providers. This type of measurement is often referred to as practice
profiling or benchmarking.17
In practice profiling the
performance of a single physician or a group is expressed as a rate, a
measure of resource use during a defined period for the population
served. A profile is created by comparing this rate to that of a
community norm based on the practices of other physicians or on other
standards such as guidelines.18
Figure 1
provides an example of the differences in the case mix-adjusted number
of relative value units per hospital admission by physician specialty
among physicians providing care to Medicare patients in Oregon vs
Florida.19
As with audits, there is also little known about how profiling affects
clinical performance. A meta-analysis of randomized trials of profiling
revealed only 12 eligible trials; many of the studies under evaluation
had notable design flaws.20
The analysis found that while
profiling had a statistically significant positive effect on
utilization, the effect was of minimal clinical importance.
Nevertheless, practice profiling is widely used and has attracted much
attention and controversy.21
22
23
In a 1994 American
Medical Association survey of practicing physicians, over half of the
respondents reported that they were subject to clinical or economic
practice profiling.24
Much of the controversy rests on the
quality, validity, and reliability of the profiling data. For
performance assessments to provide useful information, they must meet
certain methodologic criteria. Among these is the need for
well-defined, similar patient populations. It is important that
practice data are adjusted for case mix severity and for other
nonmedical care factors that are known to affect clinical performance,
and that sufficient numbers of events are measured to ensure that
differences are not due to chance alone.25
26
This last issue is particularly problematic. A study by Hofer et
al27
examined the usefulness of physician profiling for
patients with diabetes, one of the most prevalent conditions in
clinical practice. The authors conducted a study of approximately 3,600
patients with type II diabetes, under the care of 232 different
physicians. Yet, as Figure 2
illustrates, they were unable to reliably detect any true differences
in care among the physicians. They observed that " ... a
physician would need to have more than 100 patients with diabetes in
their panel for the profiles to have a reliability of 0.80 or better
(while more than 90% of all primary care physicians in the health
maintenance organization had fewer than 50 patients with
diabetes)."27
View larger version (60K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 2. Comparison of visit rate profiles for 232
physicians in three health plans, showing the relative visit rate by
physician (with 1.0 being an average profile) after adjustment for
patient demographics and detailed case mix measures. The error bars
represent a 1.4 SE confidence interval, so that overlapping confidence
intervals suggest that the overall difference between two physician
visit rates is not statistically significant (p > 0.05). In this
graph, although the overall physician effect on visits is statistically
significant, it is not possible to say that the physicians at the
extremes are significantly different in their visit rates from any of
the other physicians.
|
|
Therefore, the studies of physician profiling as a tool for changing
practice behavior present a very mixed picture. The randomized,
controlled trial literature suggests that profiling can produce a
modest, but statistically significant effect on improving physician
behavior.28
However, more recent studies on the validity
and reliability of this measurement technique have opened up new
questions about its usefulness.
|
Regulatory Oversight and Performance Indicator Systems
|
|---|
With such increased interest in attempting to improve the quality
of care through feedback of clinical data, it is perhaps no surprise
that there have been efforts to create complex systems to evaluate
clinical performance. The apparent premise behind such performance
measurement systems is to use them as administrative tools, either
voluntary or regulatory, to broadly measure quality-improvement
activities.
In the United States, one of the prototypes of these systems is the
Health Plan Employer Data Information Set (HEDIS). The HEDIS was
developed in the early 1990s by the National Committee on Quality
Assurance, a not-for-profit organization committed to evaluating and
reporting the quality of care delivered by managed care plans. Using
standardized methodology, HEDIS data are gathered from several sources
within each health plan, including administrative claims and encounter
information, medical records, and survey information. The National
Committee on Quality Assurance, which uses the information from HEDIS
as part of its accreditation program, makes the results publicly
available through a national database of HEDIS information and
accreditation results.29
Employers and consumers alike can
use this information about quality of care to make choices among health
plans. Figure 3
is an example of HEDIS data comparing mammography rates for Medicare
managed-care organizations in Orange County, CA.29
In addition to HEDIS, the United States has developed or proposed
several other performance indicator systems.30
31
32
33
Unlike
HEDIS, the role and utility of these other systems have not yet fully
evolved. Many of these newer systems do not clearly specify the primary
audience for their information. Is the primary audience the provider,
the health plan, the employer, or the consumer? Is the primary focus on
process measures or outcomes measures? What are the costs and burdens
of collecting such complex and comprehensive data? Perhaps most
importantly, what effect will such data have on changing the quality of
health care in the future?
What Is the Future for Using Performance Measurement To Evaluate
Quality Improvement Activities?
Although not always approached with rigorous research methods,
health-care performance measurement is now pervasive, and it seems
likely that these activities will continue, if not increase, in the
future. On the positive side, these efforts have helped to focus
attention on the overall importance of evaluating the quality of health
care; for the public, they have removed some of the mystery surrounding
the delivery of care. As performance measurement continues to evolve
nationally, clearer standards will emerge to define the types of
measures that are most appropriate for this field, and valid and
reliable methods will emerge for the collection, analysis, and
reporting of data.
Alternatively, there are many limitations to this evolving practice of
performance measurement that, if not adequately addressed, will
undermine its long-term credibility. There are still many unanswered
questions, such as the appropriate population size to study and the
types of data adjustments (eg, case mix, severity,
sociodemographic) that need to be applied in order to be able to make
accurate comparisons. Also, the literature has yet to determine which
clinical conditions or administrative issues benefit the most from
these types of data collection and feedback, and which methods work
best to produce positive changes in the delivery of care.
Performance measurement appears to be most useful when it is used as a
formative tool as part of a more complex set of quality-improvement
activities.34
However, this field has yet to determine
which types of quality-improvement efforts will lead to better care.
Until such questions are adequately addressed, performance measurement
should be considered to be still in the "experimental stage" in the
challenge toward reducing unintended variations in the quality of
health care.
|
Footnotes
|
|---|
Abbreviation:
HEDIS = Health Plan Employer Data Information Set
|
References
|
|---|
-
Wennberg, J, Gittelsohn, A (1973) Small area variations in health care delivery. Science 182,1102-1108[Abstract/Free Full Text]
-
Detsky, AS (1995) Regional variation in medical care. N Engl J Med 333,589-590[Free Full Text]
-
Wennberg, DE (1998) Variation in the delivery of health care: the stakes are high. Ann Intern Med 128,866-868[Free Full Text]
-
Brook, RH (1991) Appropriate and necessary. Health Manage Q 13,8-11[Medline]
-
Donabedian, A (1980) Basic approaches to assessment: structure, process and outcome. The definition of quality and approaches to its assessment: explorations in quality assessment and monitoring (vol 1) ,77-128 Health Administration Press Ann Arbor, MI.
-
Goodman, RM, Wandersman, A (1994) FORECAST: a formative approach to evaluating community coalitions and community-based initiatives. J Commun Psychol special issue,6-25
-
Ryan, MP, Buchan, IC, Buckley, EG (1979) Medical audit: a preliminary report from general practice. J R Coll Gen Pract 29,719-722[Medline]
-
LeBaron, CW, Chaney, M, Baughman, AL, et al (1997) Impact of measurement and feedback on vaccination coverage in public clinics, 1988–1994. JAMA 277,631-635[Abstract]
-
Doney, BJ (1976) An audit of the care of diabetics in a group practice. J R Coll Gen Pract 26,734-742[Medline]
-
Hewer, SM, Hambleton, G, McKenzie, S, et al (1994) Asthma audit: a multicentre pilot study. Arch Dis Child 71,167-169[ISI][Medline]
-
Verby, JE, Holden, P, Davis, RH (1979) Peer review of consultations in primary care: the use of audiovisual recordings. BMJ 1,1686-1688
-
Holmboe, E, Scranton, R, Sumption, K, et al (1998) Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Acad Med 73,901-903[ISI][Medline]
-
Curtis, P, Skinner, B, Varenholt, J, et al (1993) Papanicolaou smear quality assurance: providing feedback to physicians. J Fam Pract 36,309-312[ISI][Medline]
-
Norton, PF, Shaw, PA, Murray, MA (1997) Quality improvement in family practice: program for Pap smears. Can Fam Physician 43,503-508[ISI][Medline]
-
Reilly, PM, Patten, MP (1978) An audit of prescribing by peer review. J R Coll Gen Pract 28,525-530[Medline]
-
Palmer, RH, Hargraves, JL (1996) The Ambulatory Care Medical Audit Demonstration Project: research design. Med Care 34,S12-S28[CrossRef]
-
Lasker, RD, Shapiro, DW, Tucker, AM (1992) Realizing the potential of practice pattern profiling. Inquiry 29,287-297[ISI][Medline]
-
Lee, PR, Shapiro, DW, Lasker, RD, et al (1992) Managed care: provider profiling. J Insur Med 24,179-181[Medline]
-
Welch, HG, Miller, ME, Welch, WP (1994) Physician profiling: an analysis of inpatient practice patterns in Florida and Oregon. N Engl J Med 330,607-612[Abstract/Free Full Text]
-
Balas, EA, Boren, SA, Brown, GD, et al (1996) Effect of physician profiling on utilization: meta-analysis of randomized clinical trials. J Gen Intern Med 11,584-590[ISI][Medline]
-
Krakauer, H, Bailery, RC, Skellan, KJ, et al (1992) Evaluation of the HCFA model for the analysis of mortality following hospitalization. Health Serv Res 27,317-335[ISI][Medline]
-
Green, J, Wintfeld, N (1993) How accurate are hospital discharge data for evaluation effectiveness of care? Med Care 31,719-731[ISI][Medline]
-
Kassirer, JP (1994) The use and abuse of practice profiles. N Engl J Med 330,634-636[Free Full Text]
-
Emmons, DW, Wozniak, GD (1994) Profiles and feedback: who measures physician performance? Gonzalez, ML eds. Socioeconomic characteristics of medical practice ,11-16 American Medical Association Chicago, IL.
-
Physician Payment Review Commission Conference on Profiling Washington, DC: Physician Payment Review Commission, May 1992; Publication No. 92–2
-
Orav, EJ, Wright, EA, Palmer, RH, et al (1996) Issues of variability and bias affecting multisite measurement of quality of care. Med Care 34,SS87-SS101[ISI][Medline]
-
Hofer, TP, Hayward, RA, Greenfield, S, et al (1999) The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. JAMA 281,2098-2105[Abstract/Free Full Text]
-
Kim, CS, Kristopaitis, RJ, Stone, E, et al (1999) Physician education and report cards: do they make the grade? Results from a randomized controlled trial. Am J Med 107,556-560[CrossRef][ISI][Medline]
-
HEDIS 2000 narrative: what’s in it and why it matters. Washington, DC: National Committee for Quality Assurance, 1999; Publication No.10234–100-00
-
Palmer, RH, Peterson, LE (1995) Development and testing of performance measures for quality improvement in clinical preventive services. Am J Prev Med 11,402-406[ISI][Medline]
-
Lawtheres, AB, Palmer, RH, Edwards, FE, et al (1993) Developing and evaluating measures of ambulatory care quality: a preliminary report of the DEMPAY Project. J Qual Improv 19,552-565
-
Lansky, D (1998) Measuring what matters to the public. Health Aff 17,40-41[Abstract]
-
Braun, BI, Koss, RG, Loeb, JM (1999) Integrating performance measure data into the Joint Commission accreditation process. Eval Health Prof 22,283-297[Abstract/Free Full Text]
-
Thompson, RS (1997) Systems approach and the delivery of health services. JAMA 277,668-671
This article has been cited by other articles:
|
|
|
|
 
C. N. Sciamanna, L. R. Harrold, M. Manocchia, N. J. Walker, and S. Mui
The Effect of Web-Based, Personalized, Osteoarthritis Quality Improvement Feedback on Patient Satisfaction With Osteoarthritis Care
American Journal of Medical Quality,
May 1, 2005;
20(3):
127 - 137.
[Abstract]
[PDF]
|
|
|
TOP
Abstract
Introduction
