Ann Thorac Surg 1999;68:353-358
© 1999 The Society of Thoracic Surgeons
Data Base Panel
Measuring for improvement: from Toyota to thoracic surgery
James M. Levett, MDa,
Raymond G. Carey, PhDa
a Department of Surgery, Lutheran General Hospital, Park Ridge, Illinois, USA
Address reprint requests to Dr Levett, Department of Surgical Specialists, Physicians’ Clinic of Iowa, PC, 830 Fourth Ave SE, Cedar Rapids, IA 52403
Presented at the Thirty-fourth Annual Meeting of The Society of Thoracic Surgeons, New Orleans, LA, Jan 26–28, 1998.
Abstract
Background. Measuring quality has become a high priority in the era of managed care. Nevertheless, it can be counterproductive to use the same methods for measuring improvement in surgical procedures and processes as we use for measurement in basic research. Techniques of statistical process control have been used for many years to measure process improvement in industry and are now being applied to health care.
Methods. Examples of using statistical process control charts to monitor coronary artery bypass grafting mortality, intensive care unit admission time, and length of stay are reviewed.
Results. The major advantage of using control chart methodology is that it allows one to determine whether the process being evaluated is in fact stable and to detect when significant or special cause variation occurs.
Conclusions. Summary statistics currently provided to purchasers of care and regulatory agencies do not ensure that the processes being evaluated are stable. We need to look at data over time with statistically validated methods such as control charts to better monitor our processes of care and thereby provide accurate statistics.
In a recent article in the Joint Commission’s Journal on Quality Improvement, submitted by the Institute for Clinical Systems Integration in Minneapolis, it was emphasized that physicians "are increasingly realizing not only how critical measurement is to the quality improvement we seek, but also how counterproductive it can be to mix measurement for accountability or research with measurement for improvement" [1]. Most surgeons have been trained in basic research and in the use of random clinical trials to analyze data, but even well-designed clinical trials only establish the efficacy of various procedures and will often not tell us how effective a procedure may be in our own practice.
Currently we are being asked to provide data to many different users, including employers, managed care plans, health plans, and government agencies. To meet these requests, the entire field of so-called outcomes research is evolving, and as this occurs, it is important for clinicians to realize that meaningful data are not collected by simply taking before and after snapshot measurements at various points in time. For example, knowing that a complication rate is high does not tell us why it ishigh. What we need are useful techniques and tools to help in determining where to begin our efforts to improve. Thus, we cannot conclude anything with certainty by comparing two numbers unless we understand the stability of the processes that produced them. This insight was obtained in industry many years ago by Walter A. Shewhart, and his methodology for measuring processes was later developed by W. Edwards Deming and others [2].
Shewhart’s methods have application to surgical processes. For example, the board of directors at a local hospital is evaluating mortality rates for coronary artery bypass grafting. The bar graph in Figure 1 shows a decrease in the mortality from 5% in 1994 to 4% in 1995. When the data are analyzed monthly, however, we see that the mortality has actually increased in 1995 (Fig 2). The learning point here is that although the average mortality decreased from 5% in 1994 to 4% in 1995, it was not attributable to the new protocol, which seemed to have a negative effect. After the new protocol began, the monthly average actually increased from 2% to 6%. Thus, bar charts can be misleading unless the processes that produced the data are stable.
View larger version (14K):
[in this window]
[in a new window]
|
Fig 1. Histogram of average coronary artery bypass grafting (CABG) mortality before and after implementation of new protocol. Was the new protocol effective in reducing deaths?
|
|
View larger version (14K):
[in this window]
[in a new window]
|
Fig 2. Control chart of coronary artery bypass grafting (CABG) mortality before and after new protocol. CL = center line or mean; LCL = lower central limit; UCL = upper control limit.
|
|
This methodology of displaying data on a chart over time illustrates the concept of a time-series design. It is important to note that the technique of plotting data over time is useful in providing a way to display, think about, and analyze data in a process. The understanding derived will allow us to track changes that we make and thereby monitor improvement.
The time-series design format is further illustrated in the next three figures. Hospital A and hospital B have both succeeded in decreasing the intensive care unit admission time after open heart procedures from an average of 39 to 26 minutes (Fig 3). This admission time is defined as the time it takes to transition a patient from the operating room until he or she is stabilized in the intensive care unit with line change-over and full monitoring in place. Both hospital A and hospital B have seen improvement in this admission time as noted on the histogram. Looking at the data over time, however, suggests that two different processes are occurring. In hospital A, the change that was implemented in January 1997, resulted in a positive shift in the process as is evident from the control chart in Figure 4. Admission time improved from an average of 39 minutes in 1996 to 26 minutes in 1997. Improvement was clearly due to the change effort introduced in January 1997 where a shift in the process (special cause) is noted. Hospital B also saw an improvement, but it would appear that the changes implemented in January did not really make any difference since there was a clear trend already in place (Fig 5). The improvement was not due to the change effort in January 1997, but to some other factor that began in 1996 and continued into 1997.
View larger version (14K):
[in this window]
[in a new window]
|
Fig 3. Histogram of decrease in average intensive care unit (ICU) admission time after the 1997 implementation plan, hospitals A and B. Was the decrease due to the implementation plan?
|
|
Putting data into proper context according to the way in which the data are collected over time helps us understand when variation is significant, particularly since the most common type of error is to see a trend when none is present. To analyze variation, Shewhart [3] developed the control chart and published his methodology in 1931 in a landmark text entitled Economic Control of Quality of Manufactured Product. A simple control chart is illustrated in Figure 6 , where the key quality characteristic is represented on the y (vertical) axis, and time is represented on the x (horizontal) axis. The upper and lower control limits are generally set at three sigmas, and the mean (
) is set in the midpoint of the control limits. Data falling outside of three sigma limits are a signal that the process has significantly changed. Shewhart’s tests for a special cause, and subsequent tests, which were later described as the Western Electric Tests, have helped industrial managers to analyze variation in manufacturing processes to distinguish noise from signals [4]. Thus, one example of a significant change in the process would be when a data point falls outside of three sigma limits. These sigma limits are actually different from classic standard deviation in the sense that they measure the variability of a process over time rather than the variability of a static distribution. Another test states that when seven or eight consecutive data points fall on one side of the centerline, a significant shift in the process has occurred.
With control charts, manufacturing processes have been greatly improved and, in fact, this methodology has been used to both improve the quality and reduce variation in manufactured products since the 1950s when they were first used in Japan [5]. These techniques are now being applied in health care and, we believe, will help in analyzing how our processes work for us in our own hospitals [6].
To illustrate this point, we will review three examples from Lutheran General Hospital. In the first example, we have analyzed our elective coronary artery bypass grafting mortality over a period of time from January 1995 through August 1997 (Fig 7). The data are plotted on a control chart with the number of successful surgical procedures between mortalities on the y-axis and each point on the x-axis representing a mortality. As one can see, there was a period from March through November 1995 when 171 consecutive cases were done without a mortality, and this certainly signaled a positive change in our process, which we decided to analyze. Since that time, the process has returned to a stable, but less positive, average of 26.5 open heart procedures being done between mortalities. The next figure illustrates a way of representing coronary artery bypass grafting complications on a control chart (Fig 8). Complications that we are interested in following include reoperation for bleeding, perioperative myocardial infarction, sternal infection or dehiscence, stroke, and renal failure. The vertical axis represents the number of procedures done between complications, and the horizontal axis represents time with each point being a patient with one or more complications. The time period ranges from January through August 1997. As one can see, there were three points where around 34 to 40 patients were operated upon between one of the complications. These, however, were within the control limits of the process and were not significantly different from the other points. They are an example of "common cause" variation. We see now, however, that six consecutive points have fallen below the centerline, which represents the mean of 14.25 successful operations done between complications. These points are an example of "special cause" variation (because there are only 12 total points) and suggest a significant shift in the process. This discovery is now being analyzed, as it represents a negative finding that we had not previously identified.
View larger version (15K):
[in this window]
[in a new window]
|
Fig 7. Control chart of first/elective coronary artery bypass grafting (CABG) mortality (DRG 106 and 107).
|
|
Our final illustration is taken from a bowel surgery quality improvement team. We have included this to illustrate the technique of measuring a change in a process. Several years ago, we established a process improvement team at Lutheran General Hospital to study patients undergoing bowel resection. First, we attempted to reduce the use of resources, as measured by postoperative length of stay. The control chart in Figure 9 illustrates that the average length of stay for 18- to 64-year-old patients decreased from 9.0 days during the baseline, prepathway period to 6.8 days after the pathway was put into place.
The next question is whether the improvement in length of stay had an impact on clinical outcomes. The control chart in Figure 10 suggests that it has had a negative impact. On this chart, we are tracking a composite of complications including pulmonary embolus, pneumonia, wound infection, return to operating room, and readmission within 30 days after discharge. The number of operations between one of the complications was 8.7 during the prepathway period but decreased to 3.1 during the initial implementation of the pathway. The current pathway is a stable process with 6.4 operations being done between any one of the complications mentioned. Thus, although we have reduced our length of stay, we also see complications more frequently. The pathway is currently under evaluation by the process improvement team to ascertain whether the increase is attributable to a flaw in the pathway, or to physicians not following the pathway.
In conclusion, plotting data over time and using control chart techniques will tell us whether the variation in a surgical process is stable and predictable or whether variation signals a significant change in the process. If we really want to measure the effectiveness of our procedures with our own patients, random clinical trials and outcomes measurements alone will not accomplish this task. The summary statistics we currently provide to the purchasers of care and to regulatory agencies do not ensure that our processes are stable. In fact, such data may be misleading. We need to look at our data over time with statistically validated methods such as the control chart methodology developed by Shewhart for industry and which is now beginning to be used in health care.
References
-
Solberg L.I., Mosser G., McDonald S. The three faces of performance measurement. Joint Comm J Qual Improv 1997;23:135-147.
-
Walton M. The Deming management method. New York: Perigee Books, 1986:3-21.
-
Shewhart W.A. Economic control of quality of manufactured product. New York: Van Nostrand, 1931:351-404.
-
Carey R.G., Lloyd R.C. Measuring quality improvement in health care. New York: Quality Resources, 1995.
-
Wheeler D.J., Chambers D.S. Understanding statistical process control. Knoxville, TN: SPC Press, 1992.
-
Shahian D.M., Williamson W.A., Svensson L.G. Applications of statistical quality control to cardiac surgery. Ann Thorac Surg 1996;62:1351-1359.[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
J. Thor, J. Lundberg, J. Ask, J. Olsson, C. Carli, K. P. Harenstam, and M. Brommels
Application of statistical process control in healthcare improvement: systematic review
BMJ Qual. Saf.,
October 1, 2007;
16(5):
387 - 399.
[Abstract]
[Full Text]
[PDF]
|
|
|
