HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Gary L. Grunkemeier Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Grunkemeier, G. L. Articles by Jin, R. Search for Related Content PubMed PubMed Citation Articles by Grunkemeier, G. L. Articles by Jin, R. Related Collections Professional affairs

Ann Thorac Surg 2001;72:1845-1848
© 2001 The Society of Thoracic Surgeons

---

The statistician's page

Cardiac surgery report cards: making the grade

^a Providence Health System, Portland, Oregon, USA

* Address reprint requests to Dr Grunkemeier, 9155 SW Barnes Rd, #33, Portland, OR 97225, USA
e-mail: ggrunkemeier{at}providence.org

For several years, data-driven methodologies have been used in an attempt to improve performance in cardiac surgery programs. The article by Shahian and his colleagues [1] in this issue of The Annals provides a thoughtful and thorough comparison of "Report Cards" and continuous quality improvement (CQI) initiatives. Their conclusions are that CQI, including multi-disciplinary team site visits to identify and share processes and systems, has been proven to be effective. In contrast, "Report Cards" as currently implemented are not satisfactory and have potential to do harm. Their arguments are cogent and comprehensive. They claim that these cardiac surgery report cards are based on "sophisticated mathematical models" which engender "an exaggerated aura of scientific accuracy". Examining some of the deficiencies and limitations of the risk models used for cardiac surgery, from which the report card "Grades" are derived, provides support for their claim.

Curse of a binary outcome

The mathematical models in question are constructed using multivariable regression, which yields a formula which uses the risk factors for an individual patient to provide an estimate (expected value) of his outcome. Building such a model is not a fixed, reproducible exercise, and there are at least 9 reasons why different investigators with the same data set would produce different risk models [2]. For a continuous outcome (cost, length of stay, etc) such a model can exactly predict or at least come close to a patient’s observed value. But operative mortality is a binary outcome, and an ideal formula would result in a classification of alive or dead. Instead, logistic regression provides the expected mortality, the probability that the patient will be an operative death. This probability is always between 0% and 100%, so it will never match the observed mortality, which is either 0%, for survivors, or 100%, for deaths (Fig 1). The discrimination of such a model is measured by the C-index (area under the ROC curve). The C-index for the model in Figure 1 is 0.80. (Shahian notes that most cardiac surgery models have C-indices between 0.76–0.82.) A C-index of .50 indicates no discrimination and an index of 1.00 is perfect discrimination. So a value of 0.80 is only 60% of the way between worthless and perfect. Only in aggregate can we achieve agreement between observed and expected mortality. For example, if we have 20 patients with an expected mortality of 0.05 (5%) and one of them dies, we consider the model successful. But note that we do not say which of the 20 will die; this seems unsatisfactory, but that is the best we can do.

View larger version (16K): [in this window] [in a new window]   Fig 1. Comparison of observed to expected mortality for individual patients, based on a risk model using Providence Health System data. For a continuous variable, one can expect the data points to be clustered around the line of identity, but for a binary outcome variable, this is not possible. The points fall far from this line, yet this risk model from which the expected mortality was derived has a C-index of 0.80.

Another reason for the poor performance of CABG risk models may be the complex pathways of the outcome (death) compared to the relative few risk factors available to predict it. Most current models use a stepwise regression procedure to allow only statistically significant factors. In general the larger the sample size, the more risk factors can be found (an informal rule says the number of risk factors identified should not exceed the number of deaths in the training set divided by 10). Table 1 summarizes recently published multivariable risk models based on at least 2,000 patients [3–15]. There are 44 different risk factors cited (the original number was larger, but some similar categories were combined). The number of independent risk factors cited by any one paper varied from 5 to 29, with the largest numbers being found by the largest series.

Aggregating by provider

To grade a provider (doctor, clinic) the expected number of deaths (the sum of the expected mortality for all the patients in the group, based on the risk model) is compared to the observed number of deaths. Dividing by the number of patients yields the provider’s expected (E) and observed (O) mortality, respectively. To make comparisons, the E’s are usually considered to be without error (!), and sampling error is attached to the O’s by assuming, incorrectly, that they are statistically independent. This has the effect of identifying too many outliers (in either direction). Hierarchical models, as advocated by Shahian, extend the traditional risk model structure to compensate for this by reducing the overly optimistic precision of the estimates [20, pages 507 to 511]. Incorporating hospital or surgeon as another dimension of variability results in more realistic estimates of the provider effects, dampening or shrinking them towards the mean value for all providers.

Figure 2 illustrates the use of a heirarchical model, using data from nine hospitals. For comparison, a conventional logistic regression model was used to provide expected mortality for each hospital and the O/E ratios ranged from 0.62 to 1.62 (horizontal axis). Confidence intervals were computed for each hospital [20, page 488]. The 95% confidence intervals (solid horizontal lines) for 2 of the hospitals do not include the value 1 (vertical line), so they would be considered abnormal by this simplistic analysis, and the "significantly high" one would get a "flunking" grade. However, if 99.4% confidence intervals (dashed horizontal lines) are used instead to ensure protection against type I error (finding spurious significance due to multiple simultaneous comparisons), then all of the hospitals would get passing grades.

View larger version (23K): [in this window] [in a new window]   Fig 2. Observed over expected (O/E) mortality for coronary artery bypass grafting patients from 9 Providence Health System hospitals. Area of the symbols is proportional to the number of patients. The horizontal location of each symbol indicates the O/E ratio using conventional analysis, with 95% (solid horizontal lines) and 99.4% (dashed horizontal lines) confidence intervals. The vertical location of each symbol indicates the O/E ratio based on a heirarchical model, as recommended by Shahian and associates [1]. Note that the hierarchical hospital estimates are all closer to unity ("shrunk" towards the mean) than the conventional estimates.

Conclusion

The current methods used to determine risk factors, construct risk models and compute variability result in anti-conservative comparisons of provider effects. The report by Shahian and his colleagues provides a masterful discussion of these and related issues. It deserves a careful reading.

Acknowledgments

We thank the Providence Health System hospitals for use of their cardiac surgery data from the following institutions: Alaska—Providence Anchorage Medical Center; Washington—Providence Everett Medical Center, Providence Seattle Medical Center, Providence St. Peter Hospital (Olympia), Providence Yakima Medical Center; Oregon—Providence Portland Medical Center, Providence St. Vincent Medical Center (Portland); California—Providence St. Joseph Medical Center (Burbank), Providence Holy Cross Medical Center (Mission Hills).

References

1. Shahian M., Norman S.-L., Torchiana D.F., et al. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
2. Naftel D.C. Do different investigators sometimes produce different multivariable equations from the same data?. J Thorac Cardiovasc Surg 1994;107:1528-1529.[Free Full Text]
3. Shroyer A.L., Plomondon M.E., Grover F.L., Edwards F.H. The 1996 coronary artery bypass risk model: The Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1999;67:1205-1208.[Abstract/Free Full Text]
4. Hannan E.L., Kilburn H., Jr, Racz M., Shields E., Chassin M.R. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 1994;27:761-766.
5. Ghali W.A., Quan H., Brant R. Coronary artery bypass grafting in Canada: national and provincial mortality trends, 1992–1995. CMAJ 1998;159:25-31.[Abstract]
6. Ghali W.A., Ash A.S., Hall R.E., Moskowitz M.A. Statewide quality improvement initiatives and mortality after cardiac surgery. JAMA 1997;277:379-382.[Abstract/Free Full Text]
7. Weintraub W.S., Wenger N.K., Jones E.L., et al. Changing clinical characteristics of coronary surgery patients. Differences between men and women. Circulation 1993;88:II79-II86.
8. Grover F.L., Johnson R.R., Marshall G., Hammermeister K.E. Factors predictive of operative mortality among coronary artery bypass subsets. Ann Thorac Surg 1993;56:1296-1306.[Abstract]
9. Iyer V.S., Russell W.J., Leppard P., Craddock D. Mortality and myocardial infarction after coronary artery surgery. A review of 12,003 patients. Med J Aust 1993;159:166-170.[Medline]
10. Ivanov J., Tu J.V., Naylor C.D. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 1999;99:2098-2104.[Abstract/Free Full Text]
11. Higgins T.L., Estafanous F.G., Loop F.D., et al. ICU admission score for predicting morbidity and mortality risk after coronary artery bypass grafting. Ann Thorac Surg 1997;64:1050-1058.[Abstract/Free Full Text]
12. Mozes B., Olmer L., Galai N., Simchen E. A national study of postoperative mortality associated with coronary artery bypass grafting in Israel. ISCAB Consortium. Israel Coronary Artery Bypass Study. Ann Thorac Surg 1998;66:1254-1262.[Abstract/Free Full Text]
13. DeLong E.R., Peterson E.D., DeLong D.M., et al. Comparing risk-adjustment methods for provider profiling. Statistics in Medicine 1997;16:2645-2664.[Medline]
14. O’Connor G.T., Plume S.K., Olmstead E.M., et al. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Northern New England Cardiovascular Disease Study Group. Circulation 1992;85:2110-2118.[Abstract/Free Full Text]
15. Reich D.L., Bodian C.A., Krol M., et al. Intraoperative hemodynamic predictors of mortality, stroke, and myocardial infarction after coronary artery bypass surgery. Anesth Analg 1999;89:814-822.[Abstract/Free Full Text]
16. Bernstein A.D., Parsonnet V. Bedside estimation of risk as an aid for decision-making in cardiac surgery. Ann Thorac Surg 2000;69:823-828.[Abstract/Free Full Text]
17. Spiegelhalter D.J. Probabilistic prediction in patient management and clinic trials. Statistics In Medicine 1986;5:421-433.[Medline]
18. Steyerberg E.W., Eijkemans M.J., Harrell F.E., Jr, Habbema J.D. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 2000;19:1059-1079.[Medline]
19. Harrell F.E., Jr Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. New York: Springer-Verlag, 2001.
20. Iezzoni L.I. Risk adjustment for measuring healthcare outcomes, Second edition. Chicago: Health Administration Press, 1997.

This article has been cited by other articles:

				D. M. Shahian, S. M. O'Brien, G. Filardo, V. A. Ferraris, C. K. Haan, J. B. Rich, S.-L. T. Normand, E. R. DeLong, C. M. Shewan, R. S. Dokholyan, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery. Ann. Thorac. Surg., July 1, 2009; 88(1 Suppl): S2 - 22. [Abstract] [Full Text] [PDF]

				V. A. Ferraris, F. H. Edwards, D. M. Shahian, and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2008; 3(2008): 199 - 246. [Full Text]

				P. E. Antunes, L. Eugenio, J. Ferrao de Oliveira, and M. J. Antunes Mortality risk prediction in coronary surgery: a locally developed model outperforms external risk models Interactive CardioVascular and Thoracic Surgery, August 1, 2007; 6(4): 437 - 441. [Abstract] [Full Text] [PDF]

				A. J. Epstein Do Cardiac Surgery Report Cards Reduce Mortality? Assessing the Evidence Med Care Res Rev, August 1, 2006; 63(4): 403 - 426. [Abstract] [PDF]

				D. M. Shahian, D. F. Torchiana, R. J. Shemin, J. D. Rawn, and S.-L. T. Normand Massachusetts Cardiac Surgery Report Card: Implications of Statistical Methodology Ann. Thorac. Surg., December 1, 2005; 80(6): 2106 - 2113. [Abstract] [Full Text] [PDF]

				B. K. Nallamothu, K. A. Eagle, V. A. Ferraris, and R. M. Sade Should Coronary Artery Bypass Grafting Be Regionalized? Ann. Thorac. Surg., November 1, 2005; 80(5): 1572 - 1581. [Full Text] [PDF]

				G. L. Grunkemeier and A. P. Furnary Mandatory Database Participation: Risky Business? Ann. Thorac. Surg., September 1, 2005; 80(3): 799 - 801. [Full Text] [PDF]

				R. Jin, G. L. Grunkemeier, A. P. Furnary, J. R. Handy Jr, and for the Providence Health System Cardiovascular St Is Obesity a Risk Factor for Mortality in Coronary Artery Bypass Surgery? Circulation, June 28, 2005; 111(25): 3359 - 3365. [Abstract] [Full Text] [PDF]

				D. M. Shahian, E. H. Blackstone, F. H. Edwards, F. L. Grover, G. L. Grunkemeier, D. C. Naftel, S. A.M. Nashef, W. C. Nugent, and E. D. Peterson Cardiac Surgery Risk Models: A Position Article Ann. Thorac. Surg., November 1, 2004; 78(5): 1868 - 1877. [Abstract] [Full Text] [PDF]

				S. A M Nashef and R. J F Baskett Quality Assessment in Cardiac Surgery: Do Not Miss the Boat! Asian Cardiovasc Thorac Ann, September 1, 2004; 12(3): 187 - 189. [Full Text] [PDF]

				M. A. Herbert, S. L. Prince, J. L. Williams, M. J. Magee, and M. J. Mack Are unaudited records from an outcomes registry database accurate? Ann. Thorac. Surg., June 1, 2004; 77(6): 1960 - 1964. [Abstract] [Full Text] [PDF]

				D. E. Cutlip, K. K. L. Ho, R. E. Kuntz, and D. S. Baim Risk assessment for percutaneous coronary intervention: our version of the weather report? J. Am. Coll. Cardiol., December 3, 2003; 42(11): 1896 - 1899. [Full Text] [PDF]

				D. M. Shahian and S.-L. T. Normand The volume-outcome relationship: from Luft to Leapfrog Ann. Thorac. Surg., March 1, 2003; 75(3): 1048 - 1058. [Abstract] [Full Text] [PDF]

				V. A. Ferraris and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2003; 2(2003): 187 - 224. [Full Text]

				S. A.M. Nashef and F. Roques Risk Assessment, League Tables, and Report Cards: Where is True Quality Monitoring? Ann. Thorac. Surg., November 1, 2002; 74(5): 1748 - 1749. [Full Text] [PDF]

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Gary L. Grunkemeier Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Grunkemeier, G. L. Articles by Jin, R. Search for Related Content PubMed PubMed Citation Articles by Grunkemeier, G. L. Articles by Jin, R. Related Collections Professional affairs