Ann Thorac Surg 2008;86:1717-1720. doi:10.1016/j.athoracsur.2008.01.054
© 2008 The Society of Thoracic Surgeons
Key References
Statistical Risk Modeling and Outcomes Analysis
David M. Shahian, MDa,*,
Fred H. Edwards, MDb
a Department of Surgery, Center for Quality and Safety, and Institute for Health Policy, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
b Division of Cardiothoracic Surgery, University of Florida College of Medicine, Jacksonville, Florida
* Address correspondence to Dr Shahian, Massachusetts General Hospital, 55 Fruit St, Boston, MA 02114 (Email: dshahian{at}partners.org).
 |
General
|
|---|
These references provide a general overview of the statistical, clinical, and practical issues associated with risk-adjusted outcomes analysis and profiling.
- 1 Normand S-LT, Shahian DM. Statistical and clinical aspects of hospital outcomes profiling. Stat Sci 2007;22:206–26.
- 2 Shahian DM, Blackstone EH, Edwards FH, et al. Cardiac surgery risk models: a position article. Ann Thorac Surg 2004;78:1868–77.
- 3 Shahian DM, Normand SL, Torchiana DF, et al. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155–68.
- 4 Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group. Circulation 200624;113:456–62.
- 5 Iezzoni LI. Risk adjustment for measuring health care outcomes, 3rd ed. Chicago: Health Administration Press, 2003.
- 6 Naftel DC. Do different investigators sometimes produce different multivariable equations from the same data? J Thorac Cardiovasc Surg 1994;107:1528–9.
|
Data Source for CABG Outcomes Profiling: Administrative Versus Clinical
|
|---|
Regardless of the sophistication of a statistical risk model, the accuracy of outcomes profiling will ultimately be limited by the quality of data used.
- 7 Shahian DM, Silverstein T, Lovett AF, Wolf RE, Normand SL. Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation 2007;115:1518–27.
- 8 Mack MJ, Herbert M, Prince S, Dewey TM, Magee MJ, Edgerton JR. Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes? J Thorac Cardiovasc Surg 2005;129:1309–17.
- 9 Torchiana DF, Meyer GS. Use of administrative data for clinical quality measurement. J Thorac Cardiovasc Surg 2005;129:1223–5.
- 10 Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough? Health Serv Res 1997;31:659–78.
- 11 Hannan EL, Kilburn H Jr, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery. Does it matter? Med Care 1992;30:892–907.
- 12 Glance LG, Dick AW, Osler TM, Mukamel DB. Accuracy of hospital report cards based on administrative data. Health Serv Res 2006;41(4 Pt 1):1413–37.
- 13 Parker JP, Li Z, Damberg CL, Danielsen B, Carlisle DM. Administrative versus clinical data for coronary artery bypass graft surgery report cards: the view from California. Med Care 2006;44:687–95.
|
Predictor Variables for CABG Risk Models
|
|---|
- 14 Tu JV, Sykora K, Naylor CD. Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough? Steering Committee of the Cardiac Care Network of Ontario. J Am Coll Cardiol 1997;30:1317–23.
- 15 Jones RH, Hannan EL, Hammermeister KE, et al. Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The Working Group Panel on the Cooperative CABG Database Project. J Am Coll Cardiol 1996;28:1478–87.
|
Missing Data: Implications and Management
|
|---|
The proper management of missing data is an important issue in all data registries and the risk models derived from them.
- 16 Little RJA, Rubin DB. Statistical analysis with missing data, 2nd ed. Hoboken: John Wiley & Sons, Inc., 2002.
- 17 Horton NJ, Lipsitz SR. Multiple imputation in practice: comparison of software packages for regression models with missing variables. Am Stat 2001;55:244–54.
|
Logistic Regression: Model Development and Validation
|
|---|
Historically, multivariable logistic regression has been the technique most commonly used to develop risk models for cardiac surgery.
- 18 Harrell FE Jr. Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. New York, NY: Springer-Verlag, 2001.
- 19 Vittinghoff E, Glidden DV, Shiboski SC, McCulloch CE. Regression methods in biostatistics: linear, logistic, survival, and repeated measures models. New York, NY: Springer-Verlag, 2005.
- 20 Hosmer DW, Lemeshow S. Applied logistic regression, 2nd ed. New York, NY: Wiley, 2000.
- 21 Armitage P, Berry G, Matthews JN. Statistical methods in medical research. Hoboken, NJ: John Wiley & Sons, Inc, 2001.
- 22 Fleiss JL, Levin B, Paik MC. Statistical methods for rates & proportions, 3rd ed. Hoboken, NJ: John Wiley & Sons, Inc, 2003.
- 23 Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361–87.
- 24 Glantz SA, Slinker BK. Primer of applied regression & analysis of variance, 2nd ed. New York, NY: McGraw-Hill, 2001.
- 25 Agresti A. Categorical data analysis, 2nd ed. Hoboken: John Wiley & Sons, Inc, 2002.
- 26 Anderson RP, Jin R, Grunkemeier GL. Understanding logistic regression analysis in clinical reports: an introduction. Ann Thorac Surg 2003;75:753–7.
- 27 Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928–35.
- 28 Grunkemeier GL, Wu Y. What are the odds? Ann Thorac Surg 2007;83:1240–4.
- 29 Grunkemeier GL, Jin R. Receiver operating characteristic curve analysis of clinical risk models. Ann Thorac Surg 2001;72:323–6.
|
Hierarchical, Multilevel, and Random Effects Models
|
|---|
An important statistical advance in outcomes profiling has been the application of hierarchical models (also know as multilevel or random effects models). These references include basic to advanced presentations of this general topic, as well as its specific application to outcomes profiling in cardiac surgery.
- 30 Goldstein H, Spiegelhalter DJ. League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc (Series A) 1996;159:385–443.
- 31 Leyland AH, Goldstein H. Multilevel modelling of health statistics. Chichester: John Wiley & Sons, Ltd, 2001.
- 32 Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern Med 1997;127(8 Pt 2):764–8.
- 33 Normand S-LT, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical care: issues and applications. J Am Stat Assoc 1997;92:803–14.
- 34 Grunkemeier GL, Zerr KJ, Jin R. Cardiac surgery report cards: making the grade. Ann Thorac Surg 2001;72:1845–8.
- 35 Localio AR, Hamory BH, Fisher AC, TenHave TR. The public release of hospital and physician mortality data in Pennsylvania: a case study. Med Care 1997;35:272–86.
- 36 Austin PC, Tu JV, Alter DA. Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003;145:27–35.
- 37 Gelman A, Hill J. Data analysis using regression and multilevel/hierarchical models. Cambridge, New York, NY: Cambridge University Press, 2007.
- 38 Snijders TAB, Bosker RJ. Multilevel analysis: an introduction to basic and advanced multilevel modeling, 1st ed. London, United Kingdom: Sage Publications, Ltd, 1999.
- 39 Raudenbush SW, Bryk AS. Hierarchical linear models: applications and data analysis methods. 2nd edition. Thousand Oaks, CA: Sage Publications, 2001.
- 40 Kreft I, DeLeeuw J. Introducing multilevel modeling. London: Sage Publications, 1998.
- 41 Hannan EL, Wu C, Delong ER, Raudenbush SW. Predicting risk-adjusted mortality for CABG surgery: logistic versus hierarchical logistic models. Med Care 2005;43:726–35.
|
Bayesian and Empirical Bayes Analysis
|
|---|
Statistical analyses may be performed within a frequentist or Bayesian framework. With modern computing capabilities, the latter has become more commonly used and has numerous advantages.
- 42 Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation. West Sussex: John Wiley and Sons, Ltd, 2004.
- 43 Carlin BP, Louis TA. Bayes and empirical Bayes methods for data analysis, 2nd ed. Boca Raton, FL: Chapman & Hall/CRC, 2000.
- 44 Austin PC, Naylor CD, Tu JV. A comparison of a Bayesian vs. a frequentist method for profiling hospital performance. J Eval Clin Pract 2001;7:35–45.
- 45 Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian data analysis. Boca Raton, FL: Chapman & Hall/CRC Press, 2004.
- 46 Thomas N, Longford NT, Rolph JE. Empirical Bayes methods for estimating hospital-specific mortality rates. Stat Med 1994;13:889–903.
- 47 Edwards FH, Graeber GM. The Theorem of Bayes as a clinical research tool. Surg Gynecol Obstet 1987;165:127–9.
|
Outlier Determination
|
|---|
The determination of outliers is an important aspect of outcomes profiling, and this topic is discussed in many of the articles in the preceding sections. References in this section focus specifically on the impact of various methodological approaches on the classification of providers as outliers.
- 48 Shahian DM, Torchiana DF, Shemin RJ, Rawn JD, Normand SL. Massachusetts cardiac surgery report card: implications of statistical methodology. Ann Thorac Surg 2005;80:2106–13.
- 49 Austin PC, Alter DA, Anderson GM, Tu JV. Impact of the choice of benchmark on the conclusions of hospital report cards. Am Heart J 2004;148:1041–6.
- 50 Austin PC, Alter DA, Tu JV. The use of fixed- and random-effects models for classifying hospitals as mortality outliers: a Monte Carlo assessment. Med Decis Making 2003;23:526–39.
- 51 Austin PC, Anderson GM. Optimal statistical decisions for hospital report cards. Med Decis Making 2005;25:11–9.
- 52 Austin PC. A comparison of Bayesian methods for profiling hospital performance. Med Decis Making 2002;22:163–72.
- 53 Romano PS. Peer group benchmarks are not appropriate for health care quality report cards. Am Heart J 2004;148:921–3.
- 54 DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling. Stat Med 1997;16:2645–64.
- 55 Peterson ED, DeLong ER, Muhlbaier LH, et al. Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cooperative Cardiovascular Project. J Am Coll Cardiol 2000;36:2174–84.
|
Specific Cardiac Surgery Risk Models
|
|---|
These references provide examples of several major data registries and risk models used in cardiac surgery. Some of these articles also compare the results obtained from different risk models when applied to the same patient population. This may be a particularly important consideration when models derived from one reference population are applied to an entirely different study population.
- 56 Edwards FH, Peterson ED, Coombs LP, et al. Prediction of operative mortality after valve replacement surgery. J Am Coll Cardiol 2001;37:885–92.
- 57 Ferguson TB Jr, Dziuban SW Jr, Edwards FH, et al. The STS National Database: current changes and challenges for the new millennium. Ann Thorac Surg 2000;69:680–91.
- 58 Grover FL, Shroyer AL, Hammermeister K, et al. A decade's experience with quality improvement in cardiac surgery using the Veterans Affairs and Society of Thoracic Surgeons national databases. Ann Surg 2001;234:464–72.
- 59 Shroyer AL, Coombs LP, Peterson ED, et al. The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models. Ann Thorac Surg 2003;75:1856–64.
- 60 Nowicki ER, Birkmeyer NJ, Weintraub RW, et al. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England. Ann Thorac Surg 2004;77:1966–77.
- 61 Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9–13.
- 62 Roques F, Nashef SA, Michel P. Risk factors for early mortality after valve surgery in Europe in the 1990s: lessons from the EuroSCORE pilot program. J Heart Valve Dis 2001;10:572–7.
- 63 O'Connor GT, Plume SK, Olmstead EM, et al. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Northern New England Cardiovascular Disease Study Group. Circulation 1992;85:2110–8.
- 64 Hannan EL, Kilburn H Jr, O'Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York State. An analysis of risk factors and hospital mortality rates. JAMA 1990;264:2768–74.
- 65 Hannan EL, Kumar D, Racz M, Siu AL, Chassin MR. New York State's Cardiac Surgery Reporting System: four years later. Ann Thorac Surg 1994;58:1852–7.
- 66 Grover FL, Edwards FH. Similarity between the STS and New York State databases for valvular heart disease. Ann Thorac Surg 2000;70:1143–4.
- 67 Al-Ruzzeh S, Asimakopoulos G, Ambler G, et al. Validation of four different risk stratification systems in patients undergoing off-pump coronary artery bypass surgery: a UK multicentre analysis of 2223 patients. Heart 2003;89:432–5.
- 68 Asimakopoulos G, Al-Ruzzeh S, Ambler G, et al. An evaluation of existing risk stratification models as a tool for comparison of surgical performances for coronary artery bypass grafting between institutions. Eur J Cardiothorac Surg 2003;23:935–41.
- 69 Nilsson J, Algotsson L, Hoglund P, Luhrs C, Brandt J. Early mortality in coronary bypass surgery: the EuroSCORE versus The Society of Thoracic Surgeons risk algorithm. Ann Thorac Surg 2004;77:1235–9.
- 70 Bridgewater B, Neve H, Moat N, Hooper T, Jones M. Predicting operative risk for coronary artery surgery in the United Kingdom: a comparison of various risk prediction algorithms. Heart 1998;79:350–5.
- 71 Nashef SA, Roques F, Hammill BG, et al. Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg 2002;22:101–5.
- 72 Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or Remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 1999;99:2098–104.
|
Composite Measures of Performance
|
|---|
It is increasingly recognized that mortality alone is not an adequate measure of performance. This has led to the development of more comprehensive, multidimensional quality measures such as The Society of Thoracic Surgeons composite coronary artery bypass grafting score.
- 73 Shahian DM, Edwards FH, Ferraris VA, et al. Quality measurement in adult cardiac surgery: part 1—Conceptual framework and measure selection. Ann Thorac Surg 2007;83(4 Suppl):S3–12.
- 74 O'Brien SM, Shahian DM, Delong ER, et al. Quality measurement in adult cardiac surgery: part 2—Statistical considerations in composite measure scoring and provider rating. Ann Thorac Surg 2007;83(4 Suppl):S13–26.
- 75 Nardo M, Saisana M, Saltelli A, Tarantola S, Hoffman A, Giovannini E. Handbook on constructing composite indicators: methodology and user guide (OECD Statistics Working Paper). Organization for Economic Co-operation and Development (OECD) Statistics Working Paper, 2005.
- 76 Nardo M, Saisana M, Saltelli A, Tarantola S. Tools for Composite Indicators Building. European Commission, 2005. Available at: http://composite-indicators.jrc.ec.europa.eu/Document/EUR%2021682%20EN_Tools_for_Composite_Indicator_Building.pdf. Accessed September 29, 2008.
|
Propensity Scores
|
|---|
It is often incorrectly assumed that risk-adjusting outcomes make it possible to directly compare individual providers. In fact, indirectly standardized, risk-adjusted outcomes only compare a provider's observed results with the expected performance of an average provider presented with the same mix of patients. To directly compare one provider with another, a reasonable balance of predictor variables between providers would be required, which is often not the case in many real-life situations. Balancing scores, such as the propensity score, were developed to achieve better covariate balance when comparing different types of treatments in observational studies. Some investigators are now exploring the potential use of such approaches for comparing providers.
- 77 Blackstone EH. Comparing apples and oranges. J Thorac Cardiovasc Surg 2002;123:8–15.
- 78 Grunkemeier GL, Payne N, Jin R, Handy JR Jr. Propensity score analysis of stroke after off-pump coronary artery bypass grafting. Ann Thorac Surg 2002;74:301–5.
- 79 D'Agostino RB Jr. Propensity scores in cardiovascular research. Circulation 2007;115:2340–3.
- 80 Braitman LE, Rosenbaum PR. Rare outcomes, common treatments: analytic strategies using propensity scores. Ann Intern Med 2002;137:693–5.
- 81 Joffe MM, Rosenbaum PR. Invited commentary: propensity scores. Am J Epidemiol 1999;150:327–33.
- 82 Huang IC, Frangakis C, Dominici F, Diette GB, Wu AW. Application of a propensity score approach for risk adjustment in profiling multiple physician groups on asthma care. Health Serv Res 2005;40:253–78.
- 83 Shah BR, Laupacis A, Hux JE, Austin PC. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J Clin Epidemiol 2005;58:550–9.
- 84 Sturmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol 2006;59:437–47.
|
Algorithmic (Machine Learning) Approaches to Modeling
|
|---|
Rather than using logistic or hierarchical statistical models, some have advocated machine-learning or algorithmic approaches to modeling. Such methods leverage modern computing capabilities to better model complex biological processes without the constraints imposed by standard statistical techniques.
- 85 Lippmann RP, Shahian DM. Coronary artery bypass risk prediction using neural networks. Ann Thorac Surg 1997;63:1635–43.
- 86 Breiman L. Statistical modeling: the two cultures. Stat Sci 2001;16:199–215.
- 87 Blackstone EH. Breaking down barriers: helpful breakthrough statistical methods you need to understand better. J Thorac Cardiovasc Surg 2001;122:430–9.
|
Bootstrap Techniques
|
|---|
Bootstrap techniques have been a valuable advance in many areas of statistical modeling. These include determination of standard errors and confidence intervals, predictor selection, and model validation.
- 88 Efron B, Tibshirani R. An introduction to the bootstrap. Boca Raton, FL: Chapman & Hall/CRC Press, 1994.
- 89 Davison A, Hinkley D. Bootstrap methods and their application. Cambridge, NY: Cambridge University Press, 1997.
|
Graphical Methods for Performance Monitoring
|
|---|
There has been recent interest in the use of graphical performance monitoring techniques in cardiac surgery, many of which are derived from the field of statistical quality control. CUSM (cumulative sum) charts may provide earlier detection of performance issues. The funnel plot explicitly depicts the increasing statistical uncertainty of performance estimates at low volumes.
- 90 Shahian DM, Williamson WA, Svensson LG, Restuccia JD, D'Agostino RS. Applications of statistical quality control to cardiac surgery. Ann Thorac Surg 1996;62:1351–8.
- 91 Blackstone EH. Monitoring surgical performance. J Thorac Cardiovasc Surg 2004;128:807–10.
- 92 Grunkemeier GL, Wu YX, Furnary AP. Cumulative sum techniques for assessing surgical results. Ann Thorac Surg 2003;76:663–7.
- 93 de Leval MR, Francois K, Bull C, Brawn W, Spiegelhalter D. Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994;107:914–23.
- 94 Grigg OA, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Methods Med Res 2003;12:147–70.
- 95 Spiegelhalter D, Grigg O, Kinsman R, Treasure T. Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery. Int J Qual Health Care 2003;15:7–13.
- 96 Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;24:1185–202.
- 97 Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811–9.
- 98 Sherlaw-Johnson C, Gallivan S, Treasure T, Nashef SA. Computer tools to assist the monitoring of outcomes in surgery. Eur J Cardiothorac Surg 2004;26:1032–6.
This article has been cited by other articles:
|
|
|
|
 
T. Trauer
The Public Reporting of Organizational Performance in Mental Health: Coming Soon to a Mental Health Service Near You
Australian and New Zealand Journal of Psychiatry,
June 1, 2011;
45(6):
432 - 443.
[Full Text]
[PDF]
|
|
|
Top
General
Data Source for CABG...