Ann Thorac Surg 2005;80:799-801
© 2005 The Society of Thoracic Surgeons
The statistician's page
Mandatory Database Participation: Risky Business?
Gary L. Grunkemeier, PhD
a
,
*
,
Anthony P. Furnary, MD
b
a Providence Health System, Portland, Oregon
b Starr-Wood Cardiac Group of Portland PC, Portland, Oregon
Accepted for publication January 7, 2005.
* Address reprint requests to Dr Grunkemeier, Providence Health System, 9205 SW Barnes Rd, Suite 33, Portland, OR 97225 (Email: gary.grunkemeier{at}providence.org).
 |
Introduction
|
|---|
In this issue of The Annals, Shahian et al [1] chronicle the process of implementing a state-mandated comparison of the cardiac surgery programs in Massachusetts. They describe several "lessons" learned during their implementation experience. The results of this experiment are already available from the Massachusetts Data Analysis Center (Mass-DAC), whose web address is given in the paper [1]. One of the lessons is the mandatory statewide adoption of the Society of Thoracic Surgeons National Cardiac Database (STS NCD). The authors urge that this approach should be followed by every state. If that suggestion were implemented, it would comprise mandatory universal adoption of the STS NCD; a seemingly laudable goal ... or is it?
|
The Heart of Continuous Quality Improvement and Report Cards
|
|---|
The authors distinguish between two pathways for improving the outcomes of cardiac surgery: continuous quality improvement (CQI) and cardiac report cards. They admit that CQI has "the greatest potential for real and sustained quality improvement," but it is an internal process, and since the "demand for public accountability" must eventually be met, the report card method was selected. Risk-adjusted comparisons are the heart of both CQI and report cards. Table 1
shows that the steps involved in producing them involves collaboration among the individual hospitals (LOCAL) and the independent coordinating center (CENTRAL), and are the same for both CQI and report cards up to the last step. "Adoption of the STS NCD" could arguably be implemented at three different levels:
- (A) Use the STS data collection forms, and create new risk models with those variables to produce reports.
- (B) Use the STS data form and use their published risk models to do the analysis and reporting.
- (C) Full participation: submit data to STS database and use the Duke Clinical Research Institute (DCRI) analyses and reports.
Level C (full participation) outsources the whole project to DCRI. In this "turn-key" approach the state looses autonomy and it thus has not been adopted by any of the other states currently employing report cards, including Massachusetts. Level B, which should appeal to many, is difficult to implement given the present proprietary status of the STS risk models. In the Massachusetts implementation, the decision is to officially use level A (with CENTRAL = Mass-DAC) but to also require level C for all hospitals. Autonomy is preserved, as the analyses will be done by Mass-DAC for the official requirements, with comparisons among the Massachusetts hospitals. But the state is also requiring full participation in the STS NCD, although it is not necessary for their official reporting system. The authors note that participation in the STS NCD had some compelling political advantages during discussions with state officials.
|
Alternative Solutions
|
|---|
Many variations of the quality improvement process have already been implemented by states and other groups [2]. Some states are not large enough to go it alone. An elegant three-state solution, the Northern New England Cardiovascular Disease Study Group (NNE), is a consortium of cardiac surgery centers that has been collecting, pooling, and analyzing data since 1987. The NNE has published risk models for coronary artery surgery [3], heart valve surgery [4], and percutaneous coronary interventions (PCI) [5]. They have documented that through June 2002 their system has saved an estimated 811 fewer deaths than expected (A Regional Intervention to Improve In-hospital Mortality Associated with CABG Surgery, at www.nnecdsg.org). Thus, there is proven value in voluntary, collaborative, regional cardiac databases and risk models. The participants control the data and have intimate knowledge over the process from beginning to end.
The NNE has been a model for the development of the Providence Health System Cardiovascular Study Group (PHS), which has been collecting similar data from PHS cardiac centers in four western states since 1995. We began quarterly reports comparing hospitals in 1999, and have been moving to risk-adjusted comparisons and interhospital comparisons by developing our own risk models or adapting existing models. And, like NNE, PHS meets periodically to report overall results and share individual experiences with CQI successes.
There are alternatives to statewide mandatory public reporting, which can also achieve the ultimate goal of improved outcomes. The state of Washington is using such a system. The Clinical Outcomes Assessment Program (COAP) was endorsed in 1997 as an official quality improvement activity, sponsored by the Health Care Authority, and achieving privacy protection under Washington law [6]. COAP provides an "evidence-based mechanism to promote internal quality improvement activities while meeting a variety of external quality improvement and accountability requirements" (www.coap.org).
Thus, a regional, voluntary collaborative could arguably be considered the optimal solution for improving quality. The STS has a national reputation and immense numbers, but such a diverse enterprise must inevitably lose some of the advantages of a regional endeavor, and data quality must be more difficult to achieve. The STS NCD has established the field of cardiac surgery as leaders in the field of peer-based outcomes analysis. But Shahian and colleagues eschew other potential data solutions for state-mandated reporting, and infer that these other potential solutions are "less desirable" and "less satisfactory" than the STS NCD. We feel that the voluntary, regional solutions are still a viable solution. If universal participation in the STS NCD were made mandatory, the cost of participation would hobble the multitude of regional data banks currently in existence. A monopoly in any field, including database management, is counterproductive for growth, development, and discovery.
The STS NCD could come closer to being the best solution for public reporting in two ways. First, if the data are truly meant for the best interests of its member-surgeons and the public at large, the "proprietary" risk algorithms, which have been developed using these data should be published and made available for comparison. Regional registries that regularly publish their risk models in full do this routinely. If the "demand for public accountability is compelling, increasing, and inescapable" then the STS NCD should also respond to this demand by publishing their algorithms.
Second, perhaps our NCD would be more acceptable to both the public and our member-surgeons if the data were housed and analyzed at the National Institutes of Health and National Heart, Lung and Blood Institute (NIH/NHLBI). This would serve both the public interests and the interests of the STS as it would encourage and enhance participation in the absence of market-driven outside forces. It would also greatly enhance the reputation of the STS NCD in the eyes of the public and governmental agencies as a leader working within our National Health research system.
|
Risky Models
|
|---|
With all the excellent planning and personnel involved, as in the Massachusetts experience and that of others, it should be borne in mind that comparing hospitals is still a risky venture, no matter what system is selected. Even with the complete data, collected according to strict definitions, cleaned and audited as well as humanly possible, and with no "gaming," the best statistical models are still lacking ("All models are wrong, some models are useful"—George Box). David Naftel [7] listed nine aspects of model building which influence the result, and concluded..., "Two different serious teams will likely produce two different equations." Indeed, as shown by a comparison of large, recently published risk models for CABG mortality, thirteen different serious teams have produced thirteen different risk equations [8].
Comparisons of several risk models on the same data set have shown that, even though they fit well, they identify different outliers. For example, Peterson and colleagues [9] compared four risk models for CABG mortality to each other on an independent data set. All the models had adequate discrimination, with c-statistics (area under the receiver operating characteristic curve) ranging from 0.71 to 0.74, about average for clinical risk models. But each model identified a different set of outlier hospitals. Kizer and colleagues [10] evaluated risk models for PCI mortality and came to the same conclusion. The three models fit well, with excellent c-statistic’s of 0.85–0.89, but identified different hospitals as outliers. Hannan [11], commenting on the paper of Kizer and colleagues [10], found the same to be true using data from the state of New York to evaluate the same models.
Identifying outliers is the main purpose of the public report card, and the most important part to get correct. Hierarchical regression can be used to overcome some of the problems with comparing providers [1, 12]. This method produces more conservative models, reducing the probability of false positives (calling a normal hospital substandard), which is highly desirable. But this is done at the expense of increasing the possibility of false negatives (failing to identify a substandard hospital).
|
States of Awareness
|
|---|
We make some closing points using an interesting regression model of current interest (Fig 1), which has no connection to cardiac surgery quality improvement ... or does it? It concerns the voting patterns of 2004 United States presidential elections, and compares the percentage of Bush votes in each state to the average number of children for each white female (the data, and discussions of this and other demographic correlations, with voting preference are at www.isteve.com/babygap.htm). Several different lines could be fit to these data. The website authors used linear regression to fit the model (solid line; Fig 1). More appropriate statistical models could no doubt be fit (quadratic, logistic, etc), which would yield different regression lines. (The data plot without the regression line was shown to three MDs with extensive experience in data analysis and they were asked to independently draw the best linear fit. The lines of all three were identical, and are shown by the dashed line in Fig 1.) No matter what line (model) is used, an important point to remember is that it models the state averages at each point, and, like the regression models used for cardiac risk, may not work well for individuals. As a test, each reader is invited to find the height of the line above his or her number of children on the horizontal axis, and see if it accurately predicts her or his probability of voting for Mr. Bush.
View larger version (18K):
[in this window]
[in a new window]
|
Fig 1. Scatter diagram of the percentage of Bush voters vs the fecundity of the white population in each state. Each symbol represents one state. The circles are the "red" states, with greater than 50% Bush vote and the squares are the "blue" states, with < 50% Bush vote. The filled symbols ( ) are the states that have statewide cardiac surgery reporting systems, mandatory or voluntary, in place, including Massachusetts (*).
|
|
Finally, the cardiac connection alluded to above: 10 of the 20 "blue" states have statewide mandatory or voluntary cardiac surgery reporting in place, compared to 1 of 30 "red" states (p < 0.001) [1, 2]. It comes as no surprise that these blue, liberally minded states would be more prone to support public reporting of cardiac surgical outcomes and full public access to our data. However, it could also be anticipated that the residents of these same states would be surprised that our current STS national data and algorithms are protected from STS membership access and public scrutiny. When dealing in the risky business of assessing professional performance, the assessment tool itself should be available for all to use, in a true academic spirit, devoid of proprietary pressures—as was the original intent of the STS.
|
References
|
|---|
- Shahian DM, Torchiana DF, Normand S-LT. Implementation of a cardiac surgery report cardlessons from the Massachusetts experience. Ann Thorac Surg 2005;80:1146-1150.[Abstract/Free Full Text]
- Harlan BJ. Statewide reporting of coronary artery surgery resultsa view from California. J Thorac Cardiovasc Surg 2001;121:409-417.[Free Full Text]
- O’Connor GT, Plume SK, Olmstead EM, et al. The Northern New England Cardiovascular Disease Study Group A regional prospective study of in-hospital mortality associated with coronary artery bypass grafting JAMA 1991;266:803-809.[Abstract/Free Full Text]
- Nowicki ER, Birkmeyer NJ, Weintraub RW, et al. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England Ann Thorac Surg 2004;77:1966-1977.[Abstract/Free Full Text]
- O’Connor GT, Malenka DJ, Quinton H, et al. Northern New England Cardiovascular Disease Study Group Multivariate prediction of in-hospital mortality after percutaneous coronary interventions in 1994-1996 J Am Coll Cardiol 1999;34:681-691.[Abstract/Free Full Text]
- Maynard C, Goss JR, Malenka DJ, Reisman M. Adjusting for patient differences in predicting hospital mortality for percutaneous coronary interventions in the Clinical Outcomes Assessment Program Am Heart J 2003;145:658-664.[Medline]
- Naftel DC. Do different investigators sometimes produce different multivariable equations from the same data? J Thorac Cardiovasc Surg 1994;107:1528-1529.[Free Full Text]
- Grunkemeier GL, Zerr KJ, Jin R. Cardiac surgery report cardsmaking the grade. Ann Thorac Surg 2001;72:1845-1848.[Free Full Text]
- Peterson ED, DeLong ER, Muhlbaier LH, et al. Challenges in comparing risk-adjusted bypass surgery mortality resultsresults from the Cooperative Cardiovascular Project. J Am Coll Cardiol 2000;36:2174-2184.[Abstract/Free Full Text]
- Kizer JR, Berlin JA, Laskey WK, et al. Limitations of current risk-adjustment models in the era of coronary stenting Am Heart J 2003;145:683-692.[Medline]
- Hannan EL, Wu C. Assessing quality and outcomes for percutaneous coronary interventionchoosing statistical models, outcomes, time periods, and patient populations. Am Heart J 2003;145:571-574.[Medline]
- Normand S-LT, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical careissues and applications. J Am Stat Assoc 1997;92:803-814.
Related Article
-
Implementation of a Cardiac Surgery Report Card: Lessons From the Massachusetts Experience
- David M. Shahian, David F. Torchiana, and Sharon-Lise T. Normand
Ann. Thorac. Surg. 2005 80: 1146-1150.
[Abstract]
[Full Text]
[PDF]
Top
Introduction
The Heart of Continuous...