| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
a Division of Cardiothoracic Surgery, Oregon Health and Science University, Portland, Oregon
b Department of Surgery, Oregon Health and Science University, Portland, Oregon
Accepted for publication April 21, 2008.
* Address correspondence to Dr Welke, Division of Cardiothoracic Surgery L353, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Portland, OR 97239-3098 (Email: welkek{at}ohsu.edu).
Presented at the Forty-fourth Annual Meeting of The Society of Thoracic Surgeons, Fort Lauderdale, FL, Jan 28–30, 2008.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
Methods: Pediatric cardiac operations assigned to Risk Adjustment for Congenital Heart Surgery, version 1 (RACHS-1) categories were retrospectively identified by International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) coding from the Nationwide Inpatient Sample, 1988–2005. Hospitals were grouped by yearly pediatric cardiac surgical volume (very small, 
20; small, 21 to 100; medium, 101 to 200; large, > 200). Mortality rates were adjusted for surgical volume, case mix (RACHS-1 categories), patient age, and year of operation by logistic regression.
Results: We identified 55,164 operations from 307 hospitals; 188 (61%) performed 20 or fewer cases per year. The unadjusted mortality rate at very small hospitals was no different than at large hospitals (odds ratio, 1.0, 95% confidence interval [CI] 0.7 to 1.4). After adjustment for RACHS-1 category and age, large hospitals performed significantly better than all other volume groups. As a discriminator of mortality, volume performed significantly worse than a model with RACHS-1 category and age (receiver operating characteristic [ROC] curve area, 0.60 vs 0.81).
Conclusions: As a discriminator of mortality, volume alone was only marginally better than a coin flip (ROC curve area of 0.50). However, large-volume hospitals performed more complex operations and achieved superior results; therefore, the use of overall, unadjusted mortality rates to evaluate institution quality is misleading. Hospital comparisons and pay-for-performance initiatives must be based on robust risk-adjusted comparisons.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
Previous investigations of the volume–mortality relationship in pediatric cardiac surgery have been mixed in their conclusions. Although the earlier of these investigations demonstrated an inverse relationship between volume and in-hospital mortality, the most recent study suggested that this relationship might no longer exist [1–5]. These multiple studies have relied on a relatively limited group of data. All have used either administrative data from California and Massachusetts or clinical data from New York, with the data from California and New York having been studied for multiple time periods.
The purpose of our study was to determine the relationship between hospital surgical volume and mortality after pediatric cardiac surgery. For the investigation we used a national administrative database and methodology that accounts for both patient-level risk factors and surgical case-mix.
|
Material and Methods |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
Congenital cardiac surgical procedures performed on patients aged younger than 18 years were identified by International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes. For a patient to be included in this study, the procedure code had to match to a plausible diagnostic code. Operations were categorized by the Risk Adjustment for Congenital Heart Surgery, version 1 (RACHS-1) method [7]. This risk-stratification system groups the varied congenital cardiac surgical case-mix into six categories by similar expected short-term mortality rates. Category 1 has the lowest risk of death and category 6 the highest. The methodology has been validated and is included in the Society of Thoracic Surgeons Congenital Heart Surgery Database reports.
Mortality was defined as in-hospital mortality as indicated by the discharge disposition. Hospital volume was defined as the number of RACHS-1–categorized operations performed in a year. If a hospital appeared in the NIS in 2 or more years, each entry was treated separately. Hospital volume was not averaged for the 18-year period because all hospitals were not sampled in all years and it is likely that personnel and systems changes occurred during this time that affected volume and quality of care.
Volume was first evaluated as a continuous variable. Then, volume groups were created using the following criteria: (1) natural cut points in the data, (2) previously studied volume thresholds, and (3) maintenance of a sufficient number of hospitals in each volume group to minimize the impact of any individual hospital. The cut point of 100 cases per year was derived by dichotomizing volume and comparing odds ratios (ORs) for risk-adjusted mortality of hospitals with volumes above the cut point with those with volumes below the cut point. All volume thresholds from 1 to 300 cases per year were investigated. We found a threshold annual volume of approximately 100 cases. At this threshold and those above it, the mortality in the higher-volume cohort of hospitals was significantly lower than that in the lower-volume cohort (Fig 1). We chose the cut point of 200 cases per year from the distribution of hospital mortality rates (Fig 2). Above 200 cases per year, all hospital mortality rates were within or below the 95% confidence interval (CI) for the cohort.
|
|
2 statistic for linear trend. Second, the discrimination of volume alone as a predictor of mortality was assessed by the C statistic [8, 9] as determined from logistic regression analyses. The C statistic is numerically equivalent to the area under a receiver operating characteristic (ROC) curve. To maximize potential discrimination, volume categorizations were modeled using indicator variables for individual volume categories in lieu of a single ordinal variable. Multivariable analyses examined the increase in model discrimination (ie, C statistic values) resulting from the addition of RACHS-1 category and patient age to the models containing hospital volume. Models accounted for patients clustered within hospitals [10] and included robust methods for calculating the 95% CIs for regression coefficients [11]. Two sets of models were developed, one for the years 1988 to 2005 and the other for the years 2001 to 2005. The first range of years was chosen to include all available data and achieve the largest possible sample size and most stable relationships. The second range of years was chosen to determine the volume–mortality relationship for the most contemporary time period. All models were adjusted for year of surgery. To check for the possible effect of miscoding on the results, all models were also run excluding hospitals that performed fewer than 5 cases in a year. When this was done, there were no significant changes in the results. Our analyses used sampling weights provided within the NIS database to derive national estimates. Analyses were performed using SAS 9.1 software (SAS Institute, Cary, NC).
|
Results |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
|
|
|
|
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
Previous studies of the volume–mortality relationship in pediatric cardiac surgery have relied on state-level data. Three previous articles have used California administrative data from different time periods, 1988 (with the addition of Massachusetts administrative data from 1989), 1995–1997, and 1998–2003 [1, 4, 5]. Two other articles used data from the New York Cardiac Surgery Reporting System in overlapping time periods: 1992 to 1995 and 1990 to 1995 [2, 3]. All but the most recent of these investigations found an inverse relationship between hospital volume and mortality.
Although the numbers of patients in each of these studies were adequate, these studies were limited by the number of hospitals available for analysis (range, 16 to 52). More important, the number of hospitals in the largest volume categories ranged from 1 to 9. In contrast, our study contained 307 unique hospitals (27 in the largest volume group of > 200 cases/year) in the 1988 to 2005 cohort and 140 unique hospitals (17 in the largest volume group of > 200 cases/year) in the 2001 to 2005 cohort. A low number of hospitals in a volume group increases the relative impact of each individual hospital's mortality rate on the group mortality rate. As a result, an apparent volume–mortality relationship may instead be due to the influence of one high performer and not reflective of a "true" high performing group [5].
We did find a relationship between hospital case volume and mortality, but this finding must be interpreted with caution. First, the relationship was nonlinear. Although the ORs decreased as volume increased, there was no significant difference in risk-adjusted in-hospital mortality between the three volume categories of fewer than 200 cases per year. Second, although the threshold of 200 cases per year seems a reasonable division point based on the distribution of individual hospital mortality rates (Fig 2), it is still somewhat arbitrary. Our strategy of aggregating hospitals into volume groups gave us sufficient statistical power to analyze important relationships, but it disguised individual hospitals. One should not conclude that all hospitals performing fewer than 200 cases per year have high mortality rates. On average this was true; however, there were individual hospitals below this threshold that had low mortality rates and those that had volumes too low for any mortality rate difference to be observed.
Our analysis illustrates the inappropriateness of using volume alone as a marker of quality. The broad range of individual hospital mortality rates within volume groups resulted in volume alone being a poor discriminator of mortality. Notably, volume did not significantly improve the discrimination, as measured by the C-statistic, of a model that already included RACHS-1 category and patient age. This means that the effect of volume should be interpreted carefully, particularly from the perspective of the patient. On average, a patient's own risk characteristics and level of disease burden account for the vast majority of their mortality risk, and the impact of hospital volume on the mortality risk of an individual patient may be small. For example, the absolute difference in risk-adjusted mortality of hospitals with more than 200 cases per year and those at 200 or fewer cases per year was 1.7%, indicating that roughly 58 patients would have to undergo operations in high-volume hospitals to save one life.
We chose to use the RACHS-1 method of risk adjustment for this investigation. RACHS-1 was developed to compare the mortality for groups of patients undergoing congenital cardiac operations. Certain procedures that occur less commonly are not included in the RACHS-1 method; however, low-frequency procedures are less important when comparing overall performance. In addition, the methodology does not allow for specific conditions that may be risk factors for specific operations. However, such potential risk factors are likely to be evenly distributed in large populations. Despite these limitations, the RACHS-1 method is a widely used and validated risk-adjustment methodology for congenital heart surgery [12–14].
Our analysis has several limitations. First, although the NIS is the largest all-payer inpatient care database in the United States, with a sampling frame that comprises 90% of all hospital discharges, it is a sample and not a compete database of all hospital discharges. As a result, although it is designed to be representative of national practice, there is the possibility for error. However, the NIS is the largest collection of real data that can currently be used for addressing the present question.
Second, the NIS is an administrative database. Administrative databases were designed for claims data collection and billing, not heath care research, and can be limited by erroneous coding of congenital diagnoses [15, 16]. As a result, we may not have adequately accounted for differences in case-mix; however, there is no evidence to suggest that miscoding is related to hospital volume. Given the large sample size, random miscoding would be unlikely to significantly influence our findings. Any potential impact would likely bias the results towards the null. Thus, it is possible that volume would have had a larger effect on mortality rates that were adjusted using a more robust set of clinical variables than those that are found in administrative data. We reduced error from miscoding of data in this study by only including patients where the procedure code was matched with a plausible diagnostic code.
Notwithstanding these limitations, our study was conducted using a large, national data set with adequate power to generate current, stable mortality rates. Administrative data are widely available to payers, governmental agencies, and groups that produce Internet reports and other reports that reach patients, their families, and providers. At present, administrative data are being used to evaluate heath care quality. Such data provide a national picture of the practice of medicine. By including information from both high- and low-performing and high- and low-volume hospitals, the data can be used to evaluate the practice of hospitals that are less likely to participate in voluntary, clinical databases.
The present study found that volume alone was a poor discriminator of mortality and only marginally better than a coin flip (ROC curve area, 0.50). However, in aggregate, large-volume hospitals performed more complex operations and achieved superior results; therefore, the use of overall, unadjusted mortality rates to evaluate institution quality is misleading. Hospital comparisons and pay-for-performance initiatives must be based on robust risk-adjusted comparisons. Many factors contribute to the mortality risk of a pediatric patient undergoing a cardiac operation. Volume is not a measure of quality, but rather an easily obtained structural attribute associated with quality. It is likely a surrogate for process measures and characteristics of systems of care that lead to better outcomes, but are not currently captured in administrative or most clinical databases. As a result, institution-specific risk-adjusted outcomes are likely to be more informative than a volume threshold. Future efforts should focus on identifying the fundamental elements that are apt to explain the variation in outcomes currently attributed to volume.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
Dr Welke and his colleagues used a large data set derived from a national sample of administrative discharge data to examine the relationship between institutional volume and hospital mortality. Not surprisingly, they found that institutional volume alone was a very poor predictor of outcome. They then examined the impact of patient factors (age at surgery) and case-mix on this relationship. They identified an intriguing relationship between volume and case complexity: large institutions cared for younger, more complex patients but yet had superior results. The case complexity and the age were the most important predictors of outcome, and the addition of volume to the model added little to the predictive accuracy of the statistical model. Nevertheless, volume cannot be ignored, because the large centers operated on the sickest patients with the best results.
This study highlights the pitfalls and the risk of simple comparisons of outcomes data which are frequently used in the media. It demonstrates the need for risk or complexity adjustment. We could have a whole discussion over which method to use for that. But we need to have some method to adjust for risk or complexity when we are comparing outcomes.
The relationship between volume and case-mix is intriguing. What factors lead to a high surgical volume? Do the centers have high volume because they are doing complex cases with good results and attract patients, or do they have good results because they are larger and they have instituted structural changes and processes that improve their outcomes? Which comes first, the chicken or the egg?
I have three questions for Dr Welke. Could you identify any of the characteristics in the hospitals that might affect the outcomes, such as did a stand-alone children's hospital have different outcomes from a hospital that cares for both children or adults, or were they more likely to be represented in the high-volume programs? The confirmation of a relationship between volume and outcome would obviously have significant impact for patients, physicians, and payers, which could lead to changes in referral patterns and even closure of programs. Therefore, it is imperative to be sure that such a relationship exists.
There are significant problems with administrative data sets. They have the advantages of large numbers, but the details may be very inaccurate. At our institution, I compared the coding from the STS [Society of Thoracic Surgeons] database, which is done by the attending surgeon, with the discharge coding for 20 patients, and in this very small sample there were significant errors in coding in over 50% of the patients on the administrative data set. So what are the next steps that are needed to verify whether or not this relationship between volume and outcome really exists?
And finally, how can we identify the factors that are associated with the improved outcomes at the large volume centers and apply these to the smaller centers?
Once again I would like to thank the Society for the privilege of discussing the paper and congratulate Dr Welke.
DR WELKE: Thank you, Dr Gaynor, for your insightful comments and a thorough review of our work. I will address your questions in order.
First, it is often difficult to find associations between hospital-level characteristics and outcomes. There are a number of hospital-level characteristics available in administrative data and in the Nationwide Inpatient Sample (NIS) specifically. None of those that we examined were associated with mortality. Designation as a stand-alone children's hospital is not noted in the NIS. It is available in the Kids Inpatient Database (KID); however, we chose not to use the KID for this study for two reasons. First, the KID is only available for three nonconsecutive years, which would have limited our hospital sample size. Second, the sampling methodology of the KID is such that only 80% of cases at each sampled hospital are included. In contrast, the NIS captures all admissions to a sampled hospital, making it better suited for determining hospital volumes. Children's hospital designation is recorded in the American Hospital Association (AHA) database and from there can be linked to the NIS. We are linking AHA data with the NIS so that we can examine the question. I suspect that designation as a children's hospital may be associated with volume, but I do not have the data to support that.
Your comments about administrative data are absolutely correct. Administrative data are not perfect and coding errors do exist. Because of these issues, it is important that we restrict our analyses of administrative data to large populations rather than specific hospitals or surgeons. If coding errors are random, which is more likely with a large cohort, their impact is likely to bias the results toward the null. So, if associations exist, they are more likely to be real. The Society of Thoracic Surgeons (STS) is currently working with the Agency for Healthcare Research and Quality to compare the International Classification of Diseases, Ninth Revision, Clinical Modification codes used in administrative data with the codes used in the STS Congenital Heart Surgery Database in order to quantify and then rectify the coding problems. One way to check the validity of a finding is to test the same hypothesis in a different cohort. We have looked at the volume mortality relationship in clinical data by using the STS Congenital Heart Surgery Database and have found complementary results. We are going to present those results later this year.
Lastly, volume is a surrogate for quality, not a measure of quality in and of itself. We need to identify the structural measures and process measures that account for the apparent contribution of volume to outcome. This may be not come from analyzing large administrative or even clinical databases. Instead it may require multicenter quality improvement initiatives such as the Northern New England Cardiovascular Disease Study Group. Site visits to well-performing hospitals, regardless of size, may elucidate common factors that can be implemented at lower-performing hospitals. Some of these factors are likely to be the process measures in the list developed by this Society for presentation to the National Quality Forum. However, it will take some time to find out which of these factors are most influential. Thank you very much.
|
References |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  K. F. Welke, S. M. O'Brien, E. D. Peterson, R. M. Ungerleider, M. L. Jacobs, and J. P. Jacobs The complex relationship between pediatric cardiac surgical case volumes and mortality rates in a national clinical database. J. Thorac. Cardiovasc. Surg., May 1, 2009; 137(5): 1133 - 1140. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. F. Welke, B. S. Diggs, T. Karamlou, and R. M. Ungerleider Comparison of Pediatric Cardiac Surgical Mortality Rates From National Administrative Data to Contemporary Clinical Standards Ann. Thorac. Surg., January 1, 2009; 87(1): 216 - 223. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
