| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
a The Congenital Heart Institute of Florida, Division of Thoracic and Cardiovascular Surgery, All Children's Hospital and Children's Hospital of Tampa, University of South Florida College of Medicine, Cardiac Surgical Associates of Florida, Saint Petersburg and Tampa, Florida
b Section of Thoracic Surgery at University of Alabama at Birmingham, Division of CardioThoracic Surgery, Department of Surgery, Birmingham, Alabama
c Division of Cardiothoracic Surgery, Department of Surgery, Institute of Human Values in Health Care, Medical University of South Carolina, Charleston, South Carolina
Accepted for publication December 12, 2008.
* Address correspondence to Dr Sade, Department of Surgery, Medical University of South Carolina, 96 Jonathan Lucas St, Ste 409, PO Box 250612, Charleston, SC 29425 (Email: sader{at}musc.edu).
Presented at the Fifty-fourth Annual Meeting of The Southern Thoracic Surgical Association, Bonita Springs, FL, Nov 7–10, 2007.
Cardiothoracic surgical (CTS) outcomes have been published in the lay press for nearly 2 decades. Pressures to expand such publication come from many different areas and cannot be resisted indefinitely. Evidence exists that contemporary reporting of outcomes data is based on flawed methodologies that potentially mislead and deceive. Such deceptions may harm patients, surgeons, and hospitals in various ways, and could undermine quality of surgical care and patients' access to it. Yet, public reporting of outcomes can also be beneficial to all concerned, but only if relevant data are accurate and the formats in which they are reported are valid and easily understood.
In this essay, we review the early history of public reporting of CTS outcomes, discuss potentially negative aspects of public reporting, and suggest solutions to these problems. We then consider the positive aspects of public reporting and provide recommendations for the future. We conclude that CTS data should be collected and analyzed under the direction of professional medical societies and that reports of outcomes based on such data should be published in the lay press.
 |
The History of Public Reporting |
|---|
 TopIntroduction The History of Public... Problems with Public Reporting...Positive Aspects of Public...Implications, Inferences, and...References |
|---|
Despite these flaws, public reporting of CTS outcome data continues because of the belief that data on individual surgeons and hospitals are useful to several distinct groups: individual patients for choosing hospitals and surgeons, surgeons for improving their own outcomes and hospitals for quality improvement, and governmental entities for developing health-related laws and regulations. Critical analysis of public reporting, however, has undermined these beliefs.
|
Problems with Public Reporting and Potential Solutions |
|---|
|
TopIntroductionThe History of Public...Problems with Public Reporting...Positive Aspects of Public...Implications, Inferences, and...References |
|---|
Complexity Adjustment
The data used in outcomes reports might not be of suitable quality because they are not accurately adjusted for the complexity of the cases and patients [3, 4]. Inaccurate public information can be misleading or deceptive and therefore dangerous. Unless the quality of the data is high, public reporting of outcomes for cardiothoracic surgeons is not only useless, but might be harmful to surgeons as well as patients. The harms will remain until accurate risk-stratified data are publicly available. A great deal of published information supports this assertion [1].
No risk-adjusted model can take into account all of the complexities of individual patients, according to Dranove's review of the New York experience in 2003:
It is essential for the analysts who create report cards to adjust health outcomes for differences in patients characteristic (risk adjustment)... . But analysts can adjust for only characteristics that they can observe ... Because of the complexity of patient care, providers are likely to have better information on patient's conditions than even the most clinically detailed database. Thus, providers can improve their ranking by selecting patients on the basis of characteristics that are unobservable to the analyst but predictive of good outcomes [5].
The Society of Thoracic Surgeons (STS) Database—comprising General Thoracic, Adult Cardiac, and STS Congenital Heart Surgery Databases—includes methodology to address this problem. It currently includes 1051 participating sites and 3202 participating surgeons. More than 100 publications in professional journals and textbooks have come from the STS Database, and it has been federally funded to study quality improvement [6].
The STS Database is based on several fundamental principles that facilitate accurate comparison of outcomes and quality improvement of health care for patients undergoing cardiothoracic operations. These principles include standardized nomenclature to facilitate meaningful analysis of outcomes, a minimum database data set with precise and transparent definitions of data fields, accurate and verified mechanisms of adjustment for complexity of the patients, and data verification.
CTS outcomes analysis requires case–mix adjustment, because case–mix varies among programs, and without such adjustment, many surgeons and programs caring for high-risk patients will be inappropriately praised or criticized. Case–mix adjustment could eliminate perverse incentives for surgeons to avoid caring for the highest-risk patients—those who might need an operation the most and who might benefit the most from the procedure.
Case–mix adjustment can be variously accomplished by complexity stratification, as is currently done in the STS Congenital Database, or with formal risk modeling and risk-adjusted mortality, as is currently done in the STS Congenital Heart Surgery Databases. Table 1 compares these useful and acceptable techniques of case–mix adjustment [7, 8].
|
Data Verification
The data used in outcomes reports might not be of suitable quality because they are not complete or accurate [3, 4, 13–16]. By comparison, data in the STS Database are verified for completeness and accuracy in two ways: an intrinsic verification process designed to rectify inconsistencies of data and missing elements of data, and an on-site audit program that verifies data at its primary source. Data in the STS Adult Cardiac Surgery Database and the STS Congenital Heart Surgery Database are randomly audited as part of a formal on-site audit program conducted by an independent medical audit firm. These data verification efforts have demonstrated that the STS Database is a reliable source of complete and accurate data [13].
Reduced Access to Health Care
Anecdotal reports have suggested that public reporting can lead to referral of high-risk patients elsewhere, creating an access problem for the sickest patients who might benefit the most from CTS [17]. As already noted, outcome data that lack accurate risk assessment and adjustment are useless, but they may be worse than useless—they may be dangerous. Providers caring for the sickest patients are inappropriately penalized by the absence of proper risk adjustment. Moreover, without complexity adjustment, providers have reason to avoid caring for high-risk, high-complexity patients, sending them elsewhere instead. Severely ill patients are more likely to travel to receive their care from centers of excellence [17]. Apparently, severely ill patients believe they have more to gain by traveling to busier, higher-quality centers. This relocation allows lower-quality centers to avoid high-risk patients, thus improving their apparent ranking. Similarly, Dranove showed that "there has been a shift towards operating on healthier patients" in lower-quality centers [17]. These lower-quality centers are less likely to offer surgical intervention to high-risk patients, even though these patients may benefit from it the most.
According to Capps, the overall effect of the New York report cards was to "change the incidence from sicker patients toward healthier patents and lead to a higher cost ... and a deterioration of outcomes, especially among ill patients. We therefore conclude that the report cards were welfare-reducing" [17]. One solution to this potential problem of reduced access to health care is the use of robust clinical data sets that adjust properly for complexity of the patients so CTS outcomes can be analyzed accurately.
Sample Size and Random Variation
The limited sample size of any individual surgeon's or hospital's experience can lead to (1) wide fluctuations in outcomes from year to year and (2) reporting of outcomes as being different when they are actually statistically similar and appear to be different due only to chance rather than to true statistical differences. Without proper adjustments for sample size and random variation, a shift of one or two deaths a year can lead to a dramatic effect on the reported outcomes of a surgeon or a hospital. This effect has been clearly seen in both New York and Pennsylvania, where small sample size and low mortality rates led to wide swings in rankings from year to year, despite absences of demonstrable differences in quality of care. Failure to adjust properly for sample size and random variation can mislead and deceive.
CTS outcomes have been publicly reported using league tables, which are fundamentally flawed, unnecessary, and inappropriate. They use outcome data to rank participants, with no adjustment for sample size or random variation. League tables are commonly used in sports to rank teams or individual athletes by unadjusted outcome data, usually wins and losses. League tables always have winners and losers, with someone on the top and someone on the bottom, even when no true difference exists between the subjects being ranked [18]. Figure 1 plots the risk-adjusted 30-day mortality rates after CABG in New York between 1997 and 1999 and shows the ranked rates for individual surgeons with 95% confidence intervals. In the original publication of these data, the surgeons were named [19]. The widths of the confidence intervals in Figure 1 show few intervals that do not overlap, revealing considerable uncertainty about the true underlying mortality rates. This uncertainty, however, is not reflected in the ranks of specific surgeons [18].
|
|
|
Administrative Data and the Role of the Government
Government reports of CTS outcomes are based on severely flawed administrative databases. Furthermore, many nongovernmental organizations have advocated public reporting of CTS outcomes based on these flawed administrative data. The Consumers' Checkbook Web site (checkbook.org) uses administrative Medicare claims data to provide information about the number of procedures a particular surgeon performs for a specific type of operation. The Center for the Study of Services, which owns this Web site, has successfully sued the United States government to obtain Medicare claims data on all physicians participating in the Medicare program [24]. At this writing, the case is under appeal by the Department of Health and Human Services and the Department of Justice; chances for overturning the lower court's verdict are uncertain.
If successful again, the Consumers' Checkbook organization would publish physician-level data from Medicare on its Web site. These ratings would reflect data in the Medicare database, thus offering only a fraction of the clinical experience of many surgeons. The Medicare database is largely restricted to patients aged 65 years or older, thus biasing the analysis and misleading those who use it. Few patients who visit checkbook.org are likely to be aware of this flaw or to understand its meaning, despite the Web site's attempt to explain it.
For example, consider a patient—let's call him Joe Internet—who is searching for "the best" mitral valve surgeon to repair his 44-year-old wife's mitral valvar regurgitation. Joe and his wife are interested in valve repair instead of replacement. They visit the checkbook.org Web site and view the ranking of surgeons who perform the most mitral valve surgery in the United States. They are thrilled to tell their friends that they have identified the best and busiest surgeon. But have they? Or have they been misled?
Because many mitral valve repair operations are performed on patients aged younger than 65, a large proportion of the procedures are absent from this analysis. The site has misled—or frankly deceived—Joe and his wife about who does the most mitral valve repairs. In addition to volume, checkbook.org also intends to list the costs of a particular operation by a particular surgeon. Yet, one of the few aspects of cost surgeons can partly control is length of stay, but it is not listed. Inaccurate or incomplete data are worse than no data at all.
Government agencies cannot fully understand the complexities of medical and surgical outcomes analysis without the leadership of professional medical and surgical societies. We must help those agencies understand the best data sources and best reporting methodologies. Accurate reporting of CTS outcomes requires reliance on clinical databases rather than administrative databases. A recent study compared data on isolated CABG results from an audited and validated clinical registry with data derived from a contemporaneous state administrative database using the inclusion/exclusion criteria and risk model of the Agency for Healthcare Research and Quality [25]. This study concluded, "Cardiac surgery report cards using administrative data are problematic compared with those derived from audited and validated clinical data, primarily because of case misclassification and nonstandardized end points."
Three recent investigations compared coding of congenital cardiac disease from clinical databases with administrative databases that used the International Classification of Diseases (ICD) coding system. They demonstrated that the validity of coding congenitally malformed hearts using ICD is likely to be poor [26–28]. Several explanations of the poor diagnostic accuracy of administrative databases that use ICD codes are plausible: accidental miscoding, coding by medical records clerks who have not seen the patient, contradictory or inaccurate information in the medical record, lack of diagnostic specificity for cardiothoracic disease in the ICD codes, and inadequately trained coding clerks.
Governmental administrative databases were created to facilitate billing; they do not accurately reflect CTS outcomes with the level of detail and accuracy necessary for useful outcome analysis. Meaningful analysis of CTS outcomes requires the use of clinical databases rather than administrative databases.
The Role of the Lay Press
Lay press reports of CTS outcomes are problematic because the average health care reporter does not have the training and background to appreciate the science behind the analysis of medical and surgical outcomes. For example, on March 1, 2001, the Denver Post published a front-page article, "Children's Hospital Cardiology Chief Told to Resign." The reporter wrote, "The move follows a report that shows the average mortality rate for open-heart surgery at Children's was 4.2%, or 30 children out of 710 surgeries, between January 1997 and July 2000, while the national average was 2.7%" [29]. A follow-up Denver Post editorial the next day stated, "The hospital also found that, through last June, 4.2 percent of the children who had the procedure at Children's Hospital died. Nationally, the average mortality rate of such surgeries is 2.7 percent, according to the Society for Thoracic Surgeons [sic]" [30]. The author of the front-page article, a recent graduate of Colorado State University, failed to consider four crucial points:
The chair of the STS Congenital Heart Surgery Database at that time, Constantine Mavroudis, MD, wrote a rebuttal to the Denver Post addressing the shortcomings in their presentation of the data. The newspaper did not publish his critique. This example raises the question: Who is better prepared to present CTS outcomes, an experienced cardiothoracic surgeon or a freshman reporter who recently finished college with an undergraduate degree in journalism? Publication of CTS outcomes in the lay press requires substantial input from cardiothoracic surgeons who understand surgical complexities.
|
Positive Aspects of Public Reporting |
|---|
|
TopIntroductionThe History of Public...Problems with Public Reporting...Positive Aspects of Public...Implications, Inferences, and...References |
|---|
Stakeholders Desire Transparency
As a profession, we must recognize that, like it or not, our patients, insurance companies, news media, and several levels of government want transparency. This desire increased after recent high profile reports of medical errors and related publications from the Institute of Medicine. In 2000, it estimated that 3% to 4% of hospitalized patients experienced an adverse event, defined as an injury caused by medical management. Slightly more than half of these adverse events were attributed to errors and were classified as preventable adverse events. As many as 98,000 Americans die each year of medical errors in hospitals [31]. This report generated substantial interest in the evaluation of the quality of medical care.
In the United States, the idea of transparent disclosure of the outcomes of medical treatment is exemplified by the National Quality Forum [32]:
A not-for-profit membership organization created to develop and implement a national strategy for health care quality measurement and reporting... . The mission of the National Quality Forum is to improve the quality of American healthcare by setting national priorities and goals for performance improvement, endorsing national consensus standards for measuring and publicly reporting on performance, and promoting the attainment of national goals through education and outreach programs.
Clearly, transparency and public reporting is high priority to the National Quality Forum.
Several insurance companies, including United Health Care and Aetna, have created "data entry forms" for CTS programs to document outcomes and submit the data to the insurance companies, which use the outcomes reports to create "Centers of Excellence." Similarly, several state governments now track CTS outcomes.
The efforts of Pennsylvania and New York are well known to the CTS community, as has been discussed. Florida tracks pediatric cardiac surgical outcomes through Florida Children's Medical Services and its Children's Cardiac Program. Similar efforts by the federal government were foreshadowed in 2006 when the Centers for Medicare and Medicaid Services launched the Physician Voluntary Reporting Program, which eventually became the Physician Quality Reporting Initiative and may eventually evolve into a "pay for performance" program [33]. The desire of the federal government for public reporting is further demonstrated by federal legislation proposed in 2007: S. 1693, the "Wired for Health Care Quality Act" [34]. The bill provides "for the development of reports based on Federal health care data and private data that is publicly available or is provided by the entity making the request for the report in order to ... provide the public with reports on national, regional, and provider- and supplier-identifiable format."
The STS has expressed this opinion regarding claims data used to create outcomes reports (personal communication) [35]:
In our two decades of experience, it has become abundantly clear that claims (or billing) data are not sufficient to measure outcomes and are incapable of allowing the significant risk adjustment that can prevent patient access disparities—usually for the sickest patients. Administrative data is rife with misclassification errors, which are incredibly difficult to correct or account for in a reporting system. Only a clinical database with sufficient clinical variables can be risk adjusted enough to yield accurate findings.
The widespread desire for transparent reporting of CTS outcomes charges our profession with the responsibility to ensure that relevant data are provided in an accurate and comprehensible format.
Public Reporting Can Lead to Quality Improvement and Cost Reduction
The importance of public reporting goes far beyond the desires of stakeholders; it can improve the quality of care in cardiothoracic operations and reduce the cost of delivering care. Porter and Teisberg have argued that transparency is essential to the redefinition of health care based on value, which they define as health outcome divided by cost [36, 37]. Public reporting is likely to change referral patterns and individual surgeons' numbers of procedures. More importantly, however, it may lead to improvements in the overall quality of health care by identifying the best providers so that we can learn from them, and by identifying low performing centers that can benefit from quality improvement initiatives.
Porter and Teisberg [36, 37] state, "Value in health care is the health outcome per dollar of cost expended. If all system participants have to compete on value, value will improve dramatically." They advocate measuring risk-adjusted outcomes and costs. They also state, "If physicians fail to lead these changes, they will inevitably face ever-increasing administrative control of medicine" [36, 37].
Professional Responsibility
Public reporting promotes surgical professionalism by facilitating self-regulation and improving the state of our surgical art [38–44]. The STS Database was created for the purpose of improving the quality of care for CTS patients. As a matter of professional obligation, the STS created an excellent tool for reporting CTS outcomes. The STS Database allows for the reporting of CTS outcomes based on verified data that is adjusted and stratified for operative complexity. It also allows for the use of composite measures to assess quality based on the combination of performance and compliance with structure, process, and outcomes indicators [45, 46]. The methodologies used by the STS Database can be used to report CTS outcomes accurately and in language that patients and other laypersons can understand.
|
Implications, Inferences, and Conclusions |
|---|
|
TopIntroductionThe History of Public...Problems with Public Reporting...Positive Aspects of Public...Implications, Inferences, and...References |
|---|
Serving the medical interests of our patients entails providing or otherwise making available information that would be useful to them in deciding to whom they wish to entrust their surgical care. This idea is a corollary of informed consent for specific procedures: patients must be provided sufficient information to accept or reject a proposed treatment and, by extension, should be provided sufficient information to make decisions about their care, including relevant surgical outcome data, by surgeon and by institution. It follows, therefore, that surgeons should support public reporting of outcome data.
Mere reporting is not sufficient, however. Our obligation to serve the medical interests of our patients includes preventing avoidable harm. Current publicly available surgical outcome information, mainly from administrative databases, is misleading at best. Deceptive information of this kind threatens our patients by leading them to make misinformed choices. Publication of inaccurate negative outcome data also threatens unjustifiably to stigmatize individual surgeons and their institutions.
Another potential harm to patients is the possibility that some physicians may feel a need to protect themselves from adverse public reporting of their own results by denying interventions to patients with higher-than-average risks. Such a tactic would diminish some patients' access to the appropriate cardiothoracic operation, so is ethically unacceptable. Public reporting based on accurate and reliable risk stratification would soften or eliminate this disincentive to provide appropriate care.
A potentially important barrier to the use of the STS Database for public reporting is that it was not created with public reporting in mind. The current widespread (although not universal) participation by cardiothoracic surgeons nationally may be largely due to the promise to individual surgeons and surgical programs that their data are confidential. The only publicly available reports are combined national data; surgeon-specific and program-specific information can be reported only by or with the approval of the surgeon or program in question.
Dependence on voluntary data submission by surgeons brings with it many difficulties, the greatest of which is the likelihood that many might drop out of the STS Database if surgeon-specific or program-specific analyses based on its contents were to be publicly reported. It may be that the best way STS methodology could be used for public reporting is if reporting of CTS results were required by law and reporting agencies used STS methodology. This strategy would maintain the voluntary and confidential nature of the STS Database, but would mandate that the data submitted for public reporting be based on the same methodologies as those used in the STS Database.
The clinical detail required for the STS Database is not necessary for reimbursement purposes, but if such detail were required for reimbursement, errors and mishandling of data would almost certainly burgeon. The completeness and accuracy of information currently reported to the STS Database requires constant attention by those who submit the information and those who manage the database, including routine audits of submitted data by an independent third party. Clinical data reported for reimbursement purposes can be flawed because of incentives to maximize income by reporting erroneous or misleading information [47]. A potential consequence of mandatory reporting, therefore, is an unintended decrease in accuracy of the reported data. Data verification efforts will need to be increased and adequately funded.
In the final analysis, we as a profession should participate in solving the daunting problems associated with public reporting of CTS outcomes data, but not primarily because of its inevitability, or the threat that someone else will do it if we do not, or to protect ourselves from unjustified stigmatization. We should participate, as individuals and as professional societies, because doing so is required by our most fundamental professional commitment: to do what is best for our patients.
|
References |
|---|
|
TopIntroductionThe History of Public...Problems with Public Reporting...Positive Aspects of Public...Implications, Inferences, and...References |
|---|
This article has been cited by other articles:
|
|
 |
|
  T. L. Spray The quality conundrum. J. Thorac. Cardiovasc. Surg., November 1, 2009; 138(5): 1053 - 1059. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
