HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): David M. Shahian David F. Torchiana Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Shahian, D. M. Articles by Normand, S.-L. T. Search for Related Content PubMed PubMed Citation Articles by Shahian, D. M. Articles by Normand, S.-L. T. Related Collections Professional affairs Related Article

Ann Thorac Surg 2005;80:1146-1150
© 2005 The Society of Thoracic Surgeons

---

Review

Implementation of a Cardiac Surgery Report Card: Lessons From the Massachusetts Experience

^a Department of Surgery, Caritas St. Elizabeth’s Medical Center Boston, Massachusetts USA
^b Department of Cardiac Surgery, Massachusetts General Hospital, Boston, Massachusetts USA
^c Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts USA
^d Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts

^_* Address reprint requests to Dr Shahian, Department of Surgery, Caritas St. Elizabeth’s Medical Center, 736 Cambridge St, Boston, MA02135. (Email: david_shahian{at}cchcs.org).

	Abstract

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 
Demand is increasing for public accountability in health care. In 2000, the Massachusetts legislature mandated a state report card for cardiac surgery and percutaneous coronary interventions. During the planning and implementation of this report card, a number of observations were made that may prove useful to other states faced with similar mandates. These include the necessity for constructive, nonadversarial collaboration between regulators, clinicians, and statisticians; the advantages of preemptive adoption of The Society of Thoracic Surgeons [STS] National Cardiac Database, preferably before a report card is mandated; the support and resources available to cardiac surgeons through the STS, the National Cardiac Database Committee, and the Duke Clinical Research Institute; the value of a state STS organization; and the importance of media education to facilitate fair and dispassionate press coverage. Some important features of report cards may vary from state to state depending on the legislative mandate, local preferences, and statistical expertise. These include the choice of a statistical model and analytical technique, national versus regional reference population, and whether individual surgeon profiling is required.

	Introduction

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 
Despite the availability of other effective and less controversial approaches to improving health care outcomes, including continuous quality improvement (CQI) programs, government regulators and payers increasingly demand provider profiling ("report cards") to insure public accountability. By publishing outcomes, it is hypothesized that market forces will redirect patients to higher performing providers and compel poor performers to improve or terminate services. Such reports may also serve as the objective basis for various "pay for performance" initiatives. In some instances, physicians may themselves initiate public reporting of risk-adjusted outcomes in order to circumvent volume-based referral criteria (eg, Leapfrog). Regardless of their motivation, the processes by which report cards are implemented and the methodologies used are of critical importance. Many concerns exist regarding the conceptual, statistical, and practical aspects of report cards, including the degree to which they actually impact quality and their potential for unintended negative consequences [1].

Only a few states have actually implemented some form of public report card for cardiac surgery, including New York, New Jersey, Pennsylvania, California, Virginia, and recently Massachusetts. The process by which report cards were introduced into Massachusetts, which was facilitated by a detailed examination of prior report card initiatives, has revealed a number of lessons that may be of value to other states that face such a requirement in the future. The Massachusetts experience is also relevant to the goal of achieving more widespread utilization of The Society of Thoracic Surgeons (STS) National Cardiac Database (NCD). Report card implementation in Massachusetts led to the mandatory adoption of this database by all 14 cardiac surgery programs, most of which had not previously participated in the STS NCD.

	Evolution of the Massachusetts Report Card

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 
For several decades, a strict determination of need process limited cardiac surgery in Massachusetts to 12 programs (subsequently 11 through hospital consolidation) with approximately 50 surgeons, most of which were based in higher-volume tertiary centers with strong academic traditions. In 1997, a number of Massachusetts community hospitals petitioned the Department of Public Health to expand cardiac surgery privileges beyond tertiary centers. Beginning in August 1997 and continuing over the next 3 years, several state task forces and commissions, all of which had strong cardiac surgery representation, considered this request. Debate focused on issues such as volume and infrastructure requirements for new programs, sponsorship by academic centers, and their impact on existing programs.

At a June 1998 meeting of the Technical Advisory Committee, the commissioner of the Department of Public Health noted that rational health policy debate on this issue was hindered by the lack of a credible Massachusetts cardiac surgery database. The only available state database was administrative, and only two heart surgery programs submitted clinical data to the STS NCD. The commissioner instructed the director of the Division of Health Care Quality to engage the services of an outside consulting group to explore various database options and analytic methodologies, including those used in other states and regions.

Lesson: The credibility of cardiac surgeons is enhanced, and their commitment to quality is demonstrated, when they proactively support data collection and outcome analysis. Surgeons should foster constructive, nonadversarial, working relationships with state regulators. Surgeons and regulators share a common goal of assuring the highest-quality care for patients, although their preferred approaches to achieving this may sometimes differ.

A Massachusetts cardiac surgeon (DMS) who had recently become a member of the STS NCD Committee suggested in a subsequent letter to the commissioner that participation in the STS NCD be made mandatory for Massachusetts cardiac surgery programs. This would assist in the rational development of public health policy and would also provide credible data for academic research and continuous quality improvement activities. A separate Department of Public Health committee with cardiac surgical representatives was established to investigate this possibility, and meetings continued throughout 1999. Discussion focused on the legal ramifications of such a requirement, what types of data to collect, who would hold and have access to the data, and what would be done with the data. At one meeting, representatives from the STS NCD (Dr Ferguson) and Duke Clinical Research Institute (Dr Peterson) presented the arguments for use of the STS NCD in Massachusetts.

Lesson: Voluntary statewide adoption of the STS NCD preempts other less satisfactory database options.

While these deliberations were ongoing, Tufts Health Plan, a large Massachusetts health maintenance organization, notified cardiac surgery providers on February 18, 2000 that it had engaged an outside consultant to collect data on cardiac procedures in the state. This initiative would utilize the consultant’s own proprietary database and analytical method and would require funding by the hospitals. Because of the extensive discussions that were already taking place regarding adoption of the STS NCD in Massachusetts, the hospitals and their cardiac surgery directors indicated their opposition to this Tufts Health Plan initiative at a March 16, 2000 meeting with the payer, and the plan was abandoned.

Lesson: Statewide data collection and monitoring initiatives require the collaboration and trust of the entire cardiac surgical community. This may be facilitated by the involvement of the national STS and through formation of state STS organizations. The latter can be implemented at a relatively modest cost, even in small states.

To facilitate the introduction of the STS NCD into Massachusetts, all cardiac surgery program directors were invited to a meeting in April 2000 to discuss the state database initiative and the possible formation of a state organization. Drs Matloff and Arom, as well as database coordinator, Mary Eiken, RN, represented the national STS and the STS NCD. Surgical attendees supported the use of the STS NCD and authorized their representatives to explore formation of a Massachusetts STS modeled after the Minnesota STS. In November 2000, a second meeting of Massachusetts cardiac surgeons voted to form a state STS organization. This would establish a defined entity with which regulators, payers, and the media could interact, and which would be regarded as the expert resource on cardiac surgery. The benefits of a state database were emphasized including its use for academic research and for CQI activities. Surgeons were also reassured that data collection, analysis, and reporting would be done in an unbiased and statistically rigorous fashion, that their representatives would oversee the process to ensure fairness and accuracy, and that cardiac surgeons would be involved in the review and remediation of suspected outliers. Finally, this organization would also provide peer pressure to discourage providers from using favorable report card results for marketing. The Massachusetts Society of Thoracic Surgeons was subsequently incorporated and granted 501(c)(3) federal tax-exempt status. This was achieved with the collection of modest dues from 50 surgeons.

Lesson: Despite the well-founded preference of cardiac surgeons for confidential CQI initiatives, the demand for public accountability is compelling, increasing, and inescapable. Continuous quality improvements should be conducted in a parallel fashion with public reporting to insure real, sustainable quality improvement.

In July 2000, before a decision was reached regarding either the database or community cardiac surgery issues, and while the formation of a Massachusetts STS was in progress, the state legislature preemptively established a pilot program for community cardiac surgery as well as a new entity designated as the Massachusetts Cardiac Care Quality Advisory Commission. This commission included two surgeons who would subsequently become president (DFT) and vice-president (DMS) of the Massachusetts Society of Thoracic Surgeons, three cardiologists (one of whom had extensive experience in managing large databases for clinical trials), the director and several staff members of the Massachusetts Division of Health Care Quality, and an academic statistician (S-LTN) with special expertise in provider profiling. This commission was charged by the legislature with developing a public report card to assess and improve the quality of cardiac care in the commonwealth. Although cardiac surgery members of the commission expressed a preference for a confidential CQI approach similar to that used in northern New England [2], the chair of the Massachusetts Senate Ways and Means Committee, who had authored the original legislation, indicated that only a transparent public report card would be acceptable.

	Implementation

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 
Lesson: Utilize the resources and credibility of the STS and Duke Clinical Research Institute (DCRI) to promote use of the STS NCD as the data collection instrument.

Database Instrument
In the early deliberations of the commission, various data collection instruments were considered including the development of a "home grown" state database, use of an existing database from another state (eg, New York), the Northern New England Cardiovascular Disease Study Group Database, or the STS NCD. Although other excellent databases existed, cardiac surgical members of the commission strongly supported the STS NCD and solicited the assistance of the STS and DCRI leadership, including Drs Grover, Peterson, Ferguson, and Matloff. They emphasized the many advantages of the STS NCD including its national and international stature within the cardiac surgery community; large and growing enrollment; enhanced credibility resulting from its administration by DCRI; and its relationship with the American College of Cardiology National Cardiovascular Data Registry. The latter was relevant because the commission was also charged with developing a report card for percutaneous coronary intervention. Ultimately, at the December 12, 2000 meeting of the commission, The STS NCD was selected for use in Massachusetts.

Lesson: Choice of risk model, analytical technique, and reference population may vary from state to state depending on the expertise of statistical consultants and the preferences of local regulators and advisory boards.

Analytical Technique
Previous report card methodologies have been reviewed by Shahian and colleagues [1]. Typically, the probability of death for each patient is estimated from a logistic regression model, the expected probabilities of death for each patient cared for by a given provider are aggregated, and the resulting number of expected (E) mortalities for each provider are compared with their observed (O) mortalities (O/E ratio). Often this O/E ratio has been multiplied by the state average mortality to generate a "risk adjusted mortality rate."

Although appealingly simple and intuitive, this approach is considered inappropriate by many statisticians. It does not account for some problematic characteristics of the study populations such as clustering of observations within providers (eg, the prevalence of heart failure patients at transplant centers), which effectively reduces the number of independent observations and increases their standard errors. Furthermore, standard methods do not adequately account for disparate and often small sample sizes, which impacts how accurately the observed mortality of a provider reflects their true underlying mortality rate. The net effect may be an underestimation of random interprovider variability, overestimation of systematic interprovider variability, and an increased potential for false outlier designation. Hierarchical generalized linear models [1, 3–5] are ideally suited to address such concerns and were chosen as the analytical methodology for report cards in Massachusetts.

Reference Population
The relative merits of state versus national reference population were vigorously debated. One possibility was to have all analyses and provider comparisons performed by DCRI, using a model developed from the STS national data, comparing Massachusetts providers with both the STS national sample and the Massachusetts region. This approach would acknowledge the fact that quality standards in cardiovascular care should not vary substantially among states and regions. It had the advantage of utilizing existing data collection and analysis pathways, and it would also permit comparisons with the largest, albeit voluntary and unaudited, cardiac database in the world. However, state regulators were concerned about the release of confidential patient information to an outside, nongovernmental organization. Furthermore, there was no precedent for the STS or DCRI to provide outcome analyses directly to an agency of state government rather than to cardiac surgeons or programs. Weighing all these considerations, the commission decided to require dual submission of data to both the STS and DCRI and to the Massachusetts Data Analysis Center (see as follows). This would assure that each program would have confidential access to information regarding its performance compared with national STS results, with analyses performed by DCRI using both traditional and hierarchical statistical techniques. Customized regional reports from the STS NCD would also be available to MA hospitals. However, the public state report card would be based upon a hierarchical model developed solely from Massachusetts data that was complete, audited, and validated.

Lesson: There are persuasive arguments supporting public reporting of hospital-level outcome data, but confidential review of individual surgeon results, provided that a vigorous process is established to assure patient safety. Furthermore, clinical peer review must be performed with the involvement of cardiac surgical experts to insure that statistical outliers are not inappropriately labeled as substandard.

Individual Versus Hospital Report Cards
In New York, publication of report cards for individual surgeons was not initially planned but was forced by a lawsuit by Newsday Magazine. In general, physician reporting has a greater potential for promoting unintended negative behaviors such as high-risk case avoidance, up-coding of comorbidities, inappropriate change of operative class, and other methods for avoiding or hiding mortalities [1, 6]. Because the individual surgeon controls most of these actions, hospital-level reporting would be less likely to stimulate the anxiety that fosters such behaviors. Furthermore, because 2 to 3 years of individual surgeon data must be collected to have a valid statistical sample, followed by another year or more to clean, audit, and analyze the data, surgeon reports may not reflect current performance. Finally, it may also be argued that surgeon-specific reports place too much emphasis on one aspect of what is unquestionably a complex team endeavor. Weighing the benefits of public accountability at the individual practitioner level versus the potential negative consequences, the commission elected to publish a public report card for hospitals and conduct a confidential, legislatively-protected review process for individual surgeons.

Outliers
In conjunction with the Department of Public Health, the Massachusetts Cardiac Care Quality Commission developed mechanisms to further investigate statistical outliers. A Review Committee of the Division of Health Care Quality, with representatives from the Massachusetts STS, will investigate hospital outliers, including case-by-case analysis of their mortalities. Surgeon outliers will be investigated confidentially (Chapter 24A protection) by the Department of Public Health Cardiac Services Oversight Committee, also with cardiac surgery representation, and if necessary will be referred to the Board of Registration in Medicine. The desired outcome of this procedure is remediation, particularly if there are process or structural problems that led to low performance [7]. Suspension of privileges is an option if deemed necessary to protect the public welfare.

Lesson: Provider profiling is a complex task requiring considerable statistical expertise and judgment. When available, local health policy experts and statisticians should be involved, as well as an external advisory board to provide additional insights and national credibility.

Massachusetts Data Analysis Center
Once the commission had agreed upon the procedures and endpoints to monitor, the reference population, and the analytical technique, a request for proposal was issued by the Department of Public Health. Ultimately a contract was awarded to the Department of Health Care Policy at Harvard Medical School. Shortly thereafter the Massachusetts Data Analysis Center was established with offices at Harvard Medical School (www.massdac.org). Staff consisted of the statistical director, a nurse program manager, two statistical programmers, two clinicians (an interventional cardiologist and a radiologist), and an administrative assistant.

In order to provide expert advice and oversight to the Massachusetts Data Analysis Center, a Massachusetts Cardiac Advisory Board was formed that included representatives from other major regional and national database initiatives. These experts provided national credibility as well as valuable insights regarding data collection, coding issues, audit, analytical methods, and presentation to the public.

Lesson: Confidence in the accuracy of the data is essential to maintaining provider support and public acceptance. Regular meetings with providers enhance cooperation with the data collection effort, and involvement of cardiac surgeons in the audit and data adjudication processes provide reassurance of data integrity.

Data collection began in January 2002 with quarterly (later semiannual) data submissions. Multiple resubmissions (average, 9 per hospital) were requested because of missing, inconsistent, or out-of-range data. Data submissions were cross-checked against the Massachusetts Inpatient Acute Hospital Case Mix and Charge Database and the Massachusetts Mortality Index Database to be certain that all appropriate procedures and all 30-day mortalities had been reported. In 6 of 4,604 isolated coronary artery bypass grafting (CABG) cases submitted for 2002, patients were reported as alive at 30-days by hospitals reports and deceased according to government databases, respectively. Five of these six were confirmed dead (0.1% discrepancy), 1 patient recorded by a hospital as a 30-day mortality was found to be alive, and 1 patient from Europe could not be verified and was removed from the database.

Between August 15, 2002 and October 13, 2004, the Massachusetts Data Analysis Center conducted 14 meetings with chairs of cardiac surgery programs, members of the Cardiac Advisory Board, and data managers. Questions were addressed regarding data element definitions, and when necessary the national STS was contacted for clarification. Data managers as well as the Massachusetts Data Analysis Center program manager met on an ongoing basis in an effort to ensure consistent implementation of definitions and data collection techniques. Members of the Massachusetts Cardiac Care Quality Advisory Commission were provided with early feedback regarding data quality, interprovider variability in coding of comorbidities and complications, and frequency of missing data.

In February 2003, the Massachusetts Peer Review Organization performed an audit of 500 cases, including all deaths as well as other randomly selected cases from each program that had been reported to the Massachusetts Data Analysis Center. Cases were validated against hospital operating room logs and a state administrative database to be certain that all procedures and deaths had been reported.

After initial review of 2002 data by both the Massachusetts Cardiac Surgery Program Directors and the Cardiac Advisory Board, additional documentation was requested from many institutions regarding problematic cases. These included all cases coded as "other" or CABG plus "other", as well as records relevant to coding disparities between an individual hospital and the rest of the state. A Data Adjudication Committee consisting of 3 senior cardiac surgeons from three different programs personally reviewed 724 such records and ultimately achieved unanimous consensus in each case.

Lesson: Efforts should be made to engage and educate the media regarding the complexities, limitations, and unintended negative consequences of provider profiling.

Media Relations
In most states, report cards have generated considerable media fanfare that was not conducive to either the public’s understanding or to the acceptance of report cards by cardiac surgeons. In an attempt to avoid sensationalism and properly inform the public, discussions were held with media representatives beginning more than 1 year in advance of the first report card publication. Reporters were given extensive materials regarding the methodology of constructing report cards, their proper interpretation and uses, as well as their limitations and unintended consequences. Early media coverage has been generally accurate and unemotional [8, 9].

Public release of the first Massachusetts cardiac surgery report card occurred October 19, 2004, which corresponded to cardiac surgeries conducted during the 2002 calendar year. The data revealed no statistically significant differences among Massachusetts programs and remarkably consistent risk-adjusted and standardized mortality rates. The time interval from end of data collection to publication reflected some problems that were unique to the first data harvest. However, even for subsequent reports, a 1-year lag is anticipated due to the necessity for careful data cleansing, revision and resubmission, audit, adjudication of problematic cases, and validation against state administrative databases.

	Comment

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 
The Massachusetts experience has revealed a number of principles that we believe facilitated smooth report card implementation, as well as some issues whose resolution may vary from state to state. Massachusetts cardiac surgeons actively supported statewide adoption of the STS NCD well before this was eventually mandated by the legislature. This strengthened their credibility on the issues of quality and accountability and preempted consideration of other less desirable databases. Cardiac surgeons also had a longstanding, strong, and collegial working relationship with the Department of Public Health. This fostered mutual trust and credibility and facilitated the leadership role of cardiac surgeons on the Massachusetts Cardiac Care Quality Advisory Commission. Rather than attempting to obstruct the report card initiative, cardiac surgeons accepted the mandate to provide public accountability and worked with regulators to insure that the process was optimally implemented using the best available database and analytical techniques. If cardiac surgeons truly believe in the value of data collection and embrace the concept of public accountability, then they should actively lobby for use of the STS NCD in every state.

The leadership of the STS, the STS NCD, and DCRI provided invaluable support and national credibility to the efforts of Massachusetts cardiac surgeons. Although the development of state chapters has not been officially sanctioned or promoted by the STS in the past, this position should be reevaluated in light of the current political and economic environment in health care. In Massachusetts we believe that the incorporation of a state organization facilitated acceptance of the report card process, served as an information conduit, assured the involvement of cardiac surgeons in the audit and adjudication of data and in the identification and remediation of outliers, and helped to coordinate a parallel CQI effort that was regarded as essential to broad and sustained quality improvement.

Issues pertaining to data flow, analytical technique, and the choice of a reference population will remain largely a matter of individual state choice. Wherever possible, local health policy experts and statisticians with profiling expertise, as well as external advisory boards, should be involved in the design and implementation of state report cards. Massachusetts elected to employ a hierarchical model based on state data for the public report card, and the standard STS regional report as the basis for CQI. However, this may not be the case in other states, particularly in the future as the STS NCD becomes more universally adopted and fully audited.

Finally, in order to facilitate accurate and dispassionate public reporting, attempts should be made to educate the media regarding the development, uses, and limitations of provider outcome data.

	References

Top Abstract Introduction Evolution of the Massachusetts... Implementation Comment References

 

1. Shahian DM, Normand S-LT, Torchiana DF, et al. Cardiac surgery report cardscomprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
2. O’Connor GT, Plume SK, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group JAMA 1996;275:841-846.[Abstract]
3. Normand S-LT, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical careissues and applications. J Am Stat Assoc 1997;92:803-814.
4. Goldstein H, Spiegelhalter DJ. League tables and their limitationsstatistical issues in comparisons of institutional performance. J R Stat Soc (Series A) 1996;159:385-443.
5. Thomas N, Longford NT, Rolph JE. Empirical Bayes methods for estimating hospital-specific mortality rates Stat Med 1994;13:889-903.[Medline]
6. Landon BE, Normand S-LT, Blumenthal D, Daley J. Physician clinical performance assessmentprospects and barriers. JAMA 2003;290:1183-1189.[Abstract/Free Full Text]
7. Dziuban Jr SW. Using information from databases to improve clinical practicelessons learned under fire. Ann Thorac Surg 1997;64:S64-S67.
8. Lasalandra M. State’s experts rating hospitals for heart care. Boston Herald, February 13, 2003:35..
9. Smith S. Bypass study boosts small hospitals. The Boston Globe, October 20, 2004:B1–5..

This article has been cited by other articles:

				V. A. Ferraris, F. H. Edwards, D. M. Shahian, and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2008; 3(2008): 199 - 246. [Full Text]

				T. J. Ryan To Understand Cardiac Surgical Report Cards: Look Both Ways Circulation, March 27, 2007; 115(12): 1508 - 1510. [Full Text] [PDF]

				D. M. Shahian, T. Silverstein, A. F. Lovett, R. E. Wolf, and S.-L. T. Normand Comparison of Clinical and Administrative Data Sources for Hospital Coronary Artery Bypass Graft Surgery Report Cards Circulation, March 27, 2007; 115(12): 1518 - 1527. [Abstract] [Full Text] [PDF]

				J. Bergsland, E. Kabil, E. Mujanovic, I. Terzic, J. Roislien, J. L. Svennevig, and E. Fosse Training of Cardiac Surgeons for Bosnia and Herzegovina: Outcomes in Coronary Bypass Grafting Surgery Ann. Thorac. Surg., February 1, 2007; 83(2): 462 - 467. [Abstract] [Full Text] [PDF]

				G. L. Grunkemeier and A. P. Furnary Mandatory Database Participation: Risky Business? Ann. Thorac. Surg., September 1, 2005; 80(3): 799 - 801. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): David M. Shahian David F. Torchiana Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Shahian, D. M. Articles by Normand, S.-L. T. Search for Related Content PubMed PubMed Citation Articles by Shahian, D. M. Articles by Normand, S.-L. T. Related Collections Professional affairs Related Article