| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Ann Thorac Surg 2007;83:1237-1239
© 2007 The Society of Thoracic Surgeons
President (2006-2007), The Society of Thoracic Surgeons, Chicago, Illinois; Professor and Chair, Department of Surgery, University of Colorado Health Sciences Center, Denver, Colorado
* Address correspondence to Dr Grover, Department of Surgery, University of Colorado Health Sciences Center, 4200 East Ninth Ave, C-305, Denver, CO 80262 (Email: frederick.grover{at}uchsc.edu).
In a Supplement to this month’s The Annals of Thoracic Surgery, the Quality Measurement Task Force (QMTF) under The Society of Thoracic Surgeon’s (STS) Workforce on National Databases, chaired by Dr David M. Shahian, presents a landmark report on a very innovative and statistically rigorous new methodology for evaluating the quality of coronary artery bypass grafting procedures [1, 2]. This is "must" reading for all cardiothoracic surgeons and other practitioners because of the absolute importance of solid methodology applied to performance measurement. Your future and your program’s future depend on this. This Task Force is composed of STS members Drs Fred H. Edwards, Victor A. Ferraris, Constance K. Haan, and Jeffrey B. Rich, and STS staff Cynthia M. Shewan, PhD, along with very sophisticated statisticians, Sean M. O’Brien, PhD, and Elizabeth R. DeLong, PhD, from the Duke Clinical Research Institute (DCRI), and Sharon-Lise T. Normand, PhD, from the Department of Health Care Policy at Harvard Medical School and the Department of Biostatistics at the Harvard School of Public Health, and with DCRI representatives, Rachel S. Dokholyan, MPH, and Dr Eric D. Peterson. Finally, this Task Force is under the very able guidance of Dr Richard P. Anderson, Chair of the STS Council on Quality, Research, and Patient Safety.
In the late 1980s, the STS initiated and developed the STS Adult Cardiac Surgery Database and in subsequent years the General Thoracic Surgery and Congenital Heart Surgery Databases [3, 4]. The Adult Cardiac Database has matured and gained considerable respect over the past 18 years, now being the largest and most comprehensive single specialty clinical database in healthcare in the world. Critical in the development of this database was the concept of risk-adjusted operative mortality and morbidity. Over 780 groups and hospitals now participate in this database nationally. The initial reason for starting the database was and still is to improve the quality of patient care by allowing groups to compare their results to national and regional averages and identify whether or not problems in care exist. In the mid-1990s, an additional emphasis was placed on utilizing the databases for research and healthcare policy. Great emphasis has been placed on performance measurements beginning with the Institute of Medicine report in 1999, To Err is Human: Building a Safer Health System [5] and the Institute of Medicine’s most recent quality document titled Performance Measurement: Accelerating Improvement [6]. In addition, with the formation of the National Quality Forum (NQF) and consumer, payor and purchaser interest, there has been great interest in accountability and transparency. The STS has been integrally involved with many of these groups and is represented on the Board of the NQF and plays an active role in the AQA, the Hospital Quality Alliance and many other quality improvement groups. The STS also has testified before Congress and the Institute of Medicine, as noted by Porter and Teisberg in their book Redefining Health Care: Creating Value-Based Competition on Results [7].
Because of this interest, the STS Workforce on National Databases established the QMTF to develop innovative, objective and valid new methods for measuring performance. This is particularly important since Congress, the Centers for Medicare & Medicaid Services, and private payors have been moving in a direction for pay-for-performance methodology. This month’s Annals Supplement is the product of many months of work by this talented Task Force made up of some of the country’s leading biostatisticians, STS members, staff, and colleagues at the database warehouse, the Duke Clinical Research Institute.
The QMTF was specifically created to: (1) select a set of individual quality indicators within multiple domains of care; (2) develop methodologies for combining such measures within and across domains into composite scores; and (3) develop various methods for evaluating surgeons’ performance based on their overall composite quality scores. The Supplement, titled Quality Measurement in Adult Cardiac Surgery, is composed of two parts, the first on "Conceptual Framework and Measure Selection," and the second, "Statistical Considerations in Composite Measure Scoring and Provider Rating." In addition, a technical appendix of the statistical methodology is included.
Part 1 details how quality indicators were selected using the following principles: (1) Quality measurement will be at the level of the group or the hospital rather than the individual surgeon. (2) Initially the reports will focus on isolated coronary artery bypass grafting. (3) The quality measures will be selected from among those already endorsed by the NQF, 18/21 of which are from the STS National Database. (4) The measure selection will be consistent with the principles and criteria recommended by the 2006 Institute of Medicine’s Performance Measurement: Accelerating Improvement. (5) Data elements will be part of the STS National Adult Cardiac Database. (6) The scores should include process, structure, and outcome variables. (7) The scores should assess three temporal domains of care—perioperative, operative, and postoperative. (8) The quality scores should satisfy multiple criteria and pass validity testing. (9) These scores need to be interpretable and actionable by providers.
The Task Force selected eleven quality measures within the four domains of care. The domain of perioperative medical care includes the use of four medications—preoperative ß-blockade and discharge aspirin, ß-blockade, and lipid-lowering agents. The use of the internal mammary artery is the single process measure for the operative care domain. Finally, the two outcomes domains consist of risk-adjusted operative mortality and postoperative risk-adjusted morbidity. The complications evaluated for the morbidity domain are renal insufficiency, deep sternal wound infection, reexploration, stroke, and prolonged ventilation/intubation.
In two of these domains, individual variables were bundled. The four perioperative medical care process measures were bundled into an "all or none" compliance bundle and the five major postoperative complications were bundled into an "any or none" morbidity bundle. It was elected not to specifically include the two NQF structural measures of database participation and procedural volume. By definition, only participation in the STS Database will be eligible for this quality scoring methodology. Furthermore, procedural volume has been shown to have a very weak relationship to outcomes in coronary bypass surgery.
Part 2, "Statistical Considerations in Composite Measure Scoring and Provider Rating," is a beautifully written scientific document that physicians as well as statisticians can understand and clearly explains the statistical methodology employed. What is unique and innovative about this particular manuscript is that it addresses the challenges of weighting, standardizing, and combining various process and outcome measures when developing a composite score. The result of the QMTF statistical methodology as applied to actual STS data very much parallels clinical opinion.
The manuscript carefully takes the readers through the statistical processes of trial and error to arrive at the best and most reliable methods. The group first considered the NQF-endorsed process and outcomes measures within and across the four domains of care that the Task Force chose, ie, perioperative medical care, operative care, risk-adjusted operative mortality, and postoperative risk-adjusted morbidity. They evaluated simple or weighted averaging, a composite opportunity model, all or none scoring, scale combinations, and latent variable models. These methods were evaluated using 2004 STS data that included 133,149 coronary bypass procedures. Provider performance was estimated using Bayesian random effects analysis. Combining scores across domains was accomplished by rescaling, then adding the domain scores into an overall composite score. The QMTF then applied this methodology to the 2004 STS data and very importantly noted that a 1% improvement in mortality was equivalent in its impact on the overall composite score to an 8% improvement in morbidity, an 11% improvement in the frequency of IMA usage and a 28% change in the use of all four NQF-endorsed medications.
The QMTF then evaluated the best methods for reporting performance tiers using these composite scores. A pilot study was performed using a 99% Bayesian certainty criterion to assign the tiers and maximize accuracy. When utilizing this method, 77% of programs fell into the average performance tier, 10% were found to be higher than average performers, and 13% were lower than average performers. Most importantly, when using this scoring methodology, classification as above average or below average in performance was made with 99% certainty.
In summary, the STS Quality Measurement Task Force has made a landmark contribution to the scientific objective measurement of quality and performance. For physicians, it is extraordinarily important to read this material carefully and be well educated on this topic. There will be many performance measure methodologies developed, many of which will be flawed or at least not as rigorous or objective as this one. This statistical methodology is based on clinical data points, not inaccurate administrative databases and it avoids the subjective consensus weighting of outcomes and process variables. Furthermore, the assignment of performance tier is accomplished with 99% Bayesian certainty. It is extremely important for all of us in this era of performance ratings, which are used not only for quality improvement, but also for public reporting and for pay-for-performance, to be evaluated in the most objective and accurate way as possible. Each of us must be involved at the local, state, and national level to influence health policy so that the most rigorous and accurate performance methodologies are utilized. In order to do this, we must read and become very familiar with articles such as these so that when these issues are discussed with policy makers, in the private and public sector, and with patients and hospital administrators, we are well informed in order to successfully debate and positively influence healthcare policy.
| For related articles, see page 1240 and April 2007 Supplement (Ann Thorac Surg 2007;83:S1–26)
|
 |
References |
|---|
 Top References |
|---|
Related Article
This article has been cited by other articles:
|
|
 |
|
  A. Brunelli, G. Varela, P. Van Schil, M. Salati, N. Novoa, J. M. Hendriks, M. F. Jimenez, P. Lauwers, and on behalf of the ESTS Audit and Clinical Excellenc Multicentric analysis of performance after major lung resections by using the European Society Objective Score (ESOS) Eur. J. Cardiothorac. Surg., February 1, 2008; 33(2): 284 - 288. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
