| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Ann Thorac Surg 2005;79:1635-1649
© 2005 The Society of Thoracic Surgeons
a The Congenital Heart Institute of Florida (CHIF), University of South Florida, St. Petersburg, Florida
b Denver Children's Hospital, University of Colorado, Denver, Colorado
c St. Christophers Hospital for Children, Drexel University College of Medicine, Philadelphia, Pennsylvania
d Montreal Children's Hospital, McGill University Health Center, Montreal, Quebec, Canada
e Division of Pediatric Cardiothoracic Surgery, The Cardiac Center at The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
f The Children's Memorial Health Institute, Department of Cardiothoracic Surgery, Warsaw, Poland
g Pediatric Cardiac Surgery Unit, University of Padova Medical School, Padova, Italy
h Duke Clinical Research Institute (DCRI), Duke University Medical Center, Durham, North Carolina
i The Great Ormond Street Hospital for Children NHS Trust, London, United Kingdom
j Childrens Memorial Hospital, Northwestern University, Chicago, Illinois
Accepted for publication September 2, 2004.
* Address reprint requests to Dr Jacobs, The Congenital Heart Institute of Florida (CHIF), Pediatric Cardiac Surgery, All Children's Hospital, Children's Hospital of Tampa, University of South Florida School of Medicine, Cardiac Surgical Associates, 603 Seventh Street South, Suite 450, St. Petersburg, FL33701; (E-mail: jeffjacobs{at}msn.com).
Presented at the Fortieth Annual Meeting of The Society of Thoracic Surgeons, San Antonio, TX, Jan 26–28, 2004.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
METHODS: This data harvest represents the first STS multiinstitutional experience with software utilizing the international nomenclature and database specifications adopted by the STS and the European Association for Cardio-Thoracic Surgery (April 2000 Annals of Thoracic Surgery) and the first STS Congenital Database Report incorporating a methodology facilitating complexity adjustment. This methodology, allowing for complexity adjustment, gives each operation a basic complexity score (1.5 to 15) and level (1 to 4) based upon the work of the EACTS-STS Aristotle Committee, a panel of 50 expert surgeons. The complexity scoring, based on the primary procedure (from the EACTS-STS International Nomenclature Procedures Short List), estimates complexity through three factors: mortality potential, morbidity potential, and technical difficulty.
RESULTS: This STS harvest includes data from 16 centers reporting 12,787 cases, with discharge mortality known for 10,246 cases. The basic complexity score has been applied to the outcomes analysis of these cases and a new equation has been proposed to evaluate one aspect of performance: Aristotle Performance Index = Outcome x Complexity = (Survival) x (Mean Complexity Score)
CONCLUSIONS: The complexity analysis represents a basic complexity-adjustment method to evaluate surgical results. Complexity is a constant precise value for a given patient at a given point in time; performance varies between centers. Future STS congenital data harvests will incorporate a second step, the Comprehensive Aristotle Score, utilizing additional patient specific complexity modifiers to allow a more precise complexity adjustment.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
| Dr Jeffrey Jacobs discloses that he has a financial relationship with CardioAccess, Inc.
|
During the early and mid 1990s, both the Society of Thoracic Surgeons (STS) and the European Association for Cardiothoracic Surgery (EACTS) created congenital heart surgery outcomes databases [1–4]. A number of lessons were learned from these early multiinstitutional registries [3–5]. Most important, it became apparent that four primary requirements exist in order to allow this type of database system to facilitate meaningful multiinstitutional outcomes analysis.
The EACTS-STS Aristotle Committee (Appendices 2 and 3) was created to address the third item in the list above [11–13]. The EACTS-STS Aristotle Committee is composed of a panel of expert surgeons that started in 1999 and included 50 congenital surgeons from 23 countries, representing multiple societies including the Congenital Heart Surgeons' Society (CHSS), the EACTS, the European Congenital Heart Surgeons Association (ECHSA) (formerly the European Congenital Heart Surgeons Foundation-ECHSF), and the STS. Founded and chaired by Francois G. Lacour-Gayet, MD, the Aristotle committee sought to address the common problem that many high profile institutions appeared to be reluctant to participate in multiinstitutional outcomes registries secondary to a perceived fear that their results would be unfairly compared to other programs with less complex case-mix. Obviously, the ideal complexity-adjustment method to evaluate surgical results is based on data-driven formulas of complexity adjustment. This data-driven methodology is utilized successfully in the STS Adult Cardiac Surgery Database [14, 15]. In congenital heart surgery, such a data-driven methodology currently is not feasible because of the higher number of procedure types, the lower volume of procedures, and the lack of verified data. Because this data-driven approach currently is not feasible, the Aristotle committee has utilized the principles of the Delphi method. This approach is based upon Aristotle's concept of Doxa (Aristotle, Rhetoric, Book I, 350 BC) which states that, "When there is no scientific answer available, the opinion (Doxa) perceived and admitted by the majority has the value of truth." The conceptual approach of the Delphi method was utilized by the Aristotle committee [11–13] to create a complexity-adjustment method to evaluate surgical results based upon the nomenclature and database developed by the International Congenital Heart Surgery Nomenclature and Database Project Committee [3].
The second harvest of The STS Congenital Heart Surgery Database [16–18] followed four years after the previous first harvest of the STS Congenital Heart Surgery Database [1, 2]. The analysis of the second harvest (1998–2001) of the STS Congenital Heart Surgery Database included the first attempt by the STS to apply a complexity-adjustment method to evaluate congenital heart surgery results. The data harvest was restricted to data spanning 1998 through 2001, with 16 centers submitting their data for analysis, covering all or parts of the inclusive period. All data were submitted using either the comprehensive CardioAccess database or the free CardioAccess minimum dataset version available at the STS web site (CardioAccess Inc, St Petersburg and Fort Lauderdale, FL: http://www.cardioaccess.com). This manuscript reviews the analysis of the second harvest of the STS Congenital Heart Surgery Database and its application of the Aristotle complexity-adjustment method to evaluate congenital heart surgery results.
|
Material and Methods |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
The second harvest of the STS Congenital Heart Surgery Database is the first STS Congenital Database Report to utilize the international standardized nomenclature and minimum database dataset specifications adopted by the STS and EACTS [3].
Because of the need to determine a reliable measure of mortality, for the report of second harvest of The STS Congenital Heart Surgery Database, mortality was determined by "discharge status." Data from other mortality fields (including "30-day status," "operative mortality," and "mortality assigned to this operation") were not complete enough for meaningful analysis. If a patient had more than one operation during a hospitalization, assignment of mortality was made to the first operation of the given hospitalization. Because mortality status at discharge is the chosen measure of mortality for this Congenital Heart Surgery Database analysis report, records with a missing value for discharge mortality and sites for which the level of missing mortality status at discharge exceeded 10% were removed from mortality analyses. These records were included in all other areas of the report. For three of the 16 sites, mortality data were missing for greater than 10% of the cases. Discharge mortality was 100% complete in the data from 11 of 16 centers, greater than or equal to 99.5% complete for 12 of 16 centers, and greater than or equal to 95.9% complete for 13 of 16 centers.
Concerns about site-level comparisons led to blinding of participant identities, converting these to consecutive letters. Furthermore, in the report of second harvest of the STS Congenital Heart Surgery Database, sites are assigned to case volume categories. Each single operation is considered a case. Any single operation or single case may have multiple component procedures. Only operation types cardiopulmonary bypass ["CPB"] and "No CPB Cardiovascular" are included in site volume categorizations and in all the analyses in this report. Operations coded as operation type "CPB Standby" are converted to operation type "No CPB Cardiovascular" by the software vendor before analysis and are therefore included in site volume categorizations. Other operation types (extracorporeal membrane oxygenation ["ECMO"], cardiopulmonary support ["CPS"], "Minor Procedure," "Bronchoscopy," "Other Endoscopy," "Thoracic," "Interventional Cardiology," and "Other") are not included in site volume categorizations and in all the analyses in this report. Sites are categorized as low volume if they perform up to 100 cases per year on average, medium volume if they perform between 101 and 250 cases per year on average, and high volume if they perform greater than 250 cases per year on average.
The Aristotle Complexity-Adjustment Method to Evaluate Surgical Results
The second harvest of the STS Congenital Heart Surgery Database utilizes a methodology to facilitate complexity adjustment based upon the work of the Aristotle committee [11–13]. The Data Summary Section of this STS report incorporates the Aristotle Basic Complexity Score and Aristotle Basic Complexity Level in the discharge mortality analyses. These complexity scores and levels are reported by year, center, age group, and procedure. The complexity analysis represents a basic complexity-adjustment method to evaluate surgical results. (Complexity is a constant precise value for a given patient at a given point in time; performance varies between centers and surgeons. In other words, in the same exact patient with the same exact pathology, complexity is a constant precise value for that given patient at a given point in time. The risk for that patient will vary between centers and surgeons because performance varies between centers and surgeons.)
The Aristotle complexity scoring was based on the primary procedure of a given operation as defined by the short list of procedures of the EACTS-STS International Nomenclature [3] and was evaluated in two steps. The first step was the Aristotle Basic Complexity Score, defining, basically, the complexity through three factors: the potential for mortality, the potential for morbidity, and the technical difficulty of the operation, using a questionnaire filled out by 50 surgeons representing international centers. Only the Aristotle Basic Complexity Score (1.5 to 15) and Aristotle Basic Complexity Level (4 levels: 1 to 4) are used in the second harvest of the STS Congenital Heart Surgery Database (Table 1).
|
For each procedure, the median value of mortality, morbidity, and technical difficulty obtained from the 50 centers was calculated. The sum of these three median values gives the final Aristotle Basic Complexity Score for each procedure (Table 1). The distribution of the scoring among the centers was, in general, quite uniform, although some rare or new procedures had a large dispersion [12, 13].
In addition to assigning each procedure an Aristotle Basic Complexity Score ranging from 1.5 to 15, each procedure was next assigned an Aristotle Basic Complexity Level ranging from 1 through 4 based on the Aristotle Basic Complexity Score (basic score of 1.5 to 5.9 = basic level of 1, basic score of 6.0 to 7.9 = basic level of 2, basic score of 8.0 to 9.9 = basic level of 3, and basic score of 10.0 to 15.0 = basic level of 4). Table 1 shows that 145 procedures from the EACTS-STS procedure short list have been scored and that 29 procedures are in level 1, 46 procedures are in level 2, 45 procedures are in level 3, and 25 procedures are in level 4.
The Aristotle Basic Complexity Level provides a broad generalization of complexity by dividing surgical procedures into four complexity levels. Meanwhile, the Aristotle Basic Complexity Score can provide more precise complexity stratification. Both the score and the level are useful tools; the appropriate tool can be chosen to match the required analysis.
Future harvests hopefully will incorporate the second step, the Aristotle Comprehensive Complexity Score, which will add two sorts of complexity modifiers: procedure-dependent factors (including anatomical factors, associated procedures, and age at procedure) and procedure independent factors (including general factors, clinical factors, extracardiac factors, and surgical factors). Each factor is scored for contribution to mortality, morbidity, and technical difficulty. All complexity factors meet the following requirements: precisely quantifiable, easily available, admitted by a majority, and verifiable [12, 13]. The Aristotle committee is currently involved in ongoing research to validate this complexity adjustment scoring system on a multiinstitutional basis.
Complexity scores can incorporate only a finite number of known factors. True complexity is related to both these known factors and other factors we may not know or measure. Although complexity itself is a constant precise value for a given patient at a given point in time, the Aristotle Basic Complexity Score, the Aristotle Basic Complexity Level, and the Aristotle Comprehensive Complexity Score all represent estimates to measure complexity.
The Aristotle Basic Complexity Score was added to the analysis of this data harvest at the time the analysis was performed by DCRI based on the data in Table 1. This technique was also utilized for the 2003 harvest and will be utilized for the 2004 harvest. Although it is possible for individual database software vendors to add the Aristotle Basic Complexity Score to their software, this addition of the Aristotle Basic Complexity Score to the software performing data collection at an individual institution is not necessary to perform the type of analysis reported in this manuscript.
|
Results |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
Not all operations were eligible for mortality analysis. Only those with known discharge mortality status were part of the mortality analysis. Although this data harvest included 12,787 cases, only 10,246 had known discharge mortality status and were eligible for mortality analysis. For the purposes of this manuscript, four brief tables of outcome data will be presented. Data in Tables 2, 3, and 4 appear in the Report of The Second Harvest of The STS Congenital Heart Surgery Database [17]. Data in Table 5 are derived from data contained within the Report of The Second Harvest of The STS Congenital Heart Surgery Database [17].
|
|
Table 3 shows the number of patients and operations submitted, discharge mortality, and complexity information, by age group. As one would expect, neonates have the highest mortality and complexity, followed by infants, and then older patients.
|
|
Aristotle Performance Index = Outcome x Complexity = (Survival) x (Mean Aristotle Basic Complexity Score)
Figure 1 shows program specific overall discharge mortality plotted against program specific mean basic complexity score for the 13 participating STS sites with greater than 95% complete discharge mortality data.
|
|
|
|
|
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
Because of the lack of accurate multiinstitutional outcomes data in the field of congenital heart surgery, the Aristotle committee decided to develop a methodology of risk adjustment and complexity adjustment based on an evaluation that was partially subjective, utilizing an approach based on the opinion of a panel of experts. Plans are now being developed to validate this methodology prospectively based on verified collected outcome data.
Others have attempted to utilize this panel of experts approach. Jenkins and colleagues [21–24] have developed a consensus-based risk-adjusted scheme for congenital heart surgery named risk adjustment in congenital heart surgery-1 (RACHS-1) in order to stratify procedures for congenital heart disease. This system is procedure driven and divides all procedures into 6 groups (1 to 6, with 1 indicating easy and 6, difficult). Although the RACHS-1 represents an initial attempt to utilize this methodology, the authors of this manuscript recognize several weaknesses with the Jenkins approach which we hope to improve upon with the Aristotle methodology. First, although the RACHS-1 methodology is consensus based, the panel of experts involved in developing this consensus was quite small (11 total members including only 4 surgeons) and represents only one country (United States of America) [23]. Second, the nomenclature utilized for the procedures scored in RACHS-1 is taken from administrative databases (both Current Procedural Terminology 4 and International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes) that have extremely limited clinical utility and are not utilized in the majority of multiinstitutional and individual institutional cardiac surgery databases either world wide or in the United States of America. Third, the RACHS-1 divides procedures into 6 levels, but one of these levels is almost never used (category 5). Fourth, several important operations in the field of congenital heart surgery do not appear in the RACHS-1 system including, for example, heart transplantation, lung transplantation, and heart and lung transplantation. Fifth, the RACHS-1 system does not evaluate patients more than 18 years of age (who account for at least 25% of congenital heart surgery activity today and surely more in the future). Sixth, the value of the RACHS-1 system is quite questionable when evaluating the risk of an individual patient or the performance of an individual center. The risk of an individual patient undergoing a congenital heart operation will depend on the associated pathology present in that patient and the associated procedures performed. Although neither the RACHS-1 system nor the Aristotle Basic Complexity Score will address these individual patient variables, the Aristotle Comprehensive Complexity Score will incorporate these important factors.
In this manuscript, we address the difficult topic of quantifying surgical performance. Although this topic is not easy to discuss, most surgeons agree that clear advantages exist by maintaining physician leadership in this discussion. Our equation to measure performance represents our initial attempt to quantify this variable. Complexity is a constant and precise value for a given patient at a given point in time regardless of the center where the surgery occurs; it is a constant, at a given time for a given procedure in a given patient, whatever the center and its global location. Performance may vary from surgeon to surgeon and center to center.
Outcome is a reflection of the relationship between complexity and performance. Our initial attempt to utilize this concept of complexity and its relationship to outcome and performance is based on the Aristotle committee defining a new equation of quality of care: Complexity x Outcome = Performance [12, 13]. This equation is similar to Ohm's Law which states that [Flow = Pressure divided by Resistance] which can be medically equated with the equation [Cardiac Output = Blood Pressure divided by Systemic Vascular Resistance]. In this analogy: Performance is Pressure, Survival is Cardiac Output and Complexity is Resistance. Thus, similar to the equation [Pressure = Flow x Resistance] and the analogous equation [Blood Pressure = Cardiac Output x Vascular Resistance], one can postulate that [Performance = Outcome x Complexity].
Admittedly, the equation is a hypothesis and needs to be confirmed. Instead of [Performance = Outcome x Complexity], the equation may better be stated [Performance = Complexity FN Outcome], with FN (FN = function) still to be determined. Indeed, a coefficient might need to be placed or some other alteration to the formula might need to occur. Our current equation needs to be confirmed, and this manuscript represents our initial attempt to incorporate this yet to be validated and confirmed methodology into the STS database. We hypothesize that multiplication represents the best FN for this equation.
At this point, several limitations exist in this calculation. First, the data themselves in the STS Congenital Heart Surgery Database have yet to be verified for completeness or accuracy. This verification of the data for completeness and accuracy is currently the topic of considerable discussion. It should be remembered, however, that the STS Adult Cardiac Surgery Database has published multiple manuscripts and even developed detailed, widely utilized, data based risk stratification algorithms based upon nonverified data collected under a similar "honor system." Nevertheless, data completeness represents a potentially confounding variable. Patients not included in medical audits have a worse outcome than those included [25]. The importance of the verification of the accuracy of the data is demonstrated in a recent report from the United Kingdom Central Cardiac Audit Database (UK CCAD). The UK CCAD analyzed data from all 13 UK tertiary centers performing cardiac surgery or therapeutic cardiac catheterization in children with congenital heart disease and examined 3,666 surgical procedures and 1,828 therapeutic catheterizations performed in 2000 and 2001. Thirty day mortality was identified both by volunteered life status from the hospital databases and by independently validated life status either through the Office for National Statistics using the patient's unique National Health Service number, or through the general register offices of Scotland and Northern Ireland. Central tracking of mortality identified 469 deaths, 194 occurring within 30 days and 275 later. Forty-two of the 194 deaths within 30 days (21.6% of the 30-day mortality) were detected by central tracking but not by volunteered data. In other words, hospital-based databases underreported 30-day mortality by 21.6% even though the hospitals were aware that the data would be independently verified [26]. Second, the Aristotle Basic Complexity Score also needs to be validated. Efforts are currently underway to achieve this goal using several individual institutional and multiinstitutional outcomes registries. Third, the very definition of the word "performance" is controversial. Clearly, performance has multiple domains including the absence of mortality (survival), the absence of morbidity, cost effectiveness, long-term functional status and outcomes, patient satisfaction, the process of care utilized, adherence to appropriate practice guidelines, practicing evidence-based medicine and surgery, quality improvement, patient selection, preoperative management, intraoperative management, and postoperative management. The word performance takes on different meanings and different connotations in different arenas. In fact, this word is clearly a politically sensitive term. Although originally published as [Performance = Outcome x Complexity] [11–13], we have modified the terminology to read [Aristotle Performance Index = Outcome x Complexity = (Survival) x (Mean Complexity Score)]. Significant debate took place to agree finally on this terminology. Alternative terms for the Aristotle Performance Index were considered including the "Aristotle Metric" and the term "Complexity Adjusted Outcome." Clearly, both this project and the measurement of surgical performance in general represent "works in progress." The Aristotle Performance Index has been proposed to evaluate one aspect of performance. This concept of quantification of performance is still a theory and is yet to be confirmed. This manuscript demonstrates how this theoretical concept can be applied to the STS Congenital Heart Surgery Database.
Despite the above limitations, this incorporation of the Aristotle Basic Complexity Score into the STS database represents an important step in the development of the STS Congenital Heart Surgery Database. This methodology should help alleviate concerns from centers treating the sickest patients and potentially having a significant mortality. These centers will view this methodology as a technique by which they can avoid being unfairly penalized for treating more complex patients.
Beginning with the 2004 STS data harvest, the Aristotle Basic Complexity Score will be utilized for two new purposes in the STS Congenital Heart Surgery Database: (1) To define the primary procedure of a given operation; (2) to define allowable concomitant procedures for a given lesion specific report.
The eventual incorporation into the STS Congenital Heart Database of the Aristotle Comprehensive Complexity Score will add patient specific modifiers to the methodology of complexity adjustment. Both the RACHS-1 and the Aristotle Basic Complexity Score lack these patient specific modifiers. Clearly, this initial application of the Aristotle Basic Complexity Score to the STS database merely represents the engine that starts the process. The value of this methodology is that it "quantifies the educated guess." Obviously, this work is part of an evolving process and much further work remains. The following tasks represent the next important steps in this process.
|
Appendix 1 |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
|
Appendix 2 |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
The Aristotle Project participants were 50 congenital heart surgeons from 23 countries and multiple societies including the CHSS, EACTS, ECHSA (until 2003 known as the ECHSF), and STS.
|
Appendix 3 |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
|
Discussion |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
Simple mortality statistics can be misleading. A lower risk is to be expected for simple lesions. Jenkins and her colleagues used a consensus panel of experts to stratify congenital heart operations into six categories. Operative risk for the simplest of the six categories was less than 1% in 1996 compared to greater than 50% for the highest risk group. The Pediatric Cardiac Care Consortium reports mortality for each diagnostic lesion as a mean for the consortium institutions and the standard deviations of that mean. Institutions outside the standard deviation for the group are alerted to their performance. However, their process is not possible when the number of operations in individual diagnostic groups is small.
The Aristotle committee has assigned, by a consensus panel of experts, a complexity score for each of the primary operations in the Society of Thoracic Surgeons and European Association for Cardio-Thoracic Surgery (STS-EACTS) nomenclature system. One cannot deny the logic in assigning a greater risk to a baby having a Norwood operation compared to a child having an atrial septal defect (ASD) closed. The only question then is what is a reasonable ratio that fairly reflects the expected outcome for each operation?
The Congenital Heart Surgeons Society (CHSS) studies have taken the opposite approach to risk assessment. Rather than rely on expert opinion, data from each child in a study, including demographic, functional, procedural, and diagnostic variables, are entered into an extensive database. Multivariable analysis then determines the risk factors that impact significantly on defined outcomes. Not all babies having a Norwood operation have identical operative risk, and I suspect the same is true for a child having an ASD repair. Risk factors change over time, and they are clearly different for different operations. Hazard function analysis clearly illustrates that the early phase of mortality risk extends well beyond the arbitrary in-hospital period.
This process used by the CHSS allows risk adjustment for each patient rather than simply risk stratification of groups of presumed similar risk, as advocated by the Jenkins and Aristotle consensus panels. Importantly, the CHSS analysis allows prediction of probable outcomes with a given set of significant variables. The CHSS analyses are focused on single lesions; they are complex, time-consuming, and expensive. At the present point in time they are an impractical method of monitoring every congenital heart operation. However, one hopes in the not too distant future, when the various databases that we all use in our cardiac units can be linked to each other, true multivariable surveillance of each congenital cardiac program will be possible.
I have three questions for Dr Jacobs. Firstly, the data integrity varied among the 16 institutions considerably. Operative survival is unknown for many patients and primary operation is unknown in another 12%. The data from three institutions could not be used. Yet 11 institutions had a data completion rate of 99.5% for discharge status and I assume for the other fields. So my question is what can we learn about data collection from these 11 successful institutions that could be applied in the others? Is it a matter of resources for data managers, and if so, how did the successful units fund their data collection? Secondly, institution 1 had the lowest discharge mortality at 2.7%, yet institution 9, that had the ninth ranked operative mortality of 4.4%, had the highest (best) Aristotle score. What can you tell us about these two institutions to reassure us that the Aristotle performance score is justified? And finally, what plans do you have to validate the Aristotle score? In other words, what can you measure to determine that institutional performance is being fairly assessed? I would like to thank the Society for the privilege of discussing this paper and to Dr Jacobs for sharing the information with us.
DR JACOBS: I would like to thank Dr Williams for his thoughtful analysis and his important questions. Anyone involved in the field of outcomes analysis, and especially congenital heart surgery outcomes analysis, knows of the important contributions that Dr Williams and the team from Toronto have made to this field. I would agree that the challenge of obtaining complete and accurate data is time-consuming and costs money, and obviously we had some variability in the completeness of the data that we were able to collect. Our manuscript reports that of the 16 centers submitting data, 11 out of 16 centers were able to give us 100% complete mortality data, 12 out of 16 centers were able to give us greater than 99.5% complete mortality data, and 13 of 16 centers reported greater than 95.9% complete mortality data. Unfortunately, three centers had less than one third of their mortality data complete. Two potential etiologies exist for incomplete data: one, software problems, and two, data entry problems from the personnel and system utilized for data entry into the software within the hospital or program. In 1999 and 2000, the STS Congenital Heart Surgery Database Taskforce spent a great deal of time modifying the software specifications so that the software would encourage complete and accurate data input. I think these efforts to improve the software specifications facilitated achieving the fact that 11 out of 16 centers in this analysis have 100% complete data in this mortality field. Clearly, some centers still are not getting complete data. Part of the problem is related to data entry systems problems at the individual institutions. A number of potential solutions exist.
I will give three examples. First, developing a system with real time data entry will facilitate and support complete and accurate data entry. Second, having designated personnel responsible for data entry also helps with this problem. Third, having systems in the database that give positive reinforcement for entering data, such as having the database generate a brief operative report to put in the chart after an operation, also helps achieve complete and accurate data. It is clear that the process of obtaining complete and accurate data requires the institutional programmatic commitment of both time and money. We hope that the report of the second harvest of The STS Congenital Heart Surgery Database will alert our profession and our contributing centers to the necessity of mandatory data field completion, and help us to develop appropriate strategies for more complete data collection and presentation of the data in the future.
Now, briefly, the second question was regarding what we learned about site 9 and site 1. If one looks initially at the results shown in Table 5 from our manuscript, one might say that perhaps hospital number one (letter A in Table 5) has the best performance because they have the lowest discharge mortality of 2.7%. In reality, hospital number nine (letter I in Table 5) with the discharge mortality of 4.4% is actually performing quite well because their case mix or complexity is the highest. In other words, although site 9 had a midrange mortality, they are not penalized for their mortality by the Aristotle performance index because they are credited for their higher complexity case mix.
And finally, your very important question about data validation. We would agree that this data verification and validation process is the next step. As pointed out by David Robinson Clarke, MD, from Denver, in reality, three components of validation exist. First, validating the data within the database, which is really verifying that the data are complete and the data are accurate. The STS Congenital Heart Disease Database Taskforce plans to collaborate with the EACTS and Duke Clinical Research Institute (DCRI) to develop methods to assure and verify data completeness and data accuracy. Second, the Aristotle score needs validation. And third, our theories about the performance equation, and the concept of quantifying performance utilizing outcome and complexity, need to be confirmed. In our manuscript, Figures 3, 4, and 5 demonstrate an initial attempt at validation of the Aristotle score done by the STS Congenital Database Taskforce in collaboration with DCRI. Utilizing data that is admittedly not itself verified for completeness or accuracy, Figure 3 plots the four levels of the Aristotle system versus discharge mortality. One can see that as the complexity levels increased, the discharge mortality also increases, as we would expect. Figures 4 and 5 depict a similar relationship using the basic Aristotle complexity score instead of the levels. Certainly, further work needs to be done similar to this work, hopefully with true and validated data. I would like to thank the Society for the opportunity to present this work.
|
Acknowledgments |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentAppendix 1Appendix 2Appendix 3DiscussionAcknowledgmentsReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  A. Manrique, E. H. Jooste, B. A. Kuch, S. E. Lichtenstein, V. Morell, R. Munoz, D. Ellis, and P. J. Davis The Association of Renal Dysfunction and the Use of Aprotinin in Patients Undergoing Congenital Cardiac Surgery Requiring Cardiopulmonary Bypass Anesth. Analg., July 1, 2009; 109(1): 45 - 52. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. P. Jacobs, R. J. Cerfolio, and R. M. Sade The Ethics of Transparency: Publication of Cardiothoracic Surgical Outcomes in the Lay Press. Ann. Thorac. Surg., March 1, 2009; 87(3): 679 - 686. [Full Text] [PDF]
|
|
|
|
 |
|
 A. Ernst, M. Simoff, D. Ost, Y. Goldman, and F. J. F. Herth Prospective Risk-Adjusted Morbidity and Mortality Outcome Analysis After Therapeutic Bronchoscopic Procedures: Results of a Multi-institutional Outcomes Database Chest, September 1, 2008; 134(3): 514 - 519. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Miyamoto, N. Sinzobahamvya, J. Photiadis, A. M Brecher, and B. Asfour Survival after Surgery with Cardiopulmonary Bypass in Low Weight Patients Asian Cardiovasc Thorac Ann, April 1, 2008; 16(2): 115 - 119. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. M. O'Brien, J. P. Jacobs, D. R. Clarke, B. Maruszewski, M. L. Jacobs, H. L. Walters III, C. I. Tchervenkov, K. F. Welke, Z. Tobota, G. Stellin, et al. Accuracy of the Aristotle Basic Complexity Score for Classifying the Mortality and Morbidity Potential of Congenital Heart Surgery Operations Ann. Thorac. Surg., December 1, 2007; 84(6): 2027 - 2037. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. P. Jacobs, M. L. Jacobs, C. Mavroudis, B. Maruszewski, C. I. Tchervenkov, F. G. Lacour-Gayet, D. R. Clarke, T. Yeh Jr, H. L. Walters III, H. Kurosawa, et al. What is Operative Morbidity? Defining Complications in a Surgical Registry Database Ann. Thorac. Surg., October 1, 2007; 84(4): 1416 - 1421. [Full Text] [PDF]
|
|
|
|
 |
|
 J. Rekawek, A. Kansy, M. Miszczak-Knecht, M. Manowska, K. Bieganowska, M. Brzezinska-Paszke, E. Szymaniak, A. Turska-Kmiec, P. Maruszewski, P. Burczynski, et al. Risk factors for cardiac arrhythmias in children with congenital heart disease after surgical intervention in the early postoperative period J. Thorac. Cardiovasc. Surg., April 1, 2007; 133(4): 900 - 904. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
V. L. Vida, H. Berggren, W. J. Brawn, W. Daenen, D. Di Carlo, R. Di Donato, H. L. Lindberg, A. F. Corno, J. Fragata, M. J. Elliott, et al. Risk of Surgery for Congenital Heart Disease in the Adult: A Multicentered European Study Ann. Thorac. Surg., January 1, 2007; 83(1): 161 - 168. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. H. Artrip, D. N. Campbell, D. D. Ivy, M. C. Almodovar, K.-C. Chan, M. B. Mitchell, D. R. Clarke, and F. Lacour-Gayet Birth Weight and Complexity Are Significant Factors for the Management of Hypoplastic Left Heart Syndrome. Ann. Thorac. Surg., October 1, 2006; 82(4): 1252 - 1259. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. F. Welke, I. Shen, and R. M. Ungerleider Current assessment of mortality rates in congenital cardiac surgery. Ann. Thorac. Surg., July 1, 2006; 82(1): 164 - 170. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. Kang, V. T. Tsang, M. J. Elliott, M. R. de Leval, and T. J. Cole Does the Aristotle Score predict outcome in congenital heart surgery? Eur. J. Cardiothorac. Surg., June 1, 2006; 29(6): 986 - 988. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. Sinzobahamvya, J. Photiadis, D. Kumpikaite, C. Fink, H. C. Blaschczok, A. M. Brecher, and B. Asfour Comprehensive aristotle score: implications for the norwood procedure. Ann. Thorac. Surg., May 1, 2006; 81(5): 1794 - 1800. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. P. Jacobs, C. Mavroudis, M. L. Jacobs, B. Maruszewski, C. I. Tchervenkov, F. G. Lacour-Gayet, D. R. Clarke, T. Yeh Jr, H. L. Walters III, H. Kurosawa, et al. What is Operative Mortality? Defining Death in a Surgical Registry Database: A Report of the STS Congenital Database Taskforce and the Joint EACTS-STS Congenital Database Committee Ann. Thorac. Surg., May 1, 2006; 81(5): 1937 - 1941. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. P. Jacobs, M. L. Jacobs, B. Maruszewski, F. G. Lacour-Gayet, D. R. Clarke, C. I. Tchervenkov, J. W. Gaynor, T. L. Spray, G. Stellin, M. J. Elliott, et al. Current Status of the European Association for Cardio-Thoracic Surgery and The Society of Thoracic Surgeons Congenital Heart Surgery Database Ann. Thorac. Surg., December 1, 2005; 80(6): 2278 - 2284. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. L. Jacobs Editorial comment Eur. J. Cardiothorac. Surg., September 1, 2005; 28(3): 405 - 406. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
