|
|
||||||||
a Department of Cardiac Surgery, Unit for Clinical Research in Atherothrombosis, Centro Cardiologico Monzino IRCCS, University of Milan, Milan, Italy
b Department of Radiology, The University of Chicago, Chicago, Illinois
c Department of Cardiac Surgery, S. Orsola-Malpighi Hospital, University of Bologna, Bologna, Italy
Accepted for publication November 11, 2009.
* Address correspondence to Dr Parolari, Department of Cardiac Surgery, University of Milan Centro Cardiologico, Fondazione Monzino IRCCS, Via Parea, 4, Milan, 20138, Italy (Email: alessandro.parolari{at}cardiologicomonzino.it).
| Abstract |
|---|
|
|
|---|
Methods: We performed a meta-analysis of published studies reporting the assessment of discriminatory power of the EuroSCORE by receiver operating characteristics (ROC) curve analysis in adult valve operations. A comparison of observed and predicted mortality rates was also performed.
Results: A literature search identified 37 potentially eligible studies, and 12 were selected for meta-analysis comprising 26,621 patients with 1250 events (mortality rate, 4.7%). Meta-analysis of these studies provided an average area under the curve (AUC) value of 0.730 (95% confidence interval [CI], 0.717 to 0.743). The same results were obtained when meta-analyses were performed separately in studies categorized on reliability of uncertainty estimation: in the seven studies reporting reliable uncertainty estimation (8175 patients with 358 events; mortality rate, 4.4%), the ROC curve provided an average AUC value of 0.724 (95% CI, 0.699 to 0.749). The five studies not reporting reliable uncertainty estimation (18,446 patients with 892 events; mortality rate, 4.8%) had an average AUC of 0.732 (95% CI, 0.717 to 0.747). We documented a constant trend to overpredict mortality by EuroSCORE, both in the additive and especially in the logistic form.
Conclusions: The EuroSCORE has low discrimination ability for valve surgery, and it sensibly overpredicts risk. Alternative risk scoring algorithms should be seriously considered.
| Introduction |
|---|
|
|
|---|
Nevertheless, the EuroSCORE has been widely used to predict risk in valve surgery [3, 4], and it has been recently used to select high-risk patients for transcatheter aortic valve procedures and to compare survival after standard or transcatheter aortic valve replacement [5, 6]. In this study, we proposed a systematic review to answer the question whether it can predict early mortality after valve operations with sufficient accuracy: we assessed EuroSCORE discriminatory power using receiver operating characteristics (ROC) curves analysis and compared observed and EuroSCORE-predicted mortality rates.
| Material and Methods |
|---|
|
|
|---|
To further reduce the probability of losing any major related study, an electronic search was performed of four major cardiothoracic surgery journals in the electronic format, Interactive CardioVascular and Thoracic Surgery, The Annals of Thoracic Surgery, The European Journal of Cardiothoracic Surgery, and The Journal of Thoracic and Cardiovascular Surgery (available at http://ats.ctsnetjournals.org/search.dtl). The journals were searched from January 1999 to March 2009 for the single word "EuroSCORE" in the full text of all articles. The title of every article was considered first, then selected abstracts were searched to identify reports about risk prediction by the additive or logistic EuroSCORE, or heart valve operations. The full texts of these articles were retrieved and searched for in-hospital or 30-day mortality data and for mortality prediction by the EuroSCORE. Figure 1 reports the flow chart of paper selection.
|
Studies selected for meta-analysis were then classified as (1) studies with reliable uncertainty estimation (group A), when the figure of merit defined as the area under the curve (AUC), was reported with a reliable dispersion parameter (standard error or 95% confidence interval [CI], or both); and (2) articles without reliable uncertainty estimation (group B), if only the AUC was reported and its dispersion parameter was missing or unreliable [8]. The dispersion variables of these last studies were estimated using an approximate formula, derived from the asymptotic form of the standard error for the estimate of a population proportion [9]. Meta-analysis models were developed for all articles, and dividing articles based on the presence of uncertainty estimation to verify whether our estimate of the AUC did not seem to influence the results of the meta-analysis. Because the purpose of this meta-analysis was to gain insight in the general discriminatory characteristics of the EuroSCORE models (ie, whether they can discriminate) we included only the AUC instead of building a complete summary ROC curve. Our analysis follows the work of McClish [10] and Zhou [11]. We assessed heterogeneity using a one-sided
2 measure and estimated the overall performance using inverse variance weights, as described in Parolari and colleagues [12].
The 37 potentially eligible studies (Fig 1) were also searched for reporting the observed and predicted (by either additive or logistic model) mortality rates to assess the ratio between observed and expected mortality rates; in this case, subanalyses considering different categories of patients (eg, only valves, valves plus CABG, high-risk patients) were considered only when five or more studies reported data for different categories of patients. Observed and predicted mortality rates were compared with the
2 test with continuity correction. Multiple comparisons calculations were done following the Holm method [13].
All calculations were done with University of Chicago ROC software (http://xray.bsd.uchicago.edu/krl/), Excel 2003 (Microsoft Inc, Redmond, WA) or Mathematica 6 software (Wolfram Research Inc, Champaign, IL).
| Results |
|---|
|
|
|---|
|
|
Observed vs Expected Mortality Rates
Of the 37 potentially eligible studies, 24 reported the observed and predicted mortality rates for the additive or logistic model, or for both, in valve operations (Table 2
and Appendix
*). The following patient categories were analyzed because at least five studies were available that reported data about the observed and the expected mortality rates: (1) all patients undergoing valve operations with or without concomitant CABG; (2) high-risk patients undergoing valve operations with or without concomitant CABG; (3) patients undergoing valve surgery without concomitant CABG; (4) patients undergoing aortic valve replacement with or without concomitant CABG; and (5) patients undergoing isolated aortic valve replacement.
|
| Comment |
|---|
|
|
|---|
The discriminatory power and precision in risk prediction of the EuroSCORE in valve surgery has recently become increasingly important for two reasons. The first is that in the most centers, valve procedures—either isolated or combined—actually represent more than 50% of the total caseload; therefore, accurate risk estimation in this patient population—mainly elderly and very elderly people—has become much more important. The second reason is strictly related to the recent evolution in technical options in aortic valve operations that has led to a steady increase in the adoption of transcatheter aortic valve procedures in patients at the highest risk or in very elderly people [31]. A correct risk prediction is essential to select the patients who might benefit the most from this new and still experimental technique while at the same time managing effectively health program expenditures, because these new devices are quite expensive. Although some authors have already suggested that the EuroSCORE might be an effective tool for the selection of these patients [5, 6], the question of appropriateness of the use of these models must be addressed before such a decision process can be implemented [32].
This study has been designed mainly to define the role of the EuroSCORE in risk prediction for the current clinical cardiac surgical practice that progressively involves valve patients who are older. It was designed to attempt providing useful insights in the use of additive and logistic EuroSCORE models in defining which patients among the candidates to aortic valve replacement are at highest risk and therefore might be the best candidates for alternative and innovative transcatheter procedures.
Unfortunately, our study strongly suggests that the EuroSCORE might not be the appropriate tool for risk prediction in isolated valve operations or those combined with other cardiac procedures. The AUC derived from the meta-analysis provided estimates of 0.72 to 0.74, which are in a range of a performance considered less than satisfactory for a risk stratification algorithm [33]. EuroSCORE discrimination is also substantially lower with respect to the performance of the Society of Thoracic Surgery (STS) algorithm, which is about 0.8 for isolated valve operations [34] and about 0.75 for valves plus CABG [35]. The explanation for this is that the STS score is updated almost annually, and, for this reason, it may better follow the changes occurring in valve patient population with relative ease, whereas the EuroSCORE is now undergoing its first revision since its introduction.
That the heterogeneity test was almost significant is consistent with differences actually existing among the different procedures studies here, which should be expected, but given the large number of alternatives, it would require a large and detailed study to investigate it. Perhaps these resources would be better used to design a new more specific risk prediction scheme. Interestingly, a preliminary analysis of our data in terms of ROC analysis showed similar discriminatory performance for additive and logistic EuroSCORE models, suggesting a near equivalence for these two models from this perspective. In addition, the analysis of the ratio between observed and expected deaths by the additive and logistic models suggest overall a tendency for both models to significantly overpredict the mortality risk, which appears to be more marked for the logistic model.
In detail, the logistic EuroSCORE constantly and significantly overpredicted mortality in all five subcategories of patients used in our analysis. The older method, the additive model, showed significant overprediction in three of five subcategories, the ones with somehow smaller complexity of the case/lower risk, whereas the additive model performed at least fairly in the identification of the global risk of patient population in the most complex subcategories.
This appears to contrast with previous findings suggesting that the more recent logistic model is more appropriate for risk prediction, especially in more complex cases. In fact, it appears that we are observing a EuroSCORE paradox: the older and less sophisticated algorithm—the additive EuroSCORE—outperforms the more recent and complex method implemented just to estimate with more precision the risk in complicated cases. Our study adds further evidence that in current cardiac surgical practice, there is a compelling need for perhaps even a complete reengineering of EuroSCORE. Almost 15 years have elapsed since its development, and the clinical profile of cardiac surgical patients, especially for valve procedures, has changed substantially and so should the methods used to assess their risk.
| Appendix |
|---|
Twelve studies were then selected for meta-analysis purposes [1, 3, 9, 14, 15, 17, 21, 27, 29, 33, 35, 37]; two studies [3, 35] reported the area under the curve (AUC) for patients undergoing valve operations and valve operations plus coronary artery bypass grafting (CABG) separately, and data concerning both these European System for Cardiac Operative Risk Evaluation (EuroSCORE) performances have been included in the meta-analysis. In addition, it should also be noted that among the articles selected for meta-analysis, five [15, 17, 27, 35, 37] reported the performance of both additive and logistic EuroSCORE on the same patients subsets (Appendix Table).
|
Observed vs Expected Mortality Rates
The studies that reported the observed and predicted mortality rates for the additive or logistic model, or for both, in valve operations were references 1, 3–5, 7, 8, 10–17, 20, 22, 23, 25–27, and 32–35.
| Footnotes |
|---|
|
|
|---|
* See note at end of article regarding e-only Appendix. ![]()
* See note at end of article regarding e-only Appendix. ![]()
| References |
|---|
|
|
|---|
| References |
|---|
|
|
|---|
1. Au WK, Sun MP, Lam KT, Cheng LC, Chiu SW, Das SR. Mortality prediction in adult cardiac surgery patients: comparison of two risk stratification models Hong Kong Med J 2007;13:293-297.[Medline]
2. Berman M, Stamler A, Sahar G, et al. Validation of the 2000 Bernstein-Parsonnet score versus the EuroSCORE as a prognostic tool in cardiac surgery Ann Thorac Surg 2006;81:537-540.
3. Bhatti F, Grayson AD, Grotte G, et al. The logistic EuroSCORE in cardiac surgery: how well does it predict operative risk? Heart 2006;92:1817-1820.
4. Bose AK, Aitchison JD, Dark JH. Aortic valve replacement in octogenarians J Cardiothorac Surg 2007;2:33.[Medline]
5. Cladellas M, Bruguera J, Comin J, et al. Is pre-operative anaemia a risk marker for in-hospital mortality and morbidity after valve replacement? Eur Heart J 2006;27:1093-1099.
6. Collart F, Feier H, Kerbaul F, et al. Primary valvular surgery in octogenarians: perioperative outcome J Heart Valve Dis 2005;14:238-242discussion 42.[Medline]
7. Collart F, Feier H, Kerbaul F, et al. Valvular surgery in octogenarians: operative risks factors, evaluation of Euroscore and long term results Eur J Cardiothorac Surg 2005;27:276-280.
8. Dewey TM, Brown D, Ryan WH, Herbert MA, Prince SL, Mack MJ. Reliability of risk algorithms in predicting early and late operative outcomes in high-risk patients undergoing aortic valve replacement J Thorac Cardiovasc Surg 2008;135:180-187.
9. Florath I, Albert A, Hassanein W, et al. Current determinants of 30-day and 3-month mortality in over 2000 aortic valve replacements: Impact of routine laboratory parameters Eur J Cardiothorac Surg 2006;30:716-721.
10. Ghosh S, Jutley RS, Wraighte P, Shajar M, Naik SK. Beating-heart mitral valve surgery in patients with poor left ventricular function J Heart Valve Dis 2004;13:622-627discussion 7–9.[Medline]
11. Gibson PH, Croal BL, Cuthbertson BH, et al. The relationship between renal function and outcome from heart valve surgery Am Heart J 2008;156:893-899.[Medline]
12. Gonzalez-Vilchez F, Vazquez de Prada JA, Nistal F, et al. [Current surgical treatment of calcified aortic stenosis Rev Esp Cardiol 2008;61:84-87.[Medline]
13. Grossi EA, Schwartz CF, Yu PJ, et al. High-risk aortic valve replacement: are the outcomes as bad as predicted? Ann Thorac Surg 2008;85:102-106discussion 107.
14. Gummert JF, Funkat A, Osswald B, et al. EuroSCORE overestimates the risk of cardiac surgery: results from the national registry of the German Society of Thoracic and Cardiovascular Surgery Clin Res Cardiol 2009;98:363-369.[Medline]
15. Heikkinen J, Biancari F, Satta J, et al. Predicting immediate and late outcome after surgery for mitral valve regurgitation with EuroSCORE J Heart Valve Dis 2007;16:116-121.[Medline]
16. Kaartama T, Heikkinen L, Vento A. An evaluation of mitral valve procedures using the European system for cardiac operative risk evaluation Scand J Surg 2008;97:254-258.[Medline]
17. Karthik S, Srinivasan AK, Grayson AD, et al. Limitations of additive EuroSCORE for measuring risk stratified mortality in combined coronary and valve surgery Eur J Cardiothorac Surg 2004;26:318-322.
18. Kasimir MT, Bialy J, Moidl R, et al. EuroSCORE predicts mid-term outcome after combined valve and coronary bypass surgery J Heart Valve Dis 2004;13:439-443.[Medline]
19. Kawachi Y, Nakashima A, Toshima Y, Arinaga K, Kawano H. Risk stratification analysis of operative mortality in heart and thoracic aorta surgery: comparison between Parsonnet and EuroSCORE additive model Eur J Cardiothorac Surg 2001;20:961-966.
20. Khaladj N, Shrestha M, Peterss S, et al. Isolated surgical aortic valve replacement after previous coronary artery bypass grafting with patent grafts: is this old-fashioned technique obsolete? Eur J Cardiothorac Surg 2009;35:260-264discussion 264.
21. Langanay T, Verhoye JP, Ocampo G, et al. Current hospital mortality of aortic valve replacement in octogenarians J Heart Valve Dis 2006;15:630-637discussion 637.[Medline]
22. Levy F, Laurent M, Monin JL, et al. Aortic valve replacement for low-flow/low-gradient aortic stenosis operative risk stratification and long-term outcome: a European multicenter study J Am Coll Cardiol 2008;51:1466-1472.
23. Mascherbauer J, Rosenhek R, Fuchs C, et al. Moderate patient-prosthesis mismatch after valve replacement for severe aortic stenosis has no impact on short-term and long-term mortality Heart 2008;94:1639-1645.
24. Mestres CA, Castro MA, Bernabeu E, et al. Preoperative risk stratification in infective endocarditis. Does the EuroSCORE model work?. Preliminary results. Eur J Cardiothorac Surg 2007;32:281-285.
25. Monin JL, Monchi M, Kirsch ME, et al. Low-gradient aortic stenosis: impact of prosthesis-patient mismatch on survival Eur Heart J 2007;28:2620-2626.
26. Ngaage DL, Cowen ME, Griffin S, Guvendik L, Cale AR. Are initial valve operations in octogenarians still high-risk in the current era? J Heart Valve Dis 2008;17:227-232.[Medline]
27. Osswald BR, Gegouskov V, Badowski-Zyla D, Tochtermann U, Thomas G, Hagl S, Blackstone EH. Overestimation of aortic valve replacement risk by EuroSCORE: implications for percutaneous valve replacement Eur Heart J Eur Heart J 2009;30:74-80.
28. Pedrazzini GB, Masson S, Latini R, et al. Comparison of brain natriuretic peptide plasma levels versus logistic EuroSCORE in predicting in-hospital and late postoperative mortality in patients undergoing aortic valve replacement for symptomatic aortic stenosis Am J Cardiol 2008;102:749-754.[Medline]
29. Roques F, Nashef SA, Michel P. Risk factors for early mortality after valve surgery in Europe in the 1990s: lessons from the EuroSCORE pilot program J Heart Valve Dis 2001;10:572-577discussion 577–8.[Medline]
30. Suojaranta-Ylinen RT, Kuitunen AH, Kukkonen SI, Vento AE, Salminen US. Risk evaluation of cardiac surgery in octogenarians J Cardiothorac Vasc Anesth 2006;20:526-530.[Medline]
31. Toumpoulis IK, Chamogeorgakis TP, Angouras DC, Swistel DG, Anagnostopoulos CE, Rokkas CK. Independent predictors for early and long-term mortality after heart valve surgery J Heart Valve Dis 2008;17:548-556.[Medline]
32. Toumpoulis IK, Anagnostopoulos CE, Toumpoulis SK, DeRose Jr JJ, Swistel DG. EuroSCORE predicts long-term mortality after heart valve surgery Ann Thorac Surg 2005;79:1902-1908.
33. Toumpoulis IK, Anagnostopoulos CE. Does EuroSCORE predict length of stay and specific postoperative complications after heart valve surgery? J Heart Valve Dis 2005;14:243-250.[Medline]
34. Urso S, Sadaba R, Greco E, et al. One-hundred aortic valve replacements in octogenarians: outcomes and risk factors for early mortality J Heart Valve Dis 2007;16:139-144.[Medline]
35. van Gameren M, Kappetein AP, Steyerberg EW, et al. Do we need separate risk stratification models for hospital mortality after heart valve surgery? Ann Thorac Surg 2008;85:921-930.
36. Xu J, Ge Y, Pan S, Liu F, Shi Y. A preoperative and intraoperative predictive model of prolonged intensive care unit stay for valvular surgery J Heart Valve Dis 2006;15:219-224.[Medline]
37. Xu J, Ge Y, Hu S, Song Y, Sun H, Liu P. A simple predictive model of prolonged intensive care unit stay after surgery for acquired heart valve disease J Heart Valve Dis 2007;16:109-115.[Medline]
38. Parolari A, Pesce LL, Trezzi M, et al. Performance of EuroSCORE in CABG and off-pump coronary artery bypass grafting: single institution experience and meta-analysis Eur Heart J 2009;30:297-304.
This article has been cited by other articles:
![]() |
M. Thomas, G. Schymik, T. Walther, D. Himbert, T. Lefevre, H. Treede, H. Eggebrecht, P. Rubino, A. Colombo, R. Lange, et al. One-Year Outcomes of Cohort 1 in the Edwards SAPIEN Aortic Bioprosthesis European Outcome (SOURCE) Registry: The European Registry of Transcatheter Aortic Valve Implantation Using the Edwards SAPIEN Valve Circulation, July 26, 2011; 124(4): 425 - 433. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. V. Ariyaratne, B. Billah, C.-H. Yap, D. Dinh, J. A. Smith, G. C. Shardey, and C. M. Reid An Australian risk prediction model for determining early mortality following aortic valve replacement Eur J Cardiothorac Surg, June 1, 2011; 39(6): 815 - 821. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Rosenhek, B. Iung, P. Tornos, M. J. Antunes, B. D. Prendergast, C. M. Otto, A. P. Kappetein, J. Stepinska, J. J. Kaden, C. K. Naber, et al. ESC Working Group on Valvular Heart Disease Position Paper: assessing the risk of interventions in patients with valvular heart disease Eur. Heart J., March 15, 2011; (2011) ehr061v1. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |