HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Kenneth M. Taylor Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Omar, R. Z. Articles by Taylor, K. M. Search for Related Content PubMed PubMed Citation Articles by Omar, R. Z. Articles by Taylor, K. M. Related Collections Cardiac - other

Ann Thorac Surg 2004;77:2232-2237
© 2004 The Society of Thoracic Surgeons

---

Review

Cardiac surgery risk modeling for mortality: a review of current practice and suggestions for improvement

^a Department of Statistical Science, University College London, London, United Kingdom
^b Medical Statistics Unit, Research and Development Directorate, University College London Hospitals NHS Trust, London, United Kingdom
^c MRC Clinical Trials Unit London, London, United Kingdom
^d Department of Cardiac Surgery, British Heart Foundation, Hammersmith Hospital, Imperial College London Faculty of Medicine, London, United Kingdom

^* Address reprint requests to Dr Taylor, Department of Cardiac Surgery, Faculty of Medicine, Imperial College, Hammersmith Hospital, Du Cane Rd, London W12 ONN, UK.
e-mail: k.m.taylor{at}imperial.ac.uk

	Abstract

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 
Risk models play a vital role in monitoring health care performances. Despite extensive research and widespread use of risk models in cardiac surgery, there are methodologic problems. We reviewed the methodology used for risk models for short-term mortality. The findings suggest that many risk models are developed in an ad hoc manner. Important aspects such as selection of risk factors, handling of missing values, and size of the data used for model development are not dealt with adequately. Methodologic details presented in publications are often sparse and unclear. Model development and validation processes are not always linked to the clinical aim of the model, which may affect their clinical validity. We make some suggestions in this review for improvement in methodology and reporting.

	Introduction

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 
Clinical risk models are currently playing a vital role in monitoring healthcare performances. In cardiac surgery short-term mortality is frequently used as a measure of performance. Over the last decade several risk models such as Parsonnet, Euroscore, STS, and UK Bayes have been proposed to predict short-term mortality after accounting for differences in clinical case mix [1–21]. Despite extensive research and widespread use of risk models in cardiac surgery, there are methodologic problems. Many risk models were developed in an ad hoc manner. Some model developers provide sparse details about their methods, thus making it difficult to ascertain what was really done. Moreover, different conclusions may be reached depending on the risk model used. Therefore, it is important to ensure that risk modeling is carried out in a correct and systematic manner and that robust and accurate models are developed.

See page 1960

To fit a statistical model using a computer software is relatively straightforward. However, it is just one part of the modeling process and may not produce a model that is reliable and valid for clinical use. There are several interlinked steps in a modeling process and each of these needs to be performed in a systematic manner. For example, lack of clarity in the specification of clinical aims of the model may result in the development of a model that is not useful in clinical practice. There are many factors associated with the risk of mortality and deciding which factors to include in a model is difficult. Several strategies are available to select risk factors [22] and these may produce risk models with different sets of risk factors and different estimates of the effect of a given factor for the same patient sample. Some estimates will be more reliable than others. Presence of missing data may pose additional problems. The performance of a model should be evaluated in the light of the specified aims [23].

The objective of this study is to review the methodology used in risk modeling in cardiac surgery for mortality, highlight the weaknesses and provide some suggestions for improvement.

	Material and methods

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 
We reviewed published articles and reports on cardiac surgery that focused on at least one of the following topics:

• Development of a risk model to predict short-term (inhospital or 30-day postoperative) mortality following surgery;
  
• Updating a previously developed risk model for short-term mortality;
  
• Presentation of methodology for modeling short-term mortality illustrated with data.
  

1. Electronic databases (Medline and EMBASE);
   
2. Citation search;
   
3. Reference lists of articles found through the database searches;
   
4. Conference proceedings.
   

We excluded articles where the main purpose was to use a statistical model for adjustment for case mix and not the presentation of a risk model for prediction of short-term mortality. Articles where the primary aim was to evaluate the performance of risk models proposed by other researchers were also excluded. We developed a checklist for the review to enable us to make an objective assessment of the methodology. This was based on what we consider to be the minimum criteria for a systematic approach to risk modeling as described in Figures 1 and 2. The modeling aspects for each of the reports were independently reviewed by two of the authors (RO and GA). Where there was a discrepancy between the two reviewers the reports were reexamined by a third reviewer (JE) and the issues were resolved through discussion.

View larger version (23K): [in this window] [in a new window]   Fig 1. Steps involved in model development.

View larger version (39K): [in this window] [in a new window]   Fig 2. Steps involved in model validation.

	Results and comment

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

Model development
Clinical aim of the model
Stating clear clinical aims for the model is a vital part of the modeling process as it may influence the number and choice of risk factors included in the model (see below) and to determine how the model will be used in practice. Examples include whether the model will be used to compare surgical performances across institutions in the same country, between different countries, for patient advice or treatment management, or for all of these purposes [24]. Four articles do not specifically state any clinical aims for the risk model. The most frequently stated aim was the facilitation of fairer comparisons between institutions and surgeons to assess quality of care. Improving information available to clinicians and patients for patient advice was also mentioned.

Choice of potential risk factors for mortality
It is important that developers prepare a list of risk factors according to the specified clinical aim of the model. There needs to be a balance between the degree of complexity required for the model in terms of selection of risk factors and what can be measured in practice. For example, if the aim is institutional comparisons a model developed to predict the overall number of deaths in an institution would probably be adequate and it may not need to be as complex as a model where the aim is patient counseling. For the latter, one may wish to predict deaths within mortality based clinical risk groups and more preoperative risk factors may need to be considered. If the aim is to compare performance of surgeons within an institution, and it may be that experienced surgeons operate on more seriously ill patients, a more detailed representation of the clinical risk factors in the model is necessary. If the model's intended use is to compare performances between countries/states of a specific region that share similar demographics, socioeconomic characteristics, and health care systems (such as Western Europe/USA), it will be applied to a very large group of patients. Because the differences in patient characteristics may average out, a model with a few key risk factors that could be measured reliably and easily in a standardized manner may be adequate for this purpose. A complex model may be impractical and unnecessary.

Of course a single complicated model may satisfy all of these roles but it could prove to be difficult to develop or use in practice, particularly if information on all risk factors in the model is not collected in many of the institutions for which the model is intended. Risk factors that are rarely measured in clinical practice should only be included if the model is intended for use in specific institutions where information is collected on these factors. It is not clear whether any of the model developers selected potential risk factors with reference to their stated aims. Furthermore, the developers did not always provide the number or a list of risk factors initially chosen for examination in the model. This information should be presented as it influences the required size of the model development data set (see next section).

Size of model development data
Model developers should clearly state the number of patients (before and after exclusions) and the percentage overall mortality or the number of deaths and ensure that an adequate number of patients are available to carry out the modeling process. As a rough rule for calculating the sample size required for the developmental data, the number of deaths observed in the data should be at least ten times the number of risk factors examined for inclusion in the model [25, 26]. A risk factor with k categories contributes k–1 terms to the model. Therefore to contemplate a 15 risk factor model, assuming an inhospital or 30-day mortality rate of 3%, one would require at least 5000 patients to achieve adequate sample size. Otherwise spurious associations between the risk factors and outcome may be obtained, or the estimates of association may lack precision [25, 26]. Using this criterion, the developmental data were found to be of inadequate size for seven atricles and five reports did not provide sufficient details for assessment.

Handling of missing values
Model developers should clearly state the extent of missing observations in their data and how these were dealt with in the modeling process. Before fitting the model it is important to determine if there are systematic differences in the characteristics of patients with missing risk factors as this could introduce bias. If the extent of missing data is large, one should consider appropriate substitution of missing values and examine whether the results remain consistent. A large number of methods are available for substitution and a method should be selected with care in advance (before fitting the model) depending on the nature of the problem [27]. In ten of the papers, it is unclear how the model developers dealt with risk factors with missing values. Seven developers used only patient records that had complete information on all selected risk factors. However, it was not always clear whether the extent of bias introduced through the presence of missing data had been investigated.

Strategy used to select a final set of risk factors in a model
Sixteen developers used logistic regression, four used a Bayesian modeling technique, and one used artificial neural networks (ANN) to fit their final risk model.

Generally the variable selection strategies used by the model developers to produce their final model fall into three categories:

1. A full model approach whereby all risk factors are retained in the model regardless of their statistical significance. The disadvantage is that the model may be large and complicated, particularly if it is developed using many potential risk factors. In practice it is often difficult or expensive to measure several risk factors reliably.
   
2. A univariate analysis where the relationship between each individual risk factor and mortality is investigated separately. It makes the crucial assumption that each factor contributes independent information about the chance of mortality, which holds only if the risk factors are uncorrelated with each other and that is somewhat unlikely. If two factors convey similar information the risk of mortality may be over estimated.
   
3. Variable selection strategies such as backward elimination, forward selection, and stepwise approaches [22]. Some developers used univariate analysis as an initial screening tool to select risk factors on the basis of their statistical significance and then adopted one of these variable selection strategies to develop their final model. Univariate analysis based on statistical significance suffers from the problem of omitting risk factors that are not statistically significant individually, but achieve significance when other factors are included in the model. The other variable selection strategies also rely on statistical significance. The problem with selection based on statistical significance is that the model may be unstable with respect to the set of risk factors finally selected. There may be many models with different combinations of risk factors for the same data which fit equally well. There may also be problems with the selection of correlated risk factors, leading to an arbitrary choice among important risk factors. Statistical methods, such as resampling techniques, are available to identify risk factors that have stability and should be used in conjunction with these variable selection strategies [28]. Only one study briefly addressed the issue of model stability, but no clear description of the method used was given. In four articles the strategy used for selecting the final set of risk factors was not clearly described.
   

Construction of risk scores
Typically, risk models produce coefficients for each risk factor in the final model that represent their weights in predicting mortality. Nine model developers converted the coefficients to integer risk scores and derived an additive risk scoring system for simplicity in clinical use. Although this step can be performed with an appropriate mathematical transformation, the four developers that describe how this step was performed used a rather arbitrary and inappropriate conversion process. Scaling factors for converting coefficients to integer scores should be chosen carefully otherwise the predictive power of a model could be severely weakened. There are methods available to translate coefficients into integer scores with minimal loss of precision [29]. The process of construction of risk scores should be clearly described by the developers, or referenced.

Model validation
It is essential that a model's performance should be validated in the light of the specified clinical aims. For example, if a model aims to predict overall number of surgical deaths for institutions it should be tested how well the model can achieve this. If it has been developed to predict deaths within clinical risk groups the model should be evaluated in that context.

Type of validation
Typically, a model's performance is first evaluated on the developmental data. This is statistically termed as "goodness of fit" and is the weakest form of validation. We do not consider it further in this paper. Evaluation of a model's performance on data not used for its development is more important. It can be done using internal or external validation approaches (illustrated in Fig 2).

Ten developers carried out an internal validation using the data splitting method. In using this approach it is essential that the data are split into the model development and evaluation parts before building the model. The problems are that the size of data are reduced and the split is by definition fortuitous. With a different, equally valid split, different assessments of predictive accuracy may be obtained. This method is best suited to situations in which data on tens of thousands of patients are available. A resampling technique may be used to avoid the above mentioned problems [22].

Internal validation does not address the wider issue of the generalizability of the model. External validation on a large, completely independent test dataset is the most stringent type of validation and is an important part of the entire modeling process [23]. Model development cannot be considered complete until such evaluation has been performed. Of course, this may raise practical difficulties in obtaining suitable data. Only one article used completely independent data for validation. A less stringent approach of external validation, superior to internal validation, is to use a subsequent sample of patients from the same center or centers (temporal validation). This approach was used by four model developers. A validation process was not carried out by four of the developers, and for two it was unclear whether this step was performed.

Aspects of model validation
Calibration refers to a model's ability to predict mortality accurately. The Hosmer-Lemeshow (H-L) test was most commonly used to assess model calibration [30]. To evaluate the model, the finally selected model is applied to the test data. The H-L test compares the observed number of deaths to those predicted by the model typically in 10 equally sized risk groups (deciles). If the model is well calibrated, the observed proportion of deaths in each of the 10 groups will tend to agree with the average predicted probability of death in that risk group. If the p value from an H-L test is greater than 0.05, a current practice of the developers is to claim that the model predicts mortality accurately.

The H-L test is useful when comparing observed and predicted deaths within risk groups. However, we would like to point out some of the problems that are associated with the current use of H-L test. The traditional decile-based grouping [31] only assesses the statistical validity of the model and it may be more appropriate to use clinical risk groups. Clinical risk groups may be constructed by grouping patients into low, intermediate, and high risks of mortality or any other grouping that is considered clinically useful. However, the H-L test may be sensitive to the choice of risk groups used [30]. A particular choice may indicate good calibration, whereas another may suggest the opposite. Therefore, the clinical risk groups should be specified in advance, before starting the validation process, to avoid reaching any data driven conclusions. To quote only a p value from the H-L test is not sufficient. One may conclude from a statistically significant result that the model makes inaccurate predictions but it could still perform well for specific groups such as the low to intermediate risk groups of patients which would be the majority. A statistically nonsignificant result suggests that the model predicts accurately on average, but it may nevertheless perform badly for a particular group of patients such as a high risk group. When the validation data are large, small differences between observed and predicted numbers of deaths may come out as statistically significant although the differences are not clinically important. More insight into model "hot spots" may be gained by combining information from the H-L test with a table or a graph of observed and predicted deaths.

Discrimination is the ability of a model to distinguish patients who die from those who survive. The area under the receiver operating characteristic (ROC) curve also known as the C-index is commonly used to assess this aspect of model performance [32]. To calculate the ROC area, all possible pairs of patients in the validation data with different outcomes (died or survived) are considered. For a given pair, the predicted probability of death should ideally be higher for the patient who died than for the one who survived. The ROC area is the percentage of pairs for which this is true. For example a ROC area of 0.75 means the model correctly ranks 75% of the patient pairs according to their predicted probability. A current practice in cardiac surgery is to conclude that a model discriminates well if the ROC area is greater than 0.7.

The closer the ROC area is to 1 the greater the discriminatory ability of a model. However, the ROC measures the average discriminatory ability and is not very sensitive as it depends only on the ranks of the prediction and not the actual value [32]. Therefore, ROC results presented on their own do not provide the whole picture of a model's performance. It is possible to achieve a ROC area more than 0.7 even though the probabilities of death for all patients predicted by the model lie within a narrow range (for example: 0% to 3%) which may not be clinically useful [33]. It has been suggested that risk models with ROC areas above 0.8 have some clinical utility [22]. However, this cut-off is arbitrary and cannot be directly related to the intended use of a model.

At present the calibration and discrimination validation exercise appears to be performed routinely with little regard for the original clinical aims of the model. A typical conclusion is "the model is validated since p more than 0.05 and the ROC area more than 0.7." In two of the papers reviewed the model was considered valid on the basis of the ROC area, although the H-L test indicated poor calibration. Because the clinical aim of the model was not stated it is not possible to ascertain whether the model was clinically valid. One study used results from a ROC curve to conclude that a simple model with fewer numbers of risk factors was adequate for clinical use compared with more complex models without investigating the calibration of the models.

Depending on the aims, the validation process should take in the total picture: the ability of a model to predict mortality accurately, the range of predictions (whether these are clinically useful or not), and the ability to discriminate between high, intermediate, and low risk patients. If predictions are used to identify surgical centers or surgeons with unexpectedly high or low rates, achieving a high ROC area alone is not adequate. It is important for the model to achieve good calibration. A poorly calibrated model may cause large numbers of institutions or surgeons to reveal excessively high or low rates of mortality when in fact the fault lies with the model, not the clinical performance. If the calibration tests demonstrate poor results it may be possible to recalibrate the model. However recalibration is not straightforward and may require more than 1 set of data to obtain a reliable model, particularly if the aim is to apply the model to several different institutions. If a model is recalibrated, the recalibrated model should be presented for use. If predictions are used to stratify patients by disease severity in order to compare treatments or to decide on patient management, both calibration and discrimination aspects are important. If the purpose of the model is only to rank patients with respect to the risk of dying, then calibration is not an issue and assessing discrimination only is adequate. This situation is rather rare. Researchers should clearly specify the clinical aims of the model and draw conclusions from their validation process in relation to their stated aims.

Size of data for validation
The validation dataset should be large enough to enable precise comparison between observed and predicted outcomes and to endow statistical methods such as the H-L test with sufficient power. For the H-L test to be valid, the predicted number of events in each risk group used in the test should always be greater than 1, and for most risk groups it should be at least 5 [30]. It has been suggested as a general principle that adequate model evaluation requires at least 100 deaths in the validation sample [22]. Assuming a short-term mortality rate of 3% in cardiac surgery, 3300 patients would typically be needed to observe 100 deaths. If the size of the validation data is small the power of the statistical methods used to assess calibration may be reduced. The size of the data was small in 4 of the 16 papers in which a validation was performed.

Other aspects
Shahian and coworkers [34] have suggested the use of hierarchical models in cardiac surgery risk modeling. Statistical models, such as hierarchical models, have been developed for data that are clustered in nature. Patients within a cardiac center are likely to be more similar than patients across centers, due to similarity in treatments and perhaps socioeconomic characteristics, and may be considered as clustered within a center. Incorrect statistical inferences may be obtained if clustering is ignored in the analysis [35, 36]. These models are appropriate when data from a large number of centers are used for model development and not necessary when a model is developed using data from a single center. They may provide imprecise estimates when the data are from a small number of centers. Although some model developers have used data from a large number of centers for model development, they have not taken into account the clustering aspect.

None of the models reviewed in this study include interaction terms. These terms are important when there is prior belief on the basis of clinical experience that the effect of a risk factor on mortality may vary according to the levels of another risk factor, for example gender. If interaction terms are considered for inclusion in a risk model, they should be chosen on the basis of clinical plausibility as well as statistical significance. The size of the developmental data would need to be increased to accommodate these terms and interpretations made cautiously.

	Conclusions

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 
Risk models play an important role in health care policy. For a risk model to be used routinely in practice, the modeling methodology should be correct and robust and the proposed model must be straightforward to implement and clinically relevant. To evaluate the current cardiac risk modeling methodology for mortality, a review of papers/reports focusing on cardiac surgery risk modeling for short-term mortality was undertaken. The review suggests that there is scope for improvement in the methodology currently used to develop and validate risk models in cardiac surgery. It is important that researchers implement a structured and transparent model building process linked to the stated clinical aim, and evaluate the model's performance in the context for which it has been developed. Researchers should develop a protocol before starting a modeling process, describing how each step will be handled, and strictly adhere to this protocol. Clearer descriptions of each step of the process are required in published papers and reports. Protocols may be placed on websites.

	Acknowledgments

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 
This work was supported by a grant from the Garfield Weston Trust. We thank the editor and the referees for helpful suggestions.

	References

Top Abstract Introduction Material and methods Results and comment Conclusions Acknowledgments References

 

1. Edwards F.H., Albus R.A., Zajtchuk R., et al. Use of Bayesian statistical model for risk assessment in coronary artery surgery. Ann Thorac Surg 1988;45:437-440.[Abstract]
2. Parsonnet V., Dean D., Bernstein A.D. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79(Suppl I):I-3-I-12.
3. Higgins T.L., Estafanous F.G., Loop F.D., Beck G.J., Blum J.M., Paranandi L. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients. JAMA 1992;267:2344-2348.[Abstract/Free Full Text]
4. O'Connor G.T., Plume S.K., Olmstead E.M., et al. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Circulation 1992;85:2110-2118.[Abstract/Free Full Text]
5. Marshall G., Shroyer A.L.W., Grover F.L., Hammermeister K.E. Bayesian-logit. Model for risk assessment in coronary artery bypass grafting. Ann Thorac Surg 1994;57:1492-1500.[Abstract]
6. Hannan E.L., Kilburn H., Racz M., Shields E., Chassin M.R. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 1994;271:761-766.[Abstract/Free Full Text]
7. Tu J.V., Jaglal S.B., Naylor C.D., Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Circulation 1995;91:677-684.[Abstract/Free Full Text]
8. Tu J.V., Sykora K., David Naylor C. Assessing the outcomes of coronary artery bypass graft surgery: how many risk factors are enough?. J Am Coll Cardiol 1997;30:1317-1323.[Abstract]
9. Pons J.M.V., Granados A., Espinas J.A., Borras J.M., Martin I., Moreno V. Assessing open heart surgery mortality in Catalonia (Spain) through a predictive risk model. Eur J Cardiothoracic Surg 1997;11:415-423.[Abstract]
10. Lippmann R.P., Shahian M.D. Coronary artery bypass risk prediction using neural networks. Ann Thorac Surg 1997;63:1635-1643.[Abstract/Free Full Text]
11. Edwards F.H., Grover F.L., Shroyer A.L.W., Schwartz B.S., Bero J. The society of thoracic surgeons national cardiac surgery database: current risk assessment. Ann Thorac Surg 1997;63:903-908.[Abstract/Free Full Text]
12. Shroyer A.L.W., Grover F.L., Edwards F.H. 1995 coronary artery bypass risk model: The Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1998;65:879-884.[Abstract/Free Full Text]
13. Shroyer A.L.W., Plomondon M.E., Grover F.L., Edwards F.H. The 1996 coronary artery bypass risk model: The Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1999;67:1205-1208.[Abstract/Free Full Text]
14. Mozes B., Olmer L., Galai N.S., ISCAB Consortium. A national study of postoperative mortality associated with coronary artery bypass grafting in Israel. Ann Thorac Surg 1998;66:1254-1263.[Abstract/Free Full Text]
15. Keogh BE, Kinsman R. National Adult Cardiac Surgical Database Report. The Society of Cardiothoracic Surgeons of Great Britain and Ireland. Playhatch, Berkshire, UK: Dendrite Clinical Systems Ltd., 1998
16. Keogh BE, Kinsman R. National Adult Cardiac Surgical Database Report 1999–2000. The Society of Cardiothoracic Surgeons of Great Britain and Ireland. Playhatch, Berkshire, UK: Dendrite Clinical Systems Ltd., 2000
17. Nashef S.A.M., Roques F., Michel P., Gauducheau S., Lemeshow S., Salamon R. European system for cardiac operatic risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9-13.[Abstract/Free Full Text]
18. Eagle K.A., Guyton R.A., Davidoff R., et al. ACC/AHA guidelines for coronary artery bypass graft surgery: executive summary and recommendations. Circulation 1999;100:1464-1480.[Free Full Text]
19. Ivanov J., Tu J.V., Naylor C.D. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 1999;99:2098-2104.[Abstract/Free Full Text]
20. Bernstein A.D., Parsonnet V. Bedside estimation of risk as an aid for decision-making in cardiac surgery. Ann Thorac Surg 2000;69:823-828.[Abstract/Free Full Text]
21. Zaroff J.G., diTommaso D.G., Barron H.V., investigators in the National Registry of Myocardial Infarction 2. A risk model derived from the National Registry of Myocardial Infarction 2 Database for mortality after coronary artery bypass grafting during acute myocardial infarction. Am J Cardiol 2002;90:1-4.[Medline]
22. Harrell F.E. Regression modeling strategies. . Berlin: Springer-Verlag, 2001.
23. Altman D.G., Royston P. What do we mean by validating a prognostic model?. Stat Med 2000;19:453-473.[Medline]
24. Omar RZ, Ambler G, Taylor KM. Assessment of quality of care in cardiac surgery: an overview of risk models. In Mohan R, ed. Recent advances in cardiology 2. Kerala, India: Global Research Network, 2003:13–20
25. Concato J., Feinstein A.R., Holford T.R. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201-210.[Abstract/Free Full Text]
26. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373–9
27. Little R.J.A., Rubin D.B. Statistical analysis with missing data, 2nd edition New York: Wiley, 2002.
28. Sauerbrei W., Schumacher M. A bootstrap resampling procedure for model building: application to the Cox regression model. Stat Med 1992;11:2093-2109.[Medline]
29. Cole T.J. Statistical algorithms: scaling and rounding of regression coefficients to integers. J R Statist Soc C 1993;42:261-269.
30. Hosmer D.W., Lemeshow S. Applied logistic regression. . New York: Wiley, 1989.
31. Asimakopoulos G, Al-Ruzzeh S, Ambler G, et al. An evaluation of existing risk stratification models as a tool for comparison of surgical performances for CABG between institutions. Eur J Cardiothoracic Surg 2003;23:935–42
32. Ash AS, Shwartz M. Evaluating the performance of risk adjustment methods: dichotomous outcomes. In Iezzoni L, ed. Risk adjustment for measuring healthcare outcomes (2nd edition). Chicago: Health Administration Press, 1994:427–69
33. Al-Ruzzeh S., Asimakopoulos G., Ambler G., et al. Validation of four different risk stratification systems in patients undergoing off-pump coronary bypass surgery (OPCAB): a UK multi-centre analysis of 2,223 patients. Heart 2003;89:432-435.[Abstract/Free Full Text]
34. Shahian D.M., Normand S., Torchiana D.F., et al. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
35. Hu F.B., Goldberg J., Hedeker D., Flay B.R., Pentz M.A. Comparison of population-averaged and subject specific approaches for analyzing repeated binary outcomes. Am J Epidemiol 1998;147:694-703.[Abstract/Free Full Text]
36. Omar R.Z., Thompson S.G. Analysis of a cluster randomized trial with binary outcome data using a multi-level model. Stat Med 2000;19:2675-2688.[Medline]

This article has been cited by other articles:

				V. C. Carosella, J. L. Navia, S. Al-Ruzzeh, H. Grancelli, W. Rodriguez, C. Cardenas, J. Bilbao, and C. Nojek The first Latin-American risk stratification system for cardiac surgery: can be used as a graphic pocket-card score Interactive CardioVascular and Thoracic Surgery, August 1, 2009; 9(2): 203 - 208. [Abstract] [Full Text] [PDF]

				P. E. Antunes, J. F. de Oliveira, and M. J. Antunes Risk-prediction for postoperative major morbidity in coronary surgery Eur. J. Cardiothorac. Surg., May 1, 2009; 35(5): 760 - 767. [Abstract] [Full Text] [PDF]

				A. Parolari, L. L. Pesce, M. Trezzi, C. Loardi, S. Kassem, C. Brambillasca, B. Miguel, E. Tremoli, P. Biglioli, and F. Alamanni Performance of EuroSCORE in CABG and off-pump coronary artery bypass grafting: single institution experience and meta-analysis Eur. Heart J., February 1, 2009; 30(3): 297 - 304. [Abstract] [Full Text] [PDF]

				J. P. Dal-Bianco, B. K. Khandheria, F. Mookadam, F. Gentile, and P. P. Sengupta Management of Asymptomatic Severe Aortic Stenosis J. Am. Coll. Cardiol., October 14, 2008; 52(16): 1279 - 1292. [Abstract] [Full Text] [PDF]

				P. E. Antunes, L. Eugenio, J. Ferrao de Oliveira, and M. J. Antunes Mortality risk prediction in coronary surgery: a locally developed model outperforms external risk models Interactive CardioVascular and Thoracic Surgery, August 1, 2007; 6(4): 437 - 441. [Abstract] [Full Text] [PDF]

				R. B.M. Hosein, A. J.B. Clarke, S. P. McGuirk, M. Griselli, O. Stumper, J. V. De Giovanni, D. J. Barron, and W. J. Brawn Factors influencing early and late outcome following the Fontan procedure in the current era. The 'Two Commandments'? Eur. J. Cardiothorac. Surg., March 1, 2007; 31(3): 344 - 353. [Abstract] [Full Text] [PDF]

				J. Nilsson, M. Ohlsson, L. Thulin, P. Hoglund, S. A.M. Nashef, and J. Brandt Risk factor identification and mortality prediction in cardiac surgery using artificial neural networks J. Thorac. Cardiovasc. Surg., July 1, 2006; 132(1): 12 - 19. [Abstract] [Full Text] [PDF]

				L. D. Berrizbeitia Invited commentary Ann. Thorac. Surg., June 1, 2006; 81(6): 2088 - 2088. [Full Text] [PDF]

				B. Zingone, A. Pappalardo, and L. Dreas Logistic versus additive EuroSCORE. A comparative assessment of the two models in an independent population sample Eur. J. Cardiothorac. Surg., December 1, 2004; 26(6): 1134 - 1140. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Kenneth M. Taylor Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Omar, R. Z. Articles by Taylor, K. M. Search for Related Content PubMed PubMed Citation Articles by Omar, R. Z. Articles by Taylor, K. M. Related Collections Cardiac - other