HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Ruyun Jin Gary L. Grunkemeier Albert Starr Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Jin, R. Search for Related Content PubMed PubMed Citation Articles by Jin, R.

Ann Thorac Surg 2005;80:471-479
© 2005 The Society of Thoracic Surgeons

---

Original article: Cardiovascular

Validation and Refinement of Mortality Risk Models for Heart Valve Surgery

Providence Health System, Portland, Oregon

Accepted for publication February 23, 2005.

^_* Address reprint requests to Dr Jin, Providence St. Vincent Hospital and Medical Center, 9205 SW Barnes Rd, LL 33, Portland, OR97225 (Email: ruyun.jin{at}providence.org).

	Abstract

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
BACKGROUND: The Northern New England Cardiovascular Disease Study Group (NNE) recently published risk models for hospital mortality after heart valve surgery. The Providence Health System Cardiovascular Study Group (PHS) has been collecting similar heart valve data for 8 years, providing an ideal opportunity to both validate the NNE risk models and attempt to produce an improved model, by using some different modeling techniques.

METHODS: From 1997 to 2004, 3,324 patients aged 30 to 95 years underwent aortic valve replacement (AVR), and 1,596 underwent mitral valve replacement or repair (MVRR) at one of nine PHS medical centers. We used area under the receiver operating characteristic curve (c-index) to measure model discrimination, and Hosmer-Lemeshow statistic (H-L) to measure calibration. We modified the NNE models by ungrouping continuous variables, seeking optimal transformations of continuous variables, and imputing missing values by multiple regression.

RESULTS: The prevalence and the lethality of risk factors were similar in PHS and NNE patients. The NNE models fit PHS patients well: c-index (95% confidence interval) = 0.75 (0.70 to 0.80) for AVR and 0.81 (0.76 to 0.86) for MVRR; H-L = 3.95 (p = 0.861) for AVR and 7.10 (p = 0.526) for MVRR. A single PHS model performed slightly better for both positions: c-index = 0.79 (0.75 to 0.83) for AVR and 0.84 (0.80 to 0.88) for MVRR; H-L = 2.75 (p = 0.949) for AVR and 12.21 (p = 0.142) for MVRR.

CONCLUSIONS: The NNE models for aortic and mitral valve surgery were successfully validated using PHS patients. Using some different statistical approaches to modeling, we produced a new, unified model for both positions.

	Introduction

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
The Northern New England Cardiovascular Disease Study Group (NNE) consists of eight Northeastern state medical centers. The NNE recently published risk models for heart valve surgery derived from 5,793 aortic valve replacement (AVR) and 3,150 mitral valve replacement or repair (MVRR) operations performed during 1991 to 2001 [1]. The NNE authors acknowledged the importance of validating clinical risk models: "Applicability of these prediction models to future cases is the true test of their validity."

The Providence Health System Cardiovascular Study Group (PHS) comprises nine medical centers in four Western states. PHS collected data on 4,920 single aortic or mitral valve operations during 1997 to 2004, providing an opportunity to evaluate the NNE risk models with external data. Specifically, we addressed the following questions: (1) Are NNE and PHS patients similar with regard to the distributions of risk factors? (2) Are the risk factor effects on mortality similar between the two patient groups? (3) Do the NNE models perform well at predicting mortality for the PHS patients? (4) Can we refine the NNE models by applying some different statistical approaches?

	Material and Methods

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
Clinical Material
All PHS hospitals with cardiac surgery programs collect data according to standardized definitions and transmit it quarterly to a central coordinating center for merging into a common database. From January 1997 through March 2004, 6,578 PHS patients underwent heart valve surgery. Among them, 4,920 cases met the NNE criteria [1]: isolated AVR (n = 3,324) or MVRR (n = 1,596) with or without concomitant coronary artery bypass graft (CABG) surgery in patients between 30 and 95 years of age.

The PHS data definitions matched all but two of the NNE risk factor definitions, and we used somewhat expanded versions of those two variables. We used history of cerebrovascular disease (includes permanent, reversible, and transient events) as a substitute for prior stroke and atrial arrhythmia (includes flutter and others) as a substitute for atrial fibrillation.

Statistical Methods
Evaluating risk models
We used traditional methods to evaluate the performance of the risk models, which were developed by logistic regression [2]. We assessed discrimination by the c-index (area under the receiver operating characteristic curve) [3], and calibrated by the Hosmer-Lemeshow statistic [4] and cumulative sum analysis (Cusum) [5].

Discrimination is more critical than calibration, because one can always recalibrate a model without changing its discrimination [6]. We used a simple recalibration to make overall expected mortality equal observed mortality [7, 8]. This is done by performing a new logistic regression with a single risk factor—the risk score from the NNE model—and using the resulting mortality predictions.

Developing risk models
Many decisions are made in producing a risk model [9]. We revisited some of the decisions that NNE made, and applied some additional statistical methods.

1 In general, it is preferable to keep continuous risk factors (age, body surface area [BSA], and serum creatinine) as continuous, rather than to discard information by collapsing them into groups, as the effect of these factors on the outcome usually varies smoothly, rather than in jumps [10].

2 When an outcome variable is continuous, its relationship with a continuous predictor is natural to depict, using a scatter diagram. When the outcome variable is dichotomous (dead/alive), though, it is difficult to portray this relationship graphically. Cusum techniques can aid in visualizing, and evaluating, the effects of continuous risk factors [5].

3 Transformations of continuous variables can sometimes improve a predictive model; we used an organized methodology for seeking such transformations, called fractional polynomials [11].

4 There are many options available for imputing missing values. The NNE used a missing value indicator technique, and we used a multiple regression technique [12].

5 Reviewing published models from other large series [1, 13–21] can suggest risk factors that should be considered in the model (Table 1).

6 Finally, can we combine the risk models for the two positions (AVR, MVRR) into a single risk model? The advantage is simplicity, and larger numbers available for estimating the model coefficients.

	Results

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
Of the four questions posed at the beginning of this paper, the first three were easy to answer directly and the fourth question involved many steps.

First, are NNE and PHS patients similar with regard to the distributions of risk factors? The NNE had 5,793 AVR and 3,150 MVRR operations from 1991 to 2001. The PHS had 3,324 AVR and 1,596 MVRR operations from 1997 to 2004. The distribution of the patient risk factors were similar for AVR (Fig 1) and MVRR (Fig 2), except that NNE had more patients in New York Heart Association classes 3 and 4, and more urgent or emergent operations, whereas PHS had more patients with elevated preoperative serum creatinine.

View larger version (76K): [in this window] [in a new window]   Fig 1. Comparison of Northern New England Cardiovascular Disease Study Group (NNE) and Providence Health System Cardiovascular Study Group (PHS) patient characteristics for aortic valve replacement (AVR). Listed are 11 variables which define the 18 risk factors for mortality in the NNE risk model for AVR. *NNE and PHS use slightly different definitions for these risk factors, see text. (CABG = coronary artery bypass graft; CAD = coronary artery disease; CHF = congestive heart failure; Cr = creatinine; E = emergent; NYHA = New York Heart Association functional class.)

View larger version (77K): [in this window] [in a new window]   Fig 2. Comparison of Northern New England Cardiovascular Disease Study Group (NNE) and Providence Health System Cardiovascular Study Group (PHS) patient characteristics for mitral valve surgery (replacement or repair [MVRR]). Listed are 11 variables which define the 16 risk factors for mortality in the NNE risk model for MVRR. (CAD = coronary artery disease; CHF = congestive heart failure; Cr = creatinine; E = emergent; NYHA = New York Heart Association functional class.)

View larger version (29K): [in this window] [in a new window]   Fig 3. Comparison of Northern New England Cardiovascular Disease Study Group (NNE) and Providence Health System Cardiovascular Study Group (PHS) multivariable odds ratios for aortic valve replacement (AVR). The odds ratios (with 95% confidence intervals) are given for each risk factor in the NNE risk model for AVR mortality. (BSA = body surface area; CABG = coronary artery bypass graft; CHF = congestive heart failure; Cr = creatinine; NYHA = New York Heart Association functional class.)

View larger version (26K): [in this window] [in a new window]   Fig 4. Comparison of Northern New England Cardiovascular Disease Study Group (NNE) and Providence Health System Cardiovascular Study Group (PHS) for mitral valve surgery (MVRR). The odds ratios (with 95% confidence intervals) are given for each risk factor in the NNE risk model for MVRR mortality. (CAD = coronary artery disease; CHF = congestive heart failure; Cr = creatinine; NYHA = New York Heart Association functional class.)

	Continuous variables

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
We examined the relationships between mortality and the continuous variables that NNE collapsed into groups (age, serum creatinine, BSA).

The mortality rate by age group is shown in Figure 5, with the univariate logistic curve providing a smooth, continuous fit. Figure 6 shows a multivariate depiction of this age/mortality relationship, using Cusum plots. The cumulative sums of observed minus expected deaths from the final PHS model, with and without age included, are superimposed. When age is removed from the model (gray line), the plot drops rather uniformly until about age 72, where there is an excess of about 42 expected deaths (compared with observed), after which it rises. That means that for younger patients there are fewer deaths than expected (from the model that does not have age in it), and for the older patients there are more deaths than expected from the (age-less) model. The Cusum test for a linear trend with age is significant (p < 0.001) [5]. When age is put back into the model (black line), the Cusum plot does not show a linear trend with age (p = 0.972), only random variation about the line of equality (observed = expected), with a maximum excursion of about 7 deaths.

View larger version (19K): [in this window] [in a new window]   Fig 5. Hospital mortality by patient age. The height of the circles indicate the observed mortality for patients grouped by decade of age. The area of the circles is proportional to the number of patients in the group. The solid line shows the mortality predicted by using age as a continuous variable in a (univariate) logistic model (logistic equation).

View larger version (28K): [in this window] [in a new window]   Fig 6. Plots of the cumulative sums (Cusum) of observed minus expected deaths, by patient age. The thicker gray line uses the risk predicted with age excluded from the model. The thinner black line uses the risk predicted by the full PHS model, with age included. The horizontal line is the line of identity, representing perfect agreement between observed and expected.

View larger version (25K): [in this window] [in a new window]   Fig 7. Histogram of the logarithm of serum creatinine, showing observed values (gray bars; n = 3,492) and imputed values (black bars; n = 1428).

	Other risk factors

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
We compared the difference of other risk factors. Peripheral vascular disease, pulmonary hypertension and chronic obstructive pulmonary disease were found significant (using our combined data, from both positions) and were included in the model. These were also used in other published models (Table 1). Calendar year of surgery was not significant in this relatively time-homogeneous sample of patients, either by testing year of surgery in the logistic regression model (p = 0.34) or by consideration of the day of surgery using the cumulative sum technique (p = 0.40 for linear trend and p = 0.84 for quadratic trend). Stroke, congestive heart failure, diabetes, and mitral valve replacement were not significant and were removed. That may be because the other variables compensate for the risk due to those variables. These, and many other variables, were found significant in univariable comparisons, but dropped out of the model as other variables were entered.

	Combining AVR and MVRR models

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
As the NNE risk factors for AVR (Fig 1) and MVRR (Fig 2) were similar, we combined both positions into one model with the risk factor "valve position" included. And, somewhat unexpectedly, this potential risk factor was not significant (odds ratio = 1.11, p = 0.526). Also, interaction terms for valve position and other selected risk factors were not significant.

	Final PHS model: Comparison with calibrated NNE model

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
Accordingly, a single model, incorporating all of the modeling and risk factor decisions described above, was fit to the PHS patient data from both positions (Table 3). There were 4,561 (93%) cases without any missing values in the model (after imputing creatinine). The mortality rate was 4.6% for the patients used in the model, compared with 4.7% for all patients. The comparison of the two NNE models with the single PHS model (evaluated for both positions) shows that both perform well (Table 2).

	Comment

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

Risk-adjusted mortality models provide an individualized probability of hospital death for each patient, to give the patient and the surgeon a reasonable expectation of the true risk. These expected risks can be aggregated across providers, or across time, and compared with the observed mortality to yield performance measures. The NNE documented 74 fewer CABG deaths than expected in the first 6 years of their quality improvement program [24] and 811 fewer deaths than expected in the first 15 years [25].

The NNE has been a model and inspiration for the Providence Health System Cardiovascular Study Group, which has been collecting similar data since 1997. One of the current PHS initiatives was to acquire a risk model for hospital mortality after heart valve surgery. The NNE risk models [1] presented an ideal opportunity.

A recent paper on validating prognostic models argues for "the need to evaluate performance of a model on a new series of patients, ideally in a different location," to provide an "empirical demonstration of transportability" [26]. The study by Ivanov and colleagues for CABG mortality risk model also conclude that "Any existing index used for risk assessment in cardiac surgery should be episodically recalibrated or compared with new models derived from local subjects to ensure that its performance remains optimal" [27]. The PHS data, collected 3,000 miles away, during an overlapping time frame, provide an excellent opportunity to perform such validation on the NNE risk models.

Thus, our intention was to validate the NNE models as a prelude to possibly adopting them for PHS use.

Derivation of the PHS Model
Although the NNE models worked well on the PHS data, we proceeded to modify them by applying some different statistical strategies (see Statistical Methods section), and the considering additional risk factors used in other heart valve risk models. Table 1 lists the risk factors used in valve surgery models with more than 1,000 patients and five or more risk factors [1, 13–21]. These minimum values were not arbitrary; a mortality rate of 5% to 10% implies 50 to 100 deaths, and 10 deaths are required to support each risk factor considered in a risk model [6]. A notable omission from this table is the EuroSCORE [28], but that model is not specifically for valve surgery.

The PHS model uses age and (the logarithm of) creatinine as continuous variables. But we could not find a transformation of BMI that performed as well as a dichotomy, with the middle range having lower risk. This agrees with our previous finding (Jin R, Grunkemeier GL, Furnary AP, Handy JR Jr. Is obesity a risk factor for mortality in coronary artery bypass surgery? Circulation. Accepted 2005) that BMI from 23 to 30 seems to be the most protective range for death in CABG surgery patients.

Imputing With Missing Values
There are many ways to deal with missing values 12]. Some studies make a heroic effort to complete the missing information, in order to include all cases in the model [18]. This is the best way, but impractical for many facilities. A common method is to discard patient records with incomplete information on any of the risk factors necessary for the model; but this wastes the information for those patients on variables that are not missing and may bias the results if the excluded patients are not representative of the others. Other methods include eliminating variables with a large amount of missing data [29]; imputing the missing values with the median for continuous variables and the most prevalent value for categorical variables [13, 15]; creating an indicator variable to code missing values [1, 17]; and assuming that missing information is equivalent to absence of the risk factor [30]. In this study, we imputed preoperative creatinine using values provided by a linear regression on the other risk factors (Fig 7). A drawback to this method is that the resulting confidence intervals are too conservative (narrow), because the imputed values are used as if real, and the imprecision in their estimation is not reflected in the model. (A more complex procedure, multiple imputation, would overcome this limitation [12].)

Combined Model for AVR and MVRR
This is a major step, to consider a single valve surgery model, not separated by position. A first reaction might be that the patients and their outcomes are too dissimilar to combine. But, in fact, published risk models for aortic and mitral valve mortality have similar risk factors (Table 1), and the most recent version of The Society of Thoracic Surgeons risk models for heart valve replacement combined positions [13].

Validation of the PHS Model
The PHS model fit our data slightly better than the calibrated NNE models (Table 2), which is not surprising as the PHS model was derived from the data upon which it was evaluated, thus inflating the goodness of its fit 6]. We evaluated the models both calibrated and uncalibrated (Table 2). If we were simply adopting it as standard, it would best be used uncalibrated, with any overall differences between observed and expected being a measure of some true differences—such as unmeasured patient variables or variation in quality of care. But our goal was to compare the models in the fairest possible way; so for this comparison, we used the calibrated NNE model. The best test of the PHS model would be performed "...on a new series of patients, ideally in a different location" [26]. (Perhaps someone will do so in the future, as we have done for the NNE models in this paper.)

Limitations
As with any risk model, there are no doubt some important risk factors that are not included in the analysis. In general, the more data (deaths), the more risk factors that can enter a model. However, there is a practical advantage to parsimony in a model, especially with regard to completeness of the data in practical use. The important paper by Jones and coworkers [31] showed that seven "core" risk factors accounted for 45% of the variability and another 13 "level 1" variables accounted for another 38% of the variability. One factor that is usually not included in a risk model is the calendar time of operation. Conditions surrounding the performance of CABG are constantly changing over time, mostly for the better, and the time frame of surgery (in our case 1997 to 2004) can be a surrogate for many small factors not in the risk model. This should be born in mind when applying any risk model to future patients.

	Conclusion

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
The NNE risk models for aortic and mitral valve surgery were successfully validated using PHS patients who were similar, but located on the other side of the North American continent. Using the PHS data and some different statistical approaches to modeling, we produced a single modified model for both positions. The NNE models used 18 risk factors based on 11 variables for AVR and 16 risk factors based on 11 variables for MVRR. Our single model uses 13 risk factors based on 11 variables, and performed equally well for both positions.

	Appendix

 
Glossary of Statistical Terms

c-index [3]. The c-index is identical to the area under the receiver operating characteristic (ROC) curve for binary outcomes. The area measures discrimination, that is, the ability of the test to correctly classify those with and without the event. Its value represents the probability, from zero to one, that a randomly selected death will have a higher risk of dying than a randomly selected non-death.

Cumulative sum (Cusum) analysis [5]. The Cusum plot shows the cumulative sum of the differences of observed minus expected deaths. If the model for expected deaths has a good fit, the value of the Cusum line should vary randomly around zero. It can be used to assess the effect of continuous variables and test the calibration of a model.

Fractional polynomials [11]. This method is used to look for transformations of a continuous independent variable (risk factor). In an organized way, many combinations of two transformations (logarithm and several different powers) are tested and then an optimal combination is chosen.

Hosmer-Lemeshow (H-L) statistic [4]. The Hosmer-Lemeshow statistic is used to test the calibration of a model. It groups the ordered predictions from the logistic regression model into deciles and produces a Pearson-like statistic which has a chi-square distribution with 8 degrees of freedom.

Imputation of missing values [12]. There are several ways of imputing missing values for a given variable, including creating a multiple regression model from the other variables to estimate the missing variable (which we used) and multiple imputation, which repeats this process several times to insert the appropriate variability into the estimates (which is considered the optimal method).

Interaction term [2]. If one risk factor has a synergistic or potentiating effect on another risk factor, the two factors involved are said to interact, and a term representing this interaction (the product of the two first-level factors) may be needed in the model.

Likelihood ratio [2]. The likelihood ratio is used to test the goodness-of-fit between two models. Adding additional risk factors will always result in a higher likelihood. A chi-square test can determine if the difference in likelihood between two models is statistically significant, namely, if the additional parameters improve the model or not.

Logistic regression [2]. Logistic regression is a standard method of data analysis concerned with describing the relationship between a binomial response variable and one or more explanatory variables. The model can predict the probability of occurrence of a binary outcome, in our case, the probability of hospital death.

Odds ratio [2]. The odds (O) of an event (eg, death) is related to the probability (P) of the event by O = P/(1 – P). When P is small, they are approximately equal. Logistic regression produces an odds ratio (OR) as a measure the strength of a risk factor. The OR is the odds of the event with the risk factor present, divided by the odds of the event with the factor absent.

Recalibration [8]. Recalibration of a predictive model is done by creating a new model based on the prediction of the original model, to ensure that the number of predicted (by the new model) and observed deaths are equal. This improves the calibration of the model but does not change its discrimination.

	Acknowledgments

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 
Doctor Edward R. Nowicki kindly provided detailed additional information on the NNE valve models, and helpful discussion about risk model construction. The following Providence Health System hospitals provided essential collaboration and use of their cardiac surgery patient data: Providence Anchorage Medical Center, Alaska; Providence Everett Medical Center, Providence St. Peter Hospital (Olympia), and Providence Yakima Medical Center, Washington; Providence Portland Medical Center and Providence St. Vincent Medical Center (Portland), Oregon; Providence St. Joseph Medical Center (Burbank), Providence Holy Cross Medical Center (Mission Hills), and Providence Little Company of Mary Hospital (Torrance), California. We thank four helpful reviewers of The Annals of Thoracic Surgery for their constructive comments and suggestions that improved the paper.

	References

Top Abstract Introduction Material and Methods Results Continuous variables Other risk factors Combining AVR and MVRR... Final PHS model: Comparison... Comment Conclusion Acknowledgments References

 

1. Nowicki ER, Birkmeyer NJ, Weintraub RW, et al. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England Ann Thorac Surg 2004;77:1966-1977.[Abstract/Free Full Text]
2. Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed.. New York: Wiley-Interscience; 2000. pp. 1-373.
3. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve Radiology 1982;143:29-36.[Abstract/Free Full Text]
4. Hosmer DW, Lemeshow S. A goodness-of-fit test for the multiple regression model Commun Statistics 1980;A10:1043-1069.
5. Royston P. The use of cusums and other techniques in modelling continuous covariates in logistic regression Stat Med 1992;11:1115-1129.[Medline]
6. Harrell Jr FE, Lee KL, Mark DB. Multivariable prognostic modelsissues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361-387.[Medline]
7. Peterson ED, DeLong ER, Muhlbaier LH, et al. Challenges in comparing risk-adjusted bypass surgery mortality resultsresults from the Cooperative Cardiovascular Project. J Am Coll Cardiol 2000;36:2174-2184.[Abstract/Free Full Text]
8. DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling Stat Med 1997;16:2645-2664.[Medline]
9. Naftel DC. Do different investigators sometimes produce different multivariable equations from the same data? J Thorac Cardiovasc Surg 1994;107:1528-1529.[Free Full Text]
10. Steyerberg EW, Eijkemans MJ, Harrell Jr FE, Habbema JD. Prognostic modeling with logistic regression analysisin search of a sensible strategy in small data sets. Med Decis Making 2001;21:45-56.[Abstract/Free Full Text]
11. Royston P, Ambler G, Sauerbrei W. The use of fractional polynomials to model continuous risk variables in epidemiology Int J Epidemiol 1999;28:964-974.[Abstract/Free Full Text]
12. Little RJA. Regression with missing X’sa review. J Am Stat Assoc 1992;87:1227-1237.
13. Edwards FH, Peterson ED, Coombs LP, et al. Prediction of operative mortality after valve replacement surgery J Am Coll Cardiol 2001;37:885-892.[Abstract/Free Full Text]
14. Jamieson WR, Edwards FH, Schwartz M, Bero JW, Clark RE, Grover FL. Risk stratification for cardiac valve replacement. National Cardiac Surgery Database Database Committee of The Society of Thoracic Surgeons. Ann Thorac Surg 1999;67:943-951.
15. Gardner SC, Grunwald GK, Rumsfeld JS, et al. Comparison of short-term mortality risk factors for valve replacement versus coronary artery bypass graft surgery Ann Thorac Surg 2004;77:549-556.[Abstract/Free Full Text]
16. Florath I, Rosendahl UP, Mortasawi A, et al. Current determinants of operative mortality in 1400 patients requiring aortic valve replacement Ann Thorac Surg 2003;76:75-83.[Abstract/Free Full Text]
17. Astor BC, Kaczmarek RG, Hefflin B, Daley WR. Mortality after aortic valve replacementresults from a nationally representative database. Ann Thorac Surg 2000;70:1939-1945.[Abstract/Free Full Text]
18. Hannan EL, Racz MJ, Jones RH, et al. Predictors of mortality for patients undergoing cardiac valve replacements in New York State Ann Thorac Surg 2000;70:1212-1218.[Abstract/Free Full Text]
19. Lytle BW, Cosgrove DM, Taylor PC, et al. Primary isolated aortic valve replacement. Early and late results J Thorac Cardiovasc Surg 1989;97:675-694.[Abstract]
20. Christakis GT, Weisel RD, David TE, Salerno TA, Ivanov J. Predictors of operative survival after valve replacement Circulation 1988;78(Suppl I):I25-I34.
21. Scott WC, Miller DC, Haverich A, et al. Determinants of operative mortality for patients undergoing aortic valve replacement. Discriminant analysis of 1,479 operations J Thorac Cardiovasc Surg 1985;89:400-413.[Abstract]
22. O’Connor GT, Plume SK, Olmstead EM, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group JAMA 1996;275:841-846.[Abstract/Free Full Text]
23. O’Connor GT, Plume SK, Olmstead EM, et al. A regional prospective study of in-hospital mortality associated with coronary artery bypass grafting. The Northern New England Cardiovascular Disease Study Group JAMA 1991;266:803-809.[Abstract/Free Full Text]
24. O’Connor GT, Malenka DJ, Quinton H, et al. Multivariate prediction of in-hospital mortality after percutaneous coronary interventions in 1994–1996. Northern New England Cardiovascular Disease Study Group J Am Coll Cardiol 1999;34:681-691.[Abstract/Free Full Text]
25. A regional intervention to improve in-hospital mortality associated with CABG surgery http://www.nnecdsg.org/ Accessed February 14, 2005..
26. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453-473.[Medline]
27. Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 1999;99:2098-2104.
28. Roques F, Nashef SA, Michel P, et al. Risk factors and outcome in European cardiac surgeryanalysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 1999;15:816-823.[Abstract/Free Full Text]
29. Hassan A, Quan H, Newman A, Ghali WA, Hirsch GM. Outcomes after aortic and mitral valve replacement surgery in Canada1994/95 to 1999/2000. Can J Cardiol 2004;20:155-163.[Medline]
30. Pinna-Pintor P, Bobbio M, Sandrelli L, et al. Risk stratification for open heart operationscomparison of centers regardless of the influence of the surgical team. Ann Thorac Surg 1997;64:410-413.[Abstract/Free Full Text]
31. Jones RH, Hannan EL, Hammermeister KE, et al. Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The Working Group Panel on the Cooperative CABG Database Project J Am Coll Cardiol 1996;28:1478-1487.[Abstract]

This article has been cited by other articles:

				M. van Gameren, N. Piazza, A. J J C Bogers, J. J M Takkenberg, and A P. Kappetein How to assess risks of valve surgery: quality, implementation and future of risk models Heart, December 1, 2009; 95(23): 1958 - 1963. [Full Text] [PDF]

				V. C. Carosella, J. L. Navia, S. Al-Ruzzeh, H. Grancelli, W. Rodriguez, C. Cardenas, J. Bilbao, and C. Nojek The first Latin-American risk stratification system for cardiac surgery: can be used as a graphic pocket-card score Interactive CardioVascular and Thoracic Surgery, August 1, 2009; 9(2): 203 - 208. [Abstract] [Full Text] [PDF]

				S. M. O'Brien, D. M. Shahian, G. Filardo, V. A. Ferraris, C. K. Haan, J. B. Rich, S.-L. T. Normand, E. R. DeLong, C. M. Shewan, R. S. Dokholyan, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery. Ann. Thorac. Surg., July 1, 2009; 88(1 Suppl): S23 - S42. [Abstract] [Full Text] [PDF]

				D. M. Shahian, S. M. O'Brien, G. Filardo, V. A. Ferraris, C. K. Haan, J. B. Rich, S.-L. T. Normand, E. R. DeLong, C. M. Shewan, R. S. Dokholyan, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3--valve plus coronary artery bypass grafting surgery. Ann. Thorac. Surg., July 1, 2009; 88(1 Suppl): S43 - S62. [Abstract] [Full Text] [PDF]

				G. L. Grunkemeier, R. Jin, and Y. Wu Cumulative sum curves and their prediction limits. Ann. Thorac. Surg., February 1, 2009; 87(2): 361 - 364. [Full Text] [PDF]

				J. M. Brown, S. M. O'Brien, C. Wu, J. A. H. Sikora, B. P. Griffith, and J. S. Gammie Isolated aortic valve replacement in North America comprising 108,687 patients in 10 years: changes in risks, valve types, and outcomes in the Society of Thoracic Surgeons National Database. J. Thorac. Cardiovasc. Surg., January 1, 2009; 137(1): 82 - 90. [Abstract] [Full Text] [PDF]

				S. Roedler, M. Czerny, J. Neuhauser, D. Zimpfer, R. Gottardi, D. Dunkler, E. Wolner, and M. Grimm Mechanical Aortic Valve Prostheses in the Small Aortic Root: Top Hat Versus Standard CarboMedics Aortic Valve Ann. Thorac. Surg., July 1, 2008; 86(1): 64 - 70. [Abstract] [Full Text] [PDF]

				B. Iung Management of the elderly patient with aortic stenosis Heart, April 1, 2008; 94(4): 519 - 524. [Full Text] [PDF]

				M. van Gameren, A. P. Kappetein, E. W. Steyerberg, A. C. Venema, E. A.J. Berenschot, E. L. Hannan, A. J.J.C. Bogers, and J. J.M. Takkenberg Do We Need Separate Risk Stratification Models for Hospital Mortality After Heart Valve Surgery? Ann. Thorac. Surg., March 1, 2008; 85(3): 921 - 930. [Abstract] [Full Text] [PDF]

				M. Kuduvalli, A. D. Grayson, J. Au, G. Grotte, B. Bridgewater, B. M. Fabri, and on behalf of the North West Quality Improvement Pr A multi-centre additive and logistic risk model for in-hospital mortality following aortic valve replacement Eur. J. Cardiothorac. Surg., April 1, 2007; 31(4): 607 - 613. [Abstract] [Full Text] [PDF]

				E. L. Hannan, C. Wu, E. V. Bennett, R. E. Carlson, A. T. Culliford, J. P. Gold, R. S.D. Higgins, C. R. Smith, and R. H. Jones Risk Index for Predicting In-Hospital Mortality for Cardiac Valve Surgery Ann. Thorac. Surg., March 1, 2007; 83(3): 921 - 929. [Abstract] [Full Text] [PDF]

				Authors/Task Force Members, A. Vahanian, H. Baumgartner, J. Bax, E. Butchart, R. Dion, G. Filippatos, F. Flachskampf, R. Hall, B. Iung, et al. Guidelines on the management of valvular heart disease: The Task Force on the Management of Valvular Heart Disease of the European Society of Cardiology Eur. Heart J., January 26, 2007; (2007) ehl428v1. [Full Text] [PDF]

				M. Turina Supra-annular aortic valve replacement with a mechanical prosthesis MMCTS, November 29, 2005; 2005(1129): 646. [Abstract] [Full Text] [PDF]

				E. R. Nowicki What is the Future of Mortality Prediction Models in Heart Valve Surgery? Ann. Thorac. Surg., August 1, 2005; 80(2): 396 - 398. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Ruyun Jin Gary L. Grunkemeier Albert Starr Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Jin, R. Search for Related Content PubMed PubMed Citation Articles by Jin, R.