| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
a Cardiac Unit, Great Ormond Street Hospital, London, United Kingdom
d Clinical Operational Research Unit, University College London, London, United Kingdom
b Children's Heart Centre, Queen Silvia Children's Hospital, Gothenburg, Sweden
c Green Lane Cardiothoracic Unit, Auckland, New Zealand
Accepted for publication October 8, 2008.
* Address correspondence to Mr Tsang, Cardiac Unit, Great Ormond Street Hospital NHS Trust, London, WC1N 3JH, United Kingdom (Email: tsangv{at}gosh.nhs.uk).
 |
Abstract |
|---|
 Top Abstract IntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
Methods: We used a recently developed graphic technique to evaluate the performance of an existing risk model among those patients at a single center during 2000 to 2003 originally used in model development. We then compared the distribution of predicted risk among these patients with that among patients in 2004 to 2006. Finally, we constructed a VLAD chart of risk-adjusted outcomes for the latter period.
Results: Among 1083 patients between April 2000 and March 2003, the risk model performed well at predicted risks above 3%, underestimated mortality at 2% to 3% predicted risk, and overestimated mortality below 2% predicted risk. There was little difference in the distribution of predicted risk among these patients and among 903 patients between June 2004 and October 2006. Outcomes for the more recent period were appreciably better than those expected according to the risk model. This finding cannot be explained by any apparent bias in the risk model combined with changes in case-mix.
Conclusions: Risk models can, and hopefully do, become out of date. There is scope for complacency in the risk-adjusted audit if the risk model used is not regularly recalibrated to reflect changing standards and expectations.
|
Introduction |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
The variable life-adjusted display (VLAD) was developed for use in adult cardiac surgery to display a surgeon's or unit's perioperative mortality results, taking account of the predicted risk of death to give an accessible overview of the outcomes being achieved and to highlight any trends [2, 4] that might warrant further study. The display is a plot, against time, or typically against sequential case number, of the cumulative perioperative mortality expected at that point in the series (calculated as the sum of the individual predicted risks) minus the cumulative perioperative mortality observed at that point in the series.
Essentially, the VLAD gives a running tally of how much better (or worse) outcomes are compared with what would be expected on the basis of the underlying risk model. The trace plotted on the VLAD chart rises when a patient survives and falls when a patient dies. The degree of rise and fall is determined by the predicted risk associated with the case. For example, if a patient survives a ventricular septal defect closure with a predicted mortality risk of 2%, the trace would move up by 0.02. If the same patient dies, however, the trace would move down by 0.98. In this way, the VLAD chart provides an accessible graphic representation of the activity of an entire surgical program in terms of risk-adjusted perioperative outcome.
Using a predictive risk model based on the RACHS-1 scheme and age, we have previously shown that VLAD charts provide an effective graphic display of performance when applied to congenital heart surgery [12]. Before such methods are introduced for routine use, the underlying risk formulas and the monitoring methods both need to be tested to ensure that they are not misleading. The ultimate aim would be to validate the risk model using data sets from other institutions and also to provide an indication of any change in its appropriateness over time.
In this study we sought to evaluate recent outcomes for congenital heart operations at a single center using the previously reported risk model based on RACHS-1 and age [7] that was developed using an earlier data set from the same unit. We additionally sought to assess the extent to which observed changes in risk-adjusted outcomes over time could be attributed to the characteristics of the risk model and evolution of the case mix.
|
Patients and Methods |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
We have previously reported the development of a model of the risk of perioperative death using data pertaining to 1083 congenital open heart operations performed between April 2000 and March 2003 at the same institution [7]. For completeness, we should point out that outcomes for procedures performed between April 2003 and May 2004 inclusive have been reported elsewhere [12].
Since the congenital heart surgery risk model was developed, the Mean Adjusted Deaths Compared Against Predicted (MADCAP) graphic tool [13], has become available; this enables visual identification of patterns of systematic overestimation or underestimation of risk on the part of a risk model. As an initial step in the current study, we used this technique to assess the performance of the risk model, across the whole range of predicted risk, among the original series of 1083 patients. We used the MADCAP plot that resulted to identify ranges of predicted risk where the model performed well, underestimated risk, and overestimated risk amongst the original data set.
The risk model was then used to calculate a predicted risk of in-hospital death for each patient in the more recent series of patients. Before preparing a VLAD chart of risk-adjusted outcomes for the recent period June 2004 to October 2006, we calculated the percentage of these patients who were in each of the ranges of predicted risk previously defined according to the performance of the risk model among the original data set. A chart was then constructed to provide a visual comparison of the risk profile among the recent cases and that among the original data set. A VLAD chart was then constructed for the recent cases.
|
Results |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
Behavior of the Risk Model
Figure 1
is a MADCAP chart for the risk model among the original 1083 cases; it displays the predicted cumulative mortality and the observed cumulative mortality among patients ordered in terms of increasing predictive risk of peri-operative death.
|
Changes in the Risk Profile of the Case-Mix
Figure 2
indicates the proportion of patients in each of three predicted risk categories defined by inspection of Figure 1: less than 2%, 2% to 3%, and more than 3% for both the original data set and the more recent data set. A slightly higher proportion of patients had a predicted mortality risk exceeding 3% in the recent cohort than in the original cohort, with a correspondingly smaller proportion having risks in both the less than 2% range and the 2% to 3% range.
|
|
|
Comment |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
If there has indeed been such an improvement, this raises the question whether the risk model developed on procedures performed between 2000 and 2003 is still appropriate. Our objective is to strive for continual improvement in our performance, and we believe that the monitoring of risk-adjusted outcomes has a key role to play in this endeavor. It is conceivable that, if used in monitoring, a model that overestimates the risk of death among current patients may breed complacency and may mask runs of comparatively poor outcomes: a potential example of this is highlighted on the VLAD chart.
This report confirms that graphic monitoring tools that are widely used in the field of adult cardiac surgery [4, 14] may potentially be applied to monitor outcomes in pediatric congenital heart surgery. We are encouraged to note that the recent outcomes at Great Ormond Street Hospital are appreciably better than those predicted from a regression model developed in 2004 from the same center [7]. Our analysis suggests that this finding was not due to intrinsic bias in the risk model combined with changes in case-mix, but rather represents a genuine reduction of the risk of perioperative death associated with pediatric congenital heart surgery within this unit. It is not the purpose of this article to speculate why the apparent improvement in outcomes may have occurred. There has been greater specialization among the surgical team and a continued attention to improving outcomes across cardiology services and intensive care, but we have not attempted to evaluate the effect of these factors nor is it obvious how one would go about such an evaluation.
Before such monitoring methods could be introduced into common use in the field of pediatric congenital heart surgery, they need to be tested further in a range of contexts. When predictive risk models are to be used to monitor performance, they need to be regularly reviewed and recalibrated to reflect the true evolution in performance over time. The role of risk models in predicting risk and informing decisions at the level of the individual patient is very limited [15].
Several investigators have made significant contributions to tackling the challenging issue of risk adjustment [8, 13] and the monitoring of performance [12, 14, 16] in pediatric cardiac surgery. The RACHS-1 classification scheme has been validated in North America [9] and Europe [10, 11] and has facilitated evaluation of several important topics, including complications and mortality [17], racial and ethnic disparity [18], and congenital heart surgery in the developing world [19].
Since the Bristol Royal Infirmary Inquiry in the United Kingdom (UK), where one pediatric cardiac surgical center had an excessively high mortality rate [20], there has been a strong national interest in center- and operator-specific monitoring [21–23]. The UK Central Cardiac Audit Database (CCAD) has required the mandatory return of details pertaining to all congenital heart operations performed in the United Kingdom since 2000, and published center-specific results on the Internet for the first time in 2007 [24]. Currently, 30-day and 1-year mortality outcomes for 36 different surgical procedures are reported from 17 separate UK pediatric cardiac centers. The diversity of the data presented is an integral feature of this complex specialty and renders the information relating to a specific center rather difficult to review in its entirety. Furthermore, the use of a monitoring method that plots the accumulated difference between the observed number of adverse events and the number expected according to an accepted risk-adjustment procedure [6, 25] may be more optimally suited to detect a worrying trend in a timely fashion.
Our research group strongly supports the vision of the UK CCAD to provide high-quality risk-adjusted outcome data on all congenital heart operations performed nationally. We note that other investigators have reported improved outcomes over time for certain congenital heart procedures [26, 27]. To provide outcome data for congenital heart surgery to the highest possible standard, we advocate the use of (1) a robust method for risk adjustment validated across a number of institutions, (2) the cumulative documentation of adverse events, (3) the use of graphic tools that have proven useful in adult cardiac surgery, and (4) the regular precalibration of expected outcome to reflect true improvements in performance over time and to guard against complacency.
In conclusion, when risk-adjusted outcomes are interpreted, it is important to account for known features of the risk model. We are encouraged by the improvement in risk-adjusted outcomes but conclude that a predictive risk model requires regular recalibration over time.
|
Acknowledgments |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
|
References |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsCommentAcknowledgmentsReferences |
|---|
Related Article
This article has been cited by other articles:
|
|
 |
|
  J. M. Giroud, J. P. Jacobs, D. Spicer, C. Backer, G. R. Martin, R. C. G. Franklin, M. J. Beland, O. N. Krogmann, V. D. Aiello, S. D. Colan, et al. Report From The International Society for Nomenclature of Paediatric and Congenital Heart Disease: Creation of a Visual Encyclopedia Illustrating the Terms and Definitions of the International Pediatric and Congenital Cardiac Code World Journal for Pediatric and Congenital Heart Surgery, October 1, 2010; 1(3): 300 - 313. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 G. S. Collins and D. G. Altman An independent and external validation of QRISK2 cardiovascular disease risk score: a prospective open cohort study BMJ, May 13, 2010; 340(may13_2): c2442 - c2442. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. P. Jacobs, D. M. Shahian, M. L. Jacobs, and C. Mavroudis Invited Commentary Ann. Thorac. Surg., February 1, 2009; 87(2): 587 - 588. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
