HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Syma L. Prince Mitchell J. Magee Michael J. Mack Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Herbert, M. A. Articles by Mack, M. J. Search for Related Content PubMed PubMed Citation Articles by Herbert, M. A. Articles by Mack, M. J. Related Collections Professional affairs

Ann Thorac Surg 2004;77:1960-1964
© 2004 The Society of Thoracic Surgeons

---

Original article: cardiovascular

Are unaudited records from an outcomes registry database accurate?

^a Department of Research, Medical City Dallas Hospital, USA
^b Cardiopulmonary Research Science and Technology Institute, Dallas, Texas, USA

Accepted for publication December 12, 2003.

^* Address reprint requests to Dr Herbert, Department of Research, Medical City Dallas Hospital, 7777 Forest Ln, Suite C-740, Dallas, TX 75230, USA
e-mail: morley.herbert{at}lonestarhealth.com

	Abstract

Top Abstract Introduction Material and methods Results Comment References

 
BACKGROUND: Data from outcomes registry databases are being increasingly used for peer review and public reporting. However, administrative and clinical databases are mostly unaudited; thus, their accuracy has not been verified.

METHODS: Outcomes data from all coronary artery bypass operations from a single cardiac surgery practice were entered into The Society of Thoracic Surgeons (STS) National Cardiac Database. From our practice of 18 surgeons, we audited 247 (10%) of the clinical records of patients undergoing surgery in 2001 and correlated them with all 315 elements of the STS National Cardiac Database for verification of accuracy. Inaccuracies were defined as a disagreement with a nominal or categorical variable or, for continuous variables, as the value not being within a predetermined window. When discrepancies existed, the hospital clinical record was assumed to be accurate. Outcomes discrepancies were then analyzed by four major categories: components of the preoperative risk algorithm, operative mortality, major complications, and other outcomes.

RESULTS: Discrepancies were noted in 5% (16) or fewer of the audited fields for 98.8% of the records. Of the 32 variables in the mortality risk algorithms, discrepancies were present in fewer than 10% of the audits on 30 of the 32 variables. More than 95% of the audited charts had zero or one discrepancy in the seven most important variables in the mortality risk models. Operative mortality was determined to be completely accurate with no discrepancies between the database and the audited clinical record. Among major complications, the error rate was less than 1% for all complications except prolonged ventilation (4.0%). A higher rate of discrepancies did exist in some of the other variables, including discharge medications (14.1%) and ventilator time (36.4%).

CONCLUSIONS: A detailed audit of a clinical outcomes registry database demonstrated that the major fields within this specific database including operative mortality, major complications, and the significant factors in the risk algorithm were highly accurate. Process improvement factors were identified to further increase the accuracy of data collection.

	Introduction

Top Abstract Introduction Material and methods Results Comment References

 
Outcomes reporting of medical procedures are being increasingly used for peer review, contracting purposes, determination of "centers of excellence," and public reporting [1–5]. Outcomes for various procedures are reported from a variety of clinical and administrative databases. Although determinations of quality are made on the basis of reported data, the accuracy of these data has not been clearly substantiated [6, 7]. As a participant in The Society of Thoracic Surgeons (STS) National Cardiac Database, the Cardiopulmonary Research Science and Technology Institute (CRSTI) collects data locally and then "harvests" data to the central data warehouse at defined intervals.

To determine the accuracy of data gathered into our clinical database, each year we randomly audit 10% of the clinical records of patients whose procedural and outcomes data had been entered into our database during a 1-year period for verification of accuracy. This report covers the results of the 2001 audit.

	Material and methods

Top Abstract Introduction Material and methods Results Comment References

 
Data source
Using specialized data collection forms, surgical data were collected on all patients from a single, multicenter cardiac surgical practice; collected data were stored locally in a STS-approved database (Armus, Burlingame, CA). Since 1985, the practice has consisted of 22 cardiac surgeons practicing in 18 hospitals and the database now contains more than 33,000 patient entries.

The data were gathered by an independent not-for-profit research organization, CRSTI, which supervises the data collection and provides data entry and local warehousing. The collected data were exported to the STS for participation in the national registry and also analyzed locally. Prepared reports were used for peer review within the surgical practice. Outcomes data for individual surgeons were provided to that surgeon along with benchmarking within the practice, within the hospital in which that surgeon practiced, within the region, and compared with national standards. Data were also provided to the hospitals for performance and quality improvement initiatives.

Data collection
Data were gathered prospectively for each patient at the various stages of the clinical course. The preoperative portion of the database was collected at the time of the surgical procedure on a paper entry form by a clinical person, either a nurse practitioner, physician assistant, or perfusionist familiar with clinical data. After completion of the surgical procedure, operative data were then usually completed by the same staff member after transfer of the patient to the intensive care unit. The data elements regarding the postoperative clinical course were then generally completed by a physician assistant, clinical nurse specialist, or nurse practitioner at the time of discharge from the hospital. The individual patient data collection was sent to the clinician's office and the rest of the postoperative data was completed at the 30-day follow-up visit of the patient. Data collection and entry was under supervision of the operating physician, who generally reviewed the data record for accuracy and completeness before the record was entered into the local database. If the patient had a prolonged operative stay or did not return for follow-up, retrospective collection of missing data elements was performed.

Once the record was presented for electronic data entry, the data were subjected to a review process during which missing elements or data inconsistencies were identified. These forms were then returned to the clinicians for review, completion, or correction. After the completed forms were entered into the computerized database, weekly audits of a minimum of 10 records helped ensure integrity of the data entry process. This procedure enabled correction of data entry errors and identified procedural problems.

Database audit
On a semiannual basis an administrative person not involved with data collection or entry randomly selected 10% of the data records for audit from a list of entries generated from the previous 6 months. An outcomes database supervisory person with a background in clinical medicine then correlated the computerized data with the patient's clinical record for accuracy, using the STS National Cardiac Database definitions of each of the 315 data elements. Determination of whether a discrepancy exists depended on the variables being checked. All nominal and categorical variables (eg, yes/no) were scored as a discrepancy if there was any disagreement. For continuous variables, a definition of discrepancy was determined before the audit process was initiated. A "reasonable" window of acceptable range was determined in consultation with clinicians to determine discrepancies.

The data set included 102 variables for demographic and preoperative risk data, 113 variables containing operative procedural data, and 100 variables for postoperative data, complications, and discharge information. All discrepancies were noted and any pertinent clinical data recorded on an audit form.

The 10% sample yielded 247 data records for audit from the practices of 18 of the surgeons. The results of the audit were then divided into four major categories: preoperative risk algorithm variables, operative mortality, major postoperative complications, and other selected outcome variables.

	Results

Top Abstract Introduction Material and methods Results Comment References

 
Complete correlation of the clinical record and the database occurred for 5 (2.0%) of the patients. An additional 190 records (82.2%) had one to ten discrepancies, and 39 (16.8%) had more than ten fields of disagreement. Overall, discrepancies occurred in fewer than 16 (5%) of the data fields in 98.8% of all charts audited. No discrepancy in the data element was noted for operative mortality, with complete correlation between the database variable and the audited clinical records.

Discrepancies were found in the entries for the preoperative risk variables used in the operative mortality risk algorithm. Table 1 lists the 34 different variables used in the calculation of risk of operative mortality for the coronary artery bypass grafting, valve, and coronary artery bypass grafting + valve models, and the overall discrepancy rates measured for each variable. Procedure status and New York Heart Association class had the largest discrepancy rates (13.4% and 28.3%, respectively). Disagreement occurred in more than 5% of the records for height, weight, diabetes control method, presence of chronic obstructive pulmonary disease, timing of preoperative myocardial infarction, ejection fraction, number of diseased vessels, presence of left main disease, and mitral insufficiency.

Because variables appearing in a risk algorithm are not weighted equally, it would be helpful to know how many of these discrepancies occurred among the more important variables. Examining variables in the coronary artery bypass grafting risk model for operative mortality, we used the seven variables with the highest weighting and largest odds ratio. These included age, previous surgery, cardiogenic shock, immunosuppressive therapy, diabetes, renal failure, and operative status [8]. One hundred seventy-two (69.6%) of the charts had no discrepancies in these variables, 64 (25.9%) had one discrepancy, and 11 (4.5%) had two or three discrepancies each. Of the total of 434 discrepancies found among all the risk variables of the algorithm, only 97 (22.4%) occurred in the variables with the highest weighting or largest odds ratio. More than 95% of the charts had either zero or one discrepancy in the most important variables used in mortality risk modeling.

Table 2 lists the discrepancy rates measured for six major complications. Operative mortality and renal failure requiring dialysis was accurately reported on all patients, whereas minor discrepancies were noted for permanent stroke (0.4%), deep sternal infection (0.8%), and cardiac surgery reoperation (0.8%). Prolonged ventilation was recorded the most inaccurately, with discrepancies occurring in 4% of the records.

View larger version (36K): [in this window] [in a new window]   Fig 1. Modes of data collection used. (NP = nurse practitioner; PA = physician's assistant; RN = registered nurse.)

	Comment

Top Abstract Introduction Material and methods Results Comment References

 
The use of outcomes data as a measure of the quality of medical care is becoming increasingly important. Quality outcomes data are used for internal peer review within institutions, determination of quality compared with regional and national benchmarks, and for determination of centers of excellence—both by payers and government agencies. Yet public "report cards" on hospitals and surgeons and continuous quality improvement measures are generally based on unaudited data and outcomes [9, 10]. Frequently, outcomes reporting is based on administrative databases with an unproven correlation with actual clinical outcomes [11]. Moreover, the accuracy of clinical outcomes databases has generally not been verified. The accuracy of outcomes reporting and calculations of risk adjustment applied to the raw data are only as accurate as the data entered. Most data collection and entry are performed by individuals with varying degrees of clinical expertise and access to data sources. Yet auditing of these data for accuracy is seldom performed.

The STS National Cardiac Database is a voluntary, unaudited outcomes registry of cardiac surgical procedures performed within the United States. Since 1993, data on 2 million operative procedures performed at more than 600 sites have been entered, analyzed, and reported [12]. The data warehouse has performed ongoing internal data checks to assure data consistency and general accuracy of outcomes reporting [6, 7]. The national data warehouse cannot, however, audit the specific clinical records within the database, instead relying on the reporting centers for data accuracy.

As a whole, we found generally excellent correlation between the data our center reported to the STS National Cardiac Database and that found in the patient's clinical record. No discrepancy was noted in operative mortality, the most frequently cited outcomes measurement of "quality." Data however, are frequently reported as "risk adjusted" in order to account for varying patient preoperative risk factors. The audit of the variables contained in the risk algorithm showed generally very good correlation. The risk algorithm calculation, however, is proprietary and all variables do not count equally. Variables such as New York Heart Association classification, which showed a high discrepancy rate, will have a lesser effect on the risk calculation. However, the discrepancy rate of procedural status will have a more significant impact on the accuracy of the calculation of risk algorithm.

In a similar manner, correlation of reporting of major complications was generally good. This finding would be another endorsement of the accuracy of registry outcomes reporting, at least with the methodology used in our practice. Discrepancy rates from other selected outcomes, however, showed more variability when audited.

Table 4 summarizes our approach to maximize the accuracy of the data collection process. As a result of this audit and as part of a process improvement initiative, we have broken down the factors in the data collection process as follows: data definitions, data sources, data entry personnel, background and training, and a commitment to completeness and accuracy of data collection. In our experience, a lack of clear understanding of the definition of some variables is a problem. The variables for which the definitions were most often misunderstood included type of angina and procedural status. Specific remedies to this issue may include adding the STS definitions to the data collection forms. (See the Appendix for an example of our form including definitions.)

Another common cause for error is an inability to access the data source for determining accurate information. With same-day admissions for cardiac surgery and transfer of patients from other institutions, medical records are frequently incomplete during data collection. The greater ubiquity of electronic medical records should probably improve this situation. In the interim, however, extra effort must be expended by the staff member responsible for data collection to determine the entry information from the source data. We have entertained implementing financial incentives for data accuracy and promptness of submission to enhance the accuracy of data collection. To date, however, we have not initiated this process.

To assist in providing the data collection personnel with complete data, surgeons are being provided with "cheat sheets" containing a list of the risk variables. Using these sheets as guides, the expectation is that the surgeons' dictations will address all the necessary elements, thereby facilitating accurate data collection.

Accuracy of data entry is also enhanced by data monitoring. Training individuals involved with data collection to increase awareness of potential areas for inaccuracy and early rectification of the high areas of inaccuracy has been helpful. Although we have determined by regular auditing that data integrity is generally good, we have identified a number of areas for process improvement. Continued monitoring of the data and of the processes involved in data collection and accuracy should improve the integrity of the data. With outcomes reporting being used increasingly as a determination of quality, it behooves every individual involved in the process to strive for excellence in data collection promptness and accuracy.

	Appendix

 
Preoperative Risk Factors
Weight (kg) Height (cm)

Circle All That Apply History of Smoking: No Yes

• Current Smoker (defined as having smoked within 1 month of surgery): No Yes
  

Family History of Coronary Artery Disease (defined as any direct blood relatives with angina, myocardial infarction, or sudden cardiac death before the age of 55): No Yes Diabetes (defined as a history of diabetes, regardless of duration or need for medication): No Yes

• Diabetes Control: None, Diet, Oral Insulin, IV Insulin
  

Hypercholesterolemia (diagnosed and or treated by a physician; total cholesterol more than 200 mg/dL; low-density lipoprotein level of 130 mg/dL or higher; high-density lipoprotein level of less than 30 mg/dL): No Yes

Renal Failure (defined as a creatinine level more than 2.0 mg/dL): No Yes

• Dialysis: No Yes

• Last Preoperative Creatinine Level:

Hypertension (diagnosed with or treated for hypertension with medication, diet, or exercise; blood pressure more than 140 mm Hg systolic or more than 90 mm Hg diastolic on at least two occasions): No Yes

Cerebrovascular Accident (defined as central neurologic deficit lasting more than 24 hours): No Yes

When:

• Recent (2 weeks ago or sooner)
  
• Remote (>2 weeks ago)
  

Infectious Endocarditis: No Yes

Type:

• Treated (if no antibiotic medication, other than prophylactic medication, is being given at the time of surgery)
  
• Active (if patient is currently being treated for endocarditis)
  

Chronic Lung Disease:

• No
  
• Mild (FEV₁ 60% to 75% of predicted or on chronic inhaled or oral bronchodilator therapy)
  
• Moderate (FEV₁ 50% to 59% of predicted or on chronic steroid therapy for lung disease)
  
• Severe (FEV₁ less than 50% of predicted or room air pO₂ less than 60 mmHg or pCO₂ more than 5 mmHg)
  

	References

Top Abstract Introduction Material and methods Results Comment References

 

1. Birkmeyer J.D., Stukel T.A., Siewers A.E., Goodney P.P., Wennberg D.E., Lucas F.L. Surgeon volume and operative mortality in the United States. N Engl J Med 2003;349:2117-2127.[Abstract/Free Full Text]
2. Kizer K.W. The volume-outcome conundrum. N Engl J Med 2003;349:2159-2161.[Free Full Text]
3. New York State Department of Health. Coronary artery bypass surgery in New York State, 1997–1999. 2002 September. Available from: URL: http://www.health.state.ny.us/nysdoh/heart/1997-99cabg.pdf. Accessed January 30, 2004
4. Texas Healthcare Information Council. Available from: URL: http://www.thcic.state.tx.us/default.htm. Accessed January 30, 2004
5. HealthGrades, the Healthcare Quality Experts. Available from: URL: http://www.healthgrades.com/. Accessed January 30, 2004
6. Grover F.L., Shroyer A.L., Edwards F.H., et al. Data quality review program: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1996;62:1229-1231.[Free Full Text]
7. Shroyer A.L., Edwards F.H., Grover F.L. Updates to the Data Quality Review Program: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1998;65:1494-1497.[Abstract/Free Full Text]
8. Shroyer A.L., Coombs L.P., Peterson E.D., et al. The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models. Ann Thorac Surg 2003;75:1856-1865.[Abstract/Free Full Text]
9. Shahian D.M., Normand S.L., Torchiana D.F., et al. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
10. Grunkemeier G.L., Zerr K.J., Jin R. Cardiac surgery report cards: making the grade. Ann Thorac Surg 2001;72:1845-1848.[Free Full Text]
11. Gardner E. UB-82 forms offer wealth of information, misinformation. Mod Healthc 1990;20(38):18–9, 22–9
12. Society of Thoracic Surgeons National Cardiac Database Available from: http://www.ctsnet.org/file/STSNationalDatabaseFall2003ExecutiveSummary_Adult_Revised.pdf. Accessed April 21, 2004.

This article has been cited by other articles:

				D. M. Shahian, S. M. O'Brien, G. Filardo, V. A. Ferraris, C. K. Haan, J. B. Rich, S.-L. T. Normand, E. R. DeLong, C. M. Shewan, R. S. Dokholyan, et al. The Society of Thoracic Surgeons 2008 Cardiac Surgery Risk Models: Part 1--Coronary Artery Bypass Grafting Surgery Ann. Thorac. Surg., July 1, 2009; 88(1_Supplement): S2 - S22. [Abstract] [Full Text] [PDF]

				S. M. O'Brien, E. R. DeLong, R. S. Dokholyan, F. H. Edwards, and E. D. Peterson Exploring the Behavior of Hospital Composite Performance Measures: An Example From Coronary Artery Bypass Surgery Circulation, December 18, 2007; 116(25): 2969 - 2975. [Abstract] [Full Text] [PDF]

				K. F. Welke, E. D. Peterson, M. S. Vaughan-Sarrazin, S. M. O'Brien, G. E. Rosenthal, G. J. Shook, R. S. Dokholyan, C. K. Haan, and T. B. Ferguson Jr Comparison of Cardiac Surgery Volumes and Mortality Rates Between The Society of Thoracic Surgeons and Medicare Databases From 1993 Through 2001 Ann. Thorac. Surg., November 1, 2007; 84(5): 1538 - 1546. [Abstract] [Full Text] [PDF]

				M. J. Magee, M. A. Herbert, T. M. Dewey, J. R. Edgerton, W. H. Ryan, S. Prince, and M. J. Mack Atrial Fibrillation After Coronary Artery Bypass Grafting Surgery: Development of a Predictive Risk Algorithm Ann. Thorac. Surg., May 1, 2007; 83(5): 1707 - 1712. [Abstract] [Full Text] [PDF]

				P. E. Falcoz, M. Conti, L. Brouchet, S. Chocron, M. Puyraveau, M. Mercier, J. P. Etievent, and M. Dahan The Thoracic Surgery Scoring System (Thoracoscore): Risk model for in-hospital death in 15,183 patients requiring thoracic surgery J. Thorac. Cardiovasc. Surg., February 1, 2007; 133(2): 325 - 332. [Abstract] [Full Text] [PDF]

				R. J.F. Baskett, F. H. Cafferty, S. J. Powell, R. Kinsman, B. E. Keogh, and S. A.M. Nashef Total Arterial Revascularization is Safe: Multicenter Ten-Year Analysis of 71,470 Coronary Procedures Ann. Thorac. Surg., April 1, 2006; 81(4): 1243 - 1248. [Abstract] [Full Text] [PDF]

				M. J. Mack, M. Herbert, S. Prince, T. M. Dewey, M. J. Magee, and J. R. Edgerton Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes? J. Thorac. Cardiovasc. Surg., June 1, 2005; 129(6): 1309 - 1317. [Abstract] [Full Text] [PDF]

				R. H. Jones The Year in Cardiovascular Surgery J. Am. Coll. Cardiol., May 3, 2005; 45(9): 1517 - 1528. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Syma L. Prince Mitchell J. Magee Michael J. Mack Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Herbert, M. A. Articles by Mack, M. J. Search for Related Content PubMed PubMed Citation Articles by Herbert, M. A. Articles by Mack, M. J. Related Collections Professional affairs