| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
a Division of Cardiovascular Surgery, Mayo Clinic, Rochester, Minnesota
b Division of Cardiac Surgery, University of Alberta, Edmonton, Alberta, Canada
Accepted for publication August 20, 2008.
* Address correspondence to Dr Sundt, Mayo Clinic, 200 1st St SW, Rochester MN 55905 (Email: sundt.thoralf{at}mayo.edu).
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and MethodsResultsCommentReferences |
|---|
Methods: Questionnaires were mailed to 200 consecutive patients who underwent cardiac operations and had a complete cardiovascular consultation more than 1 month after hospital dismissal. The sensitivity and specificity for each question was compared with the electronic medical record.
Results: The median age at operation was 62.5 years (range, 19.6 to 91.1 years). The mean age of responders (n = 149, 75%) was 69 ± 13 years, and 93% lived independently in their own home. Responders and nonresponders were similar. Sensitivity and specificity of self-reporting (95% confidence intervals) included atrial fibrillation, 74% (60% to 85%) and 94% (87% to 98%); stroke, 89% (52% to 100%) and 97% (93% to 99%); bleeding requiring hospitalization, 57% (18% to 90%) and 96% (92 to 99%); permanent pacemaker implantation, 95% (75% to 100%) and 100% (97% 100%); and coronary stenting, 93% (66% to 100%) and 99% (96% to 100%).
Conclusions: Patients were reasonably accurate in reporting (high specificity) when asked about a medical condition that was not present, but were not always aware of documented medical issues (moderate sensitivity). When asked about procedures, responses were highly sensitive, specific, and accurate. Patients had difficulty discriminating among complex invasive procedures. Clinical investigators must be aware of the limitations of the data obtained from surveys, and positive responses should be confirmed.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentReferences |
|---|
|
Material and Methods |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentReferences |
|---|
We selected 200 consecutive adult cardiac surgical patients (
18 years) who met the inclusion criteria. Patients were required to have had full consultation in cardiovascular medicine at Mayo Clinic within 3 months of mailing the survey. In addition, the patients must have had a cardiovascular operation at Mayo Clinic more than 1 month before the cardiology follow-up consultation. These intervals were confirmed using institutional billing data. Questionnaires were mailed to all patients, and if no response was obtained within 1 month, patients were mailed a second survey. Professional data abstractors recorded the same data from the electronic clinical record, which was being elicited from the questions in the survey.
Responders and nonresponders were compared using a t test or 
2 test, where appropriate. Patient responses of "don't know" were considered as a negative response. If a question was not answered, it was recorded as "missing." Two variables were dichotomized: New York Heart Association (NYHA) classes III/IV vs I/II and Canadian Cardiovascular Society (CCS) classes 3/4 vs 0/1/2. The electronic medical record was used as the standard against which sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and 
(% disagreement) was calculated for each response, including a 95% confidence interval (CI). The exact binomial method was used to calculate all confidence intervals. When the medical record and the survey responses were conflicting, the medical record was analyzed to determine possible reasons for the inconsistencies.
|
Results |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentReferences |
|---|
Surveys were mailed 2 to 3 months after the cardiology consultation and 149 (75%) were returned. Two patients died during the interval between the time of cardiovascular medicine appointment at Mayo Clinic and follow-up, and 11 (5%) refused to complete the survey. Of the remaining 38 patients, 2 could not be found and 36 did not return the survey.
Surveys were completed independently by 127 patients (85%), 15 had assistance from others, and 5 surveys were answered by surrogate responders who were patients' wives. We could not determine who completed 2 surveys. A question about living status was answered as "independently in own home" for 139 patients (93%), "in own home with assistance" for 8 (5%), "in care facility" for 1 (0.5%), and as no response for 1 patient (0.5%). Health was reported as fair or poor before operation in 66 patients (44%) and was fair or poor after surgery in 27 (18%). On a 10-point scale, 85 patients (57%) rated their activity level as 6 or greater compared with others the same age (Fig 1).
|
|
, PPV and NPV comparing patient responses and the medical record are presented in Table 2. Among patients who did not report an episode of atrial fibrillation (AF), but AF was confirmed in the medical record (false-negative), 7 had perioperative AF only, 2 had atrial flutter, 1 had a permanent pacemaker (PPM) with underlying AF, and 3 had documented chronic AF. Patients who had no documentation of AF in the medical record, but responded that they had AF (false-positives) included 2 patients who had a PPM, 1 had a supraventricular tachycardia, 1 had a ventricular tachycardia, 1 had a cardiac arrest, and 1 had no documented arrhythmias. When patients were asked whether they remained in AF, no responders erroneously stated that they were. Four patients who had documented persistent or chronic AF responded that they were not in AF. One of these 4 patients had AF and flutter.
|
Four patients reported bleeding events when there was no record in the medical record (false-positives). One patient reported bleeding after a tooth extraction, 1 a nose bleed, 1 a lower gastrointestinal hemorrhage, and one a bladder infection. Two patients did not report gastrointestinal bleeding, and 1 patient did not report significant hematuria (false-negatives). No patients reported episodes of endocarditis.
Amiodarone usage was accurately reported, with only 2 patients failing to report use of this medication. Five patients failed to report aspirin usage that was found in the medical record, and 15 reported aspirin usage that was not recorded in the chart. One patient reported that they did not know if they were taking warfarin, but electronic records showed the patient was taking warfarin. However, 4 patients reported that they were taking warfarin when there was no evidence in the electronic record. One of these false-positives was likely an error in the medical record because the patient had a mechanical aortic valve.
Angiographic procedures were reported poorly, with 12 false-negative and 4 false-positive answers. No consistent reason for this poor response could be documented, although several patients who reported angiograms also had PPM insertion. Only one patient failed to report a stent insertion, and 1 patient reported stents placed before cardiac operation. No patients reported cardiac reoperations.
Reported NYHA class had a sensitivity of 33% (95% CI, 12% to 62%) and a specificity of 92% (95% CI, 86% to 96%). In most cases where disagreements existed between the medical record and the survey response, the patients rated themselves as having a better functional status (53%). For the CCS class to indicate angina, the reported sensitivity was 11% (95% CI, 28% to 48%) and specificity was 97% (95% CI, 93% to 99%). In 60% of the disagreements, the medical record rated patients' CCS class as poorer than the patient did themselves.
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentReferences |
|---|
We found that patients were reasonably accurate in reporting the absence of a specific medical condition (high specificity), but they were not always aware of documented medical issues (moderate sensitivity). Specifically, when patients were asked about procedures they had undergone after cardiac operations, including PPM implantation and coronary stent implantation, responses were highly sensitive, specific, and accurate. However, patients had difficulty discriminating between complex invasive procedures such as coronary angiography and catheter ablation procedures. It thus appears that positive responses to questions about medical conditions such as atrial fibrillation, stroke, myocardial infarction, and bleeding or thromboembolic events should be confirmed with medical documentation, but the report of some procedures may be accepted as indicated on the patient survey.
Negative responses had a good specificity (69% to 100%) and therefore could be relied on without documentation. A study by St. Sauver and colleagues [1] reported survey specificities of 93% to 99% for cardiovascular conditions and risk factors. In contrast, responders to surveys that ask about sensitive issues such as drug use or sexual behavior may deny conditions or behaviors owing to societal pressures, leading to poor specificity [2, 3]. Our study differs from others in that it was designed primarily to assess cardiovascular events rather than the prevalence of risk factors.
We did not use existing standard questionnaires, such as the Medical Outcomes Study Short-Form 36 (SF-36) Survey or the Minnesota Living with Heart Failure Questionnaire [4, 5] to assess patient functional status because they are lengthy and have not been specifically validated in cardiac surgical patients. We found, however, that despite simplification of responses, self-reported NYHA and CCS classification had very poor agreement with the medical record. This finding should be kept in mind when functional outcomes derived from patient surveys are analyzed; questions had a poor sensitivity for NYHA classes III and IV and CCS classes 3 and 4, but had good specificity.
Furthermore, although a measure such as NYHA class may have excellent prognostic ability, the interobserver reliability remains only moderate [6]. In a study of 48 patients with stable heart failure, NYHA class was not correlated with left ventricular ejection fraction, but was moderately correlated with a 6-minute walk test, peak oxygen consumption, the Minnesota Living with Heart Failure Questionnaire, and the SF-36 energy/fatigue and physical functioning scales [6]. Thus, without objective exercise testing, it is difficult to determine if the poor sensitivity of functional assessment in this investigation was caused by abstraction of the medical record or the design of questions posed to patients.
Responders to our survey tended towards being younger and male, perhaps representing a higher socioeconomic status. Most patients who returned surveys lived independently in their own homes, indicating a possible bias in which responding patients were healthier than nonresponders. This may be related to the "healthy worker effect," often seen in studies focused on a specific occupation, in which workers exhibit lower overall death rates than a general population because the severely ill and chronically disabled are generally excluded from employment [7]. One of the best ways to avoid this potential bias is to focus on achieving the highest possible response rates.
Nonresponse to surveys remains an important source of bias in many clinical studies [1]. In two meta-analyses of studies that analyzed survey response rates, the average response rates were reported as 65% [8] and 60% [9]. Our survey had a reasonably good response rate of 75%. A Cochrane meta-analysis reviewed 98 different ways of increasing response rates to all types of questionnaires [10]. The greatest improvement in response rates included monetary incentives, a suggested benefit to opening the survey on the envelope, using first-class mail, and a more interesting questionnaire topic [10]. We did not include a monetary incentive and did not suggest benefit to opening the survey in our study. The meta-analysis also found that response rates were increased with the following items that we did not include: prenotification of the survey, use of unconditional incentives, and the implication of a mandatory response. We did, however, use secondary follow-up, including a second copy of the questionnaire, a shorter questionnaire, and university sponsorship. We also included personalized questionnaires and assurances of confidentiality, which were found to have smaller improvements in response rates in the meta-analysis.
Patients today often have multiple health care providers who may not have easy access to all patient information, and providers must rely on the patient's report of his or her health status and history. As well, we are moving toward an era when patients will have ownership of their own clinical data. This means that we would rely on the patient's ability to provide information to us, and we must consider that patients may not be able to provide complete and accurate information.
No medical record is perfectly accurate because it includes subjective assessments by individual physicians and also relies on patient reports. Nevertheless, the medical record is an important source of patient information for prospective randomized and retrospective observational clinical studies and remains the gold standard for patient data. We made every effort in this study to minimize abstraction errors by using experienced abstractors who were familiar with the Mayo Clinic record and by checking all discordant answers. Also, to enhance the accuracy of this gold standard, only patients with a recent, complete cardiovascular follow-up consultation were included. It is possible, however, that a patient's condition may have changed between the time when consultation occurred and when survey was completed. In an attempt to obviate this, calendar dates were assessed in all events. We were also unable to assess if patient educational background or socioeconomic status had any effect.
Owing to the low incidence of end points, we were unable to analyze results of some survey questions, including endocarditis and reoperation. If we consider the incidence of a rare outcome to be approximately 2%, we could expect the disagreement to be at most, half of that (1%). Thus, to have a 95% CI of ±0.5%, we would have needed 1522 respondents. As well, the prevalence rate of each medical condition affects the sensitivity and specificity, and thus we have also calculated the percentage agreement ( values).
It is important to emphasize that our results may not be generalizable to other institutions because we developed our own set of questions and used our own medical record. When institutions develop in-house surveys to obtain patient follow-up, some form of validation should be considered.
Our other major limitation is the potential effect of recall bias. We did not control for the time from operation to the time of survey. This was by design, because most studies that use survey follow-up include many different years of operation. Unfortunately, our study size limits our ability to determine if there was an effect of recall bias in patients who had operations at an earlier date. Our sense is, however, that this did not affect the results because our survey focused on events subsequent to operation, which may have occurred at any time.
In conclusion, questionnaires remain a valuable tool to assess the outcomes of patients after cardiac operations. Clinical investigators must be aware of the limitations of the data obtained from surveys, and positive responses should be confirmed.
|
Appendix |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
