| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Ann Thorac Surg 1999;67:297-298
© 1999 The Society of Thoracic Surgeons
a Cardiac Surgical Research Group, Department of Surgery, Flinders Medical Centre, Bedford Park SA 5042, Australia
To the Editor
We thank Bruggemans and colleagues for their comments on our paper on the analysis of individual change on neuropsychological tests after coronary artery bypass graft surgery [1]. We concur that the Reliable Change Index (RCI) we have used to define postsurgical change has two theoretic shortcomings [2]. First, test reliability is not dealt with at both pretest and posttest. Second, by incorporating the mean change score of the control group to correct for practice effects, individual differences in the magnitude of practice effects are obscured. We note that Bruggemans and colleagues [3] have adopted a statistical approach that attempts to overcome these problems. We have attempted to reanalyze our data using their approach and encountered a number of problems.
First, to obtain separate pretest and posttest reliability measures for each test, Bruggemans and coworkers conducted a factor analysis of control subjects’ test scores obtained at four testing occasions. The estimated communalities of the scores at each of these times were taken as the measures of reliability. They generally found that reliability for each test improved from pretest to 1 week posttest. However, when a study design (such as ours) comprises only two testing occasions, the final communality estimates (ie, pretest and posttest reliability) are always identical and therefore are likely to be inaccurate estimates of reliability at one or both occasions. A modification to the RCI method to incorporate test reliability at both pretest and posttest would be beneficial.
Second, on the issue of correcting for practice effects Bruggemans and associates point out that in general the magnitude of test-retest improvement bears a strong relationship to the level of pretest performance. To take this into account when defining change in patients with varying levels of pretest performance, they matched patients with selected subgroups of control subjects with similar pretest scores. We encountered problems when we attempted this. First, a decision had to be made as to how similar the pretest scores of control subjects and patients had to be in order to be deemed a "match," thus introducing a sampling error. Second, whether we used a perfect match (identical pretest scores) or an almost perfect match (eg, ±5 seconds on Trail Making Test), we could not gather a sufficient number of control subjects per patient to "minimize the effects of irrelevant idiosyncratic variations in the change scores of individual control subjects" [3; p. 548]. Of course, this was partly a function of our 2:1 ratio of patients to controls. However, it should be noted that in the study by Bruggemans and colleagues [3], which included equal numbers of control subjects and patients (n = 63 in each group), an average of only 10 matched controls were gathered for each patient. This implies that on occasions there were even fewer than 10 controls per patient. Obviously group means derived from such small subgroups are likely to be unstable and are therefore a potential source of error. It is questionable whether this source of error is more or less serious than that associated with using the mean practice effect of a larger control group that is not individually matched with patients. Given that few studies, including ours, incorporate as many (or more) control subjects as patients by which sufficiently large matched subgroups can be created, it is clear that, as Bruggemans and coworkers state, "applying corrections for mean practice effects such as those adopted in the RSTChel index will then be the only viable option" [3; p. 557].
As a final comment, regarding the practical use of the indices of Bruggemans and associates [3], we found the process of calculating indices derived in this manner to be extremely cumbersome and time-consuming, factors that ultimately may limit the application of these indices outside of purely statistical circles. In addition, in their comparison of overestimation and underestimation of neuropsychological deficits they assume that the deficit as defined by their proposed index is correct, but do not present any evidence that their index provides a better fit of the actual distribution of difference scores.
Overall, we acknowledge the cautions of Bruggemans and colleagues against using the RCI, and we applaud their efforts to develop a statistical approach that addresses the shortfalls of existing approaches in measuring pretest to posttest change. Unfortunately, however, we believe that the theoretical advantages afforded by their approach are offset by its inapplicability to commonly used two-occasion pretest-posttest study designs, as well as its lack of practical feasibility in terms of control subject recruitment, and finally its user friendliness. We suggest that currently the method proposed by Chelune and colleagues [2], which we have used, is appropriate until a simple, broadly applicable and theoretically watertight statistical approach to define change can be developed.
References
Related Article
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
