HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Collie, A. Articles by Maruff, P. Search for Related Content PubMed PubMed Citation Articles by Collie, A. Articles by Maruff, P.

Ann Thorac Surg 2002;73:2005-2011
© 2002 The Society of Thoracic Surgeons

---

Review

Determining the extent of cognitive change after coronary surgery: a review of statistical procedures

^a Neuropsychology Laboratory, Mental Health Research Institute of Victoria, Parkville, Australia
^b Behavioural Neurology Laboratory, Mental Health Research Institute of Victoria, Parkville, Australia
^c Center for Neuroscience, The University of Melbourne, Parkville, Australia
^d School of Psychological Science, La Trobe University, Bundoora, Australia
^e Centre for Anesthesia and Cognitive Function, St. Vincent’s Hospital, Melbourne, Victoria, Australia

* Address reprint requests to Dr Collie, Neuropsychology Laboratory, Mental Health Research Institute of Victoria, Locked Bag 11, Parkville, Victoria 3052, Australia
e-mail: alex{at}neuro.mhri.edu.au

	Abstract

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
Currently, cognitive decline after coronary surgery is said to be significant if the individual’s postoperative test score is at least 1 standard deviation (SD) worse than their preoperative score. This "1-SD" technique fails to account for factors that may confound interpretation of serially acquired cognitive test scores, including regression to the mean, measurement error caused by poor test-retest reliability, and practice effects. We review the many alternative and potentially superior statistical techniques that have been described in the neuropsychologic and psychiatric literature for differentiating "true" changes in cognitive test score from changes caused by these factors.

	Introduction

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
Decreases in mortality after elective coronary artery surgery have led to the adoption of patient’s cognitive function as an important measure of surgical outcome. This follows observations over the past two decades that a large proportion of patients undergoing coronary artery bypass grafting (CABG) exhibit postoperative cognitive deficits for up to 5 years [1–4]. Investigations of perioperative cognitive function follow a common experimental protocol, whereby the patient is assessed on a short battery of cognitive tests preoperatively, and again postoperatively at intervals of days, weeks, months, or years. Any change in the patient’s cognitive status is then determined by comparing the individual’s postoperative score with their preoperative score. Considerable attention has been given to methodological issues associated with the assessment of cognitive function before and after coronary surgery, including selection of cognitive tests, the setting in which testing occurs, and the potential effects of other patient-related factors (eg, age, mood) [5]. However, much less consideration has been given to the statistical techniques used to guide decisions about the presence or absence of cognitive dysfunction after cardiac surgery.

Conventionally, statistical comparisons in coronary surgery studies are made using the 1-standard deviation (SD) method or standard deviation index (SDI), where the individual is said to have significant cognitive decline if their postoperative cognitive test score is worse than their preoperative score by more than 1 SD of an appropriate reference group [6, 7]. Another statistical technique employed commonly is the 20% change method, where significant cognitive decline is said to have occurred if postoperative test score deteriorates by greater than 20% of the preoperative score [6, 8], or on more than 20% of tests administered. A large number of studies have used these analytic techniques, leading to their general acceptance as a de facto standard for determining the significance of change in cognitive test score after cardiac surgery (for review, see reference 9). Multiple publications have utilized these methods, including a recent article in the New England Journal of Medicine [2]. Both the SDI and 20% change methods have important shortcomings that may lead to false conclusions (discussed below). This situation has arisen despite the availability of alternative and demonstrably superior statistical techniques for determining the significance of change in cognitive test score (Table 1). While many of these alternative techniques have been employed in the neuropsychological and psychiatric literature for at least a decade, only a small number of studies have investigated their validity for determining the incidence and extent of cognitive decline after cardiac surgery [10, 11]. Further, only one of these alternative techniques was discussed in an otherwise elegant review of methodological issues associated with detecting change in cognitive status after cardiac surgery [12].

The aim of the present article is to describe the statistical techniques that have been developed in the neuropsychological and psychiatric literature for differentiating "true" change in test score from measurement error, practice effects, and regression to the mean (Table 1). These techniques differ according to whether they may be applied to groups or to individuals, whether they require appropriate control data, and whether they account for the potential effects of assessment-related factors as described above. This article first briefly describes the issues associated with serial cognitive assessment, critically reviews a number of statistical techniques in relation to these issues, and then summarizes the results of some recent studies that have compared the clinical utility of these techniques. A further aim is to compare the SDI and percent change methods commonly employed in the field of cardiac surgery to other available statistical techniques. The application of these statistical techniques to cognitive test data is addressed, rather than their mathematical or theoretical aspects. It is hoped that this review will lead to the consideration of more appropriate analytical techniques in future investigations of cognitive outcomes after cardiac surgery.

	Issues associated with serial cognitive assessment

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
Most conventional cognitive tests are designed for the identification of brain dysfunction, rather than the assessment of change in brain functions over time [14]. Although such tests are suitable for the investigation of brain-behavior relationships in cognitively impaired individuals, they possess two psychometric properties that restrict their applicability for serial assessment. First, many cognitive tests have limited or nonequivalent alternate forms that may result in performance changes due to practice effects [13, 15], and low test-retest reliability resulting in increased measurement error and regression to the mean [16]. Second, many of these tests have floor or ceiling effects when they are administered to normal people, and often have only a limited range of possible scores (eg, 0 to 10 on a word learning test). These factors ensure that large changes in cognitive status are required for small changes in test score to be observed, and may mean that mild but "true" changes in cognition are not reflected as a change in test score. Other factors that may also affect test score on serial assessment include age, education, intelligence, gender, and type of disease [16]. Furthermore, assessment-related factors such as anxiety, fatigue, and stress may also affect the magnitude of change in test score on serial assessment [16].

Regression toward the mean is a statistical term that describes the phenomenon whereby an extreme test score derived from an individual at one assessment tends to revert toward the mean of the group of which that individual is a member at a follow-up assessment [16]. Thus, the test score of an individual who performs highly at one assessment is likely to decline at a subsequent assessment, while the test score of an individual who scores poorly at one assessment is likely to improve at a subsequent assessment, without any intrinsic change in that individual’s specific abilities. Regression to the mean may therefore confound interpretation of cognitive data when the SDI and percent change methods are employed. Specifically, individuals whose preoperative test score is better than the group mean (ie, extreme) are more likely to be rated as having cognitive decline postoperatively than individuals with average or poor preoperative performance [3]. For example, the effects of regression to the mean were evident in a recent study by Newman and colleagues [2], who reported that a significant predictor of postoperative cognitive decline after CABG was a high preoperative cognitive score. The magnitude of regression to the mean is exacerbated when the test used to rate cognitive status has poor reliability, as greater amounts of measurement error result in greater regression to the mean.

As mentioned above, an individual’s cognitive test score may improve with repeated assessment purely due to increased test familiarity (ie, practice effects). The magnitude of these practice effects may be modulated by the length of the test-retest interval, as longer intervals result in reduced practice effects and vice versa [17, 18]. Conventionally, investigators have adopted alternate forms of a test in order to reduce the magnitude of practice effects. However, practice effects are still observed in studies where alternate forms of the same test have been used [13]. The concurrent assessment of an appropriate control group, or the availability of control group data, is therefore essential for accurate clinical decision making. However, serially acquired normative data collected at clinically relevant testing intervals are rare for most cognitive tests. Erroneous statistical analysis may be a consequence of failure to employ an adequate control group. For example, one recent coronary surgery study [19] employed a 0.5-SD criterion for cognitive decline after CABG (compared with the conventional 1 SD) on the basis of longitudinal cognitive test data described by Mitrushina and Satz [20], who reported that the average effect of practice on cognitive tests was of the order of 0.5 SD. These authors assumed that an observed postoperative decline of 0.5 SD was therefore equivalent to a "true" decline of greater than 1 SD when the practice effect was taken into account (ie, 0.5-SD decline = 1-SD decline - 0.5-SD practice improvement). However, the control data used in this study are inadequate, being from an entirely different sample to that under study (ie, coronary surgery patients). Other methodological differences between these studies may also have affected the magnitude of the practice effect, and therefore the accuracy of any conclusions. For example, there were no intermediate assessments in the 1-year test-retest interval employed by Mitrushina and Satz [20], while the CABG patients were reassessed at 1 and 12 months [19]. The intermediate assessment may have acted to increase the magnitude of the practice effect in the CABG study, which in this case would have led to underestimation of the incidence of cognitive decline after coronary surgery.

Another common strategy for controlling practice effects is to adopt a dual baseline, and use the second assessment as the "true" baseline for subsequent comparison. This strategy assumes that practice effects operate only between the first and second administrations of a test; however, the validity of this assumption has not been validated in any systematic manner. The dual-baseline approach also fails to account for other factors, such as regression to the mean, which will continue to operate between any two assessments (eg, second baseline and postoperative assessments), regardless of the number of preceding assessments. Even this brief review reveals that there are few adequate methodological strategies for reducing error in test-retest studies. This has led to the development of statistical approaches that attempt to partial out "true" changes in cognitive test score due to an independent variable (eg, brain damage) from artificial or test-related changes and measurement error.

	Statistical techniques for determining the significance of change in cognitive test score

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
Table 1 lists some common statistical procedures for assessing change in cognitive test score, defines them statistically, and provides a reference where they have been applied to cognitive data. In this section, we discuss those techniques that provide the best correction for sources of measurement error and other systematic influences on test performance.

Reliable change index
Reliable Change Indices (RCI) are calculated by dividing the individual’s test-retest difference score by the standard error of that difference score (SE_diff [21]). In turn, the SE_diff can be calculated from the standard error of measurement (SEM). The "SE_diff describes the range of the distribution of change scores that would be expected if no actual change had occurred" [21]. RCIs are usually regarded as standardized scores, and therefore, an RCI larger than 1.96 will occur in less than 5% of cases. Advantages of RCI include that they account for test reliability at both baseline and follow-up assessments, thus allowing for regression to the mean. That is, the less reliable the test, the greater the test-retest difference score required for a significant change. Also, this RCI may be calculated for individuals without reference to control group data. However, the standard RCI does not correct for the effects of measurement error due to practice or other confounding variables. This requires manipulation of the numerator and has led to the application of modified RCIs.

Modified reliable change indices (MRCI1 and MRCI2)
In modified RCIs, a constant is placed in the numerator to reflect the extent of change expected to occur as a result of a confounding variable, or some alteration is made to the denominator to compensate for measurement error. An example is the RCI described by Chelune and associates [22], in which the RCIs calculated for epilepsy patients undergoing temporal lobectomy were corrected for the effects of practice, by first calculating the magnitude of the practice effect in a group of matched but nonsurgical epilepsy patients, and then adding this value to the numerator in the RCI (see Table 1). Although this method has the advantage of the practice effect correction, it also requires that data be available for an appropriate control group at a similar test-retest interval. Unfortunately, such data are rarely available in clinical settings. Another modified RCI is that described by Zegers and Hafkenscheid [23], who suggest replacing the raw change score in the Chelune and associates’ [22] equation (numerator) with an estimated "true" change score, and the standard error of the difference (denominator) with an estimated "true" standard error of the difference (formula in Table 1). Although this RCI provides further correction for measurement error, it requires that control data be available. Furthermore, it also requires knowledge of the reliability of the cognitive test in an appropriate control group. These modified RCIs may also be limited by their use of control group data to correct for individual practice effects, as prior research suggests that the magnitude of practice effects may vary considerably between individuals [24].

Reliability-stability index
The reliability-stability index (RSI) described by Bruggemans and colleagues [10] subtracts the output from the RCI of Zegers and Hafkenscheid [22] described above from a modified version of this RCI where the individual patient’s baseline and retest scores are replaced by the mean baseline and retest scores of a small (approximately n = 10) appropriate matched control group. More simply, the RSI of Bruggemans and colleagues represents the RCI of a matched control group subtracted from the RCI of the individual patient. Data are treated as with previously described RCIs. Being a combination of previously described RCIs, this method corrects for measurement error, individual variability, and practice effects. However, as with all other RCIs, it is not applicable to the single case, as control and appropriate test-retest reliability data are required.

Simple regression
In the simple regression method described by McSweeney and colleagues [14], a linear equation is calculated on the basis of the mean baseline and retest data from the groups of subjects described by Chelune and associates [22]. This equation is then applied to individual patient’s baseline data and a predicted retest score is obtained. The difference between the predicted and observed retest score is then divided by the standard error of the estimate (SE_est) of the control group regression equation. McSweeney and colleagues identified a significant change at retesting when this value was greater than a certain criterion. Although this method allows for both practice effects and individual variability, the raw statistic does not adequately control for measurement error. That is, like the SDI and SEM techniques described above, the simple regression method incorrectly assumes that the baseline score is perfectly reliable (ie, free from measurement error). This may be remedied by including some estimate of reliability in the regression equation.

Multiple regression
The multiple regression method described by Temkin and colleagues [25] is similar to the simple regression method of McSweeney and associates [22], in that predicted retest scores are obtained on the basis of mean baseline data collected in a group of subjects. The multiple regression method is, therefore, subject to the same limitations as the simple regression method. However, multiple regression predictions of performance attempt to account for many sources of individual variability (eg, age, level of education, gender) by including variables defining the influence of these factors in the regression equation (Table 1). The major advantage of this method is that it allows the derivation of predicted scores (and subsequently decisions regarding the normality of observed predicted-obtained score differences) for individuals of different ages, levels of education, gender, etc. However, large groups are required to formulate accurate multiple regression estimates of change.

	Examples from published reports

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
A number of recent studies have investigated the ability of these different statistical techniques to differentiate between "true" changes in cognition and changes attributable to measurement error. These studies aimed to determine the practical ability of these models to predict a follow-up score from a baseline score. For example, Temkin and colleagues [25] compared standard and practice effect–corrected RCIs with linear and multiple regression techniques as predictors of follow-up performance in a group of 384 normal adults. The corrected RCI, linear, and multiple regression methods were equally accurate at predicting follow-up score, while the standard RCI was least accurate. The generalizability of these prediction models were investigated in a later study by the same group [30], by applying them to serial cognitive data collected from a smaller, nonclinical sample (n = 124), a group of patients with schizophrenia (n = 69), a group of subjects recovering from traumatic brain injury (n = 23), and a group of subjects in whom a brain insult occurred between baseline and follow-up assessments (n = 10). All change models performed best in predicting the follow-up score of the nonclinical group, and poorly in predicting the follow-up score of the schizophrenia and brain insult groups, indicating that prediction models developed in nonclinical samples may not be generalizable to patient groups. Also, the accuracy of each model did not differ substantially, and therefore, Heaton and colleagues recommended use of the simpler RCI with practice effect correction over the more complex regression models.

Bruggemans and colleagues [10] applied six statistical techniques to test-retest cognitive data acquired from 63 patients undergoing CABG. These included the SDI, the RCI, the MRCI1 and MRCI2, the RSI, and simple regression model. Control data were collected from the spouses of the cardiac patients. These authors determined the deterioration rates in the CABG patients for each cognitive test administered according to each of these models. Techniques that correct for practice effects were observed to provide the best estimates of deterioration rates when test-retest reliability was high and when large practice effects were observed in the control group. In contrast, techniques that correct for measurement error (and therefore regression to the mean) were observed to operate best when test-retest reliability was low. These results indicate that the psychometric properties of the cognitive test employed may need to be considered when selecting a method to determine the clinical significance of an observed change in an individual performing that test.

Arndt and colleagues [26] compared the ability of simple change, simple regression, tau_a, tau_b, and the nonparametric slope (among other measures) to measure the symptom course of patients with schizophrenia and affective disorders. These authors used measures of effect size, statistical power, and Type 1 error rates derived from data sets submitted to bootstrapping techniques as their outcome variables. The ability of these change methods to detect correlations between symptom course and independent variables (eg, age, gender) was also determined. Both Kendall’s tau methods provided acceptable estimates of symptom course and provided the greatest statistical power to detect any change in course. Both tau measures were able to detect correlations with independent variables, and also recorded acceptable Type 1 error rates (approximately 5%). This important study highlights the advantages of an alternative approach to those conventionally considered in the neuropsychological literature.

	Summary and conclusions

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 
As noted above, the statistical techniques described here have arisen partly because of the psychometric limitations of many conventional cognitive and neuropsychological tests (eg, ceiling effects, poor reliability, limited range of possible scores). These techniques may be broken into three broad subtypes according to whether they attempt to account for none, some, or all possible sources of measurement error and the influence of other systematic influences on test performance. The first subtype are those techniques that do not adequately account for either measurement error or practice effects, and include the simple and percentage change methods, the SDI, Cohen’s d', Kendall’s tau, and the nonparametric slope. The second subtype are those that provide more acceptable management of measurement error and regression to the mean, but do not account for practice effects. Included in this category are the "true" change score, the standard error of measurement index (SEMI), and the simple RCI. All of these methods include in their calculation some estimate of test reliability, whether it be the r² value, the SEM, or the SE_diff. The third subtype are those that account for practice effects and measurement error. Included in this category are the simple and multiple regression methods, the modified RCIs, and the RSI. For the simple and multiple regression methods, practice effects are controlled automatically by the regression statistic, and efforts to correct for measurement are routinely made by including some estimate of test reliability in the equation. For the modified RCIs and the RSI, measurement error is controlled by the inclusion of the SE_diff or reliability coefficient in the denominator, and practice effects are managed by the inclusion of control group data.

It should be clear from this review that in research studies, including those of cognitive change subsequent to coronary surgery, the selection of a statistical technique to determine change in cognitive status must be made on the basis of the psychometric properties of the tests used, but also with respect to the methodological design of the individual study. For example, analysis of the psychometric properties of cognitive tests commonly used in CABG research suggests that the SDI is inappropriate for determining the significance of change in test score (Table 2). This was partially acknowledged very recently by Murkin [28], in an editorial where the RCI method was put forward as a possible alternative or adjunct to the SDI and percent change methods. However, we propose that the RCI method will not always be appropriate and that other statistical techniques should be given due consideration.

We have summarized here the statistical techniques currently employed in the neuropsychological literature to differentiate "true" change in test score from change due to measurement error and practice effects. Although some techniques perform quite well and facilitate accurate clinical decisions, others fail to adequately account for possible confounding factors. These techniques may be differentiated by the extent to which they account for measurement error, regression to the mean, and practice effects. The development of the more complex and competent techniques was initiated because of the unreliability and error inherent in many conventional cognitive tests. More accurate assessment of cognitive function in CABG research may be gained through implementation of these techniques to data gained from cognitive tasks that allow accurate serial assessment.(27)

	References

Top Abstract Introduction Issues associated with serial... Statistical techniques for... Examples from published reports Summary and conclusions References

 

1. Newman S., Smith P., Treasure T., et al. Acute neuropsychological consequences of coronary artery bypass surgery. Curr Psychol Res Rev 1987;6:115-124.
2. Newman M.F., Kirchner J.L., Phillips-Bute B., et al. Longitudinal assessment of neurocognitive function after coronary-artery bypass surgery. N Engl J Med 2001;344:395-402.[Abstract/Free Full Text]
3. Browne S.M., Halligan P.W., Wade D.T., Taggart D.P. Cognitive performance after cardiac operation: implications of regression to the mean. J Thorac Cardiovasc Surg 1999;117:481-485.[Abstract/Free Full Text]
4. Royse A.G., Royse C.F., Ajani A.E., et al. Reduced neuropsychological dysfunction using epiaortic echocardiography and the exclusive Y graft. Ann Thorac Surg 2000;69:1431-1438.[Abstract/Free Full Text]
5. Murkin J.M., Newman S.P., Stump D.A., Blumenthal J.A. Statement of consensus on assessment of neurobehavioural outcomes after cardiac surgery. Ann Thorac Surg 1995;59:1289-1295.[Free Full Text]
6. Blumenthal J.A., Mahanna E.P., Madden D.J., et al. Methodological issues in the assessment of neuropsychologic function after cardiac surgery. Ann Thorac Surg 1995;59:1345-1350.[Abstract/Free Full Text]
7. Mahanna E.P., Blumenthal J.A., White W.D., et al. Defining neuropsychological dysfunction after coronary artery bypass grafting. Ann Thorac Surg 1996;61:1342-1347.[Abstract/Free Full Text]
8. Stump D.A. Selection and clinical significance of neuropsychologic tests. Ann Thorac Surg 1995;59:1340-1345.[Abstract/Free Full Text]
9. Symes E., Maruff P., Ajani A., Currie J. Issues associated with the identification of cognitive change following coronary artery bypass grafting. ANZ J Psychiatr 2000;34:770-784.
10. Bruggemans E.F., Van de Vijver F.J.R., Huysmans H.A. Assessment of cognitive deterioration in individual patients following cardiac surgery: correcting for measurement error and practice effects. J Clin Exp Neuropsychol 1997;19:543-559.[Medline]
11. Kneebone A.C., Andrew M.J., Baker R.A., Knight J.L. Neuropsychologic changes after coronary artery bypass grafting: use of reliable change indices. Ann Thorac Surg 1998;65:1320-1325.[Abstract/Free Full Text]
12. Slade P., Sanchez P., Townes B., Aldea G.S. The use of neurocognitive tests in evaluating the outcome of cardiac surgery: some methodologic considerations. J Cardiothorac Vasc Anesth 2001;15:4-8.[Medline]
13. Benedict R.H.B., Zgaljardic D.J. Practice effects during repeated administrations of memory test with and without alternate forms. J Clin Exp Neuropsychol 1998;20:339-352.[Medline]
14. McSweeney A.J., Naugle R.I., Chelune G.J., Luders H. "T Scores for Change": an illustration of a regression approach to depicting change in clinical neuropsychology. Clin Neuropsychologist 1993;7:300-312.
15. McCaffrey R.J., Ortega A., Orsillo S.M., et al. Practice effects in repeated neuropsychological assessments. Clin Neuropsychologist 1992;6:32-42.
16. McCaffrey R.J., Duff K., Westervelt H.J. Practitioner’s guide to evaluating change with neuropsychological assessment instruments. New York: Kluwer Academic/Plenum Publishers, 2000.
17. Catron D.W. Immediate test-retest changes in WAIS scores among college males. Psychol Rep 1978;43:279-290.
18. Catron D.W., Thompson C.C. Test-retest gains in WAIS scores after four test-retest intervals. J Clin Psychol 1979;35:352-357.[Medline]
19. McKhann G.M., Goldsborough M.A., Borowicz L.M., et al. Cognitive outcome after coronary artery bypass: a one-year prospective study. Ann Thorac Surg 1997;63:510-515.[Abstract/Free Full Text]
20. Mitrushina M., Satz P. Effect of repeated administration of a neuropsychological battery in the elderly. J Clin Psychol 1991;47:790-800.[Medline]
21. Jacobson N.S., Traux P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12-19.[Medline]
22. Chelune G.J., Naugle R.I., Luders H., et al. Individual change after epilepsy surgery: practice effects and base-rate information. Neuropsychology 1993;7:41-52.
23. Zegers F.E., Hafkenscheid A. The ultimate reliable change index: an alternative to the Hageman & Arrindell approach. Groningen, The Netherlands: University of Groningen, 1994.
24. Matarazzo J.D., Herman D.O. Base rate data for the WAIS-R: test-retest stability and VIQ-PIQ differences. J Clin Neuropsychol 1984;6:351-366.[Medline]
25. Temkin N.R., Heaton R.K., Grant I., Dikmen S.S. Detecting significant change in neuropsychological test performance: a comparison of four models. J Int Neuropsychol Soc 1999;5:357-369.[Medline]
26. Arndt S., Turvey C., Coryell W.H., et al. Charting patients’ course: a comparison of statistics used to summarize patient course in longitudinal and repeated measures studies. J Psychiatr Res 2000;34:105-113.[Medline]
27. Heaton R.K., Temkin N., Dikmen S.S., et al. Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Arch Clin Neuropsychol 2001;16:75-91.
28. Murkin J.M. Perioperative neuropsychologic testing. J Cardiothorac Vasc Anesth 2001;15:1.[Medline]
29. Maassen G.H. Principles of defining reliable change indices. J Clin Exp Neuropsychol 2000;22:622-632.[Medline]
30. Jacobson N.S., Follette W.C., Revenstorf D. Psychotherapy outcome research: methods for reporting variability and evaluating clinical significance. Behav Therapy 1984;15:336-352.

This article has been cited by other articles:

				P. Maruff, E. Thomas, L. Cysique, B. Brew, A. Collie, P. Snyder, and R. H. Pietrzak Validity of the CogState Brief Battery: Relationship to Standardized Tests and Sensitivity to Cognitive Impairment in Mild Traumatic Brain Injury, Schizophrenia, and AIDS Dementia Complex Arch Clin Neuropsychol, March 25, 2009; (2009) acp010v1. [Abstract] [Full Text] [PDF]

				F. Hernandez Jr, J. R. Brown, D. S. Likosky, R. A. Clough, A. L. Hess, R. M. Roth, C. S. Ross, C. M. Whited, G. T. O'Connor, and J. D. Klemperer Neurocognitive Outcomes of Off-Pump Versus On-Pump Coronary Artery Bypass: A Prospective Randomized Controlled Trial Ann. Thorac. Surg., December 1, 2007; 84(6): 1897 - 1903. [Abstract] [Full Text] [PDF]

				J. Vardy, S. Rourke, and I. F. Tannock Evaluation of Cognitive Function Associated With Chemotherapy: A Review of Published Studies and Recommendations for Future Research J. Clin. Oncol., June 10, 2007; 25(17): 2455 - 2463. [Abstract] [Full Text] [PDF]

				J. R. Keith, D. J. Cohen, and L. B. Lecci Why Serial Assessments of Cardiac Surgery Patients' Neurobehavioral Performances are Misleading Ann. Thorac. Surg., February 1, 2007; 83(2): 370 - 373. [Full Text] [PDF]

				S. Al-Ruzzeh, S. George, M. Bustami, J. Wray, C. Ilsley, T. Athanasiou, and M. Amrani Effect of off-pump coronary artery bypass surgery on clinical, angiographic, neurocognitive, and quality of life outcomes: randomised controlled trial BMJ, June 10, 2006; 332(7554): 1365. [Abstract] [Full Text] [PDF]

				M. S. Lewis, P. Maruff, B. S. Silbert, L. A. Evered, and D. A. Scott Detection of Postoperative Cognitive Decline After Coronary Artery Bypass Graft Surgery is Affected by the Number of Neuropsychological Tests in the Assessment Battery Ann. Thorac. Surg., June 1, 2006; 81(6): 2097 - 2104. [Abstract] [Full Text] [PDF]

				P. D. Raymond, A. D. Hinton-Bayre, M. Radel, M. J. Ray, and N. A. Marsh Assessment of statistical change criteria used to define significant change in neuropsychological test performance following cardiac surgery Eur. J. Cardiothorac. Surg., January 1, 2006; 29(1): 82 - 88. [Abstract] [Full Text] [PDF]

				G H Maassen and A Hinton-Bayre Reliable change assessment in sport concussion research: a comment on the proposal and reviews of Collie et al * Commentary Br. J. Sports Med., August 1, 2005; 39(8): 483 - 488. [Abstract] [Full Text] [PDF]

				C. D. Hanning Postoperative cognitive dysfunction Br. J. Anaesth., July 1, 2005; 95(1): 82 - 87. [Full Text] [PDF]

				R. Motallebzadeh and M. Jahangiri Benefits of off-pump bypass on neurologic function Ann. Thorac. Surg., September 1, 2004; 78(3): 1131 - 1132. [Full Text] [PDF]

				B. S. Silbert, P. Maruff, L. A. Evered, D. A. Scott, M. Kalpokas, K. J. Martin, M. S. Lewis, and P. S. Myles Detection of cognitive decline after coronary surgery: a comparison of computerized and conventional tests Br. J. Anaesth., June 1, 2004; 92(6): 814 - 820. [Abstract] [Full Text] [PDF]

				A Collie, P Maruff, M McStephen, and D Darby Are Reliable Change (RC) calculations appropriate for determining the extent of cognitive change in concussed athletes? Br. J. Sports Med., August 1, 2003; 37(4): 370 - 372. [Full Text] [PDF]

				D. Whitaker The use of Z scores in assessing neuropsychological change after cardiac operations Ann. Thorac. Surg., March 1, 2003; 75(3): 1066 - 1066. [Full Text] [PDF]

				A. Collie, D. G. Darby, P. Maruff, and B. S. Silbert The use of Z scores in assessing neuropsychological change after cardiac operations: Reply Ann. Thorac. Surg., March 1, 2003; 75(3): 1066 - 1067. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Collie, A. Articles by Maruff, P. Search for Related Content PubMed PubMed Citation Articles by Collie, A. Articles by Maruff, P.