HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Michael A. Borger Tirone E. David Gideon Cohen Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Ivanov, J. Articles by Naylor, C. D. Search for Related Content PubMed PubMed Citation Articles by Ivanov, J. Articles by Naylor, C. D.

Ann Thorac Surg 2000;70:162-168
© 2000 The Society of Thoracic Surgeons

---

Original articles: Cardiovascular

Predictive accuracy study: comparing a statistical model to clinicians’ estimates of outcomes after coronary bypass surgery

^a Division of Cardiovascular Surgery, the Toronto General Hospital and University of Toronto, Toronto, Ontario, Canada

Address reprint requests to Ms Ivanov, Division of Cardiovascular Surgery, Toronto General Hospital, CCRW 4-803, 200 Elizabeth St, Toronto, ON, Canada M5G 2C4
e-mail: joan.ivanov{at}uhn.on.ca

	Abstract

Top Abstract Introduction Patients and methods Results Comment References

 
Background. The purpose of this study was to compare clinicians’ prior probability estimates of operative mortality (OM) and prolonged intensive care unit stay (ICU) length of stay greater than 48 hours after coronary artery bypass graft surgery (CABG) with estimates derived from statistical models alone.

Methods. Nine clinicians estimated the predicted probability of OM and ICU stay greater than 48 hours from an abstract of information for each of 100 patients selected from the 1996 to 1997 database of 1,904 patients who underwent isolated CABG. Logistic regression models were used to calculate the predicted probability of OM and ICU stay greater than 48 hours for each patient. The study sample was split into two parts; clinicians were randomly given access to a predictive rule to guide their judgements for one part of the study.

Results. Clinicians’ estimates were similar with or without access to the rule, and both parts of the study were therefore pooled. Clinicians significantly overestimated the probability of OM (model 6.3% ± 1%, clinicians 7.6% ± 3%, p = 0.0001) and ICU stay greater than 48 hours (model 25% ± 2%, clinicians 28% ± 1%, p = 0.0012). Clinicians’ estimates of OM were not significantly higher than the model’s for nonsurvivors (0.8% ± 0.7%, p = 0.2), but were significantly higher for survivors (1.4% ± 0.3%, p = 0.039).

Conclusions. Clinicians trusted their own empiric estimates rather than a predictive rule and overestimated the probability of OM and ICU stay greater than 48 hours.

	Introduction

Top Abstract Introduction Patients and methods Results Comment References

 
Several clinical predictive rules have been developed for cardiac surgery to characterize case-mix and estimate the probability of outcomes. The primary purpose of these predictive rules has been to calculate risk-adjusted outcomes for provider profiling [1–12]. Other applications for statistical models include assisting administrators in determining resource allocation, helping providers assess quality of care, and identifying high-risk subgroups for research purposes [3, 6, 7, 9, 13–16]. As well, patients may ask for probabilistic information regarding diagnosis, prognosis, and response to treatment [15, 17–19]. Statistical models can then be used to facilitate discussions between surgeons and their patients regarding the risks and benefits of a particular procedure. However, clinicians sometimes claim that their intuition and experience give them advantages over any statistical model in predicting adverse outcomes, and that the variables included in models reflect averages, rather than patient-specific clinical features that may put patients at risk. We accordingly undertook a study to: (1) compare the judgements made by a statistical model to those made by clinicians, and (2) evaluate the use of a predictive rule as an aid to clinicians in making estimates of the probability of operative mortality and prolonged intensive care unit (ICU) length of stay greater than 48 hours. To give maximum advantages to clinical judgement, we compared predictive performance for a group of patients assembled purposively to capture a cross section of both high- and lower-risk individuals who had suffered adverse events, and provided clinicians with additional clinical information not incorporated into the statistical model.

	Patients and methods

Top Abstract Introduction Patients and methods Results Comment References

 
Sampling frame and case preparation
We drew 100 cases (study sample) from the 1,904 patients who underwent isolated coronary artery bypass grafting (CABG) at the Toronto General Hospital between January 1996 and April 1997 (hereafter, the study population). We deliberately over-sampled for 70% of deaths, drawing 29 of 41 postoperative fatalities from the general patient population, covering a range of higher- and lower-risk patients who died after surgery. The remaining 71 patients were sampled from survivors. The 100 study patients were then randomly divided into two 50 patient groups, Parts I and II.

A half-page abstract was prepared for each patient. Again, to support clinical judgements, the abstract contained a table of information covering many more variables than the statistical models that were derived and validated by our group [1, 12, 16]. The variables drawn from our clinical cardiac surgery database included: age, gender, left ventricular ejection fraction, New York Heart Association functional class, anginal symptoms (ie, stable, unstable, acute coronary insufficiency), previous CABG, timing of surgery (ie, elective, same-hospitalization, urgent, emergent), left main disease, positive exercise stress test, recent myocardial infarction within the month before surgery, diabetes, peripheral vascular disease (including carotid stenoses), history of hypertension, renal insufficiency, height, and weight. Results of coronary angiography were also shown in detail as a diagram along with information regarding the quality of the vessels (ie, degree of stenosis, and vessel size, quality of distal vessels, and collateral flow), when available. A brief narrative was prepared that described each patient’s relevant history, presenting scenario, and other comorbidity.

Predictive rule
A predictive rule, developed from the combined CABG database of two Toronto hospitals for the years 1993 to 1996, was used as the format for the contemporary, site-specific guidelines. Risk weights for this rule were recalibrated in the 1993 to 1997 database for the Toronto General Hospital only. Risk weights and cutoff points of the total risk score, which defined relative risk groups (low, medium, high), can be found in Table 1. Details regarding the development of this rule have been published [12]. By presenting a simple and additive scoring system, we hoped to maximize the system’s acceptance by clinicians.

Clinicians were randomized to receive the predictive rule as an aid for Part I or Part II of the study. After a 4-month "washout" period, clinicians who received the predictive rule for Part I did not receive it for Part II of the study, and vice versa.

Outcomes
Clinicians were asked to estimate, for each patient, the probability of: (1) operative mortality (OM), defined as any postoperative, in-hospital death, and (2) prolonged ICU length of stay, defined as an ICU length of stay greater than 48 hours, which corresponded to the 83rd percentile of ICU length of stay for 1995 to 1997.

Statistical model
The model was essentially similar to the original predictive rule used to assist clinicians. Risk factors from the original rule, in addition to other important prognostic variables, were submitted to a logistic regression analysis to recalibrate regression coefficients and optimize precision [12].

In 1995, the pattern of practice changed regarding discharge from the ICU: patients were extubated earlier and discharged sooner than previous patients because of "fast-tracking" [20, 21]. Because of this changing pattern, the logistic regression model for the prolonged ICU stay outcome was rederived for only the 1995 to 1997 database of isolated CABG patients. An ICU stay of greater than 48 hours corresponded to the 83rd percentile of total ICU length of stay.

Regression coefficients from the newly derived models were used to calculate the patient-specific predicted probability of each outcome using methods previously described [12, 16, 22, 23].

Statistical analysis
Data were managed in dBASE IV datasets and analyzed using SAS for Windows Version 6.12 (SAS, Cary, NC) [24]. Results are presented as means ± standard errors. A two-tailed p value less than 0.05 indicates statistical significance unless otherwise noted.

The estimates of probability for OM and ICU stay greater than 48 hours with and without access to a predictive rule were evaluated by unpaired t test. The difference between the clinician’s estimate of the probability of OM and ICU stay greater than 48 hours and the model’s estimate was calculated for each patient (clinician minus model) and evaluated by paired t test. The null hypothesis for this analysis was that the difference in the predicted probability of an event equaled zero.

Discrimination (or predictive accuracy) was assessed by calculation of the area under the receiver-operator characteristic curve (ROC) for each set of estimates [25, 26]. The area under the ROC curve reflects the proportion of randomly paired sets of patients for which the patient experiencing the event has a higher predicted probability of having the event, compared with the patient who does not experience the event. An area under the ROC curve of 50% indicates no discriminatory ability, ROC areas above 70% represent fair discriminatory ability, and ROC areas above 80% represent good discriminatory ability.

Calibration, or precision at the group level, was evaluated by the Hosmer-Lemeshow goodness-of-fit ² statistic (HLGOF) [22]. Hosmer-Lemeshow goodness-of-fit p values less than 0.05 indicate a significantly imprecise model (ie, the model did not fit the data). Calibration is also important because, in contrast to the match between probabilities and categorical events reflected in the ROC curve area, it captures the ability of the model (or clinicians) to predict the absolute event rates on average.

	Results

Top Abstract Introduction Patients and methods Results Comment References

 
Statistical models
There were 7,313 patients who underwent isolated CABG at the Toronto General Hospital between 1993 and 1997. OM in this group was 2.3% (n = 169). In the 1995 to 1997 database of 4,572 patients undergoing isolated CABG, the prevalence of ICU stay greater than 48 hours was 17% (n = 778).

Logistic regression coefficients for both the OM model and the prolonged ICU stay model are found in Table 2. Both models demonstrated excellent discrimination and precision.

Sample characteristics
Table 3 depicts the demographics of the study population and the 100-patient study sample. The deliberate skew of the study sample is evident in both the prognostic variables and the prevalence of outcomes. There was no difference between Part I and Part II for any prognostic or outcome variable.

Difference between clinicians and the model
Predicted probabilities of outcomes
Figure 1 shows the difference by paired t test between each clinician’s predicted probability and the model’s calculations for OM (top) and ICU greater than 48 hours (bottom). The differences for OM were significant for the experienced clinicians (Nos. 1, 2, 3, 5, 6), whereas the more inexperienced clinicians (Nos. 7 to 9) demonstrated no significant difference from the model’s predictions. The junior clinicians’ (Nos. 7 to 9) estimates were significantly different from the senior clinicians (Nos. 1 to 5) by analysis of variance (p < 0.05).

View larger version (24K): [in this window] [in a new window]   Fig 1. The delta of the clinicians minus the statistical model’s estimates of the predicted probability of operative mortality (OM; top) and prolonged ICU length of stay greater than 48 hours (ICU > 48h; bottom) is depicted. The senior clinicians (Nos. 1 to 5) significantly overestimated the probability of OM compared with the statistical model. The junior clinicians’ (Nos. 7 to 9) estimates were significantly different (p < 0.05) from the seniors’ estimates of OM. There was no discernible pattern for ICU > 48h relating to seniority.

Table 4 shows the results of the pooled clinician sample compared with the statistical model. The clinicians significantly overestimated both total OM and ICU stay greater than 48 hours. However, when we evaluated probabilities for patients who died versus survivors, the differences for predictions of OM were significant only for survivors (clinicians 7.0% ± 9%, model 5.6% ± 7%, p = 0.0001); there was no significant difference in OM predictions for the nonsurvivors (clinicians 9.0% ± 10%, model 8.1% ± 9%, p = 0.23). Clinicians’ estimates of prolonged ICU time were significantly higher than the model’s for both those with short ICU stays (clinicians 22% ± 22%, model 20% ± 15%, p = 0.03) and those who did have an ICU length of stay greater than 48 hours (clinicians 39% ± 28%, model 34 ± 23%, p = 0.01).

View larger version (26K): [in this window] [in a new window]   Fig 2. The area under the ROC curve for operative mortality (OM; top) and prolonged ICU length of stay greater than 48 hours (ICU > 48 h; bottom). With the exception of one senior surgeon, discrimination was poor for OM. Discrimination was slightly better for estimates of ICU > 48 h.

	Comment

Top Abstract Introduction Patients and methods Results Comment References

In this study, we examined the comparative performance of clinicians and a statistical model in predicting adverse outcomes after CABG. We presented a group of clinicians with case abstracts for 100 patients, and deliberately skewed the sample by including 29 patients who died postoperatively. The additional information contained in the narrative section of the patient histories, coupled with the skew towards higher risk patients who experienced adverse events, was designed to give clinicians an advantage over the statistical model’s estimates of probability. We postulated that clinicians’ conceptual flexibility and intuition might allow them to identify patients at special risk more accurately than a statistical model.

The predictive rule as depicted in Table 1 was provided to each clinician for one phase of the study. The 4-month "washout" period between the two parts of the study was designed to prevent contamination. We were not particularly surprised that the majority of clinicians elected not to use the rule consistently to calculate patient-specific probabilities of each outcome. Indeed, we found no differences in predictive accuracy whether clinicians were exposed to the rule or not, and whether they claimed to use it consistently or not. These findings highlight the tendency of clinicians to rely on their own intuitive judgements rather than a statistical model.

Given the lack of impact of access to the predictive rule, we analyzed the results for both phases of the study together. Unlike previous studies, which have shown that clinicians outperformed quantitative models [27, 30, 31], a robust analysis of the paired difference of each clinician’s estimates of probability versus those made by the statistical model revealed that senior surgeons significantly overestimated the probability of operative mortality. One possible explanation for these results is that senior surgeons make typically broad predictions based on their recent experience [15], whereas the junior clinicians, lacking in experience, may rely more closely on published results.

The relatively poor performance of the statistical model in the 100-patient study sample was understandable given the deliberately skewed prevalence of outcomes in the study sample. Indeed, as noted, we expected that the approach towards a study sample with an increased prevalence of high-risk patients having outcomes would give clinicians an advantage. It did not.

One limitation of this study was that the clinicians were making estimates of mortality and prolonged ICU stay from a written synopsis of patient information rather than direct examination. However, clinicians were given the advantage by being provided with additional relevant clinical information in the narrative portion of each patient abstract. Also, the diagram of the cardiac catheterization results included distal vessel quality if it was noted to be poor. Despite these advantages, the clinicians did not out perform the statistical models, which relied solely on a limited number of variables. We doubt, therefore, that our findings would have been different if the study had taken place prospectively in practice as opposed to relying on written case abstracts.

As noted, the limited impact of access to a predictive rule was not surprising. Clinicians are trained to rely heavily on their own experience and intuitive judgements. Moreover, the literature on changing physician behavior amply illustrates that clinical decision-making is not readily influenced unless there is acceptance of a new practice norm by "opinion leaders" and a concerted local effort to address barriers to change (eg, by automating decision support systems) [32, 33]. However, surgeons and anesthetists at our center are already using an implied form of prior probability estimates to fast-track low-risk cardiac surgical patients through the ICU [20, 21]. In this regard, resources have been streamlined for those patients deemed to be at a low risk of poor outcome and, conversely, there is a greater concentration of resources for higher risk patients. We believe that with appropriate implementation strategies, clinicians will use statistical tools that enable them to make more accurate judgements of patients’ risks for adverse outcomes after CABG.

Summary
Predictive rules have the advantage of being able to integrate more information by relating both continuous and categorical variables to an outcome of interest. An accurate and precise model can be used as a concise summary of the relationships between prognostic variables and the outcome [34, 35].

Cardiac surgery clinicians, when given the option, preferred not to use a predictive rule but rather trusted their own judgements to estimate the probability of operative mortality or prolonged ICU length of stay after coronary bypass surgery. Experienced surgeons significantly overestimated the risk of operative mortality compared with their junior colleagues. Clinicians’ predictive accuracy was only fair for operative mortality, and only slightly better for estimates of prolonged ICU length of stay. Although no model can predict the specific individual who will have an adverse event, statistical models do permit reasonably accurate estimates of event rates for subgroups of patients. If methods were devised to ensure that clinicians can and do make use of predictive rules, they would be able to make more accurate judgements of the probability of adverse outcomes after CABG surgery.

Study participants
Surgeons: Tirone E. David, MD (chief), Lynda L. Mickleborough, MD, Charles M. Peniston, MD, Robert J. Cusimano, MD, Anthony C. Ralph-Edwards, MD, and Terrence M. Yau, MD. Residents: Gideon Cohen, MD, and Michael A. Borger, MD. Nurse clinician: Nancy Walton, BSc.

	Acknowledgments

 
Ms Ivanov is supported by a fellowship from the Heart and Stroke Foundation of Canada and National Health Research and Development Program. Dr Borger was supported by a fellowship from the Heart and Stroke Foundation of Canada. Dr Naylor was supported as a Senior Scientist of the Medical Research Council of Canada and is currently Dean of the Faculty of Medicine at the University of Toronto.

	References

Top Abstract Introduction Patients and methods Results Comment References

 

1. Tu J.V., Jaglal S.B., Naylor C.D., Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Circulation 1995;91:677-684.[Abstract/Free Full Text]
2. Tu J.V., Mazer C.D., Levinton C., Armstrong P.W., Naylor C.D. A predictive index for length of stay in the intensive care unit following cardiac surgery. Can Med Assoc J 1994;151:177-185.[Abstract]
3. O’Connor G.T., Plume S.K., Olmstead E.M., et al. A regional prospective study of in-hospital mortality associated with coronary artery bypass grafting. JAMA 1996;275:841-846.[Abstract]
4. Tu J.V., Sykora K., Naylor C.D. Assessing the outcomes of coronary artery bypass graft surgery. J Am Coll Cardiol 1997;30:1317-1323.[Abstract]
5. Charlson M., Ales K.L., Simon R. Why predictive indexes perform less well in validation studies. Arch Intern Med 1987;147:2155-2161.[Abstract]
6. Edwards F.H., Clark R.E., Schwartz M. Coronary artery bypass grafting. Ann Thorac Surg 1994;57:12-19.[Abstract]
7. Higgins T.L., Estafanous F.G., Loop F.D., Berck G.J., Blum J.M., Paranandi L. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients. JAMA 1992;267:2344-2348.[Abstract]
8. Hannan E.L., Kilburn H., Jr, O’Donnell J.F., Lukacik G., Shields E.P. Adult open heart surgery in New York state. JAMA 1990;264:2768-2774.[Abstract]
9. Luft H.S., Romano P.S. Chance, continuity and change in hospital mortality rates. JAMA 1993;270:331-337.[Abstract]
10. Silber J.H., Rosenbaum P.R., Schwartz S., Ross R.N., Williams S.V. Evaluation of the complication rate as a measure of quality of care in coronary artery bypass graft surgery. JAMA 1995;274:317-323.[Abstract]
11. Wasson J.H., Sox H.C., Neff R.K., Goldman L. Clinical prediction rules. N Engl J Med 1985;313:793-799.[Abstract]
12. Ivanov J., Tu J.V., Naylor C.D. Ready-made, recalibrated, remodelled? Issues in the use of risk indices for assessing mortality after coronary artery bypass graft surgery. Circulation 1999;99:2098-2104.[Abstract/Free Full Text]
13. Hannan E.L., Kilburn H., Jr, Lindsey M.L., Lewis R. Clinical versus administrative data bases for CABG surgery. Med Care 1992;30:892-907.[Medline]
14. Hannan E.L., Siu A.L., Kumar D., Kilburn H., Jr, Chassin M.R. The decline in coronary artery bypass graft surgery mortality in New York state. JAMA 1995;273:209-213.[Abstract]
15. Teres D., Lemeshow S. Why severity models should be used with caution. Crit Care Clin 1994;10:93-110.[Medline]
16. Ivanov J., Weisel R.D., David T.E., Naylor C.D. Fifteen-year trends in risk severity and operative mortality in elderly patients undergoing coronary artery bypass graft surgery. Circulation 1998;97:673-680.[Abstract/Free Full Text]
17. Poses R.M., Cebul R.D., Centor R.M. Evaluating physicians’ probabilistic judgments. Medical Decision Making 1988;8:233-240.[Abstract/Free Full Text]
18. Poses R.M., Bekes C., Winkler R.L., Scott W.E., Copare F.J. Are two (inexperienced) heads better than one (experienced) head? Averaging house officers’ prognostic judgments for critically ill patients. Arch Intern Med 1990;150:1874-1878.[Abstract]
19. Shapiro A.R. An evaluation of clinical prediction. N Engl J Med 1977;296:1509-1514.[Abstract]
20. Cheng D.C., Karski J., Peniston C., et al. Morbidity outcome in early versus conventional tracheal extubation after coronary artery bypass grafting. J Thorac Cardiovasc Surg 1996;112:755-764.[Abstract/Free Full Text]
21. Cheng D.C., Karski J., Peniston C., et al. Early tracheal extubation after coronary artery bypass graft surgery reduces costs and improves resource use. A prospective, randomized, controlled trial. Anesthesiology 1996;85:1300-1310.[Medline]
22. Hosmer D.W., Lemeshow S. Applied logistic regression. New York: John Wiley & Sons, 1989.
23. Rao V., Ivanov J., Weisel R.D., Ikonomidis J.S., Christakis G.T., David T.E. Predictors of low cardiac output syndrome after coronary artery bypass. J Thorac Cardiovasc Surg 1996;112:38-51.[Abstract/Free Full Text]
24. SAS Language Guide for Personal Computers. Cary, NC: SAS Institute Inc. 1988.
25. Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36.[Abstract/Free Full Text]
26. Hanley J.A., McNeil B.J. A method for comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839-843.[Abstract/Free Full Text]
27. McClish D.K., Powell S.H. How well can physicians estimate mortality in a medical intensive care unit?. Medical Decision Making 1989;9:125-132.[Abstract/Free Full Text]
28. Detsky A.S., Stricker S.C., Mulley A.G., Thibault G.E. Prognosis, survival, and the expenditure of hospital resources for patients in an intensive-care unit. N Engl J Med 1981;305:667-672.[Abstract]
29. Myers D.G., Lamm H. The group polarization phenomenon. Psychological Bull 1976;83:602-627.
30. Dolan J.G., Bordley D.R., Mushlin A.I. An evaluation of clinicians’ subjective prior probability estimates. Medical Decision Making 1986;6:216-223.[Abstract/Free Full Text]
31. Poses R.M., Cebul R.D., Collins M., Fager S.S. The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis. Ann Intern Med 1986;105:586-591.
32. Oxman A.D., Thomson M.A., Davis D.A., Haynes R.B. No magic bullets. Can Med Assoc J 1995;153:1423-1431.[Abstract]
33. Davis D.A., Thomson M.A., Oxman A.D., Haynes R.B. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274:700-705.[Abstract]
34. Concato J., Feinstein A.R., Holford T.R. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201-210.[Abstract/Free Full Text]
35. Laupacis A., Sekar N., Stiell I.G. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488-494.[Abstract]

This article has been cited by other articles:

				J. Nilsson, L. Algotsson, P. Hoglund, C. Luhrs, and J. Brandt Comparison of 19 pre-operative risk stratification models in open-heart surgery Eur. Heart J., April 1, 2006; 27(7): 867 - 874. [Abstract] [Full Text] [PDF]

				N. D. Desai, M. P. Pelletier, H. R. Mallidi, G. T. Christakis, G. N. Cohen, S. E. Fremes, and B. S. Goldman Why Is Off-Pump Coronary Surgery Uncommon in Canada? Results of a Population-Based Survey of Canadian Heart Surgeons Circulation, September 14, 2004; 110(11_suppl_1): II-7 - II-12. [Abstract] [Full Text] [PDF]

				L. Liao and D. B. Mark Clinical prediction models: are we building better mousetraps? J. Am. Coll. Cardiol., September 3, 2003; 42(5): 851 - 853. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Michael A. Borger Tirone E. David Gideon Cohen Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Ivanov, J. Articles by Naylor, C. D. Search for Related Content PubMed PubMed Citation Articles by Ivanov, J. Articles by Naylor, C. D.