| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Ann Thorac Surg 2004;77:12-17
© 2004 The Society of Thoracic Surgeons
a Department of Cardiovascular and Thoracic Surgery, Rush Presbyterian-St. Luke’s Medical Center, Chicago, Illinois, USA
b Virginia Commonwealth University, Medical College of Virginia Hospitals, Richmond, Virginia, USA
Accepted for publication September 23, 2003.
* Address reprint requests to Dr Higgins, Department of Cardiovascular and Thoracic Surgery, Rush Presbyterian-St. Luke's Medical Center, 1653 W Congress Parkway, Chicago, IL, USA 60612-3833
e-mail: robert_higgins{at}rush.edu
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
METHODS: Utilizing a highly customized survey tool with a group of cardiothoracic residents, we implemented a 360-degree performance assessment process based on the six Accreditation Council on Graduate Medical Education competency areas. The full spectrum of associations in a resident's sphere of interaction were surveyed (ie, supervisors, peers, direct reports, nurses, and administrative personnel).
RESULTS: Each resident received a comprehensive report that included detailed documentation of the self-evaluation and the average rating of others by category. Each resident also received a transcript of the responses to the open-ended questions and summary of the data highlighting areas of excellence, areas for improvement, and suggested goals and recommendations. The program director received copies of all of these as well as a chart documenting the average scores on each item for the whole cohort. Each resident met with the 360-degree feedback specialist and the program director to develop and commit to an action plan based on the feedback. The feedback process was repeated approximately 8 months later.
CONCLUSIONS: The 360-degree feedback results provided valuable information for the residents. It also provided our program with a reproducible, quantifiable tool to assess these competencies. Combined with other instruments, the 360-degree feedback was found to be a particularly valuable instrument.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
The ACGME provided behavioral descriptors for each of these competencies and the information on their website (www.acgme.org) [1]. Many of the methods of assessment currently used in residency programs are not ideal for supporting and gauging learning in these six performance areas [2]. During phase 2 of the Outcome Project, scheduled to last from July 2002 to June 2006, the ACGME is encouraging residency programs to develop evaluative methods that will provide "credible, accurate, reliable, and useful educational outcome data" related to performance in each of the competency areas [2].
The ACGME recommended that measurements should, of course, assess the actual achievement in the area of the competency under evaluation, but they should also foster continuous enhancement of the education of the resident and of the residency program itself. The ACGME also noted that, while providing reliable and valid data that facilitates the effectiveness of both learning and teaching in residency programs, assessment tools should achieve this in a cost and time-efficient manner [2]. The challenge for residency programs over the next few years will be to identify a set of tools, when used collectively, that will meet the full criteria set forth by the ACGME.
Utilizing a highly customized survey tool with a group of cardiothoracic residents, we implemented a 360-degree performance assessment process based on all six ACGME competency areas. It is important to note that this was a demonstration project and the number of residents involved was small. However, this article serves as a guide for how one residency site used 360-degree feedback to evaluate the performance of residents in the ACGME General Competencies. Expanding such a program to multiple residency programs would allow for the standardization and evaluation of the effectiveness of this tool across a variety of settings.
|
Material and methods |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
Three additional experimental areas were assessed as well: (1) leadership of the patient care team, (2) integrity, and (3) patient-based learning and improvement. These areas were assessed with 5 questions each accounting for the remaining 15 questions. The content of all three additional areas is consistent with the ACGME competency model and emphasizes quality patient care and the personal character of the physician.
Examples for each of the six general competency areas
1. Patient care
Shows interest in and concern for patients in daily interactions. Demonstrates sensitivity and responsiveness to patients' culture, age, gender, and disabilities.
2. Medical knowledge
Makes informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to-date scientific evidence, and clinical judgment. Works to incorporate technical advances in surgical technique into skill base and clinical practice.
3. Professionalism
Actively works toward obtaining the skills of a fully qualified and capable surgeon. Maintains personal energy and drive, even during times of stress and anxiety.
4. Systems-based practice
Uses knowledge of the hospital system to get things done. Knows how to partner with health care managers and health care providers.
5. Practice-based learning and improvement
Actively encourages patient care team to generate new ideas. Applies knowledge of study designs and statistical methods to the appraisal of clinical studies and other information on diagnostic and therapeutic effectiveness.
6. Interpersonal and communication skills
Demonstrates respect and compassion for patients and staff regardless of pressure. Listens to and hears others.
Three open-ended questions
In addition to the 45 items, the survey included 3 open-ended questions: (1) What are this person's greatest strengths? (2) What specific adjustment(s) in behavior would be most beneficial? and (3) What other comments do you have that may aid his or her growth and development?
Open-ended questions often provide a valuable context from which to interpret the quantitative data, and many participants note that the written comments are among the most helpful aspects of the 360-degree feedback, consistent with the benefits of written feedback in general [3].
Distribution of the surveys
The 360-degree feedback process was conducted in the fall and in the spring, approximately 8 months apart. All 6 residents from the cardiothoracic surgery residency program during 3 years were required to participate in the process. A total of 12 individual surveys were conducted during the 3-year period (4 surveys in fall 2001, 4 in spring 2002, and 4 in fall 2002). After an educational meeting explaining the rationale for undertaking this process, the survey was distributed to the residents and to 12 to 15 evaluators per resident. In addition to the self-rating obtained from each resident, ratings were obtained from supervisors and faculty, peers, nurses in the critical and noncritical care areas, and administrative staff. The program director selected the specific people who would evaluate each resident. All raters went through a preliminary orientation to the evaluation tool before completing the survey. Attempts were made to solicit balanced evaluations from raters. Attributes of evaluators included those with direct observational experience with residents, more senior practitioners, and those in leadership positions. Evaluators included operating room nurses, perfusionists, and anesthesiologists who had direct observational knowledge of the resident's performance in the operating environment. The confidentiality of all raters was ensured with every rater being identified only by category and by reporting all scores as averages within each rating category. Furthermore, each rater mailed his or her survey directly to an external site where the data were tabulated and the results were prepared for presentation.
Receiving the results
The program director presented the survey results to each resident who received a comprehensive report containing feedback charts for each of the 45 items of the survey broken down by rating category (self, supervisors, peers, nursing staff, and administrative staff), a transcript of the responses to the open-ended questions, and a personalized 2-page summary of the data (presenting key strengths, areas for development, and suggested goals and recommendations). Each of the 45 charts contained the question as written on the survey, the rank of the item relative to all other items on the survey, a bar graph of the resident's self rating and then, on successive lines, the average of the ratings of supervisors, peers, nursing staff, and administrative staff. Additional information provided to the residents included "gap scores" that indicate the difference between their self-rating and the average of the ratings of all others for each item, the five highest and lowest scoring items, category averages, and the average rating of their cohort group for every rating category within each item. The 2-page summaries were written by professionals with extensive experience in 360-degree feedback. In addition, a specialist in 360-degree feedback presented the results to the residents and aided them in the interpretation of the bar graphs. The specialist met with each of the residents for 1 hour during the following 2 weeks after receipt of the report to ensure accurate understanding of the feedback, to assist the resident in identifying both important strengths and focal areas for improvement, and to provide guidance in the development of a performance plan including performance improvement goals and proposed activities to achieve each goal.
The program director received a copy of all of the data and the reports (as previously determined with prior arrangement and with each resident's prior knowledge). The program director met with each resident to review the 360-degree feedback data and the performance development plan. This added a clear degree of accountability for each resident to make specific improvements. The faculty, in a parallel process, reviewed the 360-degree feedback and incorporated this information into their semi-annual evaluation of the residents. The information was also used to clarify strengths and areas for development within the resident training program.
There are two major components of this initiative that warrant further emphasis. First, the development and implementation of the surveys was completed by the program director and the Workplace Initiatives professionals. The cost to the program was approximately $500 for research and development of the surveys. The second and arguably more critical aspect of the initiative was the collection, collation, interpretation, and reporting of the results to the residents and the program director. This was completed by the Workplace Initiatives professionals at a cost of $300 per resident per year (2 evaluations per year). The total turnaround time from distribution of the surveys to the residents and raters to the submission of reports was approximately 6 weeks.
|
Results |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
Highest and lowest scoring areas
In the first administration of the survey, the residents as a group scored highest in the ACGME competencies of medical knowledge, patient care, and professionalism. Individual items of excellence related to both intellectual ability, technique as a surgeon, and excellence in physician-patient interactions. The lowest scoring ACGME competencies were systems-based practice, interpersonal and communication skills, and practice-based learning and improvement. Individual items within these categories suggesting some need for development included items related to interpersonal skills (especially around conflict) and involvement of the patient care team. There seemed to be a need, in general, for the residents to develop skills related to systems-based practice. There was also a clear indication in both the ratings and the comments that the residents would do well to make improvements in their interactions with nonphysician staff members.
Changes in scores
Interestingly, the residents demonstrated improved scores in every category when comparing the first and second administrations of the survey with a mean improvement of 4.46 on every scale
(Fig 1).
|
Responses to the feedback
Initially there was a high degree of defensiveness on the part of the residents about the 360-degree feedback process. Although not unusual within any population, it was especially pronounced with the residents. The residents expressed irritation about the tool and its use in their education. Initially, when they received the individual results, their defensiveness was high as exemplified by picking apart the questions and the particular ratings. There was a clear emphasis on feedback from supervisors and a dismissal of feedback from other sources. This highlighted a key problem, that is, the limited ability of trainees to value both systems and team-based thinking. Although the residents received overall excellence in their interactions with patients, they tended to score lower in their treatment of staff, and as a group, the residents were dismissive of the feedback from nonphysician raters whether the feedback was positive or negative. The program director emphasized the value of this process and underscored his appreciation for the personal and professional challenges that it represented. This may have eased the resistance to the process with time. The program director also provided general feedback about the survey to raters through presentations at faculty, department, and administrative meetings, as well as at educational forums (ie, grand rounds).
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
360-degree feedback: applications in nonmedical and medical settings
The 360-degree feedback has gained wide acceptance in the business community for several reasons. It is often considered to be a more balanced and accurate measure of actual performance, because the data are based on the input from a variety of people who work with the individual being evaluated, instead of just the supervisor. True to its name, when 360-degree feedback is being conducted properly, the person obtains feedback regarding his or her performance from the full circle of self-evaluation, and evaluations from supervisors, peers, direct reports, and so forth. Although no measure is above the bias of the raters, input from multiple sources tends to highlight clear patterns of behavior that are positive or problematic. Some problems may be more apparent in interactions with one level of a system, but, for example, they may be hidden from the supervisor. By its very nature, 360-degree feedback emphasizes accountability not only to an individual's supervisor, but also to the system as a whole. Perhaps most importantly, 360-degree feedback facilitates self-awareness. The comparison of an individual's self-evaluation with the perspectives of others often advances self-awareness quite rapidly, providing greater clarity regarding areas of strength and areas for development [5].
Despite the widespread use of 360-degree feedback in the business community, we found only one published report discussing the actual implementation of the 360-degree feedback with physicians [6]. Over the past 10 years, more medical facilities and practices are including 360-degree feedback as an essential part of both performance evaluation and professional development (eg, evaluation and development aspects of hospital administrators) [7]. The transition of the 360-degree feedback to physicians has emerged quite recently as small groups of physicians have begun to incorporate 360-degree feedback into their own development and evaluate and coach other physicians in practice situations to enhance performance [8]. Although these reports of its usage lack the scientific rigor to make substantive comments about validity and reliability, the tool was found to be a practical means to accumulate information about performance and behavior.
The use of 360-degree feedback for assessment of the ACGME core competencies was proposed for emergency medicine residents, but the authors do not discuss the actual implementation of 360-degree feedback into their residency program [9]. There were no published reports known to the authors of this article that discussed the actual implementation of 360-degree feedback into a residency program.
One of the significant advantages of bringing the 360-degree feedback to the residency setting is to gain information, not only on the performance of individual residents, but to ascertain the overall effectiveness of the training program. A quick review of the lowest scoring categories and specific items provides ready information to a program director as to the training needs of the residents as a group. Thus, although residents may score high in items related to patient care, they may be lacking in their understanding and execution of systems-based practice. By targeting areas for development, a program director is better able to make decisions regarding undeveloped foci of the training, make adjustments rapidly, and then assess for actual effectiveness in terms of the residents' behavior.
Uses for 360-degree feedback
Consistent with the ACGME's recommendations for use of this assessment tool, 360-degree feedback is clearly useful in assessing competencies related to the human dimensions of care within the clinical setting [10–12] and what is termed as emotional intelligence. [13] Studies of emotional intelligence as a unique construct are increasingly common and high ability in emotional intelligence has been linked to more accurate diagnoses and fewer malpractice lawsuits, possibly secondary to better listening and more accurate empathy [14]. Such studies suggest that providing residents with feedback that can assist in their interpersonal development now could be quite cost-effective in the future.
The ultimate goal of any evaluative process in the context of competency-based learning objectives is to train physicians who are better prepared to meet the healthcare needs of the public. This is particularly important in surgical specialties such as cardiothoracic surgery, in which the technical, diagnostic, and patient care demands are extraordinary. The rigors of the training process often overlook the importance of emotional development and intelligence in the overall development of cardiac surgeons in training.
Whereas 360-degree feedback can be useful in assessing medical knowledge, its use here may be complementary to other methods of assessment. However, in terms of feedback on the ACGME competencies of systems-based practice and practice-based learning and improvement, the 360-degree feedback questions served to educate the residents about these competencies and the behaviors associated with them. Although we are aware of incompetent trainees who are screened out during various points of training, it is our belief that the ACGME is attempting to document and confirm the training program's responsibility to assess these factors and also "raise the bar" in graduate medical education by improving the overall quality of those physicians we are responsible to train. Assessment can and should underscore the primary values and competencies of the ACGME. The 360-degree feedback has fulfilled these criteria over the past 2 years of implementation. The tool was implemented effectively by the program director and academic residency coordinator at a reasonable cost per resident over the 2 years of training. In our opinion the data provided accurate and valuable information and ultimately facilitated both self-awareness and actual behavioral improvements. With time our residents have responded more positively to the assessment process, and part of this may be secondary to overcoming the initial anxiety about being evaluated by the full sphere of participants in their training. Also, it is expected that resistance will be encountered whenever altering the culture of a department or organization, which tends to fade with time, especially in this case in which a positive cycle of change was spurred on by improvement in the residents' interactions with all members of the healthcare team.
Limitations of the 360-degree assessment: reliability and validity
Reliability and validity data are not yet available for the 360-degree instrument used in this study. With additional applications of the instrument, it will be possible to provide such data. At that time it will be advantageous to obtain data to ensure that the scales within the 360-degree survey contain items that are measuring the same skill area. By design there is a high degree of face validity; the 360-degree tool used in this study was closely based on the ACGME's behavioral description of each of the six competencies.
The 360-degree feedback, to some degree by its very nature is likely to fare less well when it comes to inter-rater reliability and test-retest reliability. Although pretest training of all raters may improve inter-rater reliability, the use of various rater categories allows for insight as to differences in the way that residents interact with different health professionals (ie, attendings vs nurses' aids). If a resident demonstrates good listening skills with his or her supervisory physician, it does not necessarily mean that the same will hold true when the resident is interacting with the nursing staff. The 360-degree feedback opens a window into the variability of the resident's performance.
To obtain test-retest reliability of the 360-degree feedback it would be necessary to administer the 360-degree feedback to a large number of residents twice in a 4- to 6-week period with no feedback or coaching provided to the resident until after the second administration. Although this may be ideal to test reliability, at this time it is expensive and cumbersome given the number of raters involved in the feedback survey for each resident. It may be possible to obtain this data in the future with a large cohort within one or two residency specialties.
In terms of validity, despite the wide use of 360-degree feedback in management and business, few 360-degree feedback assessments have construct validity data. Concurrent validity (the degree to which the scores of the 360-degree feedback are related to current performance) is more common, of course, but there is little data on the predictive validity of 360-degree instruments (ie, the degree to which the scores on the 360-degree instruments are related to future performance). As 360-degree feedback is incorporated into residencies and refined with time, it would be well worth conducting appropriate reliability and validity studies. At this time it may be necessary to rely upon face and concurrent validity. It is important that the standards surrounding reliability and validity of the 360-degree assessment be assessed with time as data becomes available, but applying similar rigorous standards to all assessments used in resident education is equally important.
Conclusions
We believe that a combination of tools available to thoracic training programs (ie, case portfolios, mock oral exams, in-service exams, proctored surgical technique examinations in the animal laboratory) along with 360-degree feedback provides an accurate and reliable means to assess thoracic resident competency and performance. To enhance the process there are a few suggestions that we offer based on our experience.
One suggestion is that it is widely recommended that 360-degree feedback be used solely for professional development so as to enhance the likelihood that raters will provide both balanced and honest feedback. When utilized to determine salary and position adjustments, the data are sometimes skewed to reflect the desire to either help or harm the individual. Decreasing any secondary gain of the raters is as important to the integrity of the process as is maintaining the confidentiality of the raters. The use of the tool for professional development shifts the focus to providing responses that are honest and to minimize secondary gain.
A second observation is that feedback from patients was not sought in this use of 360-degree feedback to avoid any interference in the patient–physician relationship, especially during the course of treatment. Posttreatment surveys are common, of course, and can be utilized alongside the 360-degree feedback for an even richer feedback process. However, we believe that by obtaining information from supervisors, nurses, and administrative staff, it is possible to gain a fairly accurate portrayal of each physician's interactions with his or her patients. This comprehensive but nonintrusive tool could be a significant asset to any hospital trying to enhance the quality of the physician–patient relationship without placing the patient in a possibly awkward position.
A third suggestion is that certainly not all raters are equally able or skilled enough to provide feedback to residents. For example, it is unlikely that support staff will be able to know the degree to which a surgeon's technique has progressed to meet or surpass expectation. However it is important to note that this survey did employ a wide range of evaluators from the operating room (scrub nurses and perfusionists) to intensive care (critical care staff) to nonmedical environments. Nonmedical personnel can provide substantive and valuable feedback regarding administrative and interpersonal skills, which are just as important to overall professional growth and performance as the acquisition of technical skills.
There are a few ways to address the physicians' acceptance of feedback from nonmedical personnel. One way is to provide a core survey to all raters covering areas such as professionalism, integrity, and so forth, with additional items that can be answered by appropriate raters. The second approach is to consider that all ratings are good information to a physician in terms of perception. It is fairly common for people to make assumptions regarding the ability of a physician regardless of actual ability or scientific analysis. The third way is to provide the same survey to all raters and simply review one's rating item-by-item and consider which category or categories of the rater would provide the most useful feedback. Some residents tend to devalue results from nonphysicians and fail to see how a medical team (including support staff) is valuable as a whole. Devaluation of any rater provides a supervisor with an opportunity to model and teach different behaviors.
The process of incorporating the 360-degree feedback into the culture of any organization takes time. It is reasonable that a 2- to 3-year period of adjustment may be necessary. During that time, it is typical to encounter some resistance to the process in terms of the participants' reluctance to accept feedback and the raters' reluctance to provide honest and fair assessments. As people become familiar with the 360-degree feedback process, the anxiety around it decreases and their ability to utilize it effectively increases.
|
Acknowledgments |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAcknowledgmentsReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  C. Fitch, A. Malik, P. Lelliott, D. Bhugra, and M. Andiappan Assessing psychiatric competencies: what does the literature tell us about methods of workplace-based assessment? Adv. Psychiatr. Treat., March 1, 2008; 14(2): 122 - 130. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K.-Y. Tham 360{degrees} feedback for emergency physicians in Singapore Emerg. Med. J., August 1, 2007; 24(8): 574 - 575. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Swick, S. Hall, and E. Beresin Assessing the ACGME Competencies in Psychiatry Training Programs Acad Psychiatry, August 1, 2006; 30(4): 330 - 351. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. S.V. Elkind Dr. Strangelove, or how I learned to stop worrying and love the core competencies Neurology, January 25, 2005; 64(2): E3 - E6. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
