Use of Entrustable Professional Activities and the Amount of Supervision Required in Determining Resident Achievement of Milestones
Susan D. Wolfsthal, MD
University of Maryland School of Medicine
Program Size: More than 101 residents
Academic Setting: University-Based
Clinical Setting: All
Overview
The overarching principle in developing our evaluation tools was to identify and map entrustable professional activities (EPA’s) created by the Internal Medicine community to the residents’ various rotations. To ensure faculty had the opportunity to observe resident skills in a variety of settings and rotations, our curriculum committee selected 24 EPA’s that were then mapped to all rotations and to the ACGME reporting milestones. The committee ensured that each EPA was appropriately represented across the spectrum of rotations and that each reporting milestone had an adequate number of EPA’s mapped to that milestone. Rather than using the curricular milestones, our curriculum committee chose the EPA’s as a more specific and goal directed measure of residency performance for our evaluation tools. Descriptive anchors were not provided for each EPA. Instead, faculty were trained to rate the resident as to the amount of supervision they required for each EPA. These nine supervision categories spanned from a very early learner requiring constant close supervision to a resident ready for independent practice who demonstrates high achievement or is aspirational. Clear descriptions of each supervisory class are provided at the top of each evaluation form. The 9-point numerical rating scale for each milestone is not revealed to the faculty. A general normative scale was added at the end of the form for faculty to rate the resident on the overall degree of supervision required and the overall performance compared to their peer group. Example provided is for GIM inpatient rotation; others available for all rotations.
Download the Tool
Development
- Identifying the EPA’s (curricular milestones) that would be most effectively observed and assessed during each rotation.
- Mapping the EPA’s to the reporting milestones and ensuring each EPA and reporting milestone was adequately represented across the spectrum of rotations.
- Establishing the levels of supervision that would apply to residents from the early days to the completion of their residency training.
- Training faculty in using milestones and direct observation in evaluating residents.
- Reviewing the final numeric scores semi-annually against written comments and other parameters and tools to validate the scoring system.
Lessons Learned
- The system requires close monitoring for scores that are too high or too low for a PGY level. This variation may be due to a struggling resident or one who is exceptional, but can also be due to a faculty member not attending to the supervisory levels and needing re-direction.
- Setting up the process in E-Value was extremely time consuming. Once created, the results are easy to obtain and follow.
- The global comparative score that compares residents to their peers appears to correlate with milestone achievements, thus providing some internal validation of the milestone process.
Faculty Development and Training
The faculty were offered workshops at the beginning of the process to review the concept of milestones and the structure of the new evaluation system. Suggestions were provided as to how to evaluate the milestones using direct observation. The slides were distributed to the entire clinical teaching faculty. Ongoing workshops are planned to reinforce concepts and teach skills to faculty members.
How Used to Inform Decisions about a Learner's Milestone
Low score notifications are set for each PGY level to provide real time information about residents scoring below expected milestones. After using the milestones for two years, tracking results provide critical markers in identifying residents who are either not meeting expected competency or those who are performing at a high level. The results have provided a cluster of residents for each of training year thus allowing a more clear distinction among the three classes of residents. By focusing on the EPA’s, the CCC has implemented targeted remediation plans that focus on behaviors that can be tracked over time.
For more information, please contact swolfsth@medicine.umaryland.edu.