Accreditation Council for Graduate Medical Education (ACGME) guidelines state that first-year radiology residents (R1s) are unable to take independent call. But if they only work traditional business hours in that first year, how are we sure R1s are getting enough exposure to high-acuity cases?

Linda C. Kelahan, MD, department of radiology, MedStar Georgetown University Hospital in Washington, D.C., and colleagues said this has “historically been difficult to objectively assess” and wrote about their solution in the Journal of the American College of Radiology.

Kelahan et al. said resident cases can be tracked with ACGME case logs, but those only provide some of the necessary information.

“We can know how many chest radiographs a resident reads before call or leaving residency, but we have no idea how many of those radiographs were normal or positive for pneumothorax, pneumonia, lung mass, and so on, the authors wrote. “It is unclear how many high-acuity cases residents see before beginning call.”

The team developed a customized call case dashboard that classified call cases as positive or negative and targeted high positive specificity. Making positive specificity a priority over sensitivity, they explained, “ensures that residents have actually seen a given high-acuity case.” (If a resident has only handled negative cases of a certain type, one might think they lack a certain level of needed experience.)

Kelahan and colleagues collected data from relevant radiology reports from as far back as the prior three years, making all reports anonymous and annotating them as “positive,” “negative,” “unclear,” or, in some specific cases, “old.” They then used that text to develop advanced rule-based algorithms.

“The text was parsed into sentences and tokenized into words,” the authors explained. “A rule set was applied to each of these sentences, which were categorized by their relevance. For example, the sentence ‘right lateral pontine hemorrhage’ would be a ‘relevant positive’ hematoma sentence. The sentence ‘no adjacent intraventricular or subarachnoid hemorrhage’ would be a ‘relevant negative’ hematoma sentence.”

The team’s research revealed deficiencies that were either completely unknown or had previously only been reported anecdotally.

“At our institution, we discovered several deficiencies in the high-acuity case exposure of our first-year class,” the authors wrote. “Zero of four residents had seen a positive case of ectopic pregnancy before taking call. Only two of four residents saw at least one positive case of appendicitis. One resident had never seen a positive pulmonary embolism study.”

The researchers plan to continue their research, tracking changes in R1 behavior after their dashboard has been fully implemented.