While OP-10 is a useful quality metric for radiology, it contains some significant limitations, according to a recent study published by the Journal of the American College of Radiology.

OP-10 measures the proportion of outpatient abdominal CT scans performed with and without contrast compared with scans performed with or without contrast, but not both. Both scans are typically not necessary, so the metric is commonly used to look for signs of potential overutilization.

Researchers from the department of radiology at the Virginia Mason Medical Center in Seattle decided to research the metric’s effectiveness after making a disappointing discovery.

“We were informed of a poor score on OP-10, which drove an overall score from the Consumer Reports organization for hospital quality to be below the high level we expected,” lead author Craig Blackmore, MD, MPH, and colleagues wrote. “Accordingly, we initiated quality improvement efforts to understand and improve our performance in performing CT scans, both with and without contrast.”

In its mission to improve their facility’s OP-10 score, the researchers examined their records, put together consultations between the radiology quality team and applicable clinical specialists, and identified certain scenarios where a single CT scan could replace multiple scans. They then informed radiologists and referring physicians through both meetings and email communication.

Two years later, however, Virginia Mason Medical Center received another “unsatisfactory” OP-10 score, so the team pushed forward with a second intervention. This time, the team selected  more scenarios that did not require scans with and without contrast, and more emphasis was put on reporting the results to both a safety committee and the chief operating officer

Overall, the rate of CT performed with and without contrast decreased from 21.9 percent to 16.6 percent after the first intervention in June 2012, and then down to 11.5 percent after the second intervention in June 2014.

The study included a total of more than 30,000 outpatient abdominal CT scans, including more than 13,000 scans before the first intervention, more than 10,000 scans after that first intervention, and more than 6,000 scans after the second intervention.

“The reporting of this metric drove our institution to engage in a quality improvement practice that led to demonstrable improvement in provision of appropriate imaging,” the authors wrote. “The metric itself was a powerful motivator for change, particularly because we were all aware that it was being used as the basis for public grading of the quality of our practice.”

However, the authors noted that they identified “serious limitations” with the metric. For example, use of CT with and without contrast is sometimes viewed as completely appropriate.

“We did identify several scenarios for which consensus, including the ACR Appropriateness Criteria and other guidelines, supported the use of both scans, in variance with the OP-10 specifications,” the authors wrote.

(This element of OP-10 improved in 2015, when the metric’s specifications were revised, the authors added, but this did not completely solve the issue.)

Also, the authors agree that “very high rates of use” of CT performed with and without contrast suggest overuse, but said two facilities with low usage should not necessarily be compared by their OP-10 numbers alone; there is too much uncertainty in the metric to make judgements based on relatively small differences.

“The premise that small differences in OP-10 among institutions with low usage indicate differences in quality lacks face validity for two reasons,” the authors wrote. “First, as discussed earlier, although the metric has been improved with the 2015 revision, differences in referral patterns will always exist, and these will always account for some differences in OP-10 measurements. Although valuable internally to aid us in improving care, the metric as currently formulated is sensitive to case-mix bias in evaluating centers that are like ours in being referral centers for urologic and pancreatico-hepatobiliary diseases, in which scans both with and without contrast are appropriate. Secondly, small differences in use of multiphase scanning would seem to be relatively trivial compared with the large variation in overall use of CT, radiation dosage, and compliance with appropriate-use guidelines.”

The authors did note limitations of their study. First, they wrote they were unable to assess each and every CT scan in their records to assess overall appropriateness. Also, since they only studied their one institution, one should not make larger assumptions or generalizations based on their findings.

“Other institutions with differing management structure, culture, case mix, and coding patterns, which all affect OP-10 scores, may have differing results,” the authors wrote. “We cannot define the proportion of the OP-10 score that is spurious and related to case mix; we can only define the potential at our single institution.”