In late September 2019, the Inspector General’s office of the USDOT quietly released a report evaluating FMCSA’s Compliance, Safety, Accountability (CSA) corrective action plan. STC staff read the report with great interest, and here’s the skinny. Because the Item Response Theory (IRT) approach is so complex, it has taken FMCSA longer than expected to test this new statistical scoring method. And, according to the IG report, this IRT-related delay has caused FMCSA to give itself another year (until September 2020) to make a final determination on whether it will use IRT to replace the current CSA safety measurement system (SMS). So, whether IRT will replace the current SMS methodology is “to be determined.”
It should come as no surprise to engaged stakeholders that IRT is complicated. Also, FMCSA has not yet hired an IRT expert to help them calibrate the system. Let’s face it, there aren’t that many around. Furthermore, the FMCSA is expressing considerable reservations about their ability, especially in the absence of a staff expert, to meaningfully explain the system and its results to stakeholders who need to know.
STC shares those reservations. SMS, as a measurement system, has its warts. But at least motor carriers know how their scores are calculated and how to improve them. IRT seemingly holds promise for creating more accurate scores with greater statistical confidence. But, statistical confidence isn’t actual confidence if the approach and its results can’t be explained in an understandable way.