The Invisible Rationale

An example of an output from a clinical AI tool may be to flag a patient as being at high risk, to draft a clinical note, and so on. In all these examples, the outputs appear to be reasonable. What is lacking, though, is the reasoning behind the output. The clinician sees the flag or the note and signs off on it. But, like the AI, the clinician is also responsible for the output. The reasoning is simply not available for reviewing.

This is more of a quiet problem relative to some of the more overt problems that capture attention. It is also more difficult to identify because at face value, the output is fine. The output of a clinical AI tool seems complete. The problem only becomes evident when the user of the tool needs to understand the output, and then finds out that the reasoning has not been provided.

An answer is never an explanation

Most of the clinical AI tools that are currently available synthesize information for the user without providing the user with the reasoning. An example of this may be a documenting tool that auto-generates clinical notes, or a tool that assists in clinical decision making that provides a recommendation or a score. From the perspective of a clinician, the AI tool has provided an output, but has concealed the reasoning from the clinician.

Accuracy cannot fully bridge this gap. A tool can achieve high accuracy, and still not enable the clinician to ascertain the reasoning of the tool for correctness in a particular instance. Accuracy and explainability are disparate dimensions of a tool, and a tool can possess the first, while the second may be absent. In instances when the answer is optimal, the missing rationale is an inconsequential cost. The rationale costs only in the instances the tool is incorrect, which are inherently the cases where the rationale matters the most.

The day the reasoning is demanded

Let us examine what happens following an adverse event. A decision is scrutinized, and the clinician who documented the decision is summoned to explain how the decision was made. If the decision was partially influenced by an AI tool, then the clinician is required to explain what the tool processed and why the conclusion was adequate. If the reasoning was never provided, then there is nothing to explain. The clinician is burdened with a decision for which there was no opportunity to interrogate.

This places the clinician in a difficult position. The clinician was responsible for the decision and the process, but was lacking the one critical component necessary for the process. The expectation was to trust the answer and provide justification for the decision, all without being provided the necessary components.

A different gap than the data problem

This gap should be separated from the data quality issues that garner the most attention. One thing to consider is whether the inputs to a model are correct and complete. The other thing to consider is whether the reasoning of the model is applicable, and to what extent. An organization can scrub its data and check the inputs, and still be unaware of the reasoning behind the model, because the data problem and the rationale problem are two different problems.

This is how a health system can be confident in a tool they are unable to actually explain. Good inputs and a plausible output can be present along with a complete absence of visible reasoning. An organization that has only solved the first problem will believe they have progressed farther than they actually have. The reasoning gap remains untouched after the data cleanup, because it was not regarded as a problem.

The reasoning should be a requirement

The solution is to stop treating visible reasoning as a nice to have and start treating it as a requirement. At the point where an AI output is relied on by a clinician, the reasoning should be provided in a form that is readable and not buried in an inaccessible model.

The standard should be set along the procurement path. The more serious organizations are changing what they ask from vendors by requiring performance plus explainability and not accepting accuracy alone. The focus shifts to building internal review processes to understand and examine AI-supplied data and to differentiate AI systems that can articulate reasoning from those that simply provide output. They are different systems with different trust requirements. A leader stepping into the unknown of whether their AI-augmented clinicians can view the rationale of the systems is where the inquiry should start.

The least we owe the clinician

The signature has not moved. The clinician is still the author of the record and still answers for it, the same as in the past. What changed is that we have accepted as the new status quo that systems that provide a judgment and an answer without reasoning have captured part of the record.

That does not have to be the status quo. The record has always belonged to the clinician, and the least we owe the clinician is a rationale we should reasonably make available to them.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants