The Audit Trail Illusion

If you ask a health system if it could explain an AI-assisted decision, after the fact, most would probably say yes, and do so with confidence. They maintain logs of the AI-assisted decisions. There is a timestamp on every decision, there is a confidence level associated with the output, and the system generates a digital signature. On the surface, it looks like a complete record. This record is far from complete. What the logs capture is a decision occurred. They capture almost nothing in terms of the rationale behind the decision, and this missing rationale is the essence of an audit trail.

This is what we mean when we refer to the audit trail illusion. An organization is under the impression that it can reconstruct a decision, when, in reality, it can only validate that a series of events occurred. The two can be easily confused because, in day-to-day business, the distinction is never required. It becomes painfully obvious the day someone poses the question that the logs cannot address.

A log is not an explanation

A log answers when and what. The real audit trail answers why. In the scenario that a model flagged a patient as low risk, and a clinician approved this decision, the log would document the score from the decision support system, the decision timestamp, and the clinician sign-off. What it would not explain is the rationale behind that score, the judgment behind the sign-off, the alternatives the model and the clinician considered, or the thought process on either side. These are the questions that a decision reviewer would want answered, and it is the rationale that is never documented.

The distinction matters because, following a poor outcome, the questions that need to be answered are reconstruction questions. Neither a patient, a board member, nor an external reviewer will ask whether the system ran. Instead, they will ask how the particular decision was made and if a reasonably diligent person would have identified the issue. An assortment of timestamps cannot help answer any of these questions. The system did its job, the decision was made, and the answer was lost somewhere in between.

When a human in the loop becomes a human on the hook

The reassurance is generally that someone stayed in the loop. In the majority of cases, that phrase is more indicative of the transfer of risk than it is of oversight. The clinician is placed in the position of being the named human who approved the AI output because they are asked to review and sign the AI output. However, the AI output review and approval process seldom affords clinicians the opportunity to understand the output. Consequently, they are liable for a decision they could not examine.

This does not constitute oversight. It is signing the document without understanding. If an explanation is demanded later, the person in the loop is left holding the document and the logs, which offers them no opportunity to reconstruct the reasoning any better than an external reviewer. The loop is closed, but understanding is absent.

Why the Illusion Persists Until It Does Not

This condition persists during operational success due to the absence of warning signs. All indicators are positive. All required documents are initialed. Logs are filled in the skeletal manner the system allows. An organization can operate this way for a long time and believe it is well documented because the documentation is never evaluated against the unanswerable questions.

Quietly stated in research of leaders in the hospital industry is this condition. Many of them indicate they would be unable to justify an AI decision to an outside reviewer in a timely manner and a considerable amount of them admit to being unprepared for an audit in general. Even fewer have a legitimate and enforced policy pertaining to the recording of AI decisions. The confidence that the logs are sufficient tends to remain until the first time someone asks them to do the one thing they were never structured to do.

Constructing a Record Capable of Responding to that Inquiry

This starts with the decision of what an organization would need to substantiate in the event of a challenge to document that decision prior to the deployment of the AI. The documentation would need to be captured contemporaneously and not retroactively. Each decision would need to be recorded along with the reasoning. It would not be appropriate to try to justify the decision with evidence that is merely a trail of timestamps. If the AI system is unable to justify the reasoning, that constraint would need to be articulated before deployment.

The question of ownership is as important as the technical question. A record that everybody is responsible for is a record that nobody maintains. Therefore, an organization should have the responsibility of explaining its AI decisions by assigning an individual rather than a committee. This also involves knowing all the AI systems in use within the organization, which is something that a majority of organizations cannot do. Additionally, this work should be designed to account for the failure case from the very beginning, by considering what the reconstruction will need to look like when something goes wrong. This is because this is the only case for which the audit trail was actually created.

This is contrary to the natural instinct which is to deploy the tool and build the explanation later. Retrofitting an audit trail on a system that did not record reasoning is slow and often impossible, because the information that would have explained the decision was discarded at the moment it was made.

A log you cannot explain is a liability

The unfortunate reality is that an incomplete audit trail is worse than an obvious gap. This is because an incomplete trail gives the false sense of comfort that there is documentation. An organization that actually knows it cannot explain its AI decisions will at least proceed cautiously. However, an organization that believes its logs are an audit trail will confidently proceed, despite the gap beneath it, until the day the question arises and the record has nothing to say.

An organization cannot create audit readiness on demand. It is embedded in the way decisions are documented long before the request. Systems that grasp this concept are capturing the documentation as long as the rationale exists. The others are just collecting timestamps and labeling it as a trail.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants