When Reconstruction Breaks Down

Imagine that an AI risk score that assisted in allowing a patient to go home early was validated to the point of it being accurate, and passed every necessary review. A risk score generated by a workflow that was followed exactly as it was designed. A physician looked at a score that was consistent with the other documentation and made the call. The patient was discharged, however the outcome was incorrect.

Afterward, every group involved is able to justify their decisions, and each justification is supported. The data team is able to justify that their inputs were correct and the vendor is able to justify that the model was designed and functioned as intended. The operations team is able to justify that the workflow was followed, and the physician is able to justify that their decision was appropriate based on the information that was available to them. Each explanation is true, however the outcome contradicts each group, and the investigation continues with no further avenues to discover the cause. The cause was never within the evidence of any one group. It was in the space where no one explanation is wrong and no one was assigned to investigate.

The system is not to blame

Now, AI is influencing decisions regarding patient discharges, allocation of resources, and clinical risk assessments and documentation. An AI-generated output is inserted into a workflow that multiple teams will interact with. Each team interprets and acts on the output based on their individual perspective. The teams do not collaborate to define the intended meaning of the AI output, and each interprets and acts on it to the best of their ability within their role.

The AI output did not dictate the decision. It provided a context within which the decision was made, and that context encompassed multiple roles which did not communicate regarding the decision. The output of the AI held one meaning to the team who generated it and a slightly different meaning to each of the subsequent teams. No single team was responsible for the meaning of the output and thus by the time the output reached the physician it held a sense of authority that no one team actually supported.

Every function completes its own review

As reviews are conducted, the same pattern repeats disturbingly neatly. The policy and records team confirm the use was within scope, the clinical quality team confirm the care met the standard, the technology team confirm the system functioned as designed, and the analytics team confirm the logic was sound. Four confirmations are correct, and four confirmations are incomplete in the exact same way, because none of them can explain what happened in the space where the four met to produce the single output upon which a physician acted.

That space has no owner, and the review has no place for it. The post incident review accounts for the output of each function and leaves the output of their interaction unaccounted for, because the review was designed to assign singular responsibility, not to trace collective causation. No one has ever been asked to produce a review that could do both, therefore the one thing that would explain the outcome is the one thing that the design cannot accommodate.

The meetings do not resolve it

Meetings addressing this issue tend to conclude in the same manner. One stakeholder describes their experience, and then another describes theirs. Both accounts are correct. However, the meeting fails to address the gap between the two accounts. Furthermore, no one is responsible for the gap, so it is never a meeting topic. The gap is tacitly agreed upon and no one rejects this meeting outcome, and every outcome-inducing state remains the same for the next meeting.

This is not a flaw in the design of health systems. Different functions (separated by policy, quality, technology, and analytics) are designed purposefully. The design is effective for a negative outcome that can be isolated to a single function. AI transforms this assumption. The meetings are collectively addressing the gap, but the design of the system is such that the meeting outcome is to cope with the state of interactions between functions and not the outputs of the individual functions.

The Cost is Slow, Not Loud

AI has been sold to leaders as a way to avert catastrophic time-wasting events. Rather, the cost of AI shows up through time-sinks like perpetual review cycles and unproductive meetings where stakeholders leave without a shared understanding to facilitate decision making. AI was supposed to mitigate these organizational slowdowns. Ironically, these slowdowns are the most difficult to notice.

When boards review the responsible use of AI, every function defends its use. Because there is no report that addresses the gaps and no role is tasked to drive the missing functions, no one asks if the gaps are being addressed at all, and the honest answer is that they are not being addressed.

The Better Question

Leaders often ask who is answerable for the output. This creates a clear, yet limited, response that usually reverts back to the same missing answer. A more penetrating and difficult question is the ownership behind the design and the output that integrates data quality, model behavior, workflow design, and clinical judgment.

Most health systems do not include that type of role today. Functions are not audited for their interaction, and no one is assigned ownership of the things that exist at the intersection of those functions. That absence is not an empty gap. It is already creating an output that precedes the awareness of the organization, and it operates inside systems that are functioning exactly as intended. The gap is not where the system has failed. It is where the system has not been asked to look.

Until ownership of that intersection is established, reconstructing processes will lead to the same scenario; four accurate accounts, one inaccurate outcome, and a cause that lies in the gap between them and for which no report has been created to address. The solution is not a more thorough investigation. It is assigning ownership to the intersection, in advance of the next score cycle.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants