Where Responsibility Breaks Down

Every health system articulates responsibility the same way. The pharmacy is responsible for medication safety. Infection control is responsible for infection rates. The lab is responsible for lab results. Responsibility in hospitals is assigned functionally, and for the most part, this has worked, because most outcomes are traceable to the function that produced it. AI complicates that system, because the outcomes it influences are beyond any one function, and the assigned functional responsibility has nowhere to go.

Take for instance a model that has a role in determining which patients get discharged early. The data team is responsible for the data inputs. The vendor is responsible for the model. The operations team is responsible for the workflow. The clinician is responsible for the bedside decision. All of those responsible parties are in existence and do their part. However, the outcome the patient ultimately gets is produced by all of them collectively, in combination, and that combination has no owner. Responsibility did not fail because someone dropped it. It failed because the thing that needed to be owned was not assigned to anyone.

Responsibility was built for a simpler kind of cause

Each type of harm had its place in the design of a health system. A medication error could be traced to a pharmacist, a label, or a pump. A surgical error could be traced to a team and a procedure. The design of the organization worked well to contain each type of harm. That design worked well, and that is exactly why it struggles now.

AI does not cause harm that can be contained in one place. AI brings about change that spans multiple places. One influence nudges a clinician, a workflow directs the influence, a data feed shifts in response to the model, and the outcome manifests from the engagement. None of the existing owners is responsible for the engagement, because a health system never had to assign responsibility for the engagement. The design of the organization is not wrong. It is trying to answer a question that AI no longer asks.

The handoffs are where it disappears

Responsibility tends to exist within a function and evaporate at the handoff between two. The data team hands a clean feed to the model and thinks their job is done. The vendor provides an output to the workflow and thinks their job is done. After this, a number reaches a clinician. Once again, each side of the exchange thinks the process is complete. The output loses its meaning with every exchange and no one takes ownership for the change. Once it reaches the patient, the output provides a cover of authority that no one supports.

This is also the reason why more review tends to be unhelpful. More review forces each function to be confident in their own piece. It does not help the gaps that exist between functions. An organization can review itself to the highest degree and still have no one monitoring the gaps between the functions where the actual outcome is formed.

Naming the Owner of the Seam

The fix is awkward because it does not fit the existing chart. Someone has to own the cross-functional outcome, not a single input to it. That person has to own the meaning of the output once it leaves the builders, how that meaning is retained through the various handoffs, and who is answerable when the output fails to make sense. This is a new kind of seat, and it has to be there before deployment. This has to be done before the first bad outcome happens and forces the conversation.

This is not another committee. A committee distributes the responsibility until it thins to nothing, which is essentially the same problem. What the seam needs is a single named person with the authority to look across functions and the power to stop deployment when there is no ownership of the space between functions. Until that person is named, every new model just adds another set of handoffs with no one overseeing them, and the void where responsibility should reside grows larger with each tool the system acquires.

What this asks of leaders

Instead of asking who is responsible for this output, we typically think of functional lines in a system and the empty spaces between them. We should really be asking who has ownership of the way output is shaped, as a permanent state rather than a reactive, post-incident situation. A leader that cannot identify that person has located the gap and the gap has no intention of being occupied. The gap is creating effects in a system operating as intended.

The systems that excel at this will allocate ownership of the gap in the same way they allocate ownership of medication safety, with clear intention and prior to the fact. The systems that fail to do this will discover, in a piecemeal fashion, that the most impactful AI output is the one thing no one was ever assigned responsibility for.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants