Who Should Own AI Governance Inside a Health System
AI governance fails as a binder of policy. A practitioner view on automation bias, shared accountability, and briefing a board on AI risk in one page.
Featuring Dr. Barry Chaiken on The Signal Room
Most healthcare systems push AI governance to another document collecting dust in a binder. A policy is drafted and signed off by a committee, then put on the shelf as tools are integrated into clinical workflows. The risk lies in the disconnect between the policy and practice, which cannot be assessed by a vendor checklist.
In Signal Room, Dr. Barry Chaiken presented several scenarios from clinical practice to simplify governance risk. The main point from the interview was that governance is an operation, not a policy.
Real Clinical Risk is Automation Bias
Chaiken described a clear clinical risk. A physician, who is busy and distracted, is more likely to trust an AI-generated recommendation, which is a manifestation of automation bias. In this situation, the physician is probably creating a scenario that does not exist. The risk of automation bias in any AI governance is high, especially when it does not consider this as its highest risk. The main concern with AI governance frameworks is not testing how the AI model functions in a constrained, fully controllable environment. The real focus should be on testing what constraints exist in that recommendation to decision pathway, especially for the clinician who is already fatigued after working many hours in a clinical shift.
Borrow the Cockpit, Not the Checklist
Chaiken's use of aviation is a brilliant play on the classic checklist analogy. In this case, you quickly learned the checklist was not the correct answer. Crew resource management means you need both the pilot and co-pilot's right to call out. This is where AI oversight fails most. If your governance model has one approval gate, this means one accountable signature and a prolonged silence. Crew resource management is the right to call out, given to everyone present. This is particularly needed when the system is failing, and the first to call out is the nurse, and not the CMO.
Oversight That Clinicians Trust
The term that most often dismantles governance programs is surveillance, as for clinicians this means a blame task. With Chaiken, this was the opposite. This is not to mean someone gets to disciplined. This is to mean we are making sure you are not outside the bounds of our expectations. I am sure we will find things you do that are a great idea and we have not thought of implementing. If oversight produces positive outcomes to the most common issues and goes beyond simply reporting the most common errors, this is the oversight system that most clinicians will support and from which we will get real data. If it only identifies mistakes, then it gets ignored, gamed, and starved.
Briefing the Board in One Page
The Board does not need to understand the inner workings of the model. However, there are four things they need on one page: 1) what decisions does the AI influence, 2) what decisions does the AI not influence, 3) what are the failure modes, and 4) if one of the failure modes occurs, who is held accountable. If these four things cannot be defined for a tool, then that tool is out of its governance. This is what you tell the Board. You don't show them the accuracy chart.
How Hutchins Approaches AI Governance
Rather than the model, we focus on the clinical decision that the tool influences. We assist health systems in identifying the automation bias at the point of care as a critical failure mode. We then design the intervention points and shared accountability as described by Chaiken, so the right to call out does not depend on one authorization. Governance is an operational loop with a stated owner for each deployment and an executive board view on one page.
This also aligns with the work on a just culture and data governance which any serious AI program requires. Technology advances constantly but the ability to fix accountability when things go wrong is still necessary.
These issues are also examined in The Signal Room, as clinical leaders discuss what accountability really means when AI is at the bedside.
Authoritative sources
Have a data or AI challenge like this?
A 30-minute call is enough to tell whether we're the right fit.
Continue exploring
More from Insights
Just Culture in Healthcare AI: Governing Incidents Without Blame
Just culture and trauma-informed leadership for healthcare AI governance: psychological safety, blame-free incident review, and the readiness before the model.
Data Governance in Healthcare: From Policy to Operational Reality
What healthcare data governance takes in practice: ownership, data quality, and the named owners that make clinical AI safe to deploy instead of blocking it.
Healthcare AI Consulting: What It Takes to Make AI Work
Healthcare AI consulting that begins with readiness: we assess your data, workflows, and oversight so AI reaches production and earns clinician trust.
From the AI Health Pulse
On the Signal Room podcast
Listen & watch
Read the book
Frequently asked questions
Who should govern AI in a health system?
You can't assign governance to a single sign-off. A governance model that works assigns authority to say something if they are close to the tool. There should be a designated owner who is responsible for that deployment.
What is automation bias in clinical AI?
It's a time-saving and often an over-reliance on AI to assist in verifying clinical decisions. A governance model that decides to not include automation bias as the greatest risk is misgoverned.
What does AI risk look like in a board brief?
A one page, four answer brief that expresses what decisions AI does and does not impact, the expected failure modes, and who is liable.
What do you see when you have AI oversight that clinicians trust?
Good practice is rewarded. Clinicians provide honest data to a system that aids them in improving practice. Systems that focus on error and deviation get ignored and gamed to failure.