What AI Governance Actually Is

Early in the pandemic, we had to move quickly to be certain that we had common language for things that most people would not have guessed actually had some nuance. We got stuck on two words: available bed. If we asked three people at the time, what an available bed was, we would get three answers. One meant a bed that physically existed in the building. One meant a staffed bed somebody could put a patient in that minute. One meant a bed we could put into service in a few hours if we had to. Under normal circumstances, the difference would not be material. In the pandemic, people were making capacity and staffing calls using those numbers, and the numbers had to mean the same thing so that decisions were made based on facts that everyone understood. For certain terms we had to keep a short working list of the terms that would cause the most confusion, and document each one with a definition, the logic behind it, who owned it, and the day we adopted it. It didn't make the problem go away but it gave everybody the same starting point, so that decisions could be coordinated and made quickly. That really re-enforced the need for governance that had been elusive previously. When the words don't line up, people freeze right when you need them moving.

When people hear the word governance, some will picture a committee and a binder nobody opens. In operations, it has to be simpler than that. It's the rules and definitions that everybody has agreed to, and the ones that teams were actually using, and a clear answer on who gets to decide. Organizations already have something in place but it may not be clear enough that there is confidence that it is sufficient to ensure that AI can safely use it to support operations. Almost every big health system has an AI ethics statement, fairness, transparency, explainability, and patient safety.

This is usually in a policy document but often has not been turned into anything that can operationalized. The gap between what gets published and what people do every day is where these efforts die. Doing AI responsibly is bigger than the technology. The hard part is putting the structures and the processes in place so people keep making good calls about AI over years, not just at launch.

In this case as in many others, translation was more of an obstacle than any system. Finance, the clinical side, and IT are all looking at the same problem and describing it in languages that don't match. The real work is getting each side to see what the other one is actually trying to do, and that's a lot harder than lining up the words. Years before anybody was saying AI, I spent a long stretch working to roll a dozen finance teams up into one view for an enterprise. Every team had a model that worked. Different formats, different definitions for the same measure, and every version was right in its own context. Stacked together, you couldn't reconcile them in time for the decision that needed them.

My job became working out why the numbers disagreed, why finance counted an admission one way and clinical operations counted it another, what assumption was sitting under each definition, and what broke if we changed it. What mattered was almost never whose definition was right. It was which one everybody could live with so the work could move. Most of the governance challenges in healthcare aren't about goals, they're about language. Getting people to see they're aiming at the same thing is what help to establish trust, and that's what let the work actually move.

A data club instead of data governance

Most organizations have what I call a data club instead of a data governance. They waste time bringing high-level people together to make decisions. After two or three meetings those people go away, and the only thing they end up doing is explaining why they can't do anything. It's gridlock.

Governance debt

Governance debt accumulates when organizations deploy AI faster than they build the oversight structures required to use it responsibly. Like financial debt, it does not announce itself the moment it is incurred. It compounds quietly, deployment by deployment, until it surfaces as a patient safety event, a liability exposure, or a breakdown in clinical trust that takes years to repair. Stewardship means asking hard questions before deployment, not after harm.

Someone has to be able to say stop

If something goes wrong with an AI system your organization deployed, who has the authority to stop it? If that answer is unclear, the governance architecture is incomplete, and in healthcare that is not an operational gap, it is a patient safety exposure. That authority cannot live in a committee. It must be held by named individuals with the standing to act before an incident, not after one.

The space between the verticals

I advocate for a fractional chief AI and analytics officer, because most organizations aren't ready for a full-time one but don't have anybody who understands the overlap between finance, operations, and clinical. They have experts in each vertical, but none understand enough about the others to see where things get intermingled and unintentionally put things into models that should never get there. That space in between the verticals is usually the most significant area that needs focus.

Finance is one of the first places I look, because nothing goes well if you don't have the finance people on board with a methodology they can stand behind. They're the first ones I have to get on board.

What you do not control once you sign

Remember what you do not control when you buy from a vendor. The safeguards a vendor builds reflect that vendor's assessment of acceptable risk, not the organization's. Those safeguards are a business decision the vendor can change without asking anyone's permission.

AI scribes are not covered by HIPAA, but they are contributing to PHI. So they're not under HIPAA, they're not governed by the FDA, they're all under the High Tech Act. There's no liability whatsoever for those companies that have that capability in there. But when the doctor signs off on the note, the doctor holds full liability. The models that are out there, roughly about 45% of what's being documented is accurate, which means the rest of the time there is review and editing that needs to happen and it puts it back on the clinician. So the intent was to relieve the administrative burden, but we're actually exacerbating a problem that already existed.

Readiness is where I start

I think about it in terms of AI readiness. You can have all your data in great shape, but if your cyber isn't top-notch, you've got risks you don't even know about. So the very first thing is a risk assessment. See what exposure you already have.

The first thing I do is help them understand where they are, because usually the issues are not the ones they think they have. I find them the areas they can say yes to, because they do have data in solid enough condition to trust for certain things. They're not usually the ones they want, but those are the ones they can get while we figure out the steps to get the other data sources ready.

I've talked to a lot more nurses and physicians, and they're all chomping at the bit to be involved before the design is delivered that screws up their workflow even more.

Governance does not have to be heavy to be real. In rural hospitals, governance could not look like enterprise governance. It could not be heavy-handed or slow. It had to be light, clear, and immediately useful. Rural leaders did not need thick glossaries or long deliberations. One example was a shared staffing alert system. Instead of a complex dashboard, we used a simple color-coded spreadsheet updated daily by local leaders. Green meant stable, yellow signaled potential gaps, and red triggered immediate escalation. It wasn't sophisticated, but it was consistent, and it allowed rural hospitals to act quickly without waiting for enterprise systems to catch up.

What it comes down to

We have a massive crisis of trust. It's been eroded over the last two decades to where less than 25 percent of people are going to trust anybody they work for, and that's the best case.

The culture of the organization is what employees observe happening most often over a sustained period. It is not what is displayed on the website. Whatever action you take most often becomes the interpretation of trust by the people.

What endures is not the structure of the work, but the spirit that animates it, stewardship that protects, translation that connects, clarity that steadies, equity that restores, and trust that sustains.

You already have governance, but you don't understand that you can actually manage it and control your AI. It did not eliminate complexity, but it gave people a place to start from a shared understanding.

Continue reading from Hutchins Data Strategy

The Committee Nobody Staffed → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-committee-nobody-staffed

The Chief AI Officer Mandate → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-chief-ai-officer-mandate

On the Signal Room podcast

Why AI Governance Is a Human Problem → https://signalroompodcast.com/episodes/ai-governance-human-problem

AI Governance in Healthcare: Just Culture, Emotional Readiness & Trauma-Informed Leadership | Susie Branagan → https://signalroompodcast.com/episodes/human-centered-ai-governance

Responsible AI in Healthcare: Ethical Leadership, AI Governance & the Ways of Working | Asha Mahesh → https://signalroompodcast.com/episodes/ai-ethics-ethical-leadership

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://www.chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · AI governance · data governance · AI oversight · healthcare AI governance · fractional chief AI officer · AI readiness