The Risk You Approved Once

You need to not be thinking about a specific solution as you're thinking about your strategy. You want to actually think about what partner is going to give you the most flexibility to do what you really need to do because the technologies leapfrog over each other all the time. What we know about them is changing on a daily basis. We basically treat healthcare like it's static. It's never been that, you know, the practice of medicine is an evolving science. With the introduction of this kind of capability in that environment, it's going to make it even more critical that they have the flexibility and they build that into the framework that they're using, because your governing body doesn't approve something and everything's good in perpetuity. You're going to learn some things differ from assumptions that you made and you signed off on. They may no longer be relevant. We need to look at it another way, look at it a different way now, because we've implemented some things, not anticipating drift, but there's a lot of drift. And people don't necessarily understand what that kind of drift means in the long run.

It's the understanding that these models are not like the ones that we're accustomed to, that may change when we do an upgrade once a quarter or whatever. Whenever you're putting something into production in an AI environment, that model is going to start to drift almost 100% of the time. This is not a piece of software like you're used to. It is going to change tomorrow. If you put it live today, tomorrow it is going to be different. And it's going to continue to evolve like that. And if you make a mistake and you pull the data out, you still have a problem, because now you've instantiated something based on bad data. And now you're automatically going to start to drift from that if you don't understand it and you don't address it.

It is important to remember that a vendor’s safeguards reflect their assessment of acceptable risk, which may be a different standard than what your organization holds. Understanding and documenting, and most importantly, monitoring should be part of your implementation plan given that the vendor manages this. Transparency should be expected.

When it comes to ambient listening, almost everyone I've talked to is already doing something with this technology, but they may not have any clarity around where the liability is in terms of the fact that it does not change hands. Your vendor is not accountable for any of as it is covered under the High Trust act as an administrative system.

Ambient listening warrants at least a fresh look at your policies and governance workflows. Most urgent is likely your Informed Consent forms and procedures to obtain it. Some states already require transparency and disclosure when AI is being used for patient care.

If they've got 10 different AI platforms, can you see it? Can you monitor it? Are there workflows in place that really drive the validation verification process where you've got the right subject matter experts that are signing off? Do you know what they signed off on? Do they understand what they signed off on? How are you measuring success? There is a shifting of the burden from taking notes and entering it versus reviewing something after the fact. But the net impact is we're just changing how they're using the same amount of time, and that's what they're pushing back on, because they really want us to for once maybe give some time back. Every improvement that we make with our technology just means they're spending more time reviewing things. The culture of the organization is what employees observe happening most often over a sustained period. It is not what is displayed on the website. Whatever action you take most often becomes the interpretation of trust by the people. When these tools are deployed in a clinical setting, a quick walk through a hospital ward and a peak at the nurses station may provide the most important intel regarding adoption and good system design. Do a spot check on the number of sticky notes attached to the workstations, the desk, the poster boards. That can be the clearest indicator if you've got the workflow right or not.

Who Is Accountable When a Clinical AI Model Fails?

When something goes wrong with an AI system your organization deployed, who has the authority to stop it? If that answer is unclear, governance design is incomplete and needs immediate attention. In healthcare this is not an operational gap, it is a patient safety exposure. The authority to stop or pause must live with named individuals with the standing to act inside of normal operational workflows. There have always been risks when deploying new technologies or workflows. In this new reality, a small gap can become a major incident at an unprecedented rate and scale. The significance of the human factor in this transformation cannot be overstated.

Stewardship requires asking hard questions before deployment, documenting assumptions, publishing limitations, and inviting diverse voices into the design process. While it may not be possible to totally eliminate the risks, it is very possible necessary to get these things right and minimize risk to patient care.

We have a massive crisis of trust. It's been eroded over the last two decades to where less than 25 percent of people are going to trust anybody they work for, and that's the best case.

You may already have governance, does your organization know how to manage it and control your AI?

Most organizations have what I call a data club instead of a data governance. They waste time bringing high-level people together to make decisions. After two or three meetings those people go away, and the only thing they end up doing is explaining why they can't do anything. It's gridlock.

I think about it in terms of AI readiness. You can have all your data in great shape, but if your cyber isn't top-notch, you've got risks you don't even know about. So the very first thing is a risk assessment. See what exposure you already have. The first thing I do is help them understand where they are, because usually the issues are not the ones they think they have. I find them the areas they can say yes to, because they do have data in solid enough condition to trust for certain things. They're not usually the ones they want, but those are the ones they can get while we figure out the steps to get the other data sources ready.

I want to know who I'm dealing with, and I want to know what capabilities are there, because I'm not going to gamble and put something in front of anybody unless we know for sure it's really solid. I can't gamble on things that are foolhardy. One guy going hard after a pitch said he'd send me all the details on his technology, and he sent me a document that was one hundred percent generated by AI.

Continue reading from Hutchins Data Strategy

The Oversight Debt Curve → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-oversight-debt-curve

What the CISO Now Owns → https://hutchinsdatastrategy.com/the-ai-health-pulse/what-the-ciso-now-owns

On the Signal Room podcast

Scaling Healthcare AI Beyond the Pilot | Dr. Sarah Matt → https://signalroompodcast.com/episodes/scaling-healthcare-ai-beyond-the-pilot

AI Security Risk: The Massive Mistake Companies Make | Aaron Puckett → https://signalroompodcast.com/episodes/ai-security-risk-governance-gap

The Hidden Reason Hospital AI Keeps Failing | Angel Mena, MD → https://signalroompodcast.com/episodes/hospital-ai-pilots-fail

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://www.chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI risk management · human oversight of AI · AI model drift · continuous monitoring · AI accountability gap in healthcare · AI readiness