AI Cannot Absorb Responsibility

Trust in people, not the model

We have a massive crisis of trust. It's been eroded at a rate over the last two decades where less than 25% of the people are going to trust anybody that they work for. And that's the best case scenario.

The human factor is often the last thing that's being considered when it should always be the first thing. Gaining the support and trust of the teams that experience the workflows is crucial to the success of any workflow that is a candidate for improvement or replacement.

Trust by design must be more than a slogan, it must be common practice where it can shape design decisions. Work happens much closer to the ground. Prioritizing operational relevance and partnering with clinical and operational teams is imperative to ensure that design fits how work is done, not just how systems were built historically. The people closest to work generally hold the knowledge that will make or break what gets built.

These are choices that must expressed in daily practice. They require sustained attention, humility, and responsibility. Assumptions must be understood and documented, along with trade-offs, and constraints. It requires that design teams make room for listening as decisions take shape. It means building in conditions for people to make sound decisions when the answer is not obvious.

Structure matters, but people have to trust what comes out of it. Decisions can be technically sound and still fail if they do not reflect how people experience the work.

We've got two things that are almost diametrically opposite each other right now. So the trust factor in human relationships has eroded significantly, but we're still way too quick to trust technology. And we need to find some places in the middle there.

Clinicians before the design, not after

If you talk to nurses and physicians, and they're very willing to be involved before the design is delivered that screws up their workflow even more.

The way things have gone over the last five to eight years has been influenced more by technology advocates than the voices of clinicians it seems. And it could be either the IT or innovation voice, which may be appropriate but not absent the clinical perspective. Either way, the mistake is, not thinking properly about what it takes to do some of these things.

Solutions need to be designed with clinicians, not delivered to them. We designed electronic health records to support accurate billing, not clinical workflows. We should be asking how are we going to give back the time that we have been stealing from our clinicians?

Consent means understanding, not notification

Informed consent doesn't mean you told them something that they don't understand. It means you helped them to understand something and then they agreed. Legacy agreements with minimal need for explanation are not adequate and require a fresh look. If we can't speak plainly about it in terms that a non-technical person can understand, they really can't be truly considered informed.

Oversight and the authority to stop

Oversight and governance have to be operationalized and require careful thought and planning to account for the areas where risk may not be squarely owned by one function or department. Where models will evolve at a rate that is much different than organizations and teams are accustomed to, clear answers to who has the authority to stop something become much more important. If something goes wrong with an AI system your organization deployed, the question of who can stop it is essential to answer as part of finalizing and approving architecture. Clarity in this regard can prevent or allow exposure to patient safety risks. Authority must be held by named individuals with the organizational standing to act. It requires continuous monitoring, defined escalation paths, and well-defined protocol when a risk surfaces. It's cliche, but if everyone owns it, no one owns it.

I once inherited a dashboard used by more than a dozen clinical units. It was widely used and trusted and referenced in meetings at multiple levels in the organization. It was one that was displayed in areas where leadership teams knew they could easily see it. This particular tool was well developed and had not changed for a few years. When an issue occurred and was reported to a help desk, it had been long enough that no one knew who could quickly address the issue. Time was wasted trying to find someone who knew how the tool was built, what data source(s) and other specifications that were underlying the dashboard. Eventually, it was discovered that it was designed by a developer that had since moved on from the company a couple of years prior. This is a small example of what risks may exist when clear ownership is not defined, documented, and maintained. No one knew how to maintain it, and what remained was a deeply customized tangle of nested queries, undocumented filters, and silent assumptions about the data's refresh rate. When a single column stopped updating correctly, trust in the entire tool collapsed. Not because the data was wrong, but because no one could explain how it worked, or more importantly, what it meant.

This lesson matters even more with AI. An AI system where no one knows who owns the decision or support, creates the same problem at a much larger scale and cannot be allowed to happen. AI can assist the decision. It cannot absorb responsibility.

Continue reading from Hutchins Data Strategy

We Can Do That → https://hutchinsdatastrategy.com/the-ai-health-pulse/we-can-do-that

What the CISO Now Owns → https://hutchinsdatastrategy.com/the-ai-health-pulse/what-the-ciso-now-owns

On the Signal Room podcast

Responsible AI in Healthcare: Ethical Leadership and Ways of Working | Asha Mahesh → https://signalroompodcast.com/episodes/ai-ethics-ethical-leadership

AI Governance in Healthcare: Just Culture and Emotional Readiness | Susie Branagan → https://signalroompodcast.com/episodes/human-centered-ai-governance

Why AI Tools Must Be Designed for 3 AM, Not 3 PM | Dr. Natasha Dole → https://signalroompodcast.com/episodes/ai-in-the-er-clinical-judgment

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://www.chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · responsible AI in healthcare · AI ethics frameworks for hospitals · clinical AI oversight · model ownership · informed consent · trust by design Hashtags: #HealthcareAI #ResponsibleAI #AIinHealthcare #HealthIT #DigitalHealth