The Agent You Did Not Supervise

The agent you did not authorize

One of the things I've experienced myself is that I've been working in an AI model, just building some efficiency for myself, and I'll ask it if it can do something, and all of a sudden it's gone and connected to other pieces of my model that I didn't authorize and never intended. It's about solving the problem, and it assumes that if I'm asking, I want it done.

Alignment has to come first

Give an agent the freedom to act before the people are aligned and the decisions are made, and you get chaos where you wanted empowerment. That freedom is operational: it means acting without waiting or escalation. And it is only ever as strong as the people and the decisions standing behind it.

What the alignment beneath an agent has to be

The best way to look at AI systems in healthcare is as an analog to human employees. In healthcare, not even human employees have unfettered access to patient data. There are access controls based on the level of access an individual needs, with checks and balances and supervisory control. The same philosophy should apply to autonomous systems. Just as approvals and access controls are required of human employees, so too should AI systems require approvals from human overseers.

A lot of AI agents are getting granted broad access just to make them work.

Don't cross this line, don't see these types of things, don't do these types of things. [...] Your AI needs the same boundaries you would put around an employee. It's really that simple.

You can put guardrails on this sucker. So it can't do anything except look at your data.

It will do things unless you've already thought through it and built some guardrails that it can't get past.

I think I understand it better from thinking about how we train employees, and I think that that's maybe a perspective that doesn't often get applied to agents or some of the other solutions as well.

It behooves us to really think very carefully around what are the guardrails and checkpoints that we're putting in to make sure that we're protecting ourselves from ourselves.

Building the agents versus governing them is exactly where we want to go.

What happens when there is no boundary

When you're putting things into a model, AI inherently is it's designed to go solve the problems you're asking it to solve. And so, it may actually cross over into that part of the model and start putting things together and connecting dots that you didn't know was going to connect because you only knew about what you have in there.

If it can't find what it's looking for in the place that you're telling it, it will start going in everywhere else that you have not buttoned down.

I had it go and start looking elsewhere, which kind of concerned me a little bit. I mean, it was still inside of my own little ecosystem, but it really wanted to get me the answer.

AI always is trying to solve your problem. And I discovered it on my own that at times that will mean if you don't know that it's going to go after a particular data source, it will, and you'll find out afterwards.

If you don't put the boundaries there and make it a hard stop somehow, it's going to blow right past it.

Tell it what you don't want it to do, tell it what you do want it to do. The very next thing is going to be the opposite. That's exactly what it does almost every time.

The drift you do not see

By the time you realize you've veered off, because things drift for a variety of reasons, you may be at a point where you've really gone off the rails.

These things are going to happen, and even in the administrative area it can go off the rails. It's really hard to hit the reset button once these things have been trained, they're going to remember it whether you want them to or not, which is a significant problem.

I've built my own models that I use constantly. They're very, very well tuned. But I have to stand on top of that because they just naturally will drift from the original boundaries that I put there.

People play with this stuff all the time, and you get a couple of results you believe that are good. And the drift starts to happen. You don't even detect it until it's way off the reservation.

The agent at the patient's bedside

So when the clinicians actually using it, really defining the different thought lanes that are analyzing. In their normal process of assessing a patient. And what are the things that they can actually identify that are workflow-specific enough that they can deploy agents in real time on the particular patient record?

The fact that it's being tailored in really analyzing that patient using the agentic approach to it, it gives you that guardrail and protection from some of the places it might actually veer off and do some of that wondering that becomes a little problematic because it can be a runaway train before you even realize that it's happened. And that's the part that's always been a concern to me.

Where a human stays in the loop

Build it one way and it quietly grows your business while you sleep. Build it another and it sends the wrong client the wrong message at 2 AM, and nobody notices for a week. Same tools. The difference is the choices the people behind it made, and whether someone stayed accountable for what it did.

Where are those points where the AI knows I need to hand this off to a human being?

When does the agent reach out to a patient? When does it not?

Where these things typically break is there's no escalation pathway that anyone can get comfortable with, so they just don't do it. But there's certain key words that if the thing picks up on it, it should trigger an immediate action somewhere.

The question that decides it: who can stop it

Before they start letting some of these things run wild in the environment, because the risk is just too high.

You can't access this tool or that tool unless I've already got a game plan to shut it down.

You have to come to some conclusions and build some rules around what the human escalation pathways are and what are the things that cause that to kick in. And these are decisions that have to get made.

One of the things that I've been advocating for is organizations must identify somebody. It's not a hobbyist. It's not somebody who's got a full-time job.

Who is authorized to raise the flag and say, hey, time out. We can't proceed, we have to solve some problems.

When you sign off on governance for an AI model, you need a game plan with continuity to keep that thing from going off the rails.

This is why oversight without ownership fails at scale.

I would challenge organizations to have some pretty decent guardrails and be able to answer some basic questions because ownership of outcome is ultimately on the person who signs off on the note.

The part that stays human

The agents, they're the easy part. It's the human beings that are really the irreplaceable component of what we're doing. The skill is knowing exactly which decisions to keep your hands on.

Continue reading from Hutchins Data Strategy

When the Software Stops Asking → https://hutchinsdatastrategy.com/the-ai-health-pulse/when-the-software-stops-asking

Who Answers for the Model → https://hutchinsdatastrategy.com/the-ai-health-pulse/who-answers-for-the-model

On the Signal Room podcast

AI Agents and the Patient Data Leak | Travis Garland → https://signalroompodcast.com/episodes/ai-agents-patient-data-leak

Agentic AI and Human in the Loop | Ethan Seib → https://signalroompodcast.com/episodes/agentic-ai-human-in-the-loop

Why AI Agents Need to Know When to Stop | Kaled Alhanafi → https://signalroompodcast.com/episodes/ai-agents-know-when-to-stop

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · AI agents in healthcare · agentic AI · AI oversight · autonomous AI · AI accountability · the agent you did not supervise