When the Software Stops Asking

In September 2025, the Joint Commission and Coalition for Health AI (CHAI) published the Responsible Use of AI in Healthcare guidance. Its publication had a unique significance. The Joint Commission, which accredits hospitals, and the group central to the field, do not develop guidance for a nascent technology. They develop it when the technology has progressed to the point that it is being implemented in practice.

What is advancing is the functionality of the technology. For the past two years, the focus of the discussion has been centered on generative tools that create a note or a summary of a chart and return the output to a person. The next generation of systems does not return the work. An agent, for instance, responds to a prompt, accesses a record, and executes the next step, continuing in sequence and performing the tasks as a competent assistant would. This is referred to as agentic AI, and while the degree of change the technology has introduced is small to describe, the impact it has caused is significant. The software has become autonomous.

From Drafting to Doing

Drafting tools are given information and, in return, produce a sentence, summary, or suggestion about order. The user can then decide what to do about it. An agent contains the closure of the loop. By 2026, much of the reported enthusiasm had migrated to the back office. Agents were clearing high-volume Prior Authorizations and claims with a user checking the results only at the end, if at all. The trade off is obvious. The volume of work is large, and profit margins are small. A system can finish in a matter of hours what a team is not looking forward to doing each day. The trade off is more or less clear, and someone that used to check each and every single step now only reviews a summary, which is also more likely to cover the exceptions which are generally a pain to deal with.

The progress of the clinical side has been understandably slower. An agent that re-schedules a patient is a step forward in convenience. However, an agent that modifies a prescription order or closes a care gap by acting and not simply by suggesting is of a completely different system, as the act is done before a clinician has had the chance to evaluate and approve it. The question is no longer if the model is accurate. The question is what acts a system is allowed to perform in the absence of a user.

The first mistake is assuming one answer exists when it does not. The line between propose and act is flexible for scheduling versus for medication. That line is also flexible for a large academic center compared to a rural hospital with a thin staff. An operating model that copies a vendor slide does not show where that line belongs in your building.

The Context the Agent Cannot See

I once worked with a team that designed a staffing model for an inpatient unit. The model's math was straightforward. The model even correctly identified certain shifts as overstaffed and recommended a staffing reduction. The nurse managers of those units immediately saw the error. The staffing model was not wrong. It identified the overstaffing as a patient safe response to the nursing acuity levels and the safety of the unit's operational constraints. The model was correct. It was simply unable to see the context that gave those numbers meaning.

That experience often returns to mind when I hear suggestions for allowing uncontrolled system behavior. We identified the staffing problem because the model was only suggesting, and the person remaining in the loop had the contextual knowledge to push back. An automated system behaving similarly, would have suggested a staffing reduction prior to anyone with the knowledge of the operations being able to do so. We traded the pause for where we used to have judgment, for speed. The model was not the problem, the problem was the model could not see the value of a human, which as you can see, was contextual.

What Has to Stay Human

Joint Commission and CHAI aligned guidance revolves around an established principal. The responsibility remains with the organization, not the software. The health care system remains liable for the actions of the software and thus requires that someone, in advance and in a specific way, captures where a human remains in the loop. The AMA has long maintained that automated tools are a form of augmented intelligence, created to assist and not replace a clinician, and self-acting clinical agents are a quiet perversion of that premise.

A helpful distinction is that between actions an agent may perform and actions it may only suggest. The booking of a follow-up visit is a reversible and low-harm action. The same cannot be said about the submission of a claim or a change to the clinical record because these actions become final and irreversible. Systems that assume every action is equally safe to automate have not spent time thinking about what happens when their systems are wrong.

Reversible actions are also where agents earn their worth. Systems that book appointments and chase missing documentation after-hours provide time savings for staff without a significant risk. These systems begin to lose their worth when the same time savings is offered for actions that cannot be reversed. The slow drift toward greater autonomy is rarely considered an actual decision. Instead, it is a choice that no one has really made.

What Leaders have to Build

Before automating a process, ensure the work is sorted. For the process where mistakes are costly and irreversible because they would impact a patient or payer, ensure that a person is retained regardless of how effective the agent appears in the demo. For the other processes, ensure someone is accountable for the actions taken by the agents across the system the same way a process is named after someone who is accountable for the process that is automated. Treat the agents the same way you would treat a new employee with authority, and look for process drift where the scale of the process will show you the drift.

One way to prevent this is to keep a log of what the agents do. An agent that takes actions should leave a trail that is readable to a person in the future, showing what actions were taken and why, so a mistake in the system is not discovered in a complaint. Most systems log far less than they think they do and the shortfall will only be seen once something fails. Construct the trail before you increase the agents autonomy. An action that cannot be explained is an action that cannot be justified.

The capabilities of agents will increase, and so will the quality of demos. What won’t improve on their own, however, is the decision of where humans must be positioned. That decision requires effort, and can't be entrusted to a system that profits from us placing our trust in it. An agent can perform a task. It cannot bear the responsibility of that task.

Context and Sources

This edition is informed by the Responsible Use of AI in Healthcare guidance from the Joint Commission and the Coalition for Health AI; the augmented intelligence position of the American Medical Association; the AI Risk Management Framework from the National Institute of Standards and Technology; and health technology hazard research from ECRI. It builds on themes from issue 38, Why Oversight Without Ownership Fails; issue 40, The Incident Response Fallacy; and issue 31, Shadow AI: The Symptom, Not the Threat.

Christopher Hutchins Founder & CEO, Hutchins Data Strategy Consultants