Who Answers for the Model?

During a vendor demo, a chief medical officer at a health system saw a model that could predict patient deterioration six hours earlier than the existing early warning score. The system can be configured flexibly, and her data science team is advocating to expand the scope from two to forty units. Staff in the pilot units have doubts, but the vendor is ready with a contract, and the decision is now hers.

When the time comes to consider expanding the system to forty units, there is no algorithm to guide that decision. If the alerts are wrong, how to maintain clinical trust is also a question no model can answer. This is a challenge that only people can address, and is the real problem health systems will have to confront this year, whether their leaders are prepared to take that leap.

Every clinical model comes with an output and a question, and the most difficult part comes after the output is received. Responsible use of the software described in the AI Code of Conduct published by the National Academy of Medicine is a commitment of the human while the software is in use, and will not be the property of the software.

Having a slightly differing view, the American Medical Association is identical. When it comes to identifying their work as augmented intelligence, the association deliberately made the choice of augmented over artificial. The augmented intelligence framework of the association stems from this belief that such tools ought to be applied alongside judgment in the clinical domain, and that ultimately clinicians are in charge of the care dispensed. For instance, a model may say a patient ought to be discharged, but the clinician who signs the discharge order is in control of that decision.

This burden of responsibility is not equally shared. It is principally the burden of the clinician who is at the bedside and the nurse who must decide on acting given the alert. It is also on the purchasing executive, who likely has never been to the unit. With the purchase of the model, the burden of responsibility was placed on each of these persons along with an unanswered question. The burden of responsibility will remain unassigned until it is called for.

What Tooling Cannot Resolve

Models can be evaluated for precision. However, no metric informs a leader how precise is sufficient before 40 units are at stake. The 2025 Joint Commission and Coalition for Health AI recommendations for the responsible integration of AI in health care incorporate the imperative of leadership versus engineering. Recommendations include named oversight with monitoring and validation for long durations after the launch. Each of these rests with leadership before engineering. Someone determines who will monitor the model and what the criteria for the metrics will be to stop the model from being in operation.

For the last two years, ECRI (Emergency Care Research Institute) has placed AI on their list of 10 top health technology concerns. The concern here is not with a model failing in a lab in a controlled environment. The concern is with a model performing well in a controlled environment and then, in an uncontrolled and messy environment after a model is launched, begins to drift and no one is assigned to monitor drift. The Agency for Healthcare Research and Quality has documented that models developed with inadequate data and designed to address care gaps for the most desired clientele of a health care system drift. Leaders will have to address these issues in the absence of contracts.

This example shows the dangers of inadequate oversight. In a 2019 article published in the journal Science, researchers demonstrated that a population health algorithm that had been deployed on millions of patients was likely to underestimate the health care needs of Black patients. As the researchers showed, the algorithm learned the health care expenditure patterns the health care system had already considered and made the tool to serve that purpose. It was up to the stakeholders to do the ethical assessment of the algorithm, yet for a long time, they have neglected this responsibility. The FDA, on the other hand, seems to be limiting outside authority on the software related to AI in medical applications. They are making efforts to bring government supervision to the entire process, to allow the software to maintain a constant human interface.

The Work Only a Leader Can Do

The reaction from health care systems is more distinctly defined. Several organizations have hired Chief AI Officers (CAIO) and created oversight committees to manage the limits of AI in health care solutions. While these systems will help to better the situation, the limited critical judgment of the people placed in these positions will always affect the outcome.

Candidly, there appears to be consensus among most professional bodies. The American College of Healthcare Executives identifies judgment in situations characterized by uncertainty as a core component of their model of leadership competency. As it stands, the US federal rules require that decision-support systems developed by certified and legally authorized entities must stipulate the defining features of the systems. This rule enables leaders to articulate more complex questions, but this rule is mere scaffolding. A leader must wrestle with it. This may entail reviewing the disclosure and re-engaging the nurse who has come to distrust the alerts and is thereby valuing the trust gap. Tools will undoubtedly evolve, but the decision to trust a tool and to cede authority to it will always be the province of human judgment.

This is a discipline that, when properly executed, is far from attractive. Each model has a designated owner who must periodically review the model to avoid a situation of a broken model. In advance of a warning, there is a defined threshold in the model that mandates the owner to stop when the model is in an unhealthy state. The distrust of the clinician in the alert is recorded as a comment in the log along with the frequently reported metrics. This is a trace of a basic managerial practice in the context of a new, unusual, and extremely powerful category of management tools.

What Health Systems Need to Decide

Over the next two years, all health systems will begin buying more digital health technologies. The more complicated issue is who will buy these technologies, and does that person have the authority and/or the will to say no. A health system that buys the best of the best can still not serve its population if no one is inclined to say no when the evidence suggests to say no. That is the easy part. The hard part is the subjective call and that is not developed in many systems.

Health systems have been best served when they purposefully developed this call. Within each of the clinical models, there is a person who has been named and titled the owner. The owner is not intended to be the person the committee is to be reporting to. The owner is the person with the authority and the obligation to not allow the clinical tool to be used, and to address the clinician who does not want to use the tool. The best leaders understand that the clinical tool has outputs, and each of those outputs is a question that is intended to be answered by a person.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants