Responsibility in Practice

Most health systems can document who manages their AI systems. Far fewer can answer the more important question. When an AI model acts unpredictably, who handles the fallout and what are the next steps? One answer pertains to recordkeeping. The other answer pertains to responsibility in practice, and the gap between recordkeeping and responsibility is growing.

The gap is being pressured to close from outside the system. The FDA is approving more AI medical technologies and as a result, more AI technologies are being used in clinical workflows. Laws have been enacted which require health systems to allow for a human in the AI decision-making process. The courts are getting involved due to the automated process denying health care coverage. The health care payers are implementing policies to require the health care providers they work with to comply with the same policies. The patients themselves are demanding to know who or what made the decision which most directly impacts their health care. None of the above is acting on the timelines of the health systems.

Responsibility in name only is not responsibility in practice

Walk into a large health system and oversights can be found. There is likely a committee that meets, a list that inventories the models in use, and a sign-off page that has names listed on it. While these are important, they can be easily mistaken for something more.

A document provides an answer to the question an auditor asks. The correct document provides the answer to the question a clinician asks when something is concerning. A nurse that sees deterioration notifications on the wrong patients does not need an inventory. She needs to know what to do with it, and needs to be assured that what she does is important. If the only thing she can find is a shared inbox or an antiquated name, then the document has not provided any value to the organization.

Where responsibility tends to break down

Responsibility fails when it is not clearly articulated. It fails when ownership is not clearly articulated. For example, responsibility for a model is shared among the technology group that maintains it, the analytics team that builds or tunes it, and the clinical team that relies on the model, with an oversight function above it all assuming that each of the teams is taking responsibility. Each team assumes that another team is taking responsibility.

A second issue is more insidious because there are no obvious failures. A model is validated with respect to how care occurred at a particular timestamp. Afterward, the model evolves with the workflow. A new documentation template, a different order set, an improved lab feed, and the inputs the model requires shift beneath it. Meanwhile, the output continues to arrive with the same confidence. It is possible one or more people owns the model. It is less common for someone to own how the model integrates with the workflow, which is typically where the issues start.

Then, there is the issue that does not leave. A coder or analyst detects something that seems off and brings it to the attention of the user once. The issue is either documented by someone more junior or more hesitant to document it due to the ensuing friction, and the chart review and apology are the only things that explain the lag to those who can act.

Creating findability for ownership

The first step towards ownership that is more than a formality is also the least exciting. Make ownership findable. Documented and findable are not the same, and the difference is whether the person closest to a problem has access to the person who owns it in the same shift, without an org chart or a favor.

This means that for every model currently in use, there should be an easily identifiable name and an articulated mechanism for raising a concern that a clinician on the front line would actually use. If it takes three levels of approval and a meeting request to raise a concern, then that concern will rarely, if ever, be communicated. This can be easily tested. Ask a charge nurse who is the owner of the deterioration model in her unit and what would happen if she lost trust in the model. If she does not have to look anything up, then there is real ownership. True ownership cannot be answered if it has been filed or stored somewhere.

Making answerability a part of the working rhythm

In regards to findable ownership, this gets you through the time of a problem. To keep responsibility alive, a working rhythm must be established. This means that the model owners and the model dependents must meet on a regular basis to observe how the model behaves in the working environment, rather than how the model performed before it was deployed. The validation phase of a model is a snapshot of a working environment that no longer exists.

Being open about the technology and methods used during the care of a patient should be a habit rather than a façade. When a method or technology is used, the patient should know, and the team should be able to define the purpose and limits of the method or technology. An organization can usually explain its models to a government reviewer because the clinician is the audience that is usually the most difficult to explain things to.

Audit readiness is the result of a routine rather than an emergency. Ownership, along with other responsibilities and concerns, is easily tracked and managed, resulting in the documentation that an external reviewer requests. Systems that become frantic and disorganized when a request is made are the systems that have treated responsibility as something that should be assembled on demand. The systems that remain calm built responsibility into their normal, everyday operations.

Responsibility that works is not a document and is not a committee. It is a property of how an organization operates, and like any other property of an operation, it has to be owned. Someone of sufficient authority has to be answerable for the operational property of ownership and the routing of concerns to a person who is able to address them. That person also has to have the authority to demand answerability from operational units, even if they believe they have sufficient internal controls.

Health systems cannot prepare for the increasing external pressures that will continue to build. The organizations that survive the longest will not be those with the most extensive health systems. They will be those that know which contacts to reach out to and know that that contact will be meaningful.

This edition closely reflects the direction of current United States health policy. The FDA is approving more and more AI-related medical technologies. State governments are starting to implement laws that require the disclosure of the involvement of AI in medical decisions as well as the provision of human oversight. The legal system and insurance providers are beginning to determine who is responsible when an AI system makes an automatic decision that impacts the medical coverage of a patient. The operational practices that are beginning to be implemented, i.e. defined ownership, concern pathways, real-world observation, and readiness for audits that stem from real-world observation, are concepts that are beginning to be implemented by these organizations. These concepts connect to companion editions that discuss oversight and the need to scale it, and why oversight fails when it is separated from ownership.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants