Why Oversight Without Ownership Fails

Ask a health system how it manages AI, and you are likely to be told about a committee. There is one, and they review new tools and approve deployments. The existence of such a committee is usually suggested as evidence of having laid adequate groundwork to manage the matter. Committees are good for many things, but ownership is not one of them, and it is in the ownership that the gaps in management become apparent.

The gap is more a by-product of context than anything else. The role of a committee is to take time to consider a course of action and a recommendation, then to take the next course of action. AI is not really a system in that regard. AI is continuous in its operation, functionality, and the decisions it shapes each hour of the day, and each day of the week, and the month and year and so on, long after the approving meeting has concluded. Determining a course of action takes time, but execution is continuous and unyielding. The work and the overseer are on completely different timeframes, and the committee really has no hope of ever being able to catch up and control the system.

Committees cannot own a system

So, what usually happens? A committee is formed, and after all the usual processes, the tool is approved for deployment. Deployment approved, the committee is done; the tool is in operation, and the committee has moved on, probably for good.

That is when the ownership disappears. The committee has never been intended to monitor a system on a month-by-month basis. Each of its members has a day job which is unrelated to this. The tool, therefore, operates in a supervisory vacuum, as far as formal approval is concerned, and in practice, it is unobserved. The approval established a paper trail, which gives the impression that oversight exists. However, it did not create anyone whose responsibility it is to pay attention to the tool when it starts to spiral out of control.

Paper oversight, no one answerable

The vacuum remains invisible until something goes wrong, and the questions come in an ownership form. Who approved this deployment. Which named leader agreed to accept the risk associated with this? A committee cannot answer these questions, because a committee is a group, and a group cannot be the deciding entity.

When the committee is the only answer, there is practically no answer. What the organization has is a set of meeting minutes, a record of deliberation which gives the impression of oversight, but commits no one. It is a record of deliberation, made in the language of oversight, which completely fails to inform a board or a family, who was responsible for the system which caused harm.

The law is moving toward names

There are external reasons also pushing in that direction. New rules addressing the use of AI technology in social care seek to avoid the use of the term body corporate in favor of a real person defendant, most often the social care practitioner. In such cases, the duty to answer is imposed on a person, most often through the use of the pronoun she. The duty does not rest with a committee or a board, which in practice shares the burden of duty with the best of goodwill.

This is of great importance, as it illustrates the great leap the law is making. It acknowledges that a diffuse sense of responsibility is used by organizations to avoid responsibility, and it is drafting legislation that refuses to let the board, committee, or collective be the answerable party. A health care organization that directs its oversight primarily through collective bodies is working against the existing direction of the law.

The fix is a name with real authority

The goal is not to eliminate committees, as they serve a genuine purpose and help in the consolidation of different ideas and thoughts to aid in making difficult decisions. Rather, the goal is to stop confusing committees with ownership. Any AI system used to influence and/or assist in making clinical decisions should be the responsibility of one named executive.

Keying in on continuous is critical. The owner cannot be someone who signed off and walked away. They need to carry the system the entire time it runs, meaning they need to pay attention to the system well after the approval stage. This is the type of ownership that gives the ability of traceability, because with distributed responsibility and decision making across different committees and vendors, no one has a clue who made the decision, what decision was made, and why. Ownership gives traceability to the entire chain. The authority has to be real as well, including the authority to shut down a tool while it is operational, or the ownership is just for show.

A committee can advise, but it cannot answer

The differentiator is easy to maintain. A committee can advise, and does recommend, both of which are valuable. Where a committee cannot play a role is being answerable, as this must be a person, not a group. Oversight that consists solely of a group, and never of a person, creates the illusion of control, but, in reality, does not contain any oversight. The question for any health system is not whether they have a committee, most of them do. The question is whether, for each AI system that impacts the delivery of care, there is a single person to whom the system can be attributed, and, more importantly, who has the authority to act, and the obligation to respond. If that person cannot be identified, the system operates without an owner, and oversight is nothing more than a committee, actually just a meeting.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants