The Incident Response Fallacy

Most organizations can expect how their systems will fail, and when this happens, what their response will be. An incident occurs. An alert fires. A team is mobilized, and the problem is then contained and analyzed. Sufficiently, this has served the healthcare industry for the type of failures the system was designed for. This does not accommodate AI, and this is where the entire problem lies.

The issue with AI is that it does not fail as an individual discrete event. AI fails as an accumulation. A model slightly changes what a decision is framed as. It omits a piece of information from the surrounding context, and does not produce anything that appears to be wrong. It then alters the baseline, doing so, incrementally. The influence has already permeated multiple interactions by the time anything is voiced as a concern. There is no single event that can or should be captured because the failure is not and was never a single event. Incident response is designed to react to an event that in the case of AI, will often never occur in a recognizable form.

Incident response is designed to react to an event

The most common frameworks that health systems implement are built to address observable failures that are discrete. An example would be an individual server going down, or a bedside team catching a wrong dose before it reaches a patient. A security breach is then identified and the response clock begins. The failure then signals itself. Each of the described scenarios is an example of a message that signals a failure and requires a response designed to act quickly. The entire process is built with the premise that the failure will manifest itself in a signal that can be recognized.

AI risk manifests differently. AI drift involves gradual changes in processes and decision-making that desensitize users to the changes. A framework designed to respond to discrete events cannot conceptualize a challenge that does not manifest in that way. The challenge is not a slow change. The challenge is that the response has no trigger.

There is no warning for the quiet failures

Consider what does not trigger any warning. No trigger is present when a model confidently resolves an uncertain context. No warning is registered when an output is rendered plausible by the omission of a relevant fact. No warning is triggered when a clinician begins to gradually defer more authority to the tool than he did previously, without any self-observation. These failures are paramount with AI, and none of them appear to be failures during the process.

This is the essence of the detection challenge. Incident response listens for a specific type of signal a complaint or a threshold violation. The failures that are unique to AI are silent. They do not generate a signal that the system is waiting for and therefore pass through the system.

Neither does reconstruction

Let us say the issue finally gets recognized. The next step in a typical procedure would be a root cause analysis, tracing the series of events to the point at which it failed. That step requires the existence of a path that can be determined and followed. AI does not contain such a path with any degree of reliability. The same input might produce differing outcomes, the reasoning is not preserved in a retraceable format, and the interplay between the judgment of a clinician and the output of a model is almost never documented to the degree that reconstruction would require.

At this point, the investigation has reached an impasse. The team understands that something clearly went awry and they can see the outcome, but they cannot determine how the decision was reached, if at all, or to what degree the model was responsible for it. You cannot reconstruct the decision-making process for something that was never documented; and in large part, what would account for an AI influenced decision was never documented.

The damage has already been done

There is one more, and this is the most difficult to accept, limitation. Recognizing and addressing an incident can only apply to what comes after the incident. The decisions that were made cannot be reversed. The patients whose care was influenced by the AI model were impacted long before the incident was recognized.

AI systems differ in kind from other systems that can experience momentary, contained failures. For instance, failures associated with a server outage can be deemed to be resolved once the server is restored. In contrast, once AI drift occurs, it cannot be identified until it has manifested itself in a multitude of interactions. Nothing can retroactively mitigate the damage caused by AI drift during the time it goes unidentified. In this case, the response will only be able to manage the future manifestations of drift, while the cost of the drift is already incurred.

Leadership Implications

It should not be assumed that incident response is without merit. The main concern with respect to incident response and AI is that it should be of secondary importance as a control measure, since it is only activated once an undesirable situation has occurred. In terms of a control measure, incident response should be one of multiple layers of control.

The essence of true prevention is to monitor AI systems in a proactive manner in order to mitigate drift versus the reactive approach of waiting for events that may signal drift. This means that the interface and interaction between human control and AI outputs should be adequately documented to a level of granularity that would allow for a post-hoc analysis of the control. To achieve this, leaders must accept that AI systems behave in ways that are entirely different from the systems for which the existing control measures were developed and must adjust the control measures to fit this context. Control measures that are focused on the occurrence of events will continue to report that all is well until it fails to do so.

Catch the drift, not the incident

Incident response lends itself to a real and necessary backstop for the rare circumstance where an AI failure actually does occur as a distinct event. As the primary defense, it is a delusion, as it is designed to respond to a type of question that AI typically does not ask. The organizations that manage this well will expend less effort preparing to address AI incidents, and more on developing the constant oversight to catch drift before it ever becomes one.

The worthwhile question to contend with here is this. If an AI tool in your organization was slowly but surely and quietly degrading the quality of care, what in your current system would let you know of this degradation before it was deemed an incident? When the answer is nothing at all, the incident response plan is not the answer. The response plan is merely the evidence that there is a gap.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants