When the Patient Brings the Algorithm

A patient reports an AI tool examined their recent lab work and determined an abnormality was present. The clinician is not familiar with this tool, and is unable to understand the underlying reasoning or the data upon which the AI based its conclusion. What the clinician is able to see is a patient with a legitimate concern and a heightened expectation for an answer. A few years back, this type of interaction was uncommon. Now, this type of interaction is becoming such a frequent occurrence that to consider it an exception is itself a considerable risk.

This is a novel interaction and is beyond the scope of the constructs the healthcare system established to interact and engage with AI. Those constructs were established to engage with systems that are used and operated within the confines of the healthcare system. A patient brought an AI tool to the healthcare system from outside its confines, and there was no engagement construct established for this type of interaction.

That construct was what formally defined the boundaries of the healthcare system.

Patient-brought AI alters the situation. What the patient brings are not fragments of a once read article. What the patient brings is the output from a system that generates clinical conclusions with confidence and even precision with numbers, and the clinician is expected to treat the output as a second opinion. The problem is, the output is a second opinion from a source no one can question. A clinician cannot review the model, the inputs, nor are they able to evaluate whether the output is based on sound reasoning. They are expected to embrace a colleague that is absent and unapproachable.

The clinician is exposed no matter which way they turn

This is the uncomfortable part. If a clinician finds they are in agreement with the output that the patient carried in, then they are adopting the conclusion, and the clinician is made responsible for acting on a conclusion they could not validate. To disagree creates a recorded moment of clinical judgment that will be reviewed later with the benefit of hindsight. Either way is exposed, and there is no way to make the output disappear.

Just because the tool used was unapproved does not mean the clinician does not owe a duty of care. The clinician must still consider the tool, even if it was not used by a system that the organization itself sanctioned and therefore cannot control. The liability, which previously was restricted to the organization and the sanctioned tools, now comes to the organization via the patient.

Why telling clinicians to ignore it does not work

The easiest course of action is to tell clinicians to disregard patient-sourced AIs. Unfortunately, this advice is more often than not invalid as soon as it enters the exam room. The incidence of patient use of AI tools is growing, and the tools are increasingly being used as aids in patient care. There is also the risk that disregarding the tool may, in fact, result in the clinician missing a valid clinical finding.

From the opposite side of the spectrum, engaging with no structure also presents risk. A clinician taking a completely unstructured approach to a tool, with no guidance regarding appropriate engagement, is also exposed, as the clinician is essentially using the tool in a completely unstructured way. The safest approach is to acknowledge that neither of these options is safe, and therefore the organization must provide adequate support to clinicians.

What the People in the Room with Leaders Deserve

This is a challenge leadership needs to address before the next patient comes in with one. It is about ownership of how the organization handles patient-provided AI. Currently, there is no ownership, and it usually falls on the clinician who is in the room. That owner will be able to provide support to the clinician, rather than leaving the clinician to figure it out on their own.

Support can and should be provided. Clinicians need help on how to appropriately capture the encounter, as well as the rationale for the selected course of action. Clinicians need defensible statements to support their reasoning when they choose to go against an AI recommendation, and to support the reasoning as a clinician and not as a dismissive person. Clinicians need help with the type of conversation that is going to be both clinical and relational, and that will be the first time they will be having the conversation with a patient that is anxious. None of this slows care down. It replaces the ad-hoc method with the standard method to address the challenge that is going to be normal in the future.

The AI is already there

The impulse to treat patient supplied AI as something that needs to be kept away from the clinic will not work, as this AI is the one making the decision. It is not waiting for approval before making the decision. It is already making the decision, and patient supplied AI will be back in the clinic tomorrow.

Accepting this challenge will allow organizations to provide their clinicians with a way to engage that simultaneously protects both the clinician and the patient relationship. Those that ignore this as a temporary inconvenience will force their most talented employees to act with no protection against liability for a separate incident, until the day one of those temporary actions becomes the case that defines this challenge.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants