Trust by Design: Why Clinicians Do Not Trust AI (Yet)

Trust is essential to healthcare. Without it, no one would use even the best AI models. While the healthcare industry continues to advance AI models, trust is lagging behind.

To improve efficiency, many hospitals are implementing AI models that automate administrative tasks like documentation. While many of these models promise to bring operational efficiency, there is still a lack of trust among clinicians regarding the outcomes. Most conversations associate a lack of trust as an obstacle to be solved. In reality, it is a symptom to be addressed.

This is Not a Fear of Technology

While many would consider a lack of trust as a fear of adopting new technology, in this case, it is warranted. Over the years, tools that have been implemented to improve and streamline patient care have only resulted in the opposite. Clinicians have learned to treat new systems with the warranted skepticism, earned by being burned in the past.

With the implementation of every AI system, the first question a clinician asks is always, "Do I trust this more than I trust my own judgment?" In the absence of a positive and immediate response, clinicians are left to trust their judgment, which spells failure for the AI system, irrespective of model efficacy.

The Gap Is Real

This is not conjecture. Speak to clinicians and you will find the same issues. The majority cannot articulate the reasoning behind the AI decisions, and they recognize this. Many are anxious about who will be held responsible when an AI-assisted decision results in negative consequences. The answer to this question has never been presented to them. Most have not been consulted on the assessment of a tool prior to its implementation. A survey is unnecessary to perceive the gap; it can be observed in the manner in which clinicians express their reliance on the tools they have chosen to completely disregard.

Trust Is Earned, Not Installed

You cannot install trust as a software feature. It is not an adjustable option within a tool. It is a byproduct of meticulous consideration during the design phase, and subsequently how the tool integrates within the broader context of the system in which it exists.

It is relatively simple to state what a clinician requires. It is evident in the reasoning, the data, and the uncertainty. Too many systems continue to exist as an uncommunicative mechanism providing a confident answer about a decision without the requisite foundation. This is not decision support; this is decision risk, because in the absence of justification, the clinician carries the complete responsibility and the negative consequences of the decision, which is not supported by the confidence to allow the clinician to mitigate the potential consequences of the decision.

Naming the Unnamed Asymmetry

There is an unspoken asymmetry that appears on the rare occasion an AI input informs a clinical decision that later turns out to be wrong. The clinician is the one locked into the decision once the AI input has helped shape it. The clinician deals with the fallout, the difficult conversations with the family, and the emotional toll that carries on beyond the shift, long after they are off duty. The burden is not carried by the model creators or the people who sell the model. When the AI system asks a clinician to defer to what the system decided, it is asking the person with the most burden to trust a system that is devoid of the burden. Given this situation, skepticism should not be viewed as a stubborn refusal. Rather it is a legitimate rational response to a systemic situation in which trust and burden are asymmetrically distributed. This imbalance cannot be addressed by reprimanding people for their feelings.

What Trust Requires

Usually, three things come together to build trust in a clinical tool, and a lack of any one of the three things is usually enough to destroy the tool. The first is visibility into how the tool works.

Next comes ownership. Boundaries must be defined around the application of the tool, along with a definitive answer as to who will be held responsible when an AI-assisted decision results in negative outcomes. Absent this, every clinician has been, and will be, asked to absorb a liability that no one will share.

Third is stakeholder collaboration. The people who will use the tool must be active participants in the design and evaluation process. It must not be a take-it-or-leave-it trust scenario with the tool provided in its final state. When the first two are satisfied and the third is genuinely present, trust stops being a slogan and becomes something a system can build.

Explainability as a Safety Feature

It is easy to think of explainability as the finishing touch to a job well done. It is the other way around, especially in healthcare. Having the ability to provide reasoning as to why a specific recommendation was made helps ensure safety in the system. Clear and concise reasoning gives clinicians the ability to recognize when the model is erroneously confident.

There is little need to transform every clinician into a data scientist. Sometimes, a simple visual along with a minimal description of the causative factors responsible for the output can transform a mere black box into a system that is reasonably interpretable. I have seen the most successful teams handle this seriously enough to devote time to build it. They assemble clinicians, data scientists, and people who obsess about ethics to stress-test the tool in the way one would stress-test a new colleague, to assess if the tool acts as a competent partner to collaborate with, rather than being treated as a superior being to whom one must conform. They consider more than just model accuracy. They consider if the tool, by virtue of its use, would enhance the quality of the work performed.

A Model That Knows What It Does Not Know

Trust, and especially the quieter, more understated kind, is built partly on being frank about uncertainty. A tool that is uniformly confident, returning every answer with the same crisp assurance, reads to an experienced clinician like a salesperson rather than a colleague. The colleague they trust is the one who, when the answer is uncertain, says, "I am not sure about this one." A model that is able to indicate when it is operating in the absence of sufficient data, or when the case is at the boundary of the data on which it was trained, earns something that a model that is always confident will never obtain. Calibration is the distinguishing factor between a model that knows its boundaries and one that conceals its boundaries. Most clinicians can tell the difference faster than the people building the tools expect.

Trust Debt

Like technical debt, there is a price to bypassing these steps. Lack of clarity and ownership during rollout of a tool leads to debt that I call trust debt. It accumulates, and there is no way to measure it on a balance sheet. Every decision that is made without input from clinicians, every unanswered question and every unaccounted for outcome adds to the balance.

There is a second-order cost that makes trust debt worse than the financial kind. The effects of trust debt do not remain with the tool responsible for incurring the debt. Clinicians that have had a negative experience with a tool are likely to approach the subsequent tools with the same level of distrust. Most trust debt is incurred by implementations that are thoughtless, and the burden is felt by all tools and applications that are subsequently implemented, even the beneficial and effective tools. The way that a department interacts with AI tools for the first time has a long-lasting impact on the way the department will engage with AI tools in the future.

Eventually, the debt must be paid, and it is paid in the most costly ways. It manifests as disengagement and stagnation, as adoption that never really gets started, in pilots that slowly die due to a lack of belief in the project. By the time the leadership is able to identify the issues, the debt has been accumulating for a long time. The organizations that manage to stay ahead of the issues incorporate explanation and control as a standard practice, as opposed to an afterthought, and they manage to move a lot faster because they have spent the trust in advance.

There is a limit here that technology cannot engineer its way past. It cannot outrun mistrust. Progress within a hospital is not determined by the accuracy of a model. It is determined by the belief in the model, and belief inherently is a function of the quality of the interactions that influence it.

AI can only achieve this in the healthcare sector when it is able to respect and preserve the culture of evidence and human judgment, where someone is answerable for the state of every patient. It is a slow process to establish trust, and once we have established that trust we must protect it. These tools must be explainable to be credible.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants