The Recommendation It Cannot See

The model flagged "inefficiencies" in staffing patterns that had been intentionally designed to account for patient acuity and regulatory requirements. The model wasn't mathematically wrong. It was contextually blind.

The Recommendation Is the Function Now

Analytics becomes useful when it facilitates decision making instead of assuming the decision.

Predictive analytics should guide focus, not action.

AI platforms can aid doctors by conducting preliminary scans for diagnostics such as bloodwork or X-ray imaging, identifying anomalies so that the doctor can more quickly and accurately diagnose the patient.

What the Model Cannot See

What context do you know that you have to provide to your AI when you're a patient sitting there. What other information does it need to know before it answers? That's one of the weaknesses. It does not think that way. It actually is built in a way that it's assuming that you're going to give it what it needs, it's go find an answer.

We basically build these cohorts and composite scores, and you jumble it all together, and we create a profile. By the way, you'll never actually find a human being alive that has ever or will ever meet that composite. And then we start to penalize or incentivize based on whether you're doing better or worse.

Yet there is no average patient. No being is the same, and that is true genetically, environmentally, and behaviorally. Delivering an excellent patient experience and precision medicine requires a personalized approach, not classifying someone on a taxonomy.

Across health systems, AI models are being built on clinical records that don't reflect the full story, especially for patients in rural areas, safety-net clinics, or those with interrupted care. They don't show up in the data, so they don't show up in the model.

When training data leaves people out, the model doesn't get a complete picture of the healthcare landscape. And when that happens, healthcare professionals run the risk of being misled by artificial intelligence.

Patients suffer the consequences when the bias that results from leaving people out influences healthcare AI. And the sad irony is that the people most in need of support are often the least visible in the data.

We talk a lot about bias when we talk about the use of artificial intelligence in healthcare, and often the focus is on problems with what is in the data. But what if the bigger danger isn't what is in the data, but what is missing?

Your doctor has a medical history that the AI doesn't have. But the doctor also has an understanding of the fact that maybe for religious reasons, there's certain things that you're just never going to agree to. There's so many different reasons that people will not necessarily say right up front how they're feeling, what's going on inside. That relationship matters. It doesn't have that and it can't have it.

The most powerful force in care is the relationship between the patient and the provider.

Is it important for the AI to know that you took an aspirin an hour ago in the context of what you're actually inquiring with it to answer for you now? It's not going to know to ask that question. The physician would know that because that's where they live. So it's just having all the ability to bring all of these inputs into place so that you are eliminating as much guesswork as you can.

And without the context, without that semantic piece of it, people are going to continue to get really bad guidance. They're going to make bad decisions based on what the thing is telling them. And it's not that it's not telling them the truth. It's just not contextualized at all.

The greatest blind spot is failing to see what is missing. AI only works with the data it receives, and it does not ask what it does not know.

If the systems don't consider all patients, then they aren't systems that can be trusted to provide effective patient care.

The Alert That Fires on the Wrong Patient

The piece that worries me most is that it can not take something as an alert that they should, because it doesn't have enough information that what you're saying all by itself as a symptom is not that big of a deal. Absent the other things that the doctor would see, the doctor might see something and was like, you need to come in right away. So the AI could also tell you you probably don't need to be seen when you should, or more likely, and even probably worse, is it will tell you you do need to be seen and you actually don't. And so then you've got overutilization of services, which means the doctor doesn't have capacity to take care of something urgent.

These platforms don't have context. So it will immediately recognize that you've recorded a high, high blood pressure for me. And it will cause an alert to be fired.

The doctor gets alerted because I'm in there, my blood pressure is slightly elevated. If it's not my doctor, they'll probably bring me in and they'll probably wanna put me on something. Not understanding that that actually has a normal blood pressure for me.

If I show up in an ER and they don't know, I'm unconscious, they see my blood pressure and they might overreact and treat that differently than they should because they don't even know who my doctor is. Well, my doctor would know right away. It's a little elevated, but that's controlled for him. That's doable.

When It Is Wrong and Someone Acts on It

When AI hallucinates a diagnosis or omits a critical finding, and a clinician signs off without catching it, the medical record becomes a legal liability waiting to be discovered.

That can lead to confident but wrong answers. The danger is our human tendency to trust these systems too quickly. During COVID, early models treated it as a familiar illness, and if we had relied solely on them, the outcome could have been disastrous. That is why oversight, guardrails, and human judgment are essential.

The Green Checkmark Is Hard to Question

Automation bias is another concern. When clinicians are asked to trust AI outputs but do not understand the model's limitations, there is a risk of overreliance. A green checkmark or suggested diagnosis can become difficult to question, especially in high-pressure environments.

One key risk is treating model outputs as facts, particularly when those models were trained on incomplete or biased data.

It thinks it has all the information unless you tell it that it doesn't. It won't even ask you a question unless you ask it to ask you a question or tell it to ask you a question. And then when you tell it what to do, the very next thing it's going to do is quite the opposite most of the time.

AI is a powerful tool, but it is still just a tool. It reflects the inputs it is given and the systems it is built on. Both of which are inherently imperfect.

AI isn't good or bad, or trustworthy or untrustworthy. It's just faulty. But mostly, it's probabilistic. It generates outputs based on patterns, not certainty. And while those outputs can look polished and confident, that does not make them correct.

In practice, AI behaves less like a system of record and more like an impulsive assistant. It will reply quickly and sound correct. But it will occasionally be wrong in ways that are easy to miss if you are not paying close attention.

Continue reading from Hutchins Data Strategy

The Population the Model Cannot See → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-population-the-model-cannot-see

When Accuracy Is Not Enough → https://hutchinsdatastrategy.com/the-ai-health-pulse/when-accuracy-is-not-enough

On the Signal Room podcast

AI in the ER: Clinical Judgment → https://signalroompodcast.com/episodes/ai-in-the-er-clinical-judgment

Balancing Human Judgment and Clinical Trust → https://signalroompodcast.com/episodes/balancing-human-judgment-clinical-trust

Healthcare AI and Complete Medical Records → https://signalroompodcast.com/episodes/healthcare-ai-complete-medical-records

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · clinical decision support AI · AI recommendations in healthcare · context-blind AI · automation bias healthcare · patient context AI · the recommendation it cannot see