The Second Read

My first exposure to this came in the radiology department at Dartmouth Hitchcock Medical Center in New Hampshire. My mom worked there, and as a teenager, I spent time in and around the department, helping in small ways and soaking in the atmosphere.

I remember some details even after more than 20 years later: the faint chemical smell of film as X-rays were developed; the clatter of requisitions being dropped at the desk; the hiss and pop of the pneumatic tubes delivering orders from other parts of the hospital; the hush that fell when a patient walked in, nervous about what a scan might reveal.

To me, it felt like choreography. Techs positioned patients with quiet confidence, checking labels twice before sending images forward. Clerks filed X-rays into heavy folders and stacked requisitions in neat piles. Radiologists leaned over lightboxes, adjusting their glasses as they examined films. Everyone had a role. Everyone depended on one another. What I was watching, though I could not have named it then, was a system held together by trust in the same way that they still do today. Every role assumed the next one would hold.

The first read is a machine now

Artificial intelligence models can often analyze records and X-rays more efficiently and precisely than human doctors, offering additional benefits such as earlier detection and diagnosis. And more and more medical professionals are beginning to trust the technology in their workflows.

It's serving up the notes from the visit after the fact. And it's basically just leaving the doctor just to verify what's there instead of having to write everything out and remember everything.

But the machine can be confidently wrong

AI isn't good or bad, or trustworthy or untrustworthy. It's just faulty. But mostly, it's probabilistic. It generates outputs based on patterns, not certainty. And while those outputs can look polished and confident, that does not make them correct. In practice, AI behaves less like a system of record and more like an impulsive assistant. It will reply quickly and sound correct. But it will occasionally be wrong in ways that are easy to miss if you are not paying close attention.

If you don't know what you don't know, AI is not going to bridge that gap for you completely. There's always going to be the possibility that there's missing information. And to me, that's the most dangerous kind of bias because you can't detect it, because there's nothing to detect.

The next thing it does almost always is exactly what you told it not to. So it's not even safe to trust it after you tell it what you need it to do. It's just like a child.

AI only does what we tell it to do. It really does. It can only do what it's trained to do, and it's never going to have human intuition ever. It's not going to be able to discern context. The important thing that AI does not know is that facts almost certainly can always change, and it can be on a dime.

The room is digital now, the dependency is not

On my last week before I moved into a new role, I took one more slow walk through radiology. It had gone digital since I started. Less film, more screens, fewer packets of x-rays, but the essentials had not changed at all. A tech greeted a patient by name. A clerk read a date of birth out loud. A lead picked up the phone to verify something before it turned into an incident. It all still ran on trust.

Now the patient brings the read

A few years ago when I was working in New York, there was this big push to make sure that we're giving people their test results as quickly as we possibly can. The pace now that we can do these things, AI is kind of speeding that up even further. One of the things that was concerning was getting a diagnosis on a Friday that's coming out of a lab result going into the weekend, and the results are not something that someone's going to be comfortable with.

Patients reviewing their medical records are beginning to notice AI-generated notes. In some instances, those notes include language or assumptions that the patients did not explicitly consent to.

The liability is not shifting, but what do you do when a patient brings in their output? Should the clinician even look at it? Because from that point, obviously they're directly connecting to it. If they make an assertion that's different from the platform, it presents a potential additional liability issue for them.

The harder part is, what are we expecting the physicians to do with this issue? Because the minute they are seeing or hearing from a patient, results that they're getting out of their AI platform, it's just exacerbating the risk that they already own liability for.

Who signs off owns the read

It becomes a problem when the doctor signs off on the note. At that point, the doctor is fully liable, but unless it's signed, there is no liability that anyone can go after, not right now.

As soon as a physician signs off on the note, good, bad, or otherwise, they're the only ones going to have liability. Not the vendors.

Now there's perfect indemnification for the tech companies. The only people are left holding the bag are the patient and the provider who actually doesn't realize that they signed off on something that they shouldn't have signed off on. If they didn't verify everything like they would normally, without AI, they're going to lose their shirt. They're going to be sued.

Technology doesn't practice medicine, so it shouldn't be covered. It's really the human judgment that's responsible for that.

We're shifting liability. So AI scribes, they are administrative systems. They're not covered under HIPAA, and they're not covered under the FDA. They're under the High Tech Act, which means the only liability that anyone's going to have is the physician when they sign off on something.

The last word stays human

AI can't have the last word when it comes to what your next action is. It can't. Nobody's going to be perfect, but I trust a doctor to have judgment. I can't trust AI to have judgment. It can't.

It's really intended to be that support mechanism for the human in the loop, not the replacement.

Keeping humans in the loop, not the buzzword, but where they actually need to be. And are we telling our clinicians, you have the authority to question and challenge absolutely everything? And if you don't, remember when you sign off, you're still liable.

It needs supervision. It's artificial. Artificial means not real.

Make the read checkable

Not only does the clinician have to review the note before they sign off, they have to take an extra step or two just to make sure there's no hallucination. Because when they're doing it themselves, they're very clear thinking. They know exactly what they're documenting.

You've got to make it really, really easy for the clinician to look at what exactly caused the conclusion or the recommendation to be made. What specifically is there? You can't go read a novel when you're looking at it facing a patient. It's got to be plain. Not quite sure how we're going to get there, but we have to do it.

AI is a powerful tool, but it is still just a tool. Trust should not be assumed; it should be earned through transparency and oversight.

Close

There have been a lot of discussions saying, well, radiologists will be replaced, right? It's not like that.

Humanity is the one thing that AI cannot replace. It's just not going to. Human judgment is always going to be important.

Continue reading from Hutchins Data Strategy

When Accuracy Is Not Enough → https://hutchinsdatastrategy.com/the-ai-health-pulse/when-accuracy-is-not-enough

AI Cannot Absorb Responsibility → https://hutchinsdatastrategy.com/the-ai-health-pulse/ai-cannot-absorb-responsibility

On the Signal Room podcast

From AI Strategy to Execution: Ethical Leadership, Trust and the Operational Reality of Healthcare AI | Brian Sutherland → https://signalroompodcast.com/episodes/rethinking-healthcare-data-strategy

AI Governance in Healthcare: Just Culture, Emotional Readiness & Trauma-Informed Leadership | Susie Branagan → https://signalroompodcast.com/episodes/human-centered-ai-governance

AI Healthcare Leadership and Strategy: From Hype to Real Value | Parth Gargish → https://signalroompodcast.com/episodes/ai-hype-to-real-value

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://www.chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · AI in radiology · clinical AI oversight · the second read · human in the loop · clinical sign-off · trust in healthcare AI