The Override Rate Nobody Reports

A few weeks ago I sat down on The Signal Room with Gretchen Heinen. Gretchen has been a nurse for more than 17 years, much of it in case management and utilization review. Today she's the founder and CEO of Authsnap, a company that fights clinical denials.

Where she is spending her time working on the side of health care that is probably one of the most painful for patients. It's really dealing with denials and all the crazy gyrations you have to go through to get somebody to pay for what you thought they were giving you coverage to pay for.

Gretchen told me about a cancer patient who was prescribed a three drug chemotherapy regimen but was too sick to get the third drug. Because the order didn't match the guideline exactly, the whole thing got denied, and that patient didn't get the two drugs that could have helped. Then she said this: "It's stories like that where I watch patients not be able to get the care that they needed because of a sheet of paper, not because we actually can't give the care. Like we have the doctors, we have the resources, we have the medicine, we have the technology, but a sheet of paper was preventing their care."

I remember sitting with stacks of what we call EOBs, explanation of benefits, and just having to go through them and deal with every line that was an exception. And the worst part of it was knowing that you're probably going to have to do that two or three times on the same claim before you actually get something processed. It felt like you were getting an immediate denial, and would likely deny many more times before they would actually consider it. I don't think it's quite that cut and dry, but it felt that way, and I understand why people feel frustrated by that. Because no one wants to talk to a patient and tell them that they've got a denial when they've just had a horrible, horrible thing that they've been through. They had to have a surgery that has really disrupted their life. And now this financial stuff rears its ugly head.

It's astounding to me that we're still having these kind of challenges on the insurance side. There are components of the healthcare system, where you've got insurers involved and other industries, pharmacy, pharma and all that, where the Hippocratic Oath is not the prevalent rule that everyone operates by. On the clinical side, always. As far as I've seen anyway. But on the other side of it, that's not what comes first. And I understand we do have to pay bills and all that sort of stuff.

Only now there's a new player in the middle of all this.

What came out of the records

I spent a lot of my years on the provider side of revenue cycle, so I read these records the way an operator reads a remit file. Here’s the setup. Back in January, CMS started an experiment in traditional Medicare called WISeR. In six states, Arizona, New Jersey, Ohio, Oklahoma, Texas, and Washington, outside companies using technology, AI included, now make the first call on prior authorization for a list of services. The Electronic Frontier Foundation had to sue CMS to get the records, and they’ve put out over 1,800 pages since September.

Start with how the vendors get paid. EFF's review of the records puts it plainly: WISeR vendors are paid for requests that they deny, though not for denials reversed on appeal (EFF, September 8, 2026). CMS’s own records say that setup gives vendors “an incentive to deny as many claims as possible.” A bad quality score knocks a vendor’s payment down by 5 to 10 percent. When I read that, I thought about every remit file my teams ever had to fight with.

Millions of dollars that no one wants to fight for just gives these guys license to continue to take advantage and do their underpayments. Then no one actually measures it. You have these subtle underpayments that are not that big of a deal, moving a needle on a $2,000 procedure, but do that 2,000 times, and it matters. It’s having the ability to detect that. Most organizations don’t really have a lot in that space.

There’s just too many things that are done administratively that are just not detected, and if we don’t have the resources in the provider sector, and we don’t, generally speaking, to really hold them accountable, because we’re not able to build the capabilities to measure it at the level where we can actually detect what it is that they’re doing.

Then look at the rollout. Before go-live, one of the vendors told CMS its software wasn’t ready and that it was going to approve everything automatically until it was. CMS went live anyway. Within three months, at least four of the six vendors had gotten requests for information or corrective action plans. The standard is 72 hours for an answer, and one request sat for 83 days. Providers wrote in about patients calling their offices in pain while their procedures sat in a queue.

And then there’s the part I really can’t get past. CMS says a human clinician has to review every denial, that it can’t be done by technology alone. Okay. That’s the safeguard. So how long are those reviews taking? How often does the reviewer actually disagree with the AI? What does the reviewer even get to see besides what the AI flagged? None of that is in the records. EFF says it still can’t tell whether those reviewers have the time, or the real authority, to overrule the machine. I ran production teams for years, and nobody would have accepted a quality check that couldn’t tell you how long it took or how often it caught anything. Saying there’s a human in the loop and showing me what that human is actually doing are two very different things.

We have been here before

Medicode had, 20 years ago, 25 years ago, a platform called Claims Manager. They had two versions of it: one actually for the insurance company, one for the providers. Both of them have the rules engines in them. So you're constantly monitoring your positive edits and your negative edits, so you're testing for things that should be there, and you're identifying things that shouldn't. Both cases, you're dealing with that before the claim goes anywhere.

That's how I look at WISeR. A denial is a decision about somebody's care. If nobody's counting how those decisions get reviewed, how would anyone know the review is real?

The human in the loop

You hear probably way too often the concept of human in the loop, but there's definitely some areas where that actually would be really important for people who are designing and delivering this kind of capability.

One of the things that worries me is that we get to that acceptance place too quickly in certain areas where there's a ton of risk. Healthcare is obviously one of those that's most concerning to me, just because we really, really quickly forget things.

A friend of mine's a physician, was an ophthalmologist for years, but now he's been into AI and how to use it effectively for quite a long time. His quote was, AI is actually, it's dumb. You know who's smart? The individual who's operating it that has judgment, knows when to question something, knows when to challenge it, knows when to kill it.

The most successful deployments will help clinicians and staff do their work more effectively without removing their ability to question, adjust, or override.

A human being might look at you like you're from another planet and actually question you when you should be questioned, because it's the missing context or misdirection or those type of things that human beings are going to pick up on. AI doesn't do that unless you force it.

Gretchen's company uses AI to help build appeals, and she was clear with me about where the line sits: "We have a human involved in every single case because this is clinical information. So at the minimum, a nurse is looking at this stuff before it goes out the door."

So the question I keep coming back to is pretty simple. How often does the person actually disagree with the machine? If that number is close to zero, either you've got a near perfect model or somebody stopped looking. And if it's really high, your people are redoing the work and you just bought yourself an extra step. You won't know which one you've got unless somebody's counting.

Where the work actually goes

We might actually be shifting the actual capture of the data to having to do a deep dive review with a lot of scrutiny because the models are hallucinating or they're drifting significantly. So there are scenarios where it can be net zero. We've just actually moved part of the work to a different part of the workflow, but they still have to do the same amount of effort. So that's not really relieving the burden that we were hoping to relieve in those cases.

I've heard many physicians at this point talking about the displacement of certain things that are in one part of their workflow. It just moves it to a different part. So instead of actually either using a dictaphone type of a device or keying information to the chart, at some point you're going to have to go in and review it line by line and make sure nothing's fabricated.

One of the things that I encourage people to do if they're not sure how effective they are, just go check the nursing station. If you see post-it notes all over everything, you've got a problem. That means you've got some issues that you have inserted in the middle of the workflows that are not working for them, and I would love to tell you that's an exception, but it's really not.

The clinicians have to be much more involved in this stuff than they want to be, for sure. But we're continuing to poke clinicians back into these conversations that have nothing to do with what they went to medical school for. And it gets to be a little bit painful and tedious, to say the least.

We historically have thought we were solving one thing only to introduce another challenge or an obstacle for them. I think it matters whether you're telling them what their problem is and solving it, versus hearing their problem and getting their insight into how to fix it.

Look at your own house first

Now, before we all pile on CMS, I'd turn the same questions around on ourselves. Could your organization answer them about the AI you're already running today?

It was a decision support tool that was being designed at Mass General Hospital. When I took over the department, the project had been going on for quite some time. It was actually about to go live and I discovered that there was no auditability in there. The system administrator did not have anything that they could look at to know when a physician called with an issue.

Now, all of a sudden you've got something else that's contributing to it, but you don't have the mechanism in place so that the doctor can easily see what's behind the recommendation, behind the decision.

Does a clinician, because they're the ones signing off on a note and they're accountable for that, it's their medical license, do they know that they have the ability to say, no, I'm not comfortable? I don't have enough visibility into how this recommendation was done to actually trust it. There's authority that has to go along with these people who are named as well, because they're accountable to defend the workflow and the conclusions that the AI reaches.

Count what should be there

The ones that are going to make a real significant improvement in that space are ones who are assuming nothing, looking at it and understanding in almost real time where there's a deviation.

It's a great candidate for using generative AI to understand where patterns are off and where a baseline is, and when it's changing, and being able to detect it as soon as possible.

When you're looking at the transactional flow, there's a bunch of different comments, and depending where they sit in the sequence, and it could be a very long sequence, the real issue may not be identified in the top three. And if they're not, you might still not have a great understanding or clarity on why it got denied.

Gretchen does this every week. Her team tracks how many denials they work, how many get overturned, and how many days it takes to turn an appeal around, and every customer sees that dashboard on Friday. That's what counting looks like.

It's gotta have a tangible, measurable impact on the things that really matter, which tend to be the people that are most leaned on in terms of providing care. They're the ones that are the most adversely impacted if we don't do these things right.

If you're delivering care in one of those six states, start with the denials coming in the door. Track them by service and by what happens when you appeal. How many get turned over? That's about as close to an override number as anybody outside CMS is going to get, and it's the kind of evidence you'll wish you had if this thing grows.

Who is in the room

Congress has noticed. A House committee voted back in June to cut off WISeR funding for next year. That hasn't become law, so the experiment keeps running.

I've noticed a significant gap in terms of the presence when it's time to have conversations about regulation. I kept noticing the CEOs were showing up solo from healthcare providers, the insurance industries, big pharma, all of them. There's only a couple of people who are trained to practice medicine that are there. The rest of them are primarily lawyers, and these are people we're trusting to figure out how to deal with this AI thing. When you talk about the quality of care, for example, they treat medicine like it's static, never has been.

So the insurance industry, they're not all upset. They're just not, because right now the risk still sits with the person who signs off on the note, and it's the doctor. That's why I was like, we got to get clinical people involved in the regulatory space. We have to get their voices in there so that people start to think about these things and understand it.

Gretchen has sat down with some of the biggest insurers in the country, and she has someone on her team who works with policymakers. Her view is simple: "In order to solve the problem, you have to have both people at the table."

That's the only way it's gonna work, and quite honestly, I think it's gonna take people like you and I and others to get other industry involved in collaborating. These are our families, our friends, our neighbors, our communities. It's not easy work, but it's definitely work that needs to be done.

People who are developing and designing these systems need to be challenged. I don't care how far you go; it's not going to be far enough.

And if we don't get the right voices out there where we are right now, we are going to do far more damage with this than we did with anything else we've done.

Build the exit ramp

There's got to be an exit ramp when you decide that something needs to be turfed over to a human being. And no one's built it.

You've got to build in those pause points where only a clinician will know what you have to do in that moment. And you've got to build that safety net there. And it's got to be part of the workflow.

If you are unable to capture it in a straightforward manner, it isn't ready to be deployed.

And when you build that ramp, write it down every time a case comes off the automated path. Who picked it up, and what did they decide? Keep that record and you've got your override rate, built right into the workflow.

The number I would want

CMS had to be sued to release these records, and they tell us a lot about Medicare. They also hold up a mirror. If your board has been told a clinician checks what the AI puts out, the very next question ought to be how often that clinician disagrees with it. I'd want that number in my hand before somebody else comes asking for it.

Sources

Electronic Frontier Foundation, New Records Reveal Problems with Medicare's AI Prior Authorization Experiment (September 8, 2026) → https://www.eff.org/deeplinks/2026/09/new-records-reveal-problems-medicares-ai-prior-authorization-experiment

Electronic Frontier Foundation, New Records Obtained by EFF Expose More Problems with Medicare's AI Experiment (October 27, 2026) → https://www.eff.org/deeplinks/2026/10/new-records-obtained-eff-expose-more-problems-medicares-ai-experiment

Polsinelli, House Panel Moves to Block CMS AI-Driven Prior Authorization Program → https://www.polsinelli.com/health-care/publications/house-panel-moves-to-block-cms-ai-driven-prior-authorization-program-wiser-rider-providers

Continue reading from Hutchins Data Strategy

The Second Read → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-second-read

The Agent You Did Not Supervise → https://hutchinsdatastrategy.com/the-ai-health-pulse/the-agent-you-did-not-supervise

On the Signal Room podcast

Balancing Human Judgment and Clinical Trust → https://signalroompodcast.com/episodes/balancing-human-judgment-clinical-trust

AI in the ER: Clinical Judgment → https://signalroompodcast.com/episodes/ai-in-the-er-clinical-judgment

AI Governance in Healthcare: Just Culture, Emotional Readiness & Trauma-Informed Leadership | Susie Branagan → https://signalroompodcast.com/episodes/human-centered-ai-governance

More from Chris Hutchins

Subscribe — one email gets you The AI Health Pulse, the Signal Room podcast, and a free chapter of Beneath the Signal → https://hutchinsdatastrategy.com/free-chapter

Book Chris to Speak → https://chrisjhutchins.com/

Learn about our work to improve mental health around the world → https://continuamindhealth.com/

Do you have a podcast? We offer a Free growth assessment from PodcastPull.com → https://podcastpull.com/

Read on BeeHiiv → https://aihealthpulse.beehiiv.com/

Read on Substack → https://aihealthpulse.substack.com/

Tags: AI Health Pulse newsletter · human in the loop healthcare AI · AI prior authorization · CMS WISeR model · AI oversight in healthcare · clinician override of AI · the override rate nobody reports