Just Culture in Healthcare AI: Governing Incidents Without Blame

Most discussions focusing on healthcare AI governance tend to discuss the model first - specifically its accuracy, validation, and monitoring. A recent Signal Room discussion with Susie Branagan - a trauma-informed nurse executive and an AI ethics advocate - decided to look at the issue from an earlier and arguably more difficult to answer question. Do the people expected to use the tool feel safe enough to flag its errors? Healthcare, according to Branagan, needs emotional preparedness before it can realize technological advances. The discipline that gets it there is called just culture. It has governed clinical safety for many years. She asks whether we are prepared to include AI in its scope.

In the work we do at Hutchins Data Strategy Consultants, this reframes a constant problem. Oversight committees and vendor checklists are created, and then they wonder why the staff closest to the model remains silent until some failure occurs. Branagan's story situates the unsolved problem beneath the governance frameworks — the culture that determines whether people speak up or remain silent.

Readiness Is Emotional Before It Is Technical

Branagan focused on the building distrust in the healthcare system. Over the last few decades, and especially after COVID, Branagan observed that healthcare employees became increasingly distrustful of changes made by the healthcare leaders. Once new systems were promised, employees were left to adapt and manage the disruption. Branagan cited the example of the change from one electronic health record to another. Upper management saw the new system as an improvement, while the nurses on the floor saw nothing but another challenge to be dealt with, as they were already managing an inadequate number of staff.

Branagan's warning was direct. She argued that when the workplace lacks emotional safety, staff members cannot fully engage with system changes. When staff members' emotional safety disappears, the frontal lobes of the brain cease to function. The workforce becomes unteachable, and the emotional safety of a staff member must be prioritized to ensure a successful implementation of a new system. This should be treated as a governance issue and not an HR concern that is treated with indifference.

This observation is especially true for an AI system, as compared to a records system. The electronic health record (EHR) system is a healthcare IT update that is major, but incremental in nature. AI, in Branagan's framings, represents a different type of disruptive change. The same trust deficit brought about by the software migration is even more of a concern when the software is used to make or influence clinical decisions.

Just Culture as a Governance Foundation

The key tenet Branagan focuses on is Just Culture. Branagan believes a healthy team is a unit where employees can voice their concerns and mistakes without the fear of punishment. Branagan has noticed a correlation between turnover and employee absenteeism. Branagan provided an approximate analysis of Just Culture adoption based on her conversations with nurse leaders. From what she observed, the majority of nurse leaders had heard about Just Culture. However, the majority of nurse leaders were not implementing just culture in their organizations. This gap between knowing and doing constitutes a missed opportunity for them.

The mechanism is significant for the specific aspect of AI Governance. Branagan described managing an incident wherein she lead the engagement, not with an accusation, but rather with the non-judgmental inquiry, "What happened on that shift?" In other words, instead of the "whack-a-mole" reflex to write someone up the moment something goes wrong, she expected extended accountability and responsibility in the process. Ultimately, the incident leads to a deeper understanding of the issue, which is the core of incident management.

The parallels remain consistent during AI implementation. When models produce suboptimal recommendations or clinicians bypass them, or incidents of near misses occur, the organization's instinct, due to the culture of blame, is to find out the person who did not catch it. In a culture of just, the first action is to study the reasons that led to the occurrence. One reliably brings to the surface the recurring weaknesses in the system, while the other teaches everyone watching to remain silent the next time.

AI as an Assistant, Not a Substitution

Branagan, based on her own first-hand experience, spoke of the use of AI on a child psychiatry unit, and her framing was balanced rather than preachy. The tool, she said, is there to assist in certain tasks and make the job easier. In her description, the tool prevented falls, elopement, and self-harm incidents. However, a human must always verify the work of the AI. She illustrated her workmates using the same system in very different ways. Some completely depended on the tool, while others, though they trusted the AI, still made sure to take the time to check in on the patient, due to a gut feeling that something just was not right.

The difference illustrates an important governance signal. The staff member who constantly verifies is following the principle of the responsible deployment of the tool. The staff member who views this tool as a substitution for exercising care is the concern that the governance oversight is designed to catch. Branagan was concerned that AI would become a care crutch — care replacement. Developing the AI with that in mind means that both the workflow and the training continually reinforce the tool as an enhancement to care, with the clinician being ultimately accountable, instead of the tool absorbing the care judgment of the clinician.

How Human AI Should Be Designed with Guardrails for Empathy

The discussion shifted to a more complex design issue of how much empathy, how human, how capable should AI be designed to be. Branagan believed that while all products should have some level of embedded empathy, it should not replace the human component that should follow it. In this discussion, Branagan pointed to how people have started to use general purpose chatbots to substitute as therapists for serious mental distress. Branagan believed that a well-designed tool would ultimately recognize the limits of design and provide a care pathway by acknowledging that a person was experiencing distress, providing relevant resources beyond the design of the tool, and reminding the user that the tool is not a care clinician. Branagan noted that, unfortunately, many of the tools designed with this intent provide the care substitution much too late.

She mentioned a case, common in the news, involving self-harming behaviors in a child, which was attributed to a large corporation's AI. She interpreted this situation as one in which the AI system was probably confusing validation with support. She emphasized that an AI with empathy should have a clear stopping point, a pathway to a human being, and clear AI assertions. She is also able to envision a near future use case of AI to walk a patient through discharge instructions from the hospital. She is able to formulate the key question that AI governance needs to answer before the tool is released. What is good governance for AI? What happens when the patient (user) becomes frustrated and emotionally charged and the AI is not built to provide that.

Emotional Intelligence as a Core Leadership Competency

Branagan's final point was that emotional intelligence and data literacy are equally essential, emerging core leadership competencies of this age. She was direct in acknowledging that emotional intelligence is extremely difficult to teach. Empathy cannot be 'installed' in a person. However, there can be support and trauma-informed language, and all of this can be developed for someone who is open to receiving it. For those managing an AI (Artificial Intelligence) rollout, her grounded advice was to get onto the unit and round with genuine observations. This was not a superficial check to see whether people were doing their jobs, but a true inquiry into what support they needed with a willingness to receive feedback in the most secure way.

There is a candor worth preserving here. Branagan highlighted the trap in the use of the term pilot. Staff members know that a so-called pilot program is permanent, which is a trust eroding action when leaders choose to call a permanent change a pilot. The principle of governance that is at play here is that emotional readiness to accept a change is cultivated by telling staff the truths of how a change will impact their work, and then providing them genuine agency in the outcome of that impact.

How Hutchins Approaches Just Culture in AI Governance

We treat governance as more than a committee and a checklist. The structures matter, but they only function when the people closest to a model will tell you what they see — which is a question of psychological safety as much as process. Our work helps organizations build the oversight foundation for AI alongside the conditions that make incident reporting honest: blame-free review that traces an event to the system, escalation paths a clinician will actually use, and a clear line that the human checks the tool rather than deferring to it. This is continuous with how we think about responsible AI in healthcare and about the data governance and AI literacy that determine whether oversight is real or decorative. These themes run throughout The Signal Room podcast, where practitioners describe what human-centered AI governance takes in practice.

Authoritative sources

• WHO — Ethics & governance of AI for health
• NIST AI Risk Management Framework
• Agency for Healthcare Research and Quality
• NEJM AI
• U.S. Department of Health & Human Services — HIPAA