The AI Readiness Diagnostic

A community hospital of 240 beds utilizes an AI imaging tool that flags likely pulmonary embolisms on chest CT scans. Positive flags are sent to the queues of the radiologists with priority. On the third day, a radiologist marks a false positive that delayed a different read by 40 minutes. The radiologist asks the operations team, who has the authority to pause the tool whilst the case is being reviewed. The operations team responds that they do not know. The radiologist then asks the CMIO. The CMIO escalates the case to the CIO. The CIO says since the integration was scoped and signed under his direction, the clinical halt decision is within the realm of clinical leadership. The clinical chief redirects the question to the radiologist. Six hours pass and the tool is still active, the radiologist has stopped using the outputs of the tool, and the question of who has the authority to halt the tool remains unanswered.

The question the radiologist asked was the first of five questions the leadership team would learn to ask before the next go-live.

Why the Readiness Question Remains Unanswered

The five questions emerged from the post-deployment review on Monday. Four emerged from the readiness gap questions that the leadership team did not anticipate, and one emerged from the first question asked by the radiologist. By 2026, the majority of health systems will have the required infrastructure, as real-time clinical decision support systems will have been deployed and will be collecting data. The data warehouses will be able to provide the resources needed to train the models vendors will offer. None of those resources, by themselves, will answer the critical question of the leadership team: can the health system deploy the next AI tool without creating the same nightmare answer 6 hours post go-live? The integrating diagnostic that answers that question resides at the intersection of the CIO, CMIO and Chief Data Officer, in the position that the system has not hired.

The Coalition for Health AI identifies organizational readiness as a precursor to the deployment of AI in healthcare. In their order of consideration, readiness comes prior to model selection, pre-deployment evaluation, and lifecycle monitoring. The AHA Trustees Toolkit on AI in healthcare regards the readiness issue as the primary board focus for 2024, and guided all board decisions on AI-related investments for that year. NIST AI Risk Management Framework 1.0 associates organizational readiness with the Govern function, as it relates to the organizational structure and the critical elements of senior management scoping, role clarity, and defined escalation pathways. The balance of the framework is unable to function without those elements. The literature provides a number of examples of diagnostics. Most healthcare organizations have not operationalized them.

What boards receive in lieu of a diagnostic is a survey. One of the big consulting firms produces a 60-page report that evaluates the system on a maturity continuum and prescribes additional investments. The report comes between budget cycles in a highly generalized form that hinders ability to act, and by the time the next evaluation occurs, the recommendations are obsolete and the underlying context has shifted. What is effective in lieu of a survey is a small number of operational questions that the senior management team can answer in clear, plain language, on demand, with consistency from quarter to quarter.

What Replaces the Survey

The first of many questions is can the system stop a live model in a day? Getting an answer has a lot to do with authority, process, and political reality. Most leadership teams find out the answer when they ask the question for the first time. It gives rise to an endless chain of approvals threading through legal, IT ops, contract management, and the vendor management team. There is not a single role or person authorized to approve it. ECRI Institute has characterized this type of incapacity as one of the defining reasons for placing AI on the 2025 risk list. The next question is, does the system possess validation data that is separate from what the vendor used to train the model? Without local held-out data stratified by the equity dimensions the board monitors, the system cannot conclude whether the model is servicing the patients that it is meant to assist. AHRQ patient safety publications have advocated this separation as a baseline condition for the acceptable use of AI in clinical practice. The ONC HTI-1 final ruling requires the disclosure of training data to facilitate local validation. The last question is, does each deployed model have a senior owner? Committee ownership, it turns out, is no ownership, especially when something goes wrong. Vendor ownership is essentially outsourcing. When a CIO treats a clinical model as infrastructure, that is a misfit, because most clinical models are not infrastructure.

Coalition for Health AI Blueprint v1.0 states that the lack of senior-level individual ownership is the most common reason that AI committees are unable to develop operational authority in their first year. At the same time, the work of the Robert J. Margolis Institute for Health Policy on AI committee composition shows that the committees that move beyond the first year have transitioned from a sense of collective ownership to named ownership by the ninth month.

Fourth, is the board seeing model performance on a quarterly basis, as should be the case prior to incidents that make it a requirement? Reporting that only begins after a close call makes it impossible to allocate resources to mitigate the issue before negative impact is felt. The reporting view works with a fixed cadence with the same metrics, and the same person presenting, for each quarter. AHA Trustees Toolkit on AI stated that this reporting structure is one of the four scoping decisions that influence whether the senior AI seat functions with standing or not. The fifth question, which is also the most frequently bypassed, is if there is a budget for workforce upskilling that is separate from the AI program budget. That particular budget line is the benchmark for whether the organization views AI as a tool to be leveraged or a transformation to be integrated. Budgets that have not been defined and safeguarded do not exist in the operational reality of a healthcare system. The work of the Duke-Margolis Institute on AI committee composition regards workforce upskilling as a separate budget line in the committees that have actualized operational AI, and the absence of that line is one of the most evident indicators that a system has grossly underestimated its readiness.

The Failure Pattern

The diagnostic gives a yes or no answer for each of the five elements. A system either possesses the capability or it does not. Systems that answer yes to all five have implemented at least one mature AI use case with named senior leadership. Systems that answer yes to three or four are likely to hire a Chief AI Officer in 2026 and are using the diagnostic to guide their resource planning. Systems with two or one yes responses will experience the same gaps with the next investment in AI as with the prior investment. The budget increases, but the trajectory remains the same.

At the lower end, there is a predictable sequence of events. The model is deployed because the vendor offered a persuasive sales pitch and the board wanted to have a cutting edge technology. An incident occurs six to nine months later that systematically creates the same five gaps. The model cannot be shut down, there is no local validation data, there was no assigned ownership, the board did not receive performance reports, and there was no funding for workforce upskilling. They could be addressed individually and the board would have to face them all together after the incident. The two year recovery begins after the harm is done.

ECRI Institute identified this trend when they predicted AI would be the leading health technology threat in 2025. The Joint Commission RCA2 methodology has been the dominant approach for the analysis of safety incidents since 2015, and was developed to address causes within the human, technological, and procedural domains. Events involving AI do not disaggregate in this manner.

How the Diagnostic Works

Present are the CIO, CMIO, Chief Data Officer, CFO, and if filled, the senior AI leader. This one hour intake meeting is structured around five questions that are answered in writing during the meeting. The completed document is submitted to the board during the next quarterly review. Answering the question in writing makes the meeting decision concrete. The exercise surfaces any model the system cannot halt in under 24 hours. The team must either address the model before the next quarterly review or the model is presented to the board with the condition unresolved.

The first of the quarterly meetings establishes a baseline. The same five questions are asked each of the subsequent meetings with the same expected documentation, and the board can direct investment based on the observed trend of increasing readiness over a set time. A system on a credible AI investment trajectory will go from three to four to five yes responses over a year. A system that is stagnant at two yes responses will have its gap further amplified by the next AI investment.

In Coalition for Health AI Blueprint v1.0, disciplined readiness work is considered essential to the other requirements laid out in the framework. They note that the readiness assessments conducted as part of disciplined readiness work become obsolete faster than the underlying conditions change. Health systems with mature AI portfolios adopt similar frameworks. In those frameworks, readiness is considered a constant state of evaluation. In those frameworks, the evaluation is conducted by the senior AI leader and is periodically reviewed by the board.

The five questions that came up after the community hospital go-live are the same five questions that come up in every meaningful readiness discussion. Any organization that has completed the necessary preparatory work for the next decade of AI deployment should be able to answer those questions with simple, plain-language yes or no answers. The ECRI Institute continues to elevate AI on its list of top 10 technology concerns because the readiness gap is a structural condition that is causing the top 10 concerns. Conducting the evaluation does not incur any cost. It requires an hour to complete five questions, after which the answers must be documented by the senior team.

Sources and Context

This edition refers to the Coalition for Health AI Blueprint v1.0, the 2024 Trustees Toolkit on AI in Health Care from the American Hospital Association, NIST AI Risk Management Framework 1.0, the Health Technology Hazard reports of the ECRI Institute for 2024 and 2025, the AHRQ publications on the safety of AI and its verification and lifecycle monitoring, the HTI-1 final rule of the ONC on the disclosure of training data, the RCA2 methodology of the Joint Commission for AI-related events, and the 2024 Duke University Health Policy Institute publication on AI and committee composition. Related editions: Your Board Will Ask About AI, Where Responsibility Breaks Down, The Innovation Tax, and The Chief AI Officer Mandate.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants