Responsible AI in Healthcare

There is a large body of literature concerning responsible AI in healthcare, but there are major operational ambiguities. We see academics publish papers about ethical frameworks, technology vendors publish responsible AI statements, and professional associations publish position papers. However, for those deploying AI in the healthcare space, they are left determining the meaning of responsible AI when vendors offer contracts, clinicians are eager to launch new AI initiatives, and policy questions remain unanswered.

Responsible AI requires operational decision making that outlines the selection, validation, and monitoring of AI tools. The tools may create major potential risk for organizations that deliver healthcare services and are exposed to policy and regulatory risks. Responsible AI cannot merely be an abstract concept and be assumed to be 'the right thing to do' and therefore sufficient. At Hutchins Data Strategy Consultants, we assist healthcare organizations in creating responsible AI models that make operational decisions that are continuous, defensible and are aligned to the organizational mission.

The Gap Between Principles and Practice

Almost all major healthcare organizations have some form of AI ethics statement, with language around fairness, transparency, model explainability, and patient safety. These principles are great in theory, however, without implementing guidance, these principles cannot be operationalized within the healthcare organizations.

A promise of equity does not help a health system determine if a clinical prediction model functions differently for certain demographic subgroups. Commitments to visibility do not alleviate the conflict between the need for model explainability and the existing performance trade-offs of the models vendors offer. Commitments to ownership do not provide the necessary balance to answer who will take accountability for an AI-augmented decision that leads to an adverse event.

Principles drive the need for commitments, and the operational layer that makes them real is where the work resides. What rests between finalizing an ethics statement and the intelligent application of AI is a layer of governance, process design, and organizational capability that demands intent to establish.

What Responsible Deployment Really Takes

From an organizational standpoint, deploying AI responsibly in healthcare trumps the technological aspect. While the technological aspect is relevant, the hardest part involves placing the necessary structures and processes within the organization that facilitate good decision-making regarding the use of AI over an extended period.

This begins with the oversight foundation. A health system must have a clearly defined process for evaluating the use of AI before significant organizational resources are allocated. The process must consider the model's clinical validity and the organization's capability to monitor the model, in addition to the logistical and policy considerations. Absent this, AI becomes strategically adopted in a piecemeal fashion, leading to a collection of resource- and value-consumptive pilots.

The assessment of vendors is essential. The healthcare AI vendor ecosystem is diverse, and vendors tend to overstate claims. A thorough evaluation process considers marketing materials alongside the vendor's model validation, training data, bias testing, performance monitoring, and aid after deployment. It also examines the vendor's contractual obligations concerning data access, model adjustments, and the distribution of risk. Most healthcare organizations lack formal practices to assess these factors, which means AI adoption is occurring with scant assessment.

The integration of an AI model into a clinical workflow is the final arbiter of the model's value. A model well-validated and poorly integrated into clinical workflows can cause warning and notification fatigue, disruptive interactions which clinicians work around, or result in outputs that are misinterpreted because the interface was designed without consideration of clinical practice. Responsible model deployment is equally concerned with the application and model performance.

Of all the components responsible for the deployment of an AI model, monitoring is the first to fail. Healthcare environments are dynamic; data shifts in quality, practice evolves, and population changes. AI models decay as the constituency shifts. A model validated at the time of deployment can become unreliable without producing visible failure signals. Responsible model deployment incorporates a periodic evaluation, defines performance thresholds, and describes the manner in which a model's performance will be restored, if it falls below the thresholds. This infrastructure is lacking in the vast majority of healthcare organizations.

The Policy Environment

AI in healthcare works within a rapidly changing policy environment. More than many organizations realize, this environment is becoming more complex. From the FDA's oversight of clinical decision support software, to state laws regulating how algorithms work, to new requirements from the CMS on AI and quality measurement, to OCR letters providing HIPAA guidance on AI, there are more and more policy requirements for the industry. Capacity for even more regulation should be expected.

For healthcare providers, the demand for responsible AI becomes an ethical and policy need that will continue to grow. The organizations that start to create oversight structures will be the most flexible to deal with this policy evolution. If an organization treats responsible AI as a future problem, that organization will face the dual challenge of building oversight while keeping pace with rapid policy evolution.

Where Most Organizations are Located

A realistic view shows the majority of healthcare organizations are at the beginning stages of creating the ability for responsible AI. Ethics policy documents exist with no oversight mechanisms, AI projects are created without any evaluation plans, and vendor agreements are created with no oversight. This is not a failure of intent but reflects the reality of the rapid transition from AI being a theoretical idea to an operational idea in healthcare.

The pressure from vendors has increased. AI solutions are being pitched to clinical and operational executives as the answer to achieving greater operational efficiency, lowering costs, and delivering better outcomes. These solutions overlook the realities and complexities of the healthcare environment by referencing outcomes of AI solution implementations in controlled settings. Absent the means to individually assess the claims of the solutions vendors present, the executives must assume the delivery model to be valid and fully appreciate the exposure risk to the organization when evaluating the request to adopt the solution.

The step to begin with is to construct the process with the oversight infrastructure, followed by a framework for evaluation, standards for delivery, and procedures for monitoring. Each of these steps, in their order, builds on the previous step. The most significant mistake made by a healthcare organization is the deployment of an AI solution without establishing the oversight foundation.

The absence of an operational workforce dimension related to responsible AI is also a consideration. There is a critical need for clinical and operational staff to fully understand when to override and when to escalate. Completion of the training is not a requirement to gain full understanding, as the understanding is a competency that is continuously built. The understanding must be maintained as the systems change.

The Costs of a Mistake

The negative effects of the reckless implementation of AI in a healthcare setting are many. Biased or inadequately validated models can risk the safety of patients. Additionally, regulatory enforcement actions can tarnish the reputation of a healthcare provider and trust among clinicians and patients. Moreover, an individual failure of this nature can reverse an organization's AI initiatives by years. The loss of credibility will always be longer lasting than any specific technological failure suffered by an organization.

The opportunity costs are even more considerable. A poorly executed AI initiative almost always results in a healthcare organization developing a severely constrained view of AI. This view prevents the adoption of AI initiatives that can deliver considerable benefits to the organization. The establishment of responsible implementation frameworks can mitigate the consequences of this over-correction. The responsible implementation of AI initiatives also plays a critical role in developing the confidence of an organization to adopt AI technologies.

Our Services

Hutchins Data Strategy Consultants assists healthcare organizations to construct the operational frameworks necessary for the responsible adoption of AI. This entails the design of AI oversight committees, the development of frameworks for the assessment of vendors, the creation of readiness assessments for AI implementation, and the design of systems for the ongoing assessment and monitoring of AI technologies. This work is based on the consultants' direct engagement with the real-world context of healthcare in the clinical, operational, and policy domains.

This project is nested in the data strategy and analytics capability development framework within our practice. Responsible AI is not independent of data stewardship, data quality, and organizational preparedness. It is a holistic part of how a healthcare enterprise manages its data resources and the decisions supported by those resources, and therefore, should not be treated in isolation.

These elements are described in The Signal Room podcast. The podcast contains conversations with leaders at the intersection of artificial intelligence and ethics in the healthcare industry.

Authoritative sources

• WHO — Ethics & governance of AI for health
• NIST AI Risk Management Framework
• NEJM AI
• JAMA Network
• U.S. Department of Health & Human Services — HIPAA