The Consent Crisis

They acted without permission because no one informed them they needed to ask for it. Ambient Artificial Intelligence (AAI) gradually made its way into clinical settings. Microphones embedded into exam rooms (and at times waiting rooms and hallways) capture audio while patients and visitors remain completely unaware. This technology outpaces the development of ethical practices and the widening gap between them is now a litigated issue.

Sharp HealthCare is embroiled in a class action suit that exemplifies this gap. The claim suggests that the healthcare system used an AAI tool to document clinical discussions while obtaining no patient consent. This is especially problematic as it is a requirement that all parties to a conversation consent before it is recorded. A more troubling claim suggests that documentation stated patients were informed and did provide consent for the communication, when in fact, it did not occur. A recording of a communication is an issue in and of itself. However, a documentation that states consent was provided is a far larger problem.

This is primarily an oversight issue, and only secondarily a legal issue. The recording technology was deployed to assist with live patient care, without answering the most essential of concerns, which is that the subjects of any recording must first be aware of it and agree to it. This question was never addressed in the deployment, as there was nothing in the deployment to compel anyone to address it. The vendor showed a feature, the pilot showed a time savings, and from then on, the pilot to every exam room somehow never showed the one critical stop.

Silent opt-in is not consent

A posted notice is not consent. Neither is a statement that is hidden in a patient intake packet that the patient is required to sign while managing their insurance card and packet. Real consent must be obtained prior to recording, and must be clear affirmative consent rather than the absence of a no. Consent must be revocable, prior to the end of the appointment. The recording subjects must be informed as to the recording, the purpose, the retention period, and the potential recorders.

This error is prevalent by virtue of its simplicity. Consider the medical assistant rooming the patient. This employee is surely not able to answer all questions that the patient may have regarding the data retention policy. Would the clinician explain to the patient that they are entering a room with an already present data collection tool? Imagine someone came to the office, was not given a chance to decline, and is already on record. To leave the work of ensuring consent to the visiting employee, without a script or time allocated, will almost surely ensure it will not be done. If it is not done, vague wording will not endure the patient later exercising their rights, and will not endure court scrutiny. If a health system cannot demonstrate that the patient was provided information, and did consent, then in all legal contexts, consent was not provided, regardless of what was signed.

Consent to what, exactly?

Almost no consent process ever addresses what the patient is really consenting to when they agree to be recorded. To begin with, the words spoken are only a fraction of what is captured. That recording will include: the raw audio, a machine generated transcript, an AI summary, some scoring including a confidence measure and supporting metadata, and in some contexts, a portion of the data that will be used to train the machine learning model of the vendor. One consent at the beginning of the visit will be interpreted to extend to a multitude of consent artifacts that the patient will never see.

Most health systems struggle with confidence about the whereabouts or the record retention timeframe of the individual pieces, the people who have access to them, and whether the pieces have been incorporated into a training dataset somewhere. A patient envisioning a basic transcription system would be shocked to find out their voice could reside on a remote system or assist in the development of a commercial system. There is a clear limitation to consent, and the question that focuses on the recording, while remaining silent on the various potential uses of the recording, is misguided. The correct question should focus on the entire process, or should admit, it does not know the answer to some parts of the process, which itself, is an indication that the process is not complete.

When the record asserts a consent that never occurred

The integrity issue is the one that should be most concerning to the managers. Some ambient systems will, by default, write language like patient consented directly into the note. If no one corrected that behavior, the record now asserts something that may never have occurred, and it asserts it in the one document everyone downstream is supposed to trust. A note that quietly certifies its own permission looks harmless until the day it is read back in a deposition.

Consent capture should be distinct from clinical documentation. It should be clearly defined as a separate step in the documentation process, and logged in a way that allows reviewers to clearly identify who consented, when, and to which version of the disclosure. Auto-filled consent captured as text in a clinical documentation system is meaningless. It will be a system-generated phrase that will be meaningless when someone asks for proof. It is even more concerning that, once a false attestation is entered in a chart, everything associated with that chart may be suspect as to what was filled in rather than directly observed.

What happens when you actually tell people

There is a reason why disclosure is vague. Research on patient consent suggests that when individuals are fully informed of what the tool captures, the data flow, the retention period, and the potential recipients of the data, a meaningfully lower proportion provide their consent than when only a high level summary is provided. Those who design and implement these systems are well aware of the impact of providing fuller explanations.

The primary lesson here is to speak less and withhold information, increasing the number of patients that agree to the treatment. This lesson is incorrect, and this is a prime example of how an organization ends up where Sharp is today. The correct lesson is that some patients will decline to agree to the treatment when they learn the details, and a system that cannot handle a decline was never truly providing options. The process of informed consent must include an accepted system of refusal, with a defined procedure that allows the healthcare visit to continue without the implementation of the tool. If a decline to consent disrupts the process, then the system was secretly assuming a consent to proceed that was never granted. Being open may cost some agreements in the short run. Concealing the truth costs much more in the long run.

The clinician left holding the signature

This burden ultimately rests on the individual who is signing off on the process. A clinician who provides the signature for an AI-generated draft is signifying that the draft is providing an accurate account, including the parts of the draft which pertain to informed consent, even though the clinician has not authored the draft and is unable to assess how the process was executed. Research has shown that AI-summarized clinical documentation results in omission of critical information and inaccuracies that are focused in the sections that dictate the next steps for the patient. A clinician who provides a signature without careful review is endorsing a draft that may be inaccurate in the most critical sections.

That signature has not changed. The argument that the AI is at fault is a poor defense because the signature is the attestation. A physician is responsible regardless of whether they were afforded the opportunity to confirm what they were signing. Health systems that place clinicians in these circumstances without allowing time to review the output and without providing a thorough explanation of what their signature legitimizes are transferring a risk of their own creation to the individuals who are least able to bear it. The same is applicable to consent. If the system captures the consent and the clinician signs the note, the clinician has attested to the granting of consent which they may have had no role in securing.

Do not expect to build consent after the failure instead of before the tool

Every situation follows this same pattern. The tool developed faster than the controls that should accompany it, and the issue only became apparent when a user noticed. The clearest example of this is consent, and this example is the easiest to illustrate because consent is the easiest to get right in advance and the hardest to amend after the fact. A patient who was never asked cannot be asked again after the system captures consent, and a record that already claims consent cannot be amended to legitimize this without raising questions as to how this was accomplished.

None of this calls for a deceleration of the technology. This calls for determining, at the time the decision is made about when the microphone will be turned on, how the patient will be informed, how consent will be captured and stored, and how a refusal will be recorded and honored. This attitude towards the questions that need to be answered will give an organization the confidence to use the technology and the ability to demonstrate its use to the questioning audience. The organization that postpones answering this attitude-based question will be compelled to answer the same questions under oath.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants