The accountability gap: Why health systems need governed AI, not just good AI

There is a version of the healthcare AI story that feels clean and optimistic. Intelligent agents handle routine patient calls. Staff focus on complex cases.  Satisfaction scores climb. Operating costs fall. Everyone wins.

That story is real and the outcomes data is accumulating. But there is a chapter that does not get enough attention in the boardroom, and it is the one that keeps compliance officers up at night. Health systems are deploying AI faster than they can govern it. And the regulatory environment is about to start asking hard questions.

A policy landscape in motion

In March 2026, the White House released its National Policy Framework for Artificial Intelligence, proposing that Congress adopt legislation to preempt state AI laws deemed to impose “undue burdens” on innovation. On the surface, that sounds like relief for health systems navigating an increasingly complex patchwork of state requirements.  Look closer, and the picture is more complicated.

The framework is not yet law. It may face legal challenges. State legislatures are not pausing while the federal government deliberates. More than 240 health AI bills have been introduced across 43 states in 2026 alone, and dozens have already been enacted. The themes are consistent at a high level: mandatory disclosure when AI is used in patient interactions, prohibitions on AI presenting itself as a licensed clinician, requirements for human oversight of clinical decision support, and restrictions on AI-driven claim adjudication without physician review.

But the details are not consistent at all, and the details are where health systems get exposed.

The patchwork problem

Consider a mid-to-large health system operating across three states: Texas, California, and Indiana. All three have enacted healthcare AI laws in the past six months. All three nominally require some form of disclosure and human oversight, and all three require something meaningfully different.

In Texas, under the Responsible Artificial Intelligence Governance Act, a clinician must provide the patient with written disclosure that AI is being used in their diagnosis or treatment before or at the time of the interaction. The obligation is on the provider; it is prospective, and it is patient-facing.

In California, AB 489 prohibits any language, design element, or functionality that implies the AI holds a healthcare license, or that care is being provided by a licensed human when it is not. The enforcement mechanism is notable: healthcare professional licensing boards can pursue injunctions directly, without waiting for a patient complaint. AB 2013 separately requires AI vendors to disclose information about the data used to train their systems, meaning the health system's technology procurement process is now a compliance touchpoint.

In Indiana, the focus shifts to the payer side. Effective July 1, 2026, insurers are prohibited from using AI as the sole basis to downcode a claim without reviewing the patient's medical record. Providers must also disclose when AI was involved in generating a submitted claim.

Now imagine a health system using an AI-assisted patient engagement platform. The same platform handles appointment scheduling calls in Dallas, post-discharge follow-up in Los Angeles, and prior authorization support in Indianapolis. Under Texas law, the platform needs to generate patient-facing disclosures before the interaction begins. Under California law, it cannot use language that implies clinical authority, and the vendor needs to answer questions about training data. Under Indiana law, any claim-adjacent interactions the platform touches require human review before submission.

These are not variations on the same requirement; they are structurally different obligations that affect how the AI is configured, how it communicates, what it can and cannot say, when a human must be in the loop, and who is responsible if something goes wrong. A disclosure template that satisfies Texas does not satisfy California. A workflow that passes Indiana's review requirement does not address California's licensing board exposure. And this example only covers three states. Washington passed five AI-related bills in its final legislative session.  Utah closed with nine. The pace is accelerating, not plateauing.

A gap between deployment and accountability

The “accountability gap” is the distance between what an AI agent does and what an organization can prove it did, and prove it did correctly, in each jurisdiction where it operated.

Most AI deployments in healthcare today are optimized for capability. The agents are capable. They handle scheduling, billing inquiries, pre-visit instructions, post-discharge follow-up, etc. They deflect calls that do not need a human, and they operate at a scale no workforce could match.

But when AI tells a patient about her copay and it turns out the figure is wrong, leaving her family to dispute a balance, the question is not just whether the AI worked. The question becomes whether the health system can reconstruct that interaction, identify what went wrong, and demonstrate that it has controls in place to prevent it from happening again. And increasingly, regulators in specific states will want to know whether those controls met their specific requirements.

That is a governance question that most AI solutions are not built to answer.

What governing AI requires

Accountability in AI-assisted patient engagement is not a feature; it is an architecture. It requires four things to work together.

Documentation at the interaction level. Every patient-facing AI interaction should be recorded, transcribed, and stored with the same rigor applied to calls handled by human agents. Not as an optional audit log, but as a default. The ability to reconstruct what was said, when, and in what context is the baseline requirement for any compliance or quality review, and it is foundational to demonstrating disclosure compliance under laws like Texas TRAIGA.

Real-time guardrails, not post-hoc remediation. Emerging state legislation is converging on a common principle: AI operating in clinical or patient care contexts must function under active oversight, not just retroactive review. That means the governance layer must operate in the moment. When an AI agent encounters a situation outside its defined boundaries, whether a clinical question it is not authorized to address, a disclosure it is required to make, or an escalation trigger, the platform must respond in real time. Surfacing a compliance issue in next month's report is not an acceptable answer to a California licensing board inquiry.

Configurable compliance policies. Because the regulatory environment is not static, the governance layer cannot be static either. The disclosure language required in Texas today differs from what California requires. The escalation logic appropriate for a prior authorization inquiry in Indiana needs to account for requirements that do not exist in states without downcoding legislation. Health systems need governance infrastructure they can adjust at the policy level without rebuilding the underlying AI or retraining the model. Compliance agility is now a product requirement, not a nice-to-have.

A unified audit trail across all AI touchpoints. Fragmented AI deployments create fragmented accountability. A scheduling agent from one vendor, a billing assistant from another, a post-discharge follow-up tool from a third: each carries its own interaction logs, its own compliance posture, and its own answer to the question of what happened. When a regulator or a patient asks for a complete picture of an engagement, “go check three different systems” is not an acceptable answer. Unifying AI-assisted patient engagement under a single platform with a single governance model creates operational consistency, meaningfully reduces legal risk, and makes the three-state scenario above manageable.

The trust equation

There is a reason this matters beyond compliance. Trust is the foundational asset in healthcare. Patients make decisions about whether to follow up, whether to fill a prescription, whether to show up for a procedure, based in large part on whether they trust the organization they are dealing with. AI that cannot be accounted for erodes that trust, even when it performs correctly, because the patient has no way to know what it will do next.

Health systems that close the “accountability gap” first will not just avoid regulatory exposure. They will earn something more durable: a reputation for deploying AI responsibly, standing behind every patient interaction their technology initiates, and treating governance as a prerequisite for innovation.

The question worth bringing to your next technology review is not just which AI vendor has the most capable agents; it is which platform makes those agents accountable, configurable, and auditable across every state your patients live in.

While regulatory audits and penalties are a real risk, they are not the right reason to build governed AI. Someday a patient will call your system and speak with an AI agent. She has no way to know if what it tells her is accurate or if anyone is accountable for it. Your governance posture is your answer to her.  

In healthcare, organizations that get this right will not call it compliance; they’ll call it care.

Check how your organization measures up with this AI Governance Checklist.

Watch Kristen Bell outline how CX AI can ease the burden for patients and case teams.

To learn more about how NiCE supports governed AI for patient engagement, visit our healthcare webpage.