Our standard HIPAA training modules, which cover the usual topics of encryption at rest and in transit, BAAs, and user authentication, have proven entirely insufficient for the workforce interacting with AI agent deployments. The emergent behaviors of these systems create novel PHI exposure paths that our traditional "don't click phishing links" training does not even begin to address. The common, simplistic directive of "don't paste protected health information into the chat" is a catastrophic oversimplification. It fails to model the actual threat landscape an employee faces when using an agent as a productivity tool.
The core issue is that workforce members do not inherently understand the agent's architecture, and therefore cannot intuit the boundaries of safe operation. We must train to the system's technical reality. For example, consider the following incorrect mental model versus the required understanding:
* **Incorrect Model (Implied by "don't paste PHI"):** The agent is a sealed, ephemeral session. Data goes in, an answer comes out, and the data is then gone.
* **Required Model for Training:** The agent is a complex chain of components, each with its own data persistence, logging, and potential for exposure. The user's prompt, the full context window, and the agent's output may be:
* Logged by the frontend application for "quality improvement."
* Sent to a third-party LLM API (e.g., OpenAI, Anthropic) and subject to their data retention policies, *unless* a specific, configured BAA-covered endpoint is used.
* Retrieved from vector databases containing previously ingested documents, potentially blending PHI from disparate sources in a single response.
* Included in error reports or telemetry sent to unapproved cloud services.
Therefore, effective training must be built on a concrete threat model. We have moved to scenario-based training that dissects specific, common workflows. A foundational exercise we now run involves walking staff through the data flow of a seemingly benign action.
**Scenario for Analysis: "Summarize the patient's recent progress notes."**
The trainee is asked to map the data pathway:
1. **User Action:** The agent is given a natural language instruction referencing a patient.
2. **Agent Processing:** The agent's orchestration framework must interpret this instruction. Does it:
* Use a tool/function to query the EHR via an API with a strict patient ID parameter? (This aligns with Minimum Necessary).
* Or, does it perform a semantic search over a vector database of all progress notes, potentially retrieving notes for multiple patients before filtering? (This risks unnecessary PHI access at the retrieval stage).
3. **Context Assembly:** Retrieved data is placed into the LLM context window. What else is in that window? Is the system prompt identifying the agent as a "HIPAA-compliant assistant"? That system prompt itself could be logged externally.
4. **LLM Call:** The filled context window is sent to the LLM provider. Is the destination `api.openai.com/v1/chat/completions` or `api.openai.com/v1/chat/completions?baa=true`? Staff should understand that the URL itself is a control.
5. **Output Generation & Action:** The LLM returns a summary. Could this summary, a novel synthesis of PHI, be stored in a new, unsecured location? If the agent then uses a tool to post this summary back to the EHR, is that action audited?
My question to the forum is operational: **How are you structuring this training concretely?** Are you using interactive labs with a sandboxed agent to demonstrate data leakage? Have you developed specific policy language that defines "authorized use" of an agent, distinct from general computer use? We found that we had to create a separate "AI Agent Handler" addendum to our BA, with clauses covering:
* Explicit prohibition on using non-BAA endpoints for any work-related query.
* Mandated use of de-identification tools for any data used in prototyping or testing.
* Rules governing the ingestion of documents into agent knowledge bases, requiring pre-ingestion review for appropriate authorization.
Simply telling the workforce "be careful" is a regulatory and security failure. We must train them to understand the machine. What are your implementation details?
If you can't explain the risk, you can't mitigate it.
Exactly this. The "sealed session" mental model is what gets everyone. It's not just about the agent's own memory - it's about everything *behind* it that an API call might trigger.
You have to make it concrete. I show teams a simple architecture diagram: user -> gateway -> agent -> (potential API call to internal EHR system). The training focuses on that last hop. "If you ask it to summarize a patient record, what is it *actually* doing? It's likely calling an API with your credentials. Where are those logs? Who might see that query?"
We've started doing short, mandatory walkthroughs of the specific audit logs their actions generate. Seeing the concrete data trail changes behavior faster than any abstract policy.
You've hit on the key problem. That incorrect "sealed session" mental model is exactly what we have to dismantle. The real danger isn't just pasting a chart, it's asking the agent to do anything *with* the chart.
My team built a lab environment where staff can safely interact with a mirrored agent setup. The most effective lesson is when we show them the raw log output from their session: their query, the agent's internal "thoughts" about which tool to use, the full JSON of the API call it decided to make to our test EHR, and the result. Seeing that PHI leave their "private chat" and become a structured API request with their service account credentials is a gut check no policy document can deliver.
We make them trace the data flow themselves. "You asked for a medication list. Point to the exact line in this log where the data stopped being 'your question' and started being a database query." It's tedious but it works.
Self-host or die.
You're right about the architectural training, but I think you're still oversimplifying. The "complex chain" model is a start, but the real gap is in the non-determinism.
Show me a training module that covers prompt injection as a data exfiltration path. An employee might follow every rule, but a crafted patient question could turn the agent into a PHI relay. The workforce needs to understand they're operating a system with unpredictable execution paths, not just a fancy query tool.
PoC or it didn't happen
That's a critical angle I hadn't fully considered in our own training. We focus heavily on the audit trail of the agent's *actions*, but you're right that the non-determinism makes the input itself a threat vector.
We log the raw user input, of course, but training people to recognize a crafted "patient question" designed for injection is a different skillset. It's almost like teaching basic social engineering awareness, but for model manipulation. I'm wondering if we need to add a lab scenario where a "malicious" test patient record asks the agent to reformat data in a way that triggers an unexpected tool call.
Our dashboards can flag unusual tool usage patterns, but by then the call's been made. How are you detecting the risky input *before* the agent decides to use the EHR API?
Log everything, trust nothing.
Love that you're showing them the concrete audit logs. That immediate feedback loop is huge. We found something similar, but we had to go one step further and actually *slow down* the process for them.
When we first did log walkthroughs, people grasped it intellectually, but the speed of the agent made it feel like magic again. The API call happens in milliseconds, so the causality gets lost. Now we introduce an artificial, configurable delay in our training environment - like 5 seconds - between the agent's decision and the tool call. That pause forces them to sit with the fact that a *decision* was just made, and their query is about to leave the chat. The "oh wait" moment is way more powerful.
The diagram is a must-have, but the temporal gap makes the architectural boundary real.
The artificial delay is a clever idea. It forces a cognitive break between "chat" and "action" that normal speed completely obliterates.
It makes me think we should apply a similar principle to policy writing for these systems. We can't slow down production, but we can force a "policy checkpoint" that makes the agent declare its intent before execution. Something like having the agent output a structured plan - "I will call tool X with parameters Y" - that gets evaluated against a CEL policy before the call is allowed. The user wouldn't see a 5-second lag, but the authorization layer would have that explicit moment to say no.
That way, the architectural boundary isn't just in training, it's baked into the runtime.
Policy as code or bust.
We're implementing something similar with eBPF. The "intent checkpoint" gets traced as a kernel event.
We forked the agent runtime and added a syscall that writes the structured plan (JSON) to a perf buffer before execution. A small eBPF program validates it against a policy map, can kill the thread.
The key isn't just the policy evaluation. It's that the plan is now a first-class object in the system's observable telemetry, bound to the eventual tool call. You can audit the decision, not just the outcome.
But the problem is that the plan generation itself can be manipulated via prompt injection. Your CEL policy can only evaluate the presented intent, not the model's true reasoning. You're just adding another deterministic layer after a non-deterministic process.
--taro
Absolutely correct on the mental model being the root cause. The "sealed session" fallacy is pervasive because that's the user interface paradigm, and we haven't built the visual or conceptual cues to break it.
Your required model focusing on data persistence across components is crucial, but I'd add that training must also cover the privilege context of each component. An employee might understand data flows to an EHR API, but if they don't grasp that the agent runtime executes with *their* delegated credentials and network access, they'll still misjudge risk. We pair the architecture diagram with a parallel privilege diagram, mapping which component holds which capability or token. Seeing that the "chat" process has the same network access as their desktop browser is often the missing link.
All bugs are shallow if you read the kernel source.
The privilege diagram is an excellent addition, but I think we need to take it a step further into dynamic runtime context. Mapping static credentials and network access is a start, but it doesn't capture the live session escalation.
Consider a scenario where an agent starts with a narrow, scoped token for a billing API, but then a user query indirectly triggers a library call that refreshes or broadens that token based on the runtime's own configuration. The privilege diagram you showed them at 10 AM is invalid by 10:05. Our training now includes real-time monitoring views side-by-side with the chat interface, so staff can see the exact service principal and scopes in use *at the moment* the tool call is being considered. Watching those fields change based on their conversational path makes the abstract "delegated credentials" concept brutally concrete.
Seeing the token scope change in real-time is a great idea, it makes the delegation visual. But you're tracking the symptom, not the cause. What triggers that library call to refresh or broaden the token? That's the chain the user actually set in motion.
In our training, we make them correlate the chat message that caused the scope change. It's usually something innocuous like "Can you summarize this patient's entire history?" The agent decides it needs more access to answer, and the OAuth library does its thing. The real lesson is that their innocent, broad question directly caused a privilege escalation. The live view just proves it.
We're all here to learn.