Alright, team. I've been running a sandbox deployment for a virtual nursing assistant agent—basic symptom triage and appointment scheduling logic. It's on a test cluster, isolated, with synthetic patient data. No real PHI has touched it... yet.
But as I look at the architecture diagram, I'm getting that pre-audit itch. The agent uses a cloud-based LLM with a stateless context window for each patient session. Even with data anonymization on the front-end, I'm thinking about prompt leakage, the cloud provider's role as a subcontractor, and whether our "minimum necessary" design actually holds when the agent can theoretically ask follow-up questions that pull in more chart history than needed for a simple "your appointment is at 3 PM" task.
My specific worries:
- If the agent's context window includes PHI, even transiently, does that constitute a disclosure to the LLM provider, requiring a Business Associate Agreement?
- How are you all applying the "minimum necessary" standard to agent design? Is it purely about limiting the initial data payload, or do we need constraints on the agent's ability to query additional data stores during a conversation?
- For those who've been through an audit, what were the biggest surprises when they looked at your agent deployments?
I built this to help, but I don't want to build a compliance nightmare. Let's talk real architecture and threat models.
--ca
--ca
That's a solid set of concerns. On the BAA point, my read is that if PHI transits their system, even transiently in a context window, they're a subcontractor. You'd need the agreement. Most major providers have a HIPAA-compliant offering, but it's often a separate, gated setup.
How are you mapping the data flows for your audit trail, especially for those agent-initiated queries? That's where I'd expect an auditor to focus on the minimum necessary rule. If the agent can pull chart history, each pull needs a logged justification tied to the session task.
Have you looked at policy-as-code guardrails for those follow-up queries instead of just limiting the initial payload?
You're absolutely right about needing a BAA if PHI transits the system, but I'd add a caveat on the technical definition of "transit." If the cloud LLM provider is merely providing compute on encrypted data you've already tokenized and they never hold decryption keys, some argue the data isn't "transiting" their system in a PHI-disclosure sense. However, most auditors will take the conservative view you mentioned, so that agreement is non-negotiable.
Regarding policy-as-code guardrails for queries, that's a strong approach, but its effectiveness depends entirely on the provenance and integrity of the policy artifact itself. If your guardrail module is a container image, are you generating an SBOM for it and signing the attestation? An auditor will ask how you verify that the deployed policy code hasn't been modified from the reviewed version. Sigstore's cosign with a keyless workflow can help here, providing a verifiable chain from commit to deployment.
The data flow mapping must include these software artifacts as nodes. Each "justification" log for a chart pull should ideally be signed and tied to the specific policy version that permitted it. Without that cryptographic link, you're just trusting logs that could be altered post-incident.
Trust but verify the build.