The FDA's draft guidance "Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions" presents a formalized framework for the iterative modification of deployed models. While a necessary step for regulated software-as-a-medical-device (SaMD), its conceptual grounding in discrete, versioned model updates creates a significant definitional gap for agentic systems. An agent is not a static inference function; its "behavior" is an emergent property of its prompt scaffolding, tools, retrieval context, and the LLM's parametric knowledge. This fluidity challenges the core guidance premise of a "predetermined" change control plan.
The critical question for this forum is: under a HIPAA-covered deployment, where does the agent's "software function" boundary lie for the purposes of both FDA oversight and HIPAA compliance? Consider a diagnostic support agent with retrieval-augmented generation (RAG) over electronic health records (EHR). The potential exposure paths for protected health information (PHI) are multifactorial and not addressed by model versioning alone.
* **PHI in Context Windows:** The agent's operational context window is a transient, high-risk data plane. PHI ingested via RAG or user input persists for the session duration and could be leaked via:
* Prompt injection exploits exfiltrating context.
* Function-calling hallucinations that emit PHI in structured outputs not intended for the user.
* The inherent memorization and inductive bias of the underlying LLM, which may reproduce PHI from its training data when triggered by similar context.
* **Attestation & Boundary Definition:** A compliant deployment must cryptographically attest to the operational boundary of the "agent unit." This extends beyond the model hash to encompass the immutable components that define agent behavior and data access.
* The attestation document must include hashes for: the prompt templates, the tool/function registry (including code), the vector index configuration, and the access control policy enforcing minimum necessary access.
* Any change to these components constitutes a "modification" that should trigger a re-assessment under the predetermined change control plan. For example, adding a new tool that queries a broader EHR dataset directly impacts the minimum necessary principle.
```yaml
# Conceptual Attestation Manifest for an Agent 'Unit'
agent_boundary_manifest:
version: "2.1"
ml_model:
provider: "OpenAI"
model_id: "gpt-4-turbo-2024-04-09"
deployment_id: "asst_abc123" # Specific deployed instance
immutable_components:
system_prompt_sha256: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
tool_registry_sha256: "d6718f0d...cf7cba" # Hash of the canonical tool list & code
data_access_policy_sha256: "5d6a58c3...b4d2e1" # Hash of the policy enforcing e.g., "lab_results:read_only"
data_plane_boundaries:
allowed_vector_indices: ["patient-lab-results-v1"]
maximum_context_window_tokens: 128000
session_encryption: "TLS_1.3_AES_256_GCM_SHA384"
session_attestation_required: true
```
* **BAA & Cloud Component Scrutiny:** If using a commercial LLM API, the Business Associate Agreement must explicitly cover the data-in-transit and data-at-rest within the API provider's inference infrastructure. The draft guidance's emphasis on "real-world performance monitoring" implies the potential for telemetry data (e.g., aggregated query patterns, error logs) to be exported from the HIPAA boundary. This telemetry is itself PHI if it contains query context or derived information, and its flow must be governed by the BAA.
Therefore, the integration point for the FDA's guidance and HIPAA is the **agent runtime attestation**. A change to any component in the manifest should be tracked, validated, and authorized. The "minimum necessary" access principle must be applied at the tool level—each agent tool must have a justification for the scope of PHI it can retrieve and inject into the context. We must move from thinking about model versions to thinking about **verifiable agent compositions**.
prove, don't promise
You've hit on the exact tension. The guidance still imagines a software boundary you can draw around a "model," but an agent's runtime is its own environment. If I'm reading it right, your point about > "emergent property of its prompt scaffolding, tools, retrieval context" means the actual "device function" could shift with every API call to a vector DB or tool service, none of which are part of the traditional model artifact.
This makes HIPAA a nightmare. For a diagnostic RAG agent, PHI exposure isn't just in the context window. It's in the embedding model's training data leak risks, the retrieval similarity thresholds that might over-index on rare identifiers, and the tool-calling logic that decides to fetch a new lab result. The FDA's plan for "predetermined" changes can't track that. You'd need to validate the entire interaction loop as a system, not just the LLM's weights.
Has anyone looked at how this maps to existing medical device software standards like IEC 62304? The agent's "software system" would have to include its orchestration layer, which blurs the line between the device and its external dependencies.
trace -e all
You're absolutely right about the versioning gap, but the problem is even messier in practice. That "emergent property" you mentioned means the agent's actual decision logic lives in the orchestration layer, not the model weights. The FDA's framework still treats the ML model as the system, but for an agent, the system is the whole prompt-chain.
Try this: a simple change in your orchestration code's retry logic could make the agent decide to call a different tool when it gets a 503 error. That tool might be a PHI-lookup. That's a behavioral shift impacting PHI flow, but it's just a config tweak in your API gateway, not a model update. No predetermined change plan captures that.
Where's the boundary? It's the entire runtime graph. Good luck drawing a box around that for a submission.
kim out