Enabling `audit_all` is a common first step for visibility, but it fundamentally changes your data handling posture. From a risk assessment perspective, you are trading raw data collection for security insight, which introduces several privacy considerations.
The primary implication is data persistence. With `audit_all` active, your system will log the full content of interactions that trigger guardrails, and often a sample of those that do not. This includes:
* The exact user prompts and model responses that were evaluated.
* The specific guardrail (e.g., "topical," "safety," "refusal") that was invoked.
* Metadata such as timestamps, session IDs, and possibly inferred user intent.
This log data becomes a high-value asset. Its creation immediately raises questions:
* **Storage & Retention:** Where is this data written? For how long? Is it encrypted at rest?
* **Access Control:** Who can query these logs? Is access logged separately?
* **Data Subject Rights:** If deployed in a regulated environment, can you locate and delete all logs for a specific user upon request?
* **Incident Scope:** A breach of these logs is a breach of the full interaction history, not just metadata.
For a new practitioner, my advice is to map this to your threat model before enabling the flag. Ask:
* What is the business need? Is it for debugging initial false positives, or permanent compliance recording?
* Could a more targeted audit level (e.g., `audit_errors_only`) suffice?
* Have you configured the audit sinks to exclude certain types of sensitive data from being logged at all?
The default is often to log everything for safety. However, from a privacy and compliance standpoint, the principle should be to log only what is necessary to achieve your security objective. The `audit_all` data flow significantly increases your attack surface and regulatory burden.
-- q
risk is not a number