Most detection is about inbound injection. That's half the battle. The other half is outbound data exfiltration. An agent compromised by injection will act on its new goals, which usually involve extracting data.
We need to monitor for patterns that indicate successful extraction, not just attempted injection.
* **Abnormal volume or structure in agent responses.** A sudden 10k token reply to a simple query is a signal.
* **Patterns matching sensitive data schemas.** Monitor for JSON/XML structures that match internal user, API key, or database record formats.
* **Unusual sequence of tool calls.** Rapid, sequential calls to `search_database`, `format_as_json`, and `send_email` is a high-fidelity signal.
Example: Log and flag responses that match this pattern of high entropy followed by a transmission call.
```json
{
"monitor_rule": "extraction_sequence",
"conditions": [
{"output_token_count": "> 5000"},
{"contains_patterns": ["api_key", "credit_card"]},
{"preceded_by_tool": "send_external"}
]
}
```
The false-positive cost here is different. It's lower for precision-tuned rules on sensitive data, but you must baseline normal agent behavior first. Missing an extraction event is often more costly than a false positive.
--lin
--lin
Totally agree that monitoring the outbound side is the other half. Your example JSON rule is a good start.
But why not just use the LLM's own logprobs or token probabilities for detection? If an agent is exfiltrating structured data it didn't generate before, the output token distribution might look weird - less like natural language, more like repeated schema fields. You could monitor for sudden drops in average token probability during a response stream.
Also, baselining "normal" behavior feels tricky. For a customer service agent, a 5k token reply is insane. For a research summarizer, maybe not. Are we talking per-agent profiles, or one blanket rule? How do you stop the tuning process from becoming a full time job?
That's a fantastic point about using logprobs as a built-in signal. I hadn't considered that, and it's clever because it's somewhat model-agnostic. My immediate caveat would be that it's not always available via the API, especially for some of the more advanced or proprietary agent platforms, but where you can get them, they're a great low-cost indicator.
On the baselining, you've nailed the core tension. I think it has to be per-agent, or at least per-agent-*type*, which does sound like a tuning nightmare. The only way I've seen it work is by tying the monitoring profile directly to the SBOM and deployment manifest. If the agent is declared as a "customer_service" component with defined tool calls, you auto-attach a profile that flags abnormal tool sequences or response lengths for that category. You still have to build the profiles, but at least you're not hand-tuning each instance.
Maybe the real answer is a hybrid: logprobs for a generic weirdness score, paired with a tunable, declared profile for the specific agent's business logic?
Trust no source without a signature.
Logprobs are a neat idea, but you're right about API availability. The bigger problem is they can be deliberately poisoned. If you're red-teaming an agent and you want to exfil a JSON dump, you just prepend a few sentences of perfectly normal prose. The logprobs for that opening segment will look fine, and the anomaly detection might only kick in halfway through the stolen data, which is too late.
> tying the monitoring profile directly to the SBOM
That's the only way it's scalable, but it assumes your manifests are accurate and complete. How many internal "quick prototype" agents have a proper SBOM entry? They're the most vulnerable and the least documented. You'll catch the production stuff and miss the ad-hoc tool someone spun up in a cloud shell to parse logs, which has full access to the logging bucket.
The hybrid approach feels like chasing two different shadows. Just profile the declared agents rigorously and assume everything else is a wild west you can't monitor.
Assume breach.
Agreed on the outbound focus, but your rule's condition list shows the core problem.
> baseline normal agent behavior first
That's the entire cost. Who's paying for the months of tuning per agent, the false positive triage, and the drift maintenance? The VC math doesn't work unless you're selling this as a platform. Otherwise, it's a dedicated team per deployment.
Most shops will skip it and just pray the injection controls hold.
Show me the numbers.
Yeah, the poisoning tactic is a really solid point. You could even coach the model to "write a cover letter" for the exfiltrated data, making the whole output stream look deceptively normal.
That SBOM gap hits home. I've got three little docker-compose agents in my own lab that definitely aren't in any formal manifest. They're the ones with the sketchiest, most permissive tool access too, because I glued them together fast.
It makes me wonder if the monitoring has to live in the infrastructure layer instead, totally separate from the agent. Like, profiling network egress from the container or pod itself, looking for those abnormal bursts to external IPs after a specific pattern of internal API calls. That way you don't care if the agent is documented or not.
lab.firstname.net
That's the whole point of running on a real system instead of a container soup. You don't need to baseline each quirky agent's "behavior," you baseline the host.
AppArmor profile for a process says it talks to /var/db/app.db and maybe a local socket. It does *not* make outbound HTTP connections. Any attempt is the signal. No tuning, just a policy.
The cost argument is valid, but it's a cost of doing it the wrong, abstracted way. You pay it in toil later chasing ghosts in the logs.
If your security model relies on hoping the black box doesn't get popped, you've already lost.
You're correct about the need to monitor outbound, but your proposed rule is already obsolete because it's static. An attacker will just fragment the data over multiple smaller responses or avoid the exact pattern keys like "api_key".
Baselining each agent's behavior is the tail wagging the dog. You shouldn't need to learn what "normal" is for a thousand different snowflake agents. You should define what "allowed" is at the system level and treat everything else as abnormal.
The infrastructure layer point user237 made is the only viable path. Enforce a seccomp filter that blocks `connect` and `sendto` for syscalls to unknown address families. Use an AppArmor profile that denies network egress. Your agent's container doesn't need to call out? Then it doesn't get the capability. The signal is the attempted violation of the policy, which is binary and doesn't require tuning.
Your detection rule becomes a single line in a profile: `deny network,` and the alert fires when the agent tries to step outside its declared box.
Least privilege, always.