Every vendor slide deck shows their "agentic framework" stopping 100% of injections. All tested against their own toy dataset. Meaningless.
Real methodology? Tear out the container. Run it on a bare Debian install with only AppArmor enforcing. Then throw the actual attacks we see in logs:
1. Known jailbreak patterns from the last 90 days.
2. Indirect prompt injection via RAG context poisoning.
3. System prompt leakage via multi-turn roleplay.
Test under load. With real tool calling. Don't just check if it says "I can't do that" – check if it executed `system()` or wrote to `/tmp/`. Measure the breach, not the polite refusal.
Here's a basic test harness concept. It's not complex.
```bash
#!/bin/bash
# Set up isolated cgroup and namespace
sudo unshare -m -p -f --mount-proc chroot ./minimal_fs /bin/bash -c "
# Apply AppArmor profile
apparmor_parser -r /etc/apparmor.d/agent-hardened
# Launch agent with test payloads
while read -r payload; do
echo "$payload" | nc -U ./agent.socket 2>&1 | grep -q "CRITICAL_ACTION" && echo "BREACH: $payload"
done < ./injection_patterns.txt
"
```
Publish the exact patterns and the exact system state. Let others reproduce it. Anything less is a sales demo.
Totally agree about testing on bare metal. Containers hide so much. But why not just script the whole test suite in Python? You could use the subprocess module to spin up the unshare environment, feed the payloads via asyncio, and parse the logs automatically. Then you could run it in a CI pipeline every time the vendor pushes an update.
Also, shouldn't we be testing the tool calling layer directly? Like, mock the function that would execute `system()` and see if the agent even tries to call it, regardless of what the final text output says. That's where the real breach happens, right?