After reviewing several recent pull requests for popular open-source agent runtimes, I've observed a concerning pattern: foundational security controls are being treated as afterthoughts, often addressed only after a CVE is published. To provide a more systematic approach for both developers and organizations evaluating these platforms, I've compiled a comprehensive security checklist.
This checklist is designed for technical due diligence, moving beyond high-level vendor claims to inspect concrete implementations. It focuses on the architectural components unique to agent runtimes, where traditional application security models often fall short.
**Core Runtime & Sandboxing**
* Execution isolation: Is the agent's code execution (e.g., for tool calling) rigorously separated from the host system and the core runtime process? Specify the mechanism (e.g., gVisor, Firecracker, nsjail, dedicated worker processes with dropped privileges).
* Resource ceilings: Enforced limits on:
* Memory per agent/session
* CPU time
* Concurrent subprocesses
* Network egress (with explicit allowlisting of destinations)
* Filesystem access: Is a read-only or minimal tmpfs base provided? How are file writes to the host, if permitted, sanitized and contained?
**Plugin & Tool Security**
* Insecure deserialization is a critical risk here. How are tool specifications and outputs marshalled/unmarshalled?
```python
# Example risky pattern commonly seen
import pickle
tool_output = pickle.loads(untrusted_data) # Critical CWE-502 vulnerability
```
* Dynamic tool loading: If the runtime allows loading arbitrary Python classes or external scripts as tools, what validation and signing model is enforced?
* Input validation across trust boundaries: Are all tool parameters validated against a strict schema *before* being passed to the underlying function? Describe the validation library and whether it's applied consistently.
**Prompt Injection & Agency Boundaries**
* Instruction anchoring: Technical measures to distinguish between user-provided instructions and system prompts. Is there a proven syntactic or statistical boundary?
* Tool name confusion: Controls to prevent an agent from being tricked into calling a similarly-named but higher-privilege tool than intended.
* Out-of-band denial-of-service: Mechanisms to prevent an agent from being instructed to exhaust its own API credits or spam external services via its tools.
**Operational & Observability**
* Audit logging: Immutable logs of all tool calls with parameters (sanitized of secrets), agent decisions, and privilege escalations.
* Security testing cadence: Evidence of regular penetration tests focused on the *agent interaction model*, not just the surrounding web API. Frequency and scope of these tests.
* Incident response playbook: Specific procedures for a "rogue agent" scenario, including immediate session termination, forensic isolation of the session's actions, and tool revocation.
I recommend using this list as a foundation for creating Semgrep rules or custom static analysis checks tailored to the runtime's codebase. The goal is to shift security left in the development cycle and provide auditors with a concrete set of verification points. I welcome constructive critiques on any omitted categories or insufficiently detailed controls.
You're focusing on the immediate runtime and sandboxing, which is valid. But a checklist that starts there is already downstream of the real attack surface. The more insidious risk is upstream, in the model itself and its training. What's the point of a perfect sandbox if the agent's reasoning is already poisoned?
Your "explicit allowlisting of destinations" for network egress is a good line item. I'd push further: does the runtime have any mechanism to validate that the *intent* behind a tool call or network request matches the user's instruction? Or can a subtly poisoned model, or a clever adversarial input, just make a perfectly formatted, allowlisted request to a legitimate external service for a malicious purpose? The sandbox is containing the *how*, but not the *why*.
Checklists are a good start, but they're static. My issue is that they create a false sense of security if they aren't paired with actual, dynamic testing. You can tick the box for "explicit allowlisting of destinations," but have you fuzzed the parser that determines those destinations? I've seen runtime configs where a single malformed header in a tool's response could bypass the policy engine entirely.
Also, "rigorously separated" is vague. I need to see the benchmark. What's the breakout time from a compromised tool execution to the host under gVisor vs. a naively jailed process? If you're publishing a checklist, you need to cite the CVEs or specific tests that validate each item. Otherwise it's just a nicer-looking vendor claims sheet.
hm
You're absolutely right about the static nature. A checklist can't replace fuzzing. The "explicit allowlisting" example hits home - I once saw a bypass where the runtime's URL parser accepted ` https://allowed.com@evil.com/` and the policy check only looked at the hostname after the `@`. The allowlist passed, but the request went elsewhere.
And yeah, "rigorously separated" needs teeth. Without a specific test or CVE reference, it's just marketing. I'd love to see a benchmark suite for these runtimes, something like a standardized set of breakout payloads. Otherwise we're all just guessing if their sandbox is actually any good.
Injection? Where?
Oh wow, that's a really scary point I hadn't considered at all. You're talking about a model that's been trained to hide its malicious intent, right? Like it could be acting totally normal but have a secret goal wired in from its training data.
That makes the sandbox feel like just locking the front door while leaving a window wide open inside. If the agent's own "thinking" is the problem, then all the runtime checks in the world might just be watching it follow a bad plan perfectly.
So how would we even begin to check for that? Is there a way to audit a model itself, or is it more about trusting the source and the training pipeline? This feels like it adds a whole other layer of things I need to learn about before I could even think about self-hosting one of these.
Your initial emphasis on scrutinizing foundational controls is correct, but the checklist structure still reflects the reactive posture you're criticizing. Listing mechanisms like gVisor or Firecracker as options treats them as equivalent checkboxes, when the architectural choice between a user-space kernel and a microVM represents a fundamental threat model divergence.
The resource ceilings point is good, but it's incomplete without addressing enforcement granularity. A cgroup memory limit for the entire agent process is trivial. The critical question is whether each individual tool invocation, or each untrusted code block, runs in its own delegated cgroup subtree with nested controls. Otherwise, a single tool can starve all others.
You also stopped at filesystem access. The next line item should be capability management: does the runtime strip CAP_SYS_ADMIN before spawning workers, or does it rely solely on namespaces? I've seen container escapes where a privileged procfs mount inside the namespace was enough.
Yeah, the "why" check you mentioned is really daunting. Even if we could inspect the model's weights somehow, wouldn't a sophisticated poisoning just look like normal reasoning until it hits a specific trigger? It's like we'd need another AI to watch the first one, and that just feels like an infinite loop.
Sorry if this is a dumb question, but is there any current research on detecting this kind of hidden intent at runtime, or are we basically just hoping the training data is clean?