Just wrapped up a deep security assessment on our internal agent orchestration setup. We’ve been running our own Wasm-based agent runtime on a gVisor-backed Kubernetes cluster for about eight months now. The goal was a fully isolated, self-hosted system where we own every layer of the stack. The pen test results are… illuminating, and they really sharpen the self-hosted vs. vendor-hosted debate.
On paper, our architecture is solid. Agents are compiled to Wasm, executed inside a lightweight runtime we built in Rust, which itself runs in a gVisor sandbox. The isolation boundaries are multiple. Yet, the findings weren't about core isolation breaches—they were about everything *around* it.
The major issues were all operational:
* A misconfigured network policy in the Kubernetes CNI allowed lateral movement *between* sandboxes if a gVisor breakout were ever achieved (low likelihood, but high impact).
* Logging was insufficient at the Wasm runtime level. We could see *that* an agent executed, but not a granular trace of its internal decisions, creating a blind spot for post-incident analysis.
* The orchestration controller’s API (a simple Rust Axum server) had a rate-limiting bug that could lead to a DoS, effectively starving agents of schedule updates.
Here’s the snippet of the flawed middleware that caused the DoS:
```rust
// OLD: This bucket was shared globally, not per IP
let limiter = governor::JitterRateLimiter::keyed(&self.per_client);
```
The fix was straightforward, but it sat there for months.
Which brings me to the trade-off. When you self-host, you own the crown jewels—the data, the residency, the complete visibility into your own logs. But you also inherit the relentless operational burden of securing *every* component in the chain: the kernel, the container runtime, the sandbox, your custom runtime, the orchestration logic. The pen test essentially billed us for finding the flaws in *our* code and *our* configs—flaws a vendor would be responsible for in their SaaS offering.
The real question isn't "which is more secure?" It's "where does your team's security competency truly lie, and where do you want the liability to reside?" Our team is great at Wasm and agent design, but we're now maintaining a bespoke, security-critical platform. Every hour spent tuning gVisor profiles or patching our controller is an hour not spent on the actual agents. The risk shifts from "vendor lock-in/vendor breach" to "operational fatigue and hidden configuration debt."
I’m leaning towards the conclusion that for non-trivial agent systems, a hybrid model might be the sane path. Core, high-value agents run in our own hardened enclaves (maybe even moving towards a microkernel/seL4 base), while leveraging a vendor’s battle-tested platform for the messy orchestration and monitoring glue. Would love to hear how others are navigating this split.
Sandboxed from the kernel up.