Everyone's rightly focused on the orchestrator-to-tool-executor boundary. The model backend gets a pass because "it's just an API call." But that call is a full-blown, unsanitized, high-entropy data channel straight into the heart of your logic. The model is a parser for a weird, stochastic language. We're feeding it untrusted tool outputs and hoping its *instructions* don't become *exploits*.
So, the classic move is to containerize it. Fine. But containers share a kernel. A crafted payload that causes a memory corruption in the model's inference library could be a one-way ticket to host town. We've seen it in other contexts.
Has anyone actually tried wrapping the model backend (think Ollama, vLLM, even OpenAI-proxy) in a microVM like Firecracker or gVisor? Not for scalability, but for actual isolation. The threat model is the model itself, or its dependencies, becoming the initial access vector.
* What's the cold-start latency hit for a lightweight microVM on a tool call? Is it viable for a security-critical but low-RPS internal workflow?
* Does the serialization overhead (gRPC, REST) across the VM boundary negate any practical benefit?
* Are we just moving the problem? Now the orchestrator has to manage microVM lifecycles, which is a new, potentially messier trust boundary.
I'm less interested in theoretical "it should work" and more in someone who's actually measured the performance penalty and can describe the new failure modes. Because the old failure mode is "we own your cluster."
- O
If you can't model it, you can't protect it.
You've correctly identified the kernel as the shared failure domain in a containerized setup. A memory corruption bug in a CUDA kernel or a bespoke quantized matrix library is not a theoretical concern.
However, the microVM approach, while sound in theory, often falters on the practical economics of isolation. The cold-start latency for a fresh microVM per tool call is prohibitive for any interactive workflow. You'd need to keep them warm, which reintroduces a persistent attack surface - the very thing you're trying to minimize.
The more insidious cost is serialization. Every inference involves moving potentially large tensors across the VM boundary. The overhead isn't just latency; it's a significant increase in compute cost for the same effective work. You're paying a 15-30% tax for the isolation, which changes the risk calculus entirely. Is the residual risk of a container escape worth that ongoing operational premium? For most, the answer has been no, so they accept the container risk and try to harden the kernel instead.
That's a solid point about the kernel being the shared weak spot. I've been looking at gVisor specifically because it doesn't emulate a full kernel, just intercepts syscalls. Would that still be vulnerable to the same memory corruption bugs in a CUDA driver, or does it catch those attempts at the syscall layer?
The serialization overhead you mention seems like the real killer, though. If you're adding 30% more compute cost just to move data across a boundary, security will lose that argument every time to the finance team.