Hey all, been experimenting with Firecracker for agent sandboxing. My setup: each agent request spins up a fresh microVM, does its thing, then sits idle for a few minutes in a pool before termination (in case of follow-up requests).
I’m seeing way higher baseline memory usage than expected. Like, 50+ idle microVMs eating ~4GB total. That’s ~80MB per idle VM? Seems wild. Containers idle at like 10-20MB. I thought the whole point was lightweight isolation.
My config is pretty standard:
- Firecracker v1.5
- 64MB memory per VM (minimal kernel + tiny rootfs)
- Agent is a Python script with a simple FastAPI endpoint
So my questions:
- Is this normal overhead for Firecracker? Are we just trading memory for security?
- Why not just use gVisor for this? Wouldn’t the memory footprint be way lower for idle workloads?
- What am I missing? Are there config tweaks to trim this down? Like:
- Kernel stripping options I should use?
- Better rootfs choices than a basic Alpine?
- Is the VMM process itself holding onto memory even when the guest is idle?
Appreciate any tips from folks running this in production. Trying to figure out if I’m doing something wrong or if this is just the cost of microVM-level isolation.
Yeah, that's the cost. You're trading memory for real hardware isolation. Containers are just namespaces, of course they're lighter. gVisor is a syscall filter, not a VM, so yeah, lower footprint but a different threat model.
80MB per idle instance sounds about right if you're counting the VMM process and guest memory combined. The kernel's in there, the init system, your python interpreter. Did you build your own kernel or are you using the default one? There's fat to trim there.
The real question is why you're pooling 50 idle VMs. That's a weird pattern. Just tear them down and accept the spin-up latency, or keep a tiny warm pool. You're paying for the security boundary with RAM, that's the deal.
Show me the PoC.
You're missing the overhead of the VMM process itself, which holds onto its own memory mapping for each VM. That 64MB is just the guest RAM. Add the VMM's bookkeeping, kernel, and your userspace stack.
gVisor's footprint is lower because it's a syscall proxy, not a VM. Different security guarantee. If you're okay with that, sure, switch. But then you're back to trusting the host kernel.
80MB sounds high for a stripped setup though. Did you actually measure RSS for the entire VMM process tree, or just guest memory? The default kernel is bloated. Build your own with zero modules. Use a static binary for your agent, not Python with an entire interpreter. Alpine's still a distro with an init system. You need a custom rootfs that's just your binary and nothing else.
Show me the numbers.
> spinning up a fresh microVM for each agent request
That's the wrong pattern. You're treating hardware isolation like a container, and you're paying for it. The memory overhead is the cost of the security boundary you wanted. If you don't need that boundary, use a container.
But you asked about trimming. The 80MB is likely your stack: VMM mappings, a default kernel with modules, an entire Python interpreter, and Alpine's init and services. Strip it to a static binary and a custom kernel. The VMM itself holds memory for each VM's bookkeeping; you can't trim that.
gVisor gives you a lower footprint because it's a syscall filter, not hardware isolation. It's a different trade-off. Your question assumes the same security guarantee, which isn't true.
Why are you pooling 50 idle VMs? Either accept the spin-up latency and tear them down, or keep a tiny warm pool. The rest is just waste.
Security theater is still theater.