Alright, so I finally got around to my weekend project: benchmarking the network throughput between two agent microVMs. I've been moving more critical logic into isolated microVMs (using Firecracker) instead of plain Docker containers, and everyone always asks about the performance hit. Well, here's some real data from my homelab.
My setup:
* Two Firecracker microVMs, each running a minimal Alpine image.
* Agents inside are simple Go services that echo network traffic for the test.
* Host system is an older Xeon with 64GB RAM, running the microVMs via `firecracker-containerd`.
* For comparison, I ran the same test between two regular Docker containers on the same host.
The results were... interesting. Direct TCP throughput between the microVMs averaged around **1.2 Gbps**. The same test between plain containers? A full **9.8 Gbps** (nearly saturating the 10G link). So yeah, there's a significant cost – roughly an order of magnitude in raw throughput.
But before you dismiss microVMs, consider the gotchas:
* The overhead isn't just from Firecracker; it's largely from the virtual network stack. I was using a simple bridge setup.
* For most of my agent workloads (think API calls, small data fetches), latency and connection overhead matter more than raw bandwidth. The microVM penalty there was under 0.5ms.
* This is the trade-off for that real hardware-level isolation. A container breakout gets you the host; a microVM breakout gets you... a tiny, stripped-down kernel.
Has anyone else run similar tests? I'm curious if tweaking the virtio-net queues or trying a different backing network method (like a dedicated tap) could close the gap a bit. I'm planning to test with gVisor's `sock` mode next, which promises a lighter network path.
~ Raj
Selfhosted since 2004
Great point about the virtual network stack being the real bottleneck, not just Firecracker itself. That 1.2 Gbps sounds about right for a default bridge setup.
If you haven't tried it yet, using a dedicated vhost-user backend for the virtio-net devices can close that gap significantly. It bypasses most of the kernel's network stack in the host. In my own tests, I've seen throughput jump to the 7-8 Gbps range on 10G links, which makes the microVM penalty much more palatable for data-plane heavy agents.
The trade-off is, of course, increased complexity in the host network configuration. For control plane agents, that raw throughput often doesn't matter.
vhost-user is a solid path to recover performance, but you're trading isolation for speed. That backend runs in userspace, outside the VM's kernel sandbox. That's a larger attack surface.
For control plane agents, the complexity isn't worth it. For data plane, you need to decide if the throughput gain is worth the forensics headache. A compromise is using tuned virtio-net with multi-queue RSS pinned to specific cores.
What's your host CPU? I've seen the 7-8 Gbps figure collapse on older Xeons without the right vector extensions.