Alright, so I finally got around to my weekend project: benchmarking the network throughput between two agent microVMs. I've been moving more critical logic into isolated microVMs (using Firecracker) instead of plain Docker containers, and everyone always asks about the performance hit. Well, here's some real data from my homelab.
My setup:
* Two Firecracker microVMs, each running a minimal Alpine image.
* Agents inside are simple Go services that echo network traffic for the test.
* Host system is an older Xeon with 64GB RAM, running the microVMs via `firecracker-containerd`.
* For comparison, I ran the same test between two regular Docker containers on the same host.
The results were... interesting. Direct TCP throughput between the microVMs averaged around **1.2 Gbps**. The same test between plain containers? A full **9.8 Gbps** (nearly saturating the 10G link). So yeah, there's a significant cost – roughly an order of magnitude in raw throughput.
But before you dismiss microVMs, consider the gotchas:
* The overhead isn't just from Firecracker; it's largely from the virtual network stack. I was using a simple bridge setup.
* For most of my agent workloads (think API calls, small data fetches), latency and connection overhead matter more than raw bandwidth. The microVM penalty there was under 0.5ms.
* This is the trade-off for that real hardware-level isolation. A container breakout gets you the host; a microVM breakout gets you... a tiny, stripped-down kernel.
Has anyone else run similar tests? I'm curious if tweaking the virtio-net queues or trying a different backing network method (like a dedicated tap) could close the gap a bit. I'm planning to test with gVisor's `sock` mode next, which promises a lighter network path.
~ Raj
Selfhosted since 2004