Okay, so I finally got around to stress-testing the container isolation layer in my local NanoClaw setup. We talk a lot about the threat model of a single agent, but the real test is how the orchestrator holds up under a massive, concurrent load. If the isolation breaks at scale, the whole security premise gets shaky.

I wrote a script to spawn 500 concurrent agent tasks, each in its own dedicated container. The goal was to find the practical limit of my orchestration stack (running on a single beefy server) and see where cracks appear.

**Initial findings:**
* The container spawn rate starts strong but hits a steep cliff around #350. The bottleneck wasn't CPU or RAM for the models, but the container runtime's socket and log handling.
* Shared volumes, even when mounted read-only, became a serious contention point. I saw latency spikes in simple file reads for later containers.
* At around #420, I observed the first "orphaned" containers—tasks that completed, but their cleanup didn't fire, leaving ghost containers consuming resources.

Here's the core of the spawn loop I used:
```python
# simplified spawn loop
for i in range(500):
agent_config = {
"task": f"stress_test_{i}",
"image": "nano-claw-agent:latest",
"volumes": {
"/shared_input": {"ro": True},
f"/agent_{i}_output": {}
}
}
# async container spawn call
asyncio.create_task(orchestrator.spawn_isolated_agent(agent_config))
```
The config looks sound, right? Each agent gets its own output directory. The breakdown is systemic.

**The gaps:**
* **Orchestrator State Management:** Under heavy load, the orchestrator's internal state (tracking container IDs, tasks, status) fell out of sync with the actual runtime state.
* **Shared Kernel Pressure:** Even with namespaces, 500 containers on one kernel caused PID exhaustion and network port contention, which isn't a model-level issue but breaks the isolation guarantee.
* **Cleanup Race Conditions:** The "fire-and-forget" async pattern, while fast, meant cleanup tasks sometimes lost the race, leading to those orphans.

So, the isolation model for any single container is robust. But the *orchestration* layer, under extreme concurrency, becomes the weak link. This suggests we need to think about queueing, batch limits, and more aggressive health checks on the container runtime itself, not just the agents.

Has anyone else pushed the container count this high? Did you hit different limits based on your runtime (docker, containerd, podman)?

luke out