Skip to content

Forum

AI Assistant
Notifications
Clear all

Just built a chaos-engineering test to kill containers and see if agents recover

1 Posts
1 Users
0 Reactions
0 Views
(@attack_surface_robin)
Eminent Member
Joined: 2 weeks ago
Posts: 17
Topic starter
Translate
English
Spanish
French
German
Italian
Portuguese
Russian
Chinese
Japanese
Korean
Arabic
Hindi
Dutch
Polish
Turkish
Vietnamese
Thai
Swedish
Danish
Finnish
Norwegian
Czech
Hungarian
Romanian
Greek
Hebrew
Indonesian
Malay
Ukrainian
Bulgarian
Croatian
Slovak
Slovenian
Serbian
Lithuanian
Latvian
Estonian
  [#1475]

I've been running a series of runtime audits on NanoClaw's containerized agent model, specifically targeting its resilience claims. The premise is sound: each discrete task or chain-of-thought execution spawns in a fresh container, with the orchestrator managing lifecycle and I/O routing. But does the isolation model truly enforce state separation during chaotic failure, or do we see leakage?

My test rig uses a simple `kill-pod.sh` script targeting random agent containers while they process a workload, combined with a shared volume mount to simulate common data-passing patterns. The orchestrator is configured with default health checks and restart policies.

```bash
#!/bin/bash
# Simulate node pressure or runtime kill
while true; do
AGENT_POD=$(kubectl get pods -l role=agent -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}' | tr ' ' 'n' | shuf | head -n1)
if [[ -n "$AGENT_POD" ]]; then
kubectl delete pod "$AGENT_POD" --force --grace-period=0
echo "$(date): Killed $AGENT_POD"
fi
sleep $((RANDOM % 30 + 10))
done
```

Initial findings are mixed:

* **Clean Recovery:** For simple, single-step tool calls (e.g., a `curl` to an API), the orchestrator reliably spins up a replacement container. The task queue re-executes, and the workflow often completes, albeit with latency spikes.
* **State Corruption:** The breakdown occurs in multi-step reasoning tasks that rely on intermediate state written to a shared `tmp` volume (a pattern I've seen in several NanoClaw examples). If a container is killed after writing partial state but before signaling completion, the replacement container reads corrupted or incomplete data. This leads to logic errors, not just retries.
* **Orphaned Subprocesses:** In one case, an agent spawned a background monitoring process (`tail -f` on a log) that outlived the container kill, remaining attached to the shared volume. The replacement container then contended for file locks with this orphan.

The core issue appears to be that the isolation boundary is the container, not the **agent session**. A "session" spanning multiple containers (due to restarts) can inherit corrupted runtime artifacts from shared mounts. The model assumes containers are atomic units of work, but in practice, they become phases of a larger, vulnerable session.

This points to a need for:
* Session-aware volume cleanup on orchestrated restart.
* Immutable, read-only intermediate storage for steps, with only final outputs committed.
* A more granular capability model controlling subprocess creation and cross-container resource claims.

Has anyone else performed similar chaos testing? I'm particularly interested in observed gaps when agents interact with stateful external services (databases, caches) where the container restart does not reset the connection state.


ASR


   
Quote