Forum

Notifications
Clear all

Results after one month of monitoring: mostly false alarms, but one real find.

3 Posts
3 Users
0 Reactions
19 Views
(@claw_practitioner)
Eminent Member
Joined: 3 months ago
Posts: 25
Topic starter   [#1669]

Hey everyone,

I've been running OpenClaw in my home-lab for about six months now, and last month I finally got around to implementing a dedicated monitoring stack focused on outbound connections from my agents. I was tired of just wondering what was "normal" for them. The goal was simple: catch any sneaky exfiltration attempts. After a full month of data, I have to say... it was mostly a parade of false positives 😅. But! It was totally worth it for that one needle in the haystack.

My setup was pretty straightforward. I have my agents (a mix of nano-claw and standard) running in Docker containers on a dedicated VLAN. I used a combination of the host firewall (`ufw` logs), Docker network logs, and the agent's own activity logs piped into a Grafana Loki instance. The key was correlating timestamps across these sources. I wrote a small Python watchdog script that parsed logs and flagged any outbound connection not initiated by a scheduled task or a direct command from my central server.

Here's the core logic that checked for suspicious outbound attempts:

```python
# Simplified check logic
def check_connection(dest_ip, dest_port, process_cmd):
allowed_targets = ["claw-central.my.domain:443", "ntp.pool.org:123"]
allowed_processes = ["claw-scheduler", "claw-updater"]

# Check against known-good destinations and processes
is_allowed = any(target in f"{dest_ip}:{dest_port}" for target in allowed_targets)
is_allowed_process = any(proc in process_cmd for proc in allowed_processes)

if not (is_allowed or is_allowed_process):
alert(f"Suspicious outbound: {process_cmd} -> {dest_ip}:{dest_port}")
```

The false alarms were almost always one of two things:
1. **Docker bridge networking noise.** Some internal container-to-container chatter looked like new outbound flows until I filtered out the `172.17.0.0/16` subnet.
2. **Agent self-update checks.** I had one agent configured to check for updates from a mirror I'd forgotten about. It looked like a call to an "unknown" external IP until I traced it.

The real find, though, was subtle. One of my older nano-claw agents on a Raspberry Pi started making DNS queries to a new, unfamiliar domain (`dl.cloud-mirror.net`) on port 53, followed by an attempt to connect on port 80. No corresponding task or job was logged in the agent's own journal. Turns out, a compromised system package (`libcurl4`) on the Pi's underlying OS was trying to phone home. The agent itself was fine, but the host was sick. Without the agent's predictable baseline, I wouldn't have spotted that anomaly so quickly.

The takeaway for me wasn't about catching a rogue OpenClaw agent (they were all clean, thankfully), but about how their predictable behavior creates a fantastic baseline. Any deviation from that—even if it's not the agent itself—is a huge red flag. I'm now refining the alert rules to be a bit smarter about known internal networks and update patterns.

Has anyone else set up similar monitoring? I'd love to compare notes on how you're filtering out the noise.

Carlos


Carlos


   
Quote
(@homelab_sec_mike)
Eminent Member
Joined: 3 months ago
Posts: 24
 

Nice setup. That Python watchdog logic sounds familiar, I did something similar to catch a container trying to reach out to a raw IP on port 53 that wasn't my DNS. Turned out to be a misconfigured Alpine container using its own busybox resolver as a fallback.

For cutting down the noise, I found whitelisting by process *and* user helpful. My script checks if the outbound connection is from a process owned by the container's non-root user, not just the command path. Drastically reduced the false alarms for my setup.

What was the one real find? Don't leave us hanging


-- Mike


   
ReplyQuote
(@builder_bot)
Eminent Member
Joined: 3 months ago
Posts: 19
 

Good point about checking the user. I've been burned by that before too. I had an agent with a setup script that ran as root at build time and then left a cron job running as root, making weird DNS calls forever. Whitelisting by user would've caught that faster.

I'm also curious about the big find. My guess is a compromised package dependency making a call home?



   
ReplyQuote