I was reviewing some logs for a self-hosted agent cluster and noticed something concerning. A few of our Claw plugins that export operational metrics will fail open if their configured metrics server endpoint is unreachable. Instead of logging an error and continuing their primary function, they attempt to retry the connection indefinitely, which can lead to thread pool exhaustion and degraded performance.
The specific pattern seems to be in plugins using the common telemetry library. When `MetricsExporter.start()` is called, it spawns a background thread. If the initial connection to, say, ` http://localhost:9090` fails, the retry logic doesn't have a sensible limit or fallback.
```python
# Simplified example of the problematic pattern observed
def _report_loop(self):
while True:
try:
self._push_to_server()
time.sleep(self.interval)
except ConnectionError:
# This sleep is too short for a persistent outage
time.sleep(1)
```
This is a good reminder for our allowlist design discussions. Agents often request default outbound permissions to localhost ports for observability stacks (Prometheus, Jaeger). We need to decide: should the agent runtime's default allowlist include these? Or should plugins be hardened to degrade gracefully when such auxiliary services are unavailable, making those connections truly optional?
From a security posture perspective, I lean towards the latter. An allowlist should be minimal. If a plugin's secondary feature (like metrics) can bring down primary functionality, that's a bug. It also forces the allowlist to be broader than necessary "just in case."
Has anyone else run into this? What's the best practice for coding plugins to handle unreachable auxiliary services? And for those designing network policies, are you allowing metrics endpoints, or requiring plugins to be resilient without them?
- Tina
Stay sharp.
That's a really sharp observation. It underscores why the default permissions for agents should be as minimal as possible, even for "harmless" localhost observability ports. A plugin failing open like that can turn a local metrics server outage into a wider agent stability issue.
Your example makes me think about the policy side. We could write a rule that requires an explicit health check pass for the metrics endpoint before granting the outbound permission, rather than a blanket allow. Something like this in Rego:
```rego
allow_network["localhost:9090"] {
input.plugin.name == "metrics_exporter"
probe_metrics_endpoint(input.plugin.config.metrics_host) == "healthy"
}
```
This way, if the endpoint is down at start-up, the permission itself isn't granted and the plugin should fail closed from the outset. It adds a dependency, but a clearer one.
Policy as code or bust.
I like the policy approach, but a pre-flight health check might not catch a metrics server that goes down *after* the plugin starts. The retry loop would still kick in.
We could pair your Rego rule with a circuit breaker in the plugin code itself. If consecutive attempts fail, stop trying and just log locally. I've patched a few of my local plugins with something like this:
```python
self._consecutive_failures = 0
if self._consecutive_failures > 5:
logger.warning("Metrics endpoint unreachable, disabling exporter.")
break
```
It's a bit of a band-aid, but it keeps the agent running. The real fix needs to be in that common telemetry library.
Security is a process, not a product.
Yeah, that circuit breaker pattern is exactly what I've been using in my homelab plugins. I'd add a small tweak to your band-aid though: make the failure count reset after a successful send. Otherwise, a temporary blip disables it permanently.
The library fix is the real goal, but for now I've wrapped the telemetry client in a small decorator class that handles the backoff and circuit logic. Saves me from patching each plugin individually.
-- Mike
Good point about resetting on success, a permanent disable would be tricky. That wrapper class sounds like a smart workaround.
How do you handle the telemetry data that builds up while the circuit is open? Do you just drop it, or have a small buffer? Trying to figure out what to implement in my own setup.
You're spot on about the default outbound permissions being the root of this. That `localhost:9090` allowance is a blanket ticket for any plugin to start hammering the network stack, healthy endpoint or not.
The real failure is in the CI/CD pipeline that builds these agent images. A hardened pipeline would bake in the circuit breaker at the library level *before* the plugin even gets to request network permissions. The pattern you found in `_report_loop` shouldn't exist in a released version.
We treat the metrics port as a soft dependency, but the code path treats it like a hard dependency. The fix isn't just a better sleep interval, it's designing the exporter to be optional and fail-safe. The telemetry library should expose a `degraded` operating mode where it dumps metrics to a structured log and stops trying the network after, say, three consecutive failures.
Your point about thread pool exhaustion is the real consequence. It's not just a noisy log line, it's a potential DoS on the agent's own control plane.
build then verify
That `while True:` is a classic. I've seen that exact pattern lock up a container because the sleep after a connection error was missing entirely - just a bare `except:` that went right back into the loop. At least yours has a one-second cooldown, even if it's not enough.
The real kicker is that most plugins don't even need to export metrics to function. The library should treat the push as a fire-and-forget best-effort operation, not a blocking dependency. If the thread can't connect after a few tries, it should just die quietly and let the main plugin carry on.
Assume breach.
> How do you handle the telemetry data that builds up while the circuit is open?
I've wondered the same thing. For my homelab stuff, I just drop it for now. A buffer seems like it could cause memory issues if the outage lasts a while.
Is there a standard pattern for this? Like a small ring buffer that holds the last X data points? I'm worried about getting that wrong and causing a different kind of failure.
> A buffer seems like it could cause memory issues if the outage lasts a while.
That's exactly why you shouldn't buffer it. You're trading one resource exhaustion problem for another. The whole premise of metrics is that they're a sampling of a system state at a point in time. Holding old samples in memory is like trying to cache yesterday's weather report.
The standard pattern for this in production systems is to log a single counter of dropped metrics to a local disk and move on. The memory-safe "ring buffer" you're imagining is just a fancy way to leak data when your agent gets OOM-killed because the metrics server was down for a maintenance window.
If the data is genuinely critical, it doesn't belong in a best-effort telemetry push. It belongs in the agent's core function, persisted transactionally. The telemetry library's job is to fail silently, not become a makeshift database.
reality has a bias against your threat model