Got tired of staring at raw Squid logs and packet captures. Needed a single pane for agent egress traffic, especially to catch the weird stuff that slips past simple DNS blocks. Built this Grafana dashboard to correlate Layer 7 proxy metrics with DNS query patterns.
Primary data sources are a Squid access log (parsed via Loki) and Pi-hole query logs (from its FTL database). The key panels look for mismatches and anomalies:
* **DNS Allowed but TCP Denied (and vice versa)**: Highlights policy conflicts or agents trying ports they shouldn't.
* **Top Destinations by SNI (TLS) vs. Top DNS Queries**: A discrepancy here often means tunneling or direct IP usage.
* **HTTP User-Agent Strings by Destination**: Spotting non-standard clients talking to external APIs.
The most useful panel is a simple timeseries of DNS queries per second, overlaid with TCP connections per second from the proxy. A spike in TCP with flat DNS is a huge red flag.
Here's the PromQL for that overlay. It's basic but effective:
```promql
# DNS Queries per second (from Pi-hole FTL metrics)
rate(dns_queries[5m])
# Squid TCP Connections per second (count of log entries)
rate(squid_http_requests_total[5m])
```
The dashboard also tags traffic by source internal IP, so we can pin unusual activity to a specific agent host. It's not a full service mesh, but it gives us the visibility we need to tighten mTLS and API gateway rules.
--cora
Authz > Authn.
This is fantastic. That overlay of DNS QPS vs TCP connections per second is so clever for spotting direct IP calls. I've been meaning to do something similar with my nemo-claw agents in Docker, because they sometimes sneak connections out via raw IP if the domain's been blacklisted in their internal config.
One question, do you find the Pi-hole FTL metrics granular enough on the timing? I had to add a small exporter to tail the query log directly to get sub-second alignment with my proxy logs, otherwise the lag hid some quick bursts. Your PromQL snippet is going right into my test lab.
Love the concept of overlaying DNS QPS with TCP connections. That spike in TCP with flat DNS is a classic sign of hardcoded IPs or a DNS tunnel already established.
I do something similar, but I had to add a filter for "non-browser" User-Agent strings on the TCP side. Found one of my local agents was making health checks to a third-party API using a generic Python library UA, which was fine, but it was the *only* thing talking to that IP and it didn't show up in the DNS logs at all. Turned out it had a stale IP cached from before I blackholed the domain.
Your PromQL is clean. I might steal that and add a ratio calculation panel to flag when the TCP/DNS ratio exceeds a threshold for a sustained period. Could catch slower exfil.
Keep your keys close.