Alright, let's get this out of the way. The prevailing wisdom is that you must log every agent heartbeat, tool call, and token count to your SIEM. So we did it. We've been piping our entire agent runtime event stream into Datadog for a month. The verdict? The visibility is genuinely useful. The alerting, however, is a clunky mess that makes me question the whole "log everything, figure it out later" mantra.
The good part is obvious. You can finally see patterns. Spotting an agent stuck in a loop because of a recursive tool call chain is trivial when you graph it. Watching token consumption spike on certain user queries gives you a real performance baseline. It's the difference between guessing why an agent timed out and having a concrete event log showing it was trying and failing to parse a malformed JSON payload for 90 seconds.
The problem starts when you try to build actionable alerts. Datadog's log-based alerting, and I suspect this is true for Splunk or Elastic too, is built for traditional infra. Trying to craft a rule that says "alert me if an agent uses tool X more than 5 times in a minute, but only if the conversation context contains keywords Y and Z, and ignore it if it's user ID from the QA team" becomes a configuration nightmare. You end up with either deafening noise from overly broad rules or silent failures because your condition logic is too brittle to capture the actual anomalous behavior.
We're creating a new class of software here—agents that reason and act. Their "abnormal" behavior isn't a failed login or a 500 error; it's a logical or operational drift within their allowed autonomy. Our SIEMs aren't built for that nuance. So we get great dashboards for post-mortems and terrible tools for real-time intervention. Maybe instead of forcing the square peg of agent logic into the round hole of SIEM alerting, we need to think about building the detection logic closer to the agent runtime itself and just ship the *alerts* to the SIEM.
Jack
Security theater is still theater.
I'm not surprised. That mantra has always been cargo cult logic. You've hit the exact problem: you're trying to use a monitoring system designed for servers and network packets to understand semantic, stateful behavior.
The alerting is clunky because it is. It's a square peg in a round hole. You're now spending more time building brittle log queries to approximate agent intent than you would just instrumenting a few key business logic events. Log everything for debugging, fine. But alerting needs to be on *meaning*, not raw event volume.
You'll drive yourself insane trying to make Datadog understand conversation context. That's an application-level concern. Build those thresholds and heuristics directly into your agent's supervision layer, then emit a simple, clear event when something's actually wrong. Let the SIEM catch that.
Security theater is still theater.
That's really helpful to hear, thanks for sharing your experience. The visibility benefits make sense, but that's a great warning about the alerting trap.
I'm just starting to set up basic logging and was already worried about alert noise. Your example about trying to catch a specific tool use with conversation context is exactly the kind of thing I'd waste a week trying to build.
If the alerting is that clunky, maybe the better move is to only use the SIEM for exploratory debugging and dashboards, like you said. Let the agent handle its own meaningful alerts internally and just log a simple "anomaly detected" event for the SIEM to catch. Is that the direction you're leaning now, using it more for forensics than real-time alerts?