That "why" question hits home. I've definitely been guilty of logging everything just because I could.
Your three fields make sense for cutting through the noise. But as someone just starting out, I'm already worried about that step from "non-zero exit" to "something to alert on." How do you even begin to write a rule for "persistent" failures without creating more complexity? Is it just a simple count over five minutes, or does it depend on the agent type? Feels like the first big tuning hurdle after getting the fields right.
You've got the right three. Your bit about a persistent non-zero exit is key, and it's where most new setups fall over.
They'll implement the basic alert without thinking about agent execution frequency. A cron job that fails every minute for an hour shouldn't blow up the pager 60 times. The rule logic needs to coalesce those failures into a single "agent X is unhealthy" alert, resetting only after a clean success.
For the identifier, you need the logical check name *and* the unique agent instance ID. The check name ("nginx_config_audit") tells you what's broken. The instance ID ties it to a specific host or deployment so you can actually find it. Without both, you're stuck.
Pin your deps or go home.
You're absolutely right to focus on the 'why' first, and those three fields are the bare, unvarnished minimum. But I'll add a twist from the adversarial side: your identifier field is a juicy target.
If I'm trying to blind your monitoring, I'm not going after your SIEM. I'm poisoning the *agent identifier*. A simple data-only prompt injection against a poorly-sanitized agent that pulls its check name from a config file can turn "webhook_check_443" into "webhook_check_443_exited_cleanly". Your logs now show a beautiful stream of successes for a check that's actually dead.
The rule isn't to add more fields. It's to sign the identifier, or at least hash it with a runtime nonce before emission. Otherwise, you've built a detection pipeline that trusts the very thing it's supposed to monitor. Seen it happen with AI agents, but the principle's the same for any automated runner.
pwn responsibly