> catch every exception and log it as a warning to keep the app running
That's the classic trap, and it kills visibility. For logging dashboards, a sea of "ERROR" logs from caught-and-continued exceptions drowns out the real, actionable failures.
My rule is: treat logs as your last line of defense for monitoring. If you swallow the exception after logging, you're just creating noise. Let it crash, and let your alerting catch the *process* failure. That's a much cleaner signal.
I'd add a caveat though: the "edge" you validate at should be as early as possible, but also as specific as possible. Don't just catch `Exception`. Catch `json.JSONDecodeError` or your own `ValidationError`, log it with the raw input for forensics, *then* crash. That way your log still has the diagnostic info, but the system properly fails.
--Em
I'm with you on the specific catches and logging for forensics. That's the only way to get a useful trace.
But I have to push back a little on "let your alerting catch the process failure." That works for a service you control, but a lot of these AI tool-calling patterns are embedded inside a larger, stateful session, like a chatbot. The orchestrator crashing might just kill a single user's thread, leaving the main app running. That's a softer failure, but it can still be an availability problem or a weird user experience. The alerting often misses those degraded states entirely.
The real trouble starts when the malformed result isn't caught at parsing. If it's valid JSON but semantically poisoned, it slips through and becomes part of the model's context. That's where fuzzing the content, not just the syntax, gets ugly.
That's a really smart question about recovery, I've been wondering the same thing. I think a lot of it depends on the orchestrator's design philosophy, like some folks mentioned above about "dying loudly." If it's built with that in mind, it probably just stops completely to preserve the boundary.
But I've seen some configs where you can set a fallback behavior in the policy, like "on_malformed_result: reject" vs. "retry" or maybe even "use_default". It's scary to think about a default being used there, though. What if the default itself is wrong or unsafe?
Your last point about isolating the fuzzer hits home. I'm still figuring out my own lab setup. Are people usually running this in a separate container or VM? Or do you just point it at a test instance of your whole stack?