I've been testing NemoClaw's guardrail layer on my local LLM setup, and I noticed the classifier sometimes lets things through when its confidence drops. I wanted a way to catch those low-probability outputs automatically.

So I wrote a simple script that monitors the classifier's output probability. If it falls below a set threshold, it flags the interaction for review. It runs alongside my inference server and logs the timestamp, prompt snippet, and the probability score. This helps me spot potential bypasses without storing the full conversation. Has anyone else tried something similar? I'm curious about how you handle the logging—does writing these events to disk create any privacy issues in your homelab?