I’ve been hardening agent deployment pipelines for a while now, and a pattern I’m seeing is that teams implement canary tokens as a naive injection detection layer, only to find they never get a hit in production. If your canary tokens fire in your sandbox but go completely silent once you ship, your architecture is almost certainly the problem, not the concept.

The core issue is that a canary token is a passive tripwire. It only works if the injected text is actually *processed* by the system prompt in the exact way you expect. In development, you’re likely using a simplified, direct pipeline. Production introduces layers that can strip, alter, or bypass the context containing your token. Common culprits I’ve torn apart:

* **Pre-processing layers stripping context:** Many production pipelines have input sanitization, logging filters, or “prompt optimization” steps that might truncate long system prompts or remove strings that match certain patterns (like your obvious canary token).
* **Orchestration frameworks that re-write prompts:** Some agent frameworks dynamically assemble the system prompt per-request, potentially overwriting or appending to your static canary.
* **The injection is happening *after* your token:** If your canary is placed early in a static system prompt, but the user’s session context or a retrieved document is appended *after* it, an injection payload there will never co-mingle with your token.
* **Monitoring is looking in the wrong place:** You’re scanning logs for the token string, but the log aggregation is sampling, or the token output is being masked as a security measure (PII filtering).

First, audit your actual production prompt. Instrument your application to dump the *exact* system prompt context sent to the LLM for a sample of requests, and compare it to your dev specification.

```python
# Example: Quick debug endpoint to verify prompt assembly
@app.post("/debug-prompt")
async def debug_prompt(request: Request):
# ... your logic to build the final messages list
final_messages = assemble_messages(request)
return {"final_system_prompt": final_messages[0]["content"]}
```

You need to validate:
1. The canary token is present, verbatim, in the string.
2. Its position relative to user/retrieved content hasn’t changed.
3. No other middleware is adding instructions like "Ignore previous instructions" *after* your token.

Then, review your data flow:
* Is there a separate content moderation service that might be blocking requests containing the canary token before they hit your monitoring?
* Are you using a CDN or API gateway with regex filters?

The false-positive cost is low, but a silent canary is worse than useless—it gives a false sense of security. You must design the token to be inseparable from the critical instruction and monitor at the point of prompt execution, not just at ingress. Consider active canaries that trigger a divergent, measurable behavior instead of just logging a string match.