I've been conducting a deep-dive analysis of credential exfiltration pathways in multi-agent workflows, specifically focusing on the persistence of sensitive strings in tool outputs, intermediate LLM context, and—most insidiously—in plaintext logs. While network-level egress filtering is a baseline, the real challenge is detecting the *presence* of a leak within the complex dataflow of an agent runtime.
The concept of credential canaries, or canary tokens, is highly relevant here. In a kernel context, we might think of `dmesg` canaries for detecting unwanted root activity. For agents, we need to embed decoy credentials with high entropy and unique properties into the environment, then monitor for their appearance anywhere outside the intended, memory-isolated sandbox.
My current prototype involves a two-layer system:
1. **Canary Injection:** A kernel module (or a privileged sidecar in a containerized setup) that places canary files, environment variables, and even fake process arguments into the agent's namespaced environment. These are generated per-session and logged in a secure, out-of-band database.
2. **Distributed Sniffing:** A series of eBPF probes attached at critical syscall exit points (`write`, `sendto`, `sendmsg`) and at the interface between the agent runtime and its logging library (e.g., hooking into `fprintf` to the log file descriptor). The probes perform simple pattern matching against the known canary set.
Here's a simplified eBPF sketch for the syscall monitor component:
```c
// Pseudo-code for eBPF kprobe on sys_write
SEC("kprobe/sys_write")
int trace_write(struct pt_regs *ctx) {
int fd = PT_REGS_PARM1(ctx);
const char *buf = (const char *)PT_REGS_PARM2(ctx);
// ... fetch current task credentials, check if it's in an agent cgroup
if (is_agent_task()) {
// Compare buffer slice against a per-cpu map of active canaries
struct canary *c = bpf_map_lookup_elem(&canaries, &canary_id);
if (c && memmem(buf, count, c->token, c->length)) {
bpf_ringbuf_output(&alerts, &event, sizeof(event), 0);
}
}
return 0;
}
```
The architectural challenges are significant:
* Canary tokens must be indistinguishable from real secrets to the agent's own processing logic, which may involve string transformations.
* The monitoring system must have visibility into all possible output channels: stdout/stderr, log files, network sockets opened by the agent, and even shared memory or IPC if used.
* There's a risk of the LLM itself "learning" the canary pattern and avoiding its output, though this is less likely with per-session unique tokens.
I'm particularly interested in whether anyone has tackled the namespace/cgroup isolation aspect—ensuring the canary sniffing logic resides outside the agent's container or user namespace but can still inspect its syscall arguments. Have you implemented a similar canary system? What was your detection vector (logs, network, both)? And how did you handle the performance overhead of constant syscall argument inspection?
Syscalls don't lie.
Your focus on the dataflow within the agent runtime is the critical angle. A canary token in a log is only useful if you can cryptographically verify its origin and ensure it wasn't placed there by the monitoring system itself.
Your two-layer system's security depends entirely on the integrity of the secure, out-of-band database. How are you managing the keys for that? If the canary generation and logging system uses a single compromised credential, the entire scheme is transparent to the attacker.
Also, consider embedding canaries with a specific structure, like a PGP-signed token containing the session ID and a timestamp. This allows any sniffing probe to validate the token's authenticity on the spot, without a centralized query, reducing the risk of the alert channel being blocked. The signature key must, of course, reside in an HSM.
Don't roll your own crypto. Unless you have a spec.
The eBPF angle for distributed sniffing is the right call. Kernel modules are a pain to maintain across kernel versions. I've done something similar with a set of eBPF programs attached to `tracepoint/syscalls/sys_enter_write` and `tracepoint/syscalls/sys_enter_read` to catch canaries hitting stdout or being read from certain file descriptors.
The biggest operational hurdle is the canary database itself. You need that out-of-band logging to be rock solid, as user62 points out. I use a separate, air-gapped logging cluster that only accepts connections from the canary injector host on a non-routable interface. The sniffers generate events locally but only ship alerts, not the tokens, to a separate SIEM channel.
Have you considered the noise from false positives? If your canary token generation isn't perfectly unique, you can get alerts from your own monitoring tools or even from developers who accidentally copy a fake credential into a test script. It becomes a trust exercise in your own alerting pipeline.
Log everything, alert on anomalies.
That's a solid approach, focusing on the environment itself. The kernel module for injection is key for bypassing user-space monitoring that a sophisticated agent might employ.
One nuance with the namespace injection: if your agents can spawn sub-processes or lateralize, you need to ensure the canaries propagate. A static injection at agent launch might miss a child process with a different, cleaner environment. Your out-of-band logging will need to track the entire process tree, not just the initial PID.
Also, placing canaries in process arguments is clever, but be mindful of process listing commands that might truncate them. You'll want to ensure your eBPF probes are catching the full `execve` arguments, not just what `ps` would show.
I like the two-layer approach, especially the kernel module for injection. It gets around a lot of user-space visibility problems.
How are you handling the entropy source for generating the canary tokens themselves? If they're predictable or derived from something in the agent's accessible environment, an attacker could fingerprint and avoid them.
Also, for the eBPF probes, are you filtering at the syscall level or deeper in the stack? I'm thinking about agents that might use something like `sendmsg` with ancillary data to bypass simple `write` checks.
Due diligence.
That's a fantastic point about child processes. I was so focused on the initial agent container that I hadn't fully considered lateral movement within the PID namespace. My current docker-compose setup spawns a few supervisor processes, and you're right, they'd have a clean slate.
I'm relying on an environment variable injection via the kernel module at container launch, but if the main agent spawns a child with something like `os.system()` or `subprocess.Popen(..., env=clean_env)`, that canary is gone. I'd need to hook deeper than just the initial `execve` for the container entry point, wouldn't I? Maybe tying the canary to the namespace itself, so any new process in that namespace inherits it, regardless of the parent's `env`? I'm still getting my head around the namespace semantics.
Also, the `ps` truncation warning is a lifesaver. I just tested with a dummy long argument, and sure enough, it gets chopped. My eBPF probe is grabbing the full buffer from the syscall, but if I ever had to manually check from a host shell, I'd be blind. Definitely need to document that for anyone else trying to debug the system.
- Liam
Injecting into the namespace directly is the right idea. Set a canary in the `init` process's environment. Every forked child in that PID namespace inherits a copy of its parent's environment, but a child can explicitly clear it. You can't force inheritance.
Hook `execve` in the kernel. When a new binary loads, your module can write your canary token into the new process's `envp` array before it starts, regardless of the caller's arguments. This catches `Popen` with a clean env.
On the truncation: that's why your eBPF probe is essential. Kernel sees the full argument vector. `ps` and most user-space tools are useless for this. Rely on your own tooling.
Static injection at container launch is basically security theater for anything but the most naive agent setups. The whole point of these canaries is to detect exfiltration, and exfiltration rarely happens from the pristine parent process.
Your point about `execve` arguments is valid, but filtering there is still a high-friction approach. You're now in the business of maintaining a kernel module that hooks a major syscall across multiple kernel versions. That's a huge ops burden, and one mistake bricks your nodes.
What's the actual detection rate of a canary appearing in a truncated `ps` output versus a full `execve` capture? If an attacker is dumping process lists, they've already lost. You're optimizing for a scenario that probably doesn't matter.
The bigger hole is encrypted exfil. If the token hits a `write` to a TLS socket, your eBPF probe sees ciphertext. Game over.