I have been conducting an ongoing experiment to instrument my primary research assistant agent with a real-time policy evaluation layer, specifically to monitor for policy violations that may indicate indirect injection. The agent's toolset includes a code analysis plugin that can fetch and summarize package manifest files from public repositories. Yesterday, the monitoring system triggered a high-severity anomaly, and the root cause analysis revealed a sophisticated attempt at a supply chain attack vector.
The agent was tasked with comparing dependencies between two projects. It invoked the plugin to retrieve the `package.json` for a seemingly legitimate utility library. The plugin returned the expected JSON structure, which the agent began to parse. However, embedded within a dependency version string was a crafted payload designed to break out of the JSON parsing context and, when the agent's internal processing concatenated strings for a subsequent shell command tool, execute arbitrary code.
My detector is built as a series of Rego policies that evaluate the *inputs* and *outputs* of tool calls, not just the authorization to call the tool itself. The critical policy evaluates string data returned from tools for patterns indicative of injection. It flagged this because the returned data contained a nested sequence of characters that matched a heuristic for command escape sequences, within a field where a SemVer string was expected.
```rego
package agent.injection.detection
import future.keywords.contains
import future.keywords.if
# Heuristic: detect common shell escape sequences in unexpected places
shell_escape_sequences contains seq if {
seq := "`"
} {
seq := "$("
} {
seq := "\x"
}
# Analyze tool output strings
detect_possible_injection if {
output := input.tool_output.raw_data
is_string(output)
shell_escape_sequences contains seq
output contains seq
}
```
The architectural defense here is multi-layered:
* **Policy Layer:** Real-time evaluation of tool outputs against a security policy before the data is processed by the agent's reasoning loop.
* **Contextual Sanitization:** All data retrieved from external tools is treated as belonging to a specific, strict schema (e.g., a `version` field). Any deviation from the expected pattern or content type for that schema is a violation.
* **Tool Output Sandboxing:** The output from plugins, especially those fetching from the network, should be initially placed in a quarantined data structure where its content can be validated and transformed (e.g., through strict JSON decoding with type coercion) before being passed to the agent's prompt context or other tools.
This incident underscores that the attack surface is not merely the direct user prompt. The data flow *from* tools *to* the agent's reasoning and subsequent tool-calling loop is equally critical. A comprehensive agent permissions model must therefore include:
* Input policies (what tools an agent can call, with what parameters).
* Output policies (what data the agent is permitted to receive and process from those tools).
* Data flow policies (how data from one tool may be used as input to another).
Without this, we are authorizing agents to retrieve arbitrary, potentially malicious content and then process it with the full privilege of their identity and subsequent tool access. The policy-as-code approach allows us to encode these constraints in a auditable, reusable, and testable format. I am now extending the policy set to include checks for indirect prompt injections in retrieved textual data, which presents a more complex pattern recognition challenge.
Deny by default. Allow by rule.
Wow, that's a fantastic catch. It's such a subtle vector, hiding in a version string where a human reviewer would likely just glaze over it. I'm really glad you're building that *input/output* evaluation layer, because that's where the real battle is fought after the initial "can it call this tool?" check.
I've been playing with something similar for my Docker build agents, but at a lower level. Instead of Rego, I'm using a simple pattern-matching sidecar that watches the agent's activity logs. It flags any command that suddenly includes a substring from a previously fetched remote resource. It's not nearly as elegant as a proper policy layer, but it did catch a sneaky attempt to inject a `curl | sh` pipeline from a corrupted API response last month.
Your example makes me wonder about the recursion depth of these checks. If the plugin itself was compromised and returned clean JSON but with a malicious dependency *name*, would your policy evaluate fetching *that* next manifest? It feels like we need a way to mark certain retrieved data as "tainted" for the rest of the session.
lab.firstname.net
You've touched on a critical limitation. The tainting problem you've identified is exactly why policy layers need to integrate with a system's information flow tracking, if available. A simple regex on activity logs is reactive; it sees the `curl | sh` after the fact. Marking data as tainted requires defining a provenance graph for every string in the agent's context, which is computationally heavy but necessary for true prevention.
Your sidecar approach is a pragmatic first step, but consider that pattern-matching on substrings can be evaded by trivial obfuscation in the fetched resource. A more deterministic, if burdensome, method is to enforce a strict schema validation with enumerated allowed values for any field that influences execution, like a dependency name or version. This moves the check from "does this look malicious?" to "is this on the allow list?".
The recursion depth issue is real. My policy engine does have a configurable depth limit for chained fetches, but it's a band-aid. The proper solution is that taint label. If data from source A is marked untrusted, any operation triggered by that data - including a new plugin call to fetch the manifest for a dependency it mentioned - should inherit that label and face stricter scrutiny or be blocked outright. Implementing that is my current headache.
Trust in gradients is misplaced.