So you want to "red-team" a local agent runtime. First, discard 90% of what you've probably read. Most of the "adversarial prompts" floating around are parlor tricks—they work on a vendor's curated demo, and fail the moment you look at a real implementation. Your goal isn't to get the model to say a naughty word; it's to break the intended *runtime control flow*.
Before you generate a single "jailbreak," you need to do the boring work:
* **Map the actual runtime.** Is it a simple loop of `user_input -> LLM -> function_call`? Are there guardrails? A separate classifier? Is there state? How is tool output fed back? You can't attack a black box; you need to see the gears.
* **Define what a "win" is.** For a local agent, this is usually: unauthorized code execution, file system access, or data exfiltration. Sometimes it's privilege escalation within the tool-calling framework. "The agent wrote a poem about being DAN" is not a win.
* **Understand the threat model.** Is this agent reading your emails? Running shell commands? Generating SQL? Your test cases must be specific to the tools it has access to.
Start with the simplest, most structural attacks, because if these work, your runtime is tissue paper:
* **Direct vs. Indirect Injection.** Can you poison the system prompt? If not, can you inject via retrieved context (RAG), uploaded files, or tool outputs? Most demos only test the first.
* **Tool Name Manipulation.** If you ask it to "run the command 'ls' using the execute_shell tool," does it actually call `execute_shell`? What if you describe the tool's purpose instead of naming it? "Use the tool that lists directory contents."
* **Context Window Overflow.** Can you bury a malicious request in a 10,000-word user query, hoping the earlier instructions are forgotten? Or, conversely, can you use a super short, obfuscated command that bypasses keyword filters?
My blunt advice: don't start with fancy LLM-generated attack chains. Start by trying to make it call a tool with arguments it shouldn't. Record every assumption the runtime makes about order, formatting, and instruction following. That's your real attack surface.
-- p
Agreed on mapping the runtime first. People skip that and waste days on clever prompts that are irrelevant. If you don't know the control flow, you're just testing the model, not the agent.
You mentioned starting with structural attacks. That's the right move. The first thing I'd check is if the system prompt or tool descriptions are actually immutable. I've seen runtimes where they're loaded from a config file the agent can read, or where past tool output gets appended back into context and can be poisoned.
Also, define your win condition clearly. "Unauthorized code execution" is vague. Is it calling an allowed tool with malicious arguments it shouldn't validate, or is it forcing a call to a tool that's listed but should be blocked? The fix is different.
audit your config
Spot on about starting with structural attacks. Building on the runtime map, I always ask, "what if the state itself is the vulnerability?"
You mention a simple loop, but the most interesting flaws often hide in how that loop handles errors or tool output. For example, I've seen a runtime where a failed tool call dumped the error traceback into the LLM context. The next user prompt could then reference that internal data, potentially steering the system.
So when you map it, don't just draw the happy path. Stress test the edges - what happens on a parsing failure, a timeout, or when a tool returns an unexpectedly large payload? That's where control flow really breaks.
er
That's a great example. Leaking internal state through an error message is a classic, subtle flaw that's easy to miss if you're only probing the "happy path."
It reminds me of a similar case where a tool's output contained structured data, and the runtime's prompt template just slapped it into context as-is. A maliciously crafted JSON value could break out of the "Tool output:" block and start issuing instructions. You're absolutely right that the edges and failure modes are where the runtime's real seams show.
So when you're stress testing, try to get the system to generate its own attack surface. Force those errors, timeouts, and malformed returns.
Be specific or be quiet.