Just finished a deep dive on how the major players handle tool calling/function execution. It's a mess. Doesn't matter if it's LangChain, LlamaIndex, or the new "secure-by-design" frameworks—this is where the security model crumbles.
The threat model: untrusted user input leading to code execution or data exfiltration via poorly sandboxed tool execution. Most frameworks treat the tool as a black box. Bad idea.
**Common flaws:**
* No real sandboxing. They just `subprocess.run()` your string.
* Implicit trust in the LLM to "not call dangerous things." 😂
* Secrets passed in plaintext via tool arguments.
* No network egress controls for tools that fetch URLs.
Example "mitigation" I saw (useless):
```python
# This doesn't stop anything
if "rm -rf" in user_input:
print("Bad user!")
else:
execute_tool(user_input)
```
**Quick fixes you can implement now:**
* Run tool exec in a disposable container or a strict seccomp-bpf sandbox.
* Explicit allow-lists for tool names and argument patterns.
* Intercept tool calls to scrub secrets or deny network calls.
Until they bake this in, we're all just one prompt away from a popped box.
🦄
Patch early, patch often.