Forum

Notifications
Clear all

Step-by-step: Auditing the supply chain of an AutoGen plugin from PyPI to runtime

1 Posts
1 Users
0 Reactions
12 Views
(@bob_hardcase)
Eminent Member
Joined: 2 months ago
Posts: 31
Topic starter   [#228]

Hey everyone, been lurking for a bit but wanted to jump in. I've been experimenting with a bunch of AutoGen plugins from PyPI lately, and it got me thinking: how do we *really* know what we're installing? The docs say one thing, but the code could be doing another, right?

I'm trying to build a repeatable audit process for myself, but I feel like I might be missing steps or overcomplicating things. Here's my current checklist when I find a promising plugin, say `autogen-plugin-awesome-tool`:

* First, I check PyPI for the package page and the listed dependencies. But the metadata there is just what the author submitted.
* So I pull the source distribution (.tar.gz) and the wheel, diff them if they exist. Look for hidden scripts or odd binaries.
* Then I scan the `setup.py` or `pyproject.toml` for custom build steps or post-install hooks.
* I review the actual source, especially the `__init__.py` and any module that handles external API calls. Look for obfuscated code or weird `eval()` calls.

But is that enough? Why not just rely on the hash from PyPI? I feel like a determined actor could still slip something past this. For example, what about:
* The plugin's own dependencies? Do I need to audit *their* supply chain too? That's a rabbit hole.
* Dynamic code loading from the web? I've seen `requests.get()` + `exec()` patterns in other contexts.

My main skills are in Python scripting and API integration, so maybe I'm overestimating how deep I need to go for a typical agent framework plugin. What steps do you all take before you `pip install` something that's going to have access to your LLM calls and potentially external tools? Are there any automated scanners you'd recommend, or is manual review still the only real way?



   
Quote