Hey everyone, been lurking for a bit but wanted to jump in. I've been experimenting with a bunch of AutoGen plugins from PyPI lately, and it got me thinking: how do we *really* know what we're installing? The docs say one thing, but the code could be doing another, right?
I'm trying to build a repeatable audit process for myself, but I feel like I might be missing steps or overcomplicating things. Here's my current checklist when I find a promising plugin, say `autogen-plugin-awesome-tool`:
* First, I check PyPI for the package page and the listed dependencies. But the metadata there is just what the author submitted.
* So I pull the source distribution (.tar.gz) and the wheel, diff them if they exist. Look for hidden scripts or odd binaries.
* Then I scan the `setup.py` or `pyproject.toml` for custom build steps or post-install hooks.
* I review the actual source, especially the `__init__.py` and any module that handles external API calls. Look for obfuscated code or weird `eval()` calls.
But is that enough? Why not just rely on the hash from PyPI? I feel like a determined actor could still slip something past this. For example, what about:
* The plugin's own dependencies? Do I need to audit *their* supply chain too? That's a rabbit hole.
* Dynamic code loading from the web? I've seen `requests.get()` + `exec()` patterns in other contexts.
My main skills are in Python scripting and API integration, so maybe I'm overestimating how deep I need to go for a typical agent framework plugin. What steps do you all take before you `pip install` something that's going to have access to your LLM calls and potentially external tools? Are there any automated scanners you'd recommend, or is manual review still the only real way?