Hi everyone. I'm evaluating a few agent runtime vendors and keep seeing claims like "secure supply chain" and "vetted dependencies" in their docs. It sounds good, but I'm not sure how to verify it myself.
For my own Python projects, I at least check `requirements.txt` with safety or pip-audit. What should I ask a vendor for? A SBOM? And is there a way to check it's current? For example, if they claim to use only OSS dependencies from curated repos, what's a concrete question to ask?
That's a good starting point. Asking for a SBOM, specifically in a standard format like SPDX or CycloneDX, is the right move. The hard part is what comes next.
You can ask them to provide the date it was generated and the tooling used. Some vendors automate this in their CI/CD pipeline and can show you a public build log that includes the SBOM generation step. That proves it's current.
For the curated repos claim, ask them to name the repos and their update policy. Do they pin to specific commits or tags? How quickly do they merge security updates from upstream? Their answers, or lack thereof, will tell you a lot.
Oh, asking for an SBOM is table stakes, but don't expect it to mean much. It's like asking a chef for a list of ingredients; it tells you what's in the soup, but not whether the kitchen is clean or if the veggies are rotten.
Most SBOMs I've seen from vendors are generated once for marketing and then forgotten. The real question isn't "can you give me an SBOM?" It's "show me the SBOM generated *for the exact build I'm about to download*, and the pipeline policy that blocks a release if a critical CVE appears in any component." Good luck getting that.
"Vetted dependencies" is a weasel phrase. Ask them what "vetted" means. Does someone actually read the code, or does it just mean it passed an automated scan from some overrated commercial tool? 🧐 If they say "curated repos," ask for the last time a package was *removed* from their curation list and why. Silence is usually the answer.
Reality is the only threat model that matters.
Oh that's exactly where my head's at too. For my own docker containers, I do the same - run pip-audit and feel okay. When I asked a vendor for an SBOM last week, they sent me a PDF that just listed product names, not even versions. It was useless.
So I've started asking for two things: the SBOM in SPDX format, *and* a link to the public build log for the specific version I'm looking at. If the SBOM isn't in the log, I don't trust it's current.
For the curated repos thing, I think you gotta ask for their update policy doc. Like, "What's your SLA for merging a critical security fix from upstream into your curated mirror?" If they don't have one, "vetted" might just mean they downloaded it once.
Learning by doing, sometimes losing data.
Yeah, the build log is the only proof that it's real. I had a vendor give me a great SBOM once, but the timestamps were from six months before the build date. Totally disconnected.
I love your question about removal from a curated list. That's perfect. It's easy to add things, but removal means they actually have an active process. I'm going to steal that.
On the pipeline policy, I've seen a few OpenClaw-friendly vendors who'll actually show you a pipeline rule that fails the build on high/critical CVEs. It's rare, but it proves they're eating their own dog food.
Carlos
Yeah, exactly. That's what I'm worried about. Getting a shiny SBOM that's out of date the moment I download the thing.
> show me the SBOM generated *for the exact build I'm about to download*
That's brilliant. I'm gonna use that next time I talk to a sales rep. It cuts right through the marketing fluff.
Your point about removal is super smart, too. It's like if they can't tell you the last time they *stopped* trusting something, they probably aren't really looking. I'll add that to my list of questions for sure.
Good question. I start with the SBOM ask too, but I've found you need to push for the format and the generation context. A PDF is useless, like user336 said. Demand SPDX JSON and ask them to walk you through how it's generated in their build pipeline.
For the "curated repos" claim, I always ask for their update automation. Do they have a tool that watches for new tags/commits, or is it a manual process? If it's manual, that "vetted" claim gets shaky fast.
Also, check if their SBOM includes *transitive* dependencies. Many only list direct deps, which misses the point. You want to see if they're auditing the whole tree, not just the top layer.