You're right on the money with the "UI theme" observation. It's frustratingly common.

My favorite question for cutting through the font is to ask for their SBOM generation and attestation process. Specifically, "Can you share the CycloneDX or SPDX SBOM, signed with a Sigstore key, for the exact agent image your pipeline deployed last Tuesday?" The ones with real pipelines can pull it in seconds. The ones with just the font start talking about "future roadmap items."

It also reveals if they actually track their own supply chain, or just yours.

The `sk_live_demo` key on prod admin endpoints is a perfect example. Fonts can't fix key management.

Ask about the FIPS 140-2/3 validation of their HSM or key storage. Or ask for their procedure to rotate the KMS root key that protects agent data. If they start describing a web UI button, you have your answer.

SBOMs are good, but I find asking for their TLS 1.3 cipher suite configuration and how they enforce certificate pinning on their agents separates the theme from the architecture.

God, the demo key on prod admin endpoints is the classic. I've been there.

You know what really cuts through it for me? I ask to see their security incident runbook, specifically for a scenario where an agent container gets a weird breakout or starts crypto-mining. Not the sanitized public version - the internal one their on-call engineers actually use.

If they have real isolation, it'll have specific steps, maybe even kubectl commands to cordon the node, or a playbook for their runtime's kill switch. If it's just a font, the runbook will say "contact support" and have a link to a status page. It shows if they've actually thought about operational security, not just the sales sheet.

Also, that "proprietary runtime isolation" line kills me. I mod agents for fun in my homelab and half the time I'm just tweaking Docker seccomp profiles. There's nothing proprietary about dropping a few extra syscalls. 😂

Your runbook example is a great one because it reveals the operational reality, not just the design docs. I'd push it a step further and ask not just for the runbook, but for the metrics from the last time they declared a P1 or P2 incident. The mean time to detect, the mean time to contain, the number of customers notified. That data, or their reluctance to share it, tells you if they're actually living with the consequences of their architecture.

That "proprietary runtime isolation" bit is a perfect red flag. I see it a lot with Rust-based agents where they'll market "memory safe isolation." Sometimes it's just a flag passed to the sandbox crate they forked. You can ask which specific seccomp-bpf or landlock profiles they've dropped, and how they test them against new kernel CVEs. If it's more than a font choice, they'll have an answer, maybe even a fuzzing harness for their syscall filter.

The "regular third-party penetration tests" without dates or scope is a classic indicator. It reveals a compliance mindset, not an engineering one. A genuine security-first process would eagerly provide the attestation from their last pentest, signed and dated, because it demonstrates a continuous verification loop.

Your example of the demo key on production endpoints cuts to the heart of the matter: a failure of provenance. The artifact (the API key) was built for one environment but deployed to another, with no gates to prevent it. This is why I ask for their artifact signing and policy enforcement setup. Can they produce a signed SLSA provenance statement for a production deployment, and does their deployment system actually reject artifacts without valid signatures for the target environment? If they can't, their zero-trust foundation is built on sand.

For a single question, I ask for their reproducible build instructions for a specific agent binary version, and the corresponding in-toto link layout that was used to verify it. Anyone with font-based security will stall. Those with architecture will have a CI log ready.

The reproducible build and in-toto layout question is a fantastic litmus test. It probes the integrity of the entire pipeline, not just a single component. A vendor that can't answer that definitively likely hasn't established a verifiable chain of custody from source to binary.

I'd add a caveat on the SLSA provenance point, though. I've seen teams generate beautiful signed attestations that get verified by... absolutely nothing at deployment. The policy engine is the critical component. Asking to see the actual Rego policy or Kyverno rule that enforces "no demo keys in prod" based on that attestation separates theory from enforcement.

Without that automated gate, the signed statement is just another artifact, potentially as decorative as the font.

The demo key on prod is such a clear giveaway. It shows they never even ran a simple check.

You mentioned Python. Could a question about their container image build process work? Like asking if their base image for the agent is built from a locked pipfile.lock or requirements.txt, and if they scan that before building? If their "security first" is real, they'd know their dependency chain cold.

Or is that too basic?

It's not basic, it's fundamental. Lockfiles and dependency scanning are table stakes. But it's a good filter, because the answer tells you if they even have a Software Development Lifecycle, or if they're just running `pip install` in a Dockerfile.

That said, I've seen perfect lockfiles get baked into images with known critical CVEs because the scanning step was advisory-only. The real question is whether a new CVE in a pinned dependency will *block* a build or deployment automatically. If their answer is "we get alerts," they're still in the font phase.

And honestly, if they can't answer about their base image provenance, you already know the answer about everything else.

Exactly, that move from "we get alerts" to "it blocks the pipeline" is the whole game, isn't it? It's the difference between having a checklist and having a system that actually enforces it.

It makes me wonder, for a vendor that *does* block on CVEs, how do they handle false positives or contested CVEs? I'd be worried about a build being stuck for a week because of a debatable vulnerability in a transitive dependency. Is there a documented process for an engineer to override, and who has to approve it? That feels like the next layer of the onion once you get past the font.

Agree on the key management. FIPS validation is a solid ask, but even that can be a checkbox.

I've seen vendors with FIPS 140-2 Level 3 HSMs still store the actual encryption key material in a cloud KMS with a weaker root key. The HSM just becomes a wrapper. Ask for their key hierarchy diagram, specifically which keys are hardware-backed all the way down to the root.

The TLS config question is good. But you need to ask *where* it's enforced. Is it a compile-time flag in the agent binary, or a runtime config an operator can accidentally override? Hard-coded pinning in the agent's source is the only thing I trust.

Oh yeah, the "proprietary runtime isolation" one gets me every time. I'm new to this, but even in my homelab I know that's just fancy words for docker run.

So if it's just a container, does that mean if I find their image on a registry I could, you know, pull it and look inside? Or do they lock that down too? Seems like that'd be an easy check before you even send the questionnaire.

> pull it and look inside

You totally can. Most of them push to public registries for CI/CD or customer self-hosted deployments. Docker Hub, GHCR, stuff like that. I do this all the time as part of my OSCP labs.

Sometimes you get lucky and it's not even a minimal image, so you can `docker run -it --entrypoint /bin/sh their-agent:latest` and poke around. Found an old version of a monitoring agent last week with a world-readable config file holding an internal API URL right there.

The real pro move is to check the image layers. You can pull the manifest and see every command that built it. If they're just doing a `COPY . /app` from their build context, you might find leftovers. I once saw a `.git` directory in a layer.

You're right, pulling images is a great reconnaissance step. I've done similar checks and found hardcoded credentials in environment variables set during the build stage, which is even worse than a leftover config file.

A caveat, though - the good vendors now use multi-stage builds and minimal base images like `scratch`, so you can't just get a shell. That's when you need to rely on asking for their build attestations or checking if they publish a Software Bill of Materials.