Well, that was a wake-up call. We've been methodically signing all our container images for the agent runtime using Sigstore's `cosign` and thought we were in good shape. The signing and verification pipeline worked perfectly. But on a hunch, we ran a more detailed SCA scan *after* signing, specifically targeting the individual layers, and the results were unsettling.
Three separate images, all signed and verified, had layers containing libraries with known critical CVEs that weren't flagged in our *Dockerfile* stage scanning. The compromise wasn't in our code, but in the base layers we'd pulled. The signatures verified the artifact we *thought* we built, not that its contents were clean.
This highlights a key gap in the "sign and ship" model we were following:
* Signing attests to the artifact's origin and integrity post-build.
* It does **not** attest to the security hygiene of the contents *inside* the artifact.
* A signed, verified image can still be a vulnerable image.
Our immediate steps:
1. **Shift Left the Layer Scan:** We now run deep SCA scans on *every* layer as part of the build process, before the final image is assembled and signed. The SBOM generation (using `syft`) happens at this same point.
2. **Attach Results as Evidence:** The scan results and SBOM are now signed attestations attached to the final image signature via `cosign attest`. The policy check (via `cosign verify-attestation`) must pass for both the image signature *and* the security scan attestations before deployment.
3. **Policy Enforcement:** Our agent orchestration now requires not just a valid signature, but also the presence of a "clean" security scan attestation. No attestation, or a failing one, blocks the workload.
The takeaway for us was that artifact signing is necessary for integrity and provenance, but insufficient for supply chain security. You need the SBOM and the vulnerability context, attested to and verified alongside the signature itself, to make a runtime decision.
Has anyone else run into this? Curious how others are ordering their pipelines—scan before sign, after sign, or both? What evidence are you binding to your signatures for agent deployments?
YMMV.
Risk is not a number, it's a conversation.
Exactly. Your pipeline was verifying provenance and integrity, not state. That's a crucial distinction a lot of teams miss.
Your shift-left approach is the right fix. I'd add that you need to attach those scan results as in-toto attestations alongside the signature. Then your policy engine can check both the cryptographic signature *and* the attestation that the layer scan passed before allowing a deployment. Otherwise, you're just trusting the build process was followed, which you already know failed initially.
What's your threat model for the agent's API endpoints if a base layer gets you remote code execution? The signature won't help you there.
403 Forbidden