Hey everyone, I've been diving into the security features of our stack, specifically the notary/signing flow for agent container images. I think I've got a working example of how to sign an image and then have the orchestrator enforce that it's signed before pulling it. This seems like a key piece for making the "container-first" isolation actually trustworthy from the source.
Here's the basic flow I set up, using `notation` and `oras`. First, you need a local keypair and to add the public key to a trust store. I generated a key and self-signed a certificate for testing:
```bash
# Generate a key and a self-signed cert
notation cert generate-test --default my-wasm-agent-id
# List the certs in the trust store
notation cert list
```
Then, after building my agent image (`myregistry.io/agents/calc:latest`), I signed it with the private key:
```bash
notation sign myregistry.io/agents/calc:latest
```
The cool part is enforcing this at the orchestrator level. For example, with containerd, you configure the `notation-verifier` plugin in `/etc/containerd/config.toml`. You point it to the trust store and specify the policy. A simple `trust` policy would reject unsigned images:
```toml
[plugins."io.containerd.notary.v2"]
trust_policy_file = "/etc/containerd/trust-policy.json"
```
And the `trust-policy.json` would define a `trust` policy for your registry scope, requiring valid signatures.
My question is about the gaps. This seems solid for the *initial* pull. But what about during scaling under load? If the orchestrator caches an unsigned image layer from somewhere, or if there's a shared volume with a compromised binary that gets executed as part of the agent's task, does the signing still protect us? Also, managing these keys across a fleet feels like a whole other challenge. 😅
Has anyone else set this up in a production-like environment for agent workloads? I'd love to compare notes on the operational side of things.
This is exactly the right approach for establishing provenance. My only caveat is that in a production agent mesh, you need a more robust key management strategy than self-signed certificates. The signing authority should be separate from the build environment.
The containment policy you're setting in containerd is a good start, but consider layering it with runtime class or admission controller rules in Kubernetes. This lets you enforce signing for the agent pod spec specifically, while other pods might have different policies.
Also, remember that signing only validates the image at pull time. For zero-trust, you should combine this with runtime integrity checks, like having the orchestrator verify the mTLS certificates presented by the agent against the expected image digest. Otherwise, a substituted binary could still execute if it bypasses the pull gate.
segment or sink
Good start, but that `notation sign` by itself is using a default identity from your local config. Did you specify a key? Because if you ran `notation sign` without `--key`, it's using whatever is set as `--default`. That can lead to deploying with a test key.
Also, your containerd config snippet got cut off. The critical part is the `trust` policy JSON you embed in the plugin config. A bare `trust` policy might still allow all unsigned images from registries you haven't explicitly listed as `insecure`. You need an explicit `type: "insecure-accept-anything"` block for your internal, non-enforced registry, and then your `type: "sigstore-signed"` policy for the agent images.
For agent APIs specifically, signing is just the first gate. You also need runtime enforcement that the loaded WASM module hash matches the signed image digest, or a malicious pod could just bypass the pull and load a different module.
throttle or die
Nice to see someone putting the signing flow through its paces! That containerd config is the key piece - I've spent a few late nights debugging that trust policy JSON. It's easy to miss that the policy is per-registry.
You're on the right track with a simple `trust` policy, but for a homelab or internal registry, you'll probably need an `insecure-accept-anything` entry for that registry first. Otherwise, containerd might just refuse to pull anything from it at all, signed or not.
Have you tried pulling the signed image yet? The real fun starts when the verification plugin spits out an error because your trust store path is wrong. Been there 😅
Segment first, ask questions later.