Hey folks. Ran into this one last night while trying to deploy a new Nano Claw agent to my isolated workcell network. The artifact was signed with Sigstore's `cosign` and the verification step on the deployment host threw this:
```
Error: invalid signature: crypto/rsa: verification error
```
I'm self-hosting the entire pipeline, so my immediate thought was a mismatch between the public key used for verification and the one that actually signed the artifact. But I'd stored the key pair in my secrets manager and was sure I was pulling the right one. 🤔
Here's my basic flow and the command that failed:
1. **Signing** (on the build box, separate subnet):
```bash
cosign sign --key k8s://production/agent-signing-key agent-image:v1.2.3
```
2. **Verification** (on the deployment host inside the workcell):
```bash
cosign verify --key cosign.pub agent-image:v1.2.3
```
*This is where it barked.*
**What I've checked already:**
* The public key (`cosign.pub`) is definitely the one paired with the private key used to sign.
* The artifact digest hasn't changed (pulled by digest for verification).
* No trailing whitespace in the key file (a classic).
My current suspicion is around **key formats**. I exported the public key from the K8s secret for easier distribution to the deployment host. Could there be a PEM encoding issue? Or does `cosign verify` expect the key in a specific format when not using the keyless flow?
Has anyone else wrestling with a full zero-trust, self-hosted agent deployment hit this? I'll post my solution once I nail it down, but curious if the community has seen this `crypto/rsa: verification error` before and what the root cause was.
Lee
Isolation is freedom.
That pubkey file path looks suspect. You're signing with a key from a K8s secret, but verifying with a local `cosign.pub`. How'd you get that file onto the deployment host? If you extracted it from the secret and maybe re-saved it, you could have introduced a PEM encoding glitch.
Try verifying directly from the secret store to cut out the middleman:
```bash
cosign verify --key k8s://production/agent-signing-key agent-image:v1.2.3
```
If that works, your file is the culprit. I've seen this happen when someone `cat`s the key, copies the output, and pastes it into a new file - it can subtly mangle the format. Use `kubectl get secret -o jsonpath` and redirect to file instead.
Also, check your cosign versions. Mismatch between sign/verify versions can cause this exact rsa error, especially if you're not on the latest. Self-hosted pipelines tend to drift.
do
Great catch on the key file path. That mismatch is almost certainly the root cause. When you're pulling from a secrets manager, always verify using the same reference you used to sign. The file-based verification introduces a whole extra layer where things can break, especially with encoding.
While you're at it, double-check that the deployment host has network access to your container registry for that specific artifact. A network timeout during the signature fetch can sometimes manifest as a cryptic verification error, not a clean network failure. It's a red herring, but I've seen it happen on air-gapped segments.
Be excellent to each other.
That network timeout red herring is a real gotcha. It's the kind of thing that sends you chasing phantom key issues for hours. In my homelab, I once got a weird TLS error during verification that turned out to be my registry's certificate chain being incomplete on the isolated host. It wasn't a crypto error, but it felt just as vague.
So, if a network blip can cause this, is there a way to make cosign more explicit about the failure mode? Or is it just a side effect of how it fetches the signature artifact?
Still learning.
Oh man, that verification command looks exactly like the trap I fell into last week! I was using a `cosign.pub` file too, and it drove me nuts because I'd *swear* the key was right.
Your step 1 says `--key k8s://production/agent-signing-key` but step 2 uses a local file. Did you actually export the public key from that same K8s secret to create `cosign.pub`? Because if you generated the key pair elsewhere and just uploaded the private part to the secret, the public key in the secret might be different... or not even there. I learned that K8s secrets don't magically generate a public key for you, you have to put both parts in.
Maybe try verifying straight from the secret like `user150` said? If that fails, then at least you know it's not the file's fault.