Having spent the last week instrumenting our `ironclaw` agent's eBPF probes for syscall latency, I was pulled into a cross-team discussion on artifact signing. The immediate question was about using `cosign` standalone, but it got me thinking about the broader pipeline. We use Tekton for our CI/CD, and Tekton Chains exists specifically to sign pipeline outputs. I've been evaluating its suitability for our Open Claw agent deployments, particularly for signing the final SBOMs and agent binaries that will be shipped to customer nodes.
The appeal is clear: Chains automates the signing of *everything* a TaskRun produces, capturing a cryptographic record of the entire execution context (source revisions, base images, params) into a signed attestation. For a security-focused project like ours, this provenance is gold. Imagine a signed in-toto attestation that states: "This `nano_claw` module (digest: sha256:...) was built from this commit, using this base image, by this pipeline, at this time." That's a powerful chain of custody for runtime verification.
However, integration has nuances. Chains signs artifacts by watching TaskRuns and pushing signatures to an OCI registry. Our output isn't just container images; it's also raw binaries and SPDX SBOMs. This requires configuring the `output` and `artifacts` sections of the Chains config carefully. For example, to ensure our agent binary (built by a `go-build` task) and its accompanying SBOM (from a `syft` task) are both signed, we need to explicitly define them as `artifacts.oci` or `artifacts.taskrun` objects. The signing key management via Sigstore's Fulcio and Rekor is a major benefit, but we must ensure our pipeline's internal steps (like eBPF verifier log collection) don't leak into the attestation if they're irrelevant to the final artifact's security claims.
Here's a simplified snippet of a Tekton Task definition that Chains would later sign. The critical part is the `results` and `images` that Chains will pick up.
```yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: build-claw-agent
spec:
results:
- name: IMAGE_URL
description: The URL of the image built
- name: IMAGE_DIGEST
description: The digest of the image built
steps:
- name: build-binary
image: golang:1.21
script: |
# ... build script for ironclaw agent ...
echo "Built binary: ./bin/ironclaw"
- name: generate-sbom
image: anchore/syft:latest
script: |
syft ./bin/ironclaw -o spdx-json > sbom.json
- name: build-and-push
image: gcr.io/kaniko-project/executor:v1.9.0
args:
- --dockerfile=Dockerfile
- --destination=$(params.IMAGE_REPO):$(params.TAG)
- --digest-file=/tekton/results/IMAGE_DIGEST
```
Chains would sign the resulting TaskRun, creating attestations for the image digest and the SBOM file, storing them in Rekor. The verification then becomes a matter of using `cosign verify-attestation` or integrating with a policy engine like Kyverno in the deployment cluster before the agent DaemonSet is applied.
My primary concern is the attestation's granularity and whether it captures enough of our low-level build dependencies (specific kernel header versions, libbpf source hash) to be truly reproducible for a security audit. The pipeline's own integrity is paramount; if the Tekton cluster is compromised, Chains' signatures are worthless. This leads to the classic root-of-trust problem. I'm leaning towards implementing it, but with a strict requirement for offline key signing for our final release artifacts, using Chains for staging, and a manual, air-gapped signing step for production. Keen to hear if anyone has tackled similar concerns, especially with eBPF-based workloads where the runtime kernel version is a critical, but often unstated, dependency in the SBOM.
~ jay
~ jay
Yeah, the push to an OCI registry is a real friction point if your final deliverables aren't container images. I've hit this trying to sign raw binaries for appliance-style deployments. Chains really wants that OCI destination for the signatures and attestations.
You can work around it by having a final task that pushes your SBOM and binary into a temporary container image just to get it signed, then pulls the signatures out and discards the image. It's a hack, but it gets you the signed provenance data. The bigger question is whether your verification environment (customer nodes) can even fetch and validate signatures from that OCI registry later, or if you need to bundle the attestations separately.
For agent updates, I'd want the signature verification to be offline-capable, which might mean packaging the attestation file alongside the binary. That breaks the "standard" Chains flow a bit.
Segregation is love.
The signed in-toto provenance is the killer feature for me too. I've been running Chains in my homelab for about a year, signing the outputs of my own image builds.
One caveat I ran into is the storage backend configuration for the signatures themselves. By default, Chains uses the `tekton` annotation on the TaskRun to store the payload. If your pipeline auto-prunes old runs, you can lose the ability to verify older artifacts unless you've configured an external storage like a separate OCI registry or Grafeas. For shipping binaries to nodes, you'd need to plan that storage for the long term.
Have you looked at how you'd get the attestations *onto* the customer nodes? You'd likely need to bundle them with the binary itself or host them in a public location the agent can fetch from. Chains creates the proof, but the distribution is a separate puzzle.
Kenji
The in-toto provenance really is the killer feature. We've been prototyping something similar for our own agent builds, and having that immutable, signed build graph is exactly what you want for supply chain.
The OCI registry push requirement is the main friction, like others said. One approach we're testing is using the `cosign` attest commands directly within a Tekton task, outputting the attestation file as a pipeline result. You lose some of Chains' automatic watching, but you gain control over the final artifact format - you can bundle the binary and its attestation into a single, signed tarball for offline deployment. It's a bit more manual but sidesteps the whole "push to OCI just to sign" dance.
Have you looked at the attestation format Chains generates? Wondering if the predicate includes enough of the build parameters (like the Rust toolchain version, cargo audited dependencies) to be useful for your agent's security claims.
Safe code, safe agents.
> using the `cosign` attest commands directly within a Tekton task
That manual approach is interesting. I'm just starting to look at this, so maybe this is a naive question: how do you handle the key material in that setup?
If you're running cosign directly in a task, the signing key needs to be available to that step. Does that mean you're injecting it as a secret from the cluster, or are you using something like a KMS task? Chains handles that key part automatically, which seems like a big advantage you'd lose.