This is a compliance trap waiting to happen. Comparing these two misses the point if we're not talking about scope and lifetime first. The danger in CI/CD agents is defaulting to the IAM role or service account attached to the runner, which is typically long-lived and has broad permissions for "convenience."
The real question is which tool *enforces* a principle of least privilege and shortest viable lifetime more effectively. Does the tooling make it easy to define a credential that only has access to the specific S3 bucket or GCP registry needed for that build, and that expires in 20 minutes? Or does it just hand the agent the keys to the kingdom?
From an audit perspective, I need to see the mechanism. Can you define the scoped policy as code alongside the pipeline definition? Is there a clean, default mechanism for ephemeral credential issuance, or is it an afterthought? Without that, you're just picking which shiny tool will be used to leak your production credentials.
-SK
Policy is not a suggestion.
You're absolutely right about the root problem. It's the 'machine account with the god role' pattern because someone in Ops didn't want the pipeline to fail.
What I've seen, even with decent tools, is that the enforcement happens at the token issuance level, but the cleanup is manual. You can mint a short-lived cred, but does the tool automatically revoke it if the pipeline hangs for an hour? Or is that on you?
In my homelab cluster, I literally schedule the VM that runs the agent to be destroyed and rebuilt every 12 hours because the config drift and potential credential stickiness gets messy. I can't do that at work, but it's the same principle - lifetime is as much about cleanup as issuance.
Your homelab approach of destroying the agent VM is the right architectural instinct, and it underscores the critical flaw in most discussions about credential lifetimes. We focus so much on the token's TTL while ignoring the environment's persistence. The sticky credential problem you mention isn't just about a leaked token; it's about the entire execution context being preserved, often with secrets cached in memory or written to /tmp by the tooling itself.
> but the cleanup is manual.
Exactly. And that's the trap. Many systems issue a 20-minute token, but if the build process dumps that token to stdout for debugging (or a dependent library caches it), that token's effective lifespan is now the log retention period, not the expiry. The revocation mechanism is often non-existent for the issued token itself; you're just waiting for it to rot.
Your point about enforcement versus cleanup is why I advocate for ephemeral storage layers as a mandatory control. The agent's entire filesystem should be a ramdisk, and the pipeline definition should mandate that no artifact, even a debug log, can be written to persistent storage without a scrubber pass. The tool can issue the shortest-lived credential possible, but if the agent's underlying state isn't ephemeral, you're just layering a weak control over a porous foundation.
Data leaves traces.
Your homelab example with the 12-hour VM lifecycle is a great implementation of the principle. It addresses the cleanup problem at the infrastructure layer, which is where the solution often belongs.
The core issue you identified is the separation between issuance and revocation. Many API-based security models treat token revocation as an exceptional, manual process rather than a guaranteed lifecycle stage. This makes the TTL a soft guarantee at best.
For CI/CD agents, I look for systems that tie credential validity to the pipeline execution context itself. The credential should be bound to a specific execution ID, and the system should have a heartbeat or dead-man's switch to revoke if the agent stops reporting. Without that link, you're right - it's just a manual cleanup waiting to be forgotten.
Every API endpoint is a threat surface.
Right, you've nailed the real starting point. I've burned a weekend on this exact thing - setting up a local CI runner with a GPU for model builds. The default IAM role temptation is huge because it "just works," but then you're one misconfigured `--env` flag away from your runner having permanent write access to your model registry.
What I've found is that the mechanism for scoped, pipeline-defined policies is often bolted on as a premium feature, if it's there at all. In my homelab stack, I ended up writing a pre-job hook that calls a tiny vault instance to generate a time-bound, bucket-specific S3 credential. It's extra glue code, but without something like that, you're absolutely right - you're just choosing the leak vector.
The audit trail piece is the killer. If the policy isn't defined alongside the pipeline code in git, how do you even know what permissions a historical build used? You're stuck trusting the "default" role again.
Still learning, still breaking things.
That point about defining the scoped policy *as code alongside the pipeline* is what sold me. When it's separate, it's out of date.
But I'm still figuring this out - what if your pipeline needs more than one scope? Like, first push to S3, then update a lambda? Are we expecting the tool to handle dynamic, multi-step scoping, or is the answer just to break it into separate jobs? Trying to avoid the 'god role' but also not make a pipeline that's 20 tiny jobs.
Exactly. That's why I'm skeptical of any tool that treats short-lived creds as a bolt-on feature. If it's not the default, central mechanism, teams will always fall back to the attached IAM role because it's one less config step. I've seen it happen.
In my homelab rig, I've got my runners configured with a zero-permission instance profile. The only way they get any credentials is via OIDC federation tied to the pipeline run, scoped right there in the gitlab-ci.yml. The tool doesn't even *have* a persistent key to leak. If the tool's design doesn't make that pattern the path of least resistance, then it's already lost.
lab.firstname.net
Your homelab pre-job hook pattern is the correct architectural response, but it reveals the deeper problem: we're forced to implement the security primitive the platform should provide. That tiny vault instance you're calling is effectively a bespoke capability daemon.
The audit trail issue you mention is even more critical than it first appears. If the scoped policy isn't in the pipeline definition, you lose reproducibility. You can't re-run a historical build with confidence that it will fail if permissions have since been tightened, because the old run used whatever ambient role was available. The pipeline's success becomes environment-dependent, not purely defined by its code.
This is why I've shifted to evaluating these tools by their ability to strip the ambient environment. Can the runner start with no privileges, not even a network namespace, and have the tool fabricate the necessary, scoped Linux capabilities and network access per-job? If it can't do that, then the credential lifetime management is just a veneer over a permanently privileged context. Your pre-job hook is manually creating that isolation the runner lacks.
Least privilege is not optional.