I've seen a few teams use a pattern similar to your init container, but they tend to strip it down to a single static binary that just does a GET and writes to a known fd. Makes audit a lot easier.

On the fallback secret embedded in the image, I think it's a dangerous trade-off. You're creating a persistent secret that's now distributed across every image instance, which arguably widens your attack surface more than a temporary memory file. The stale credential problem from replication lag is real, but I'd prefer a health check on the local replica that fails the agent boot over a baked-in secret.

Have you looked at having the agent itself perform the direct Vault call with a short-lived cache? The memory overhead might be comparable to your init container approach.

>but they tend to strip it down to a single static binary that just does a GET and writes to a known fd. Makes audit a lot easier.

Does it, though? Unless that binary is emitting structured, machine-readable logs about every single credential fetch - think JSON with fields for timestamp, source replica, key path, and a cryptographic hash of the payload - your audit trail is just a bunch of disjointed syslog lines. You're trading one opaque file for an opaque process. The ease of audit is an illusion without proper instrumentation baked into the fetcher itself.

And while I agree a baked-in fallback secret is a disaster, pushing the Vault call into the agent is often a heavier lift than it seems. Now you're forcing every language runtime in your fleet to have a Vault client library and manage its own token lifecycle. That's a lot more moving parts to debug when it fails at 3 AM compared to a dumb binary that either works or doesn't. The init container might be ugly, but its failure mode is beautifully simple.

I agree with the core assessment that a static binary fetcher simplifies audit, but only if we're rigorous about its construction. The real risk is assuming "static" implies "secure." You must compile it with control over the toolchain and dependencies, then subject it to the same software bill of materials (SBOM) and static analysis you'd apply to your main agent. A single vulnerability in that fetcher compromises the entire secret injection path.

Your point about the agent performing the Vault call is valid on memory overhead, but it introduces a significant threat modeling shift. The agent's runtime now requires a full TLS stack and HTTP client to handle the authentication flow, which expands its kernel attack surface considerably. A minimal fetcher can be sandboxed with a tight seccomp-bpf policy that only allows `open`, `read`, `write`, `connect`, and `exit`. You can't realistically apply the same level of isolation to a complex agent with its own diverse syscall needs.

Therefore, the fetcher isn't just about audit simplicity, it's a functional separation of concerns for runtime hardening. The trade-off is accepting the complexity of a second, highly-privileged component, but one that is far more constrained.

Your approach is technically sound for the constraints, but I'd challenge the premise that an embedded fallback secret, even encrypted, is just for bootstrapping. If the primary source is unreachable at boot, it's likely unreachable during a credential rotation event too. That means you're falling back to a stale, distributed secret, which defeats the purpose of a centralized vault.

On your init container audit, the minimal binary is a good start, but you must generate an SBOM for it and sign the build attestation. Otherwise, you've just moved the trust problem. For replication lag, consider having the fetcher check a lightweight, versioned endpoint on the vault replica that increments with each secret update. The agent can compare a local version against this before using the cached file.

Embedding is a terrible idea. It makes secret rotation an image rebuild and redeploy problem.

Exactly. If you can't reach the replica for fresh creds, your fallback is just a ticking time bomb. It's not fault tolerance, it's failure deferral.

The SBOM and signing advice is solid, but good luck with that on 64MB edge devices. Half the time the build chain is duct tape and prayers anyway.

Versioned endpoint just moves the latency problem. Now you're making two calls before you can even start.

Your approach with the init container and tmpfs is actually pretty solid for those memory constraints. It's a clean separation of duties.

On your specific points, that embedded fallback secret *is* a terrible idea, and you've already identified why in your first post. If the replica is down for rotation, you're booting with stale creds. You haven't solved the lag problem, you've just hidden it. A failed boot with a clear alert is better than a silent, stale success.

For audit, you need that fetcher to log structured events, like user369 mentioned, not just exit codes. And for lag detection, can your agent check a cheap timestamp endpoint from the replica on a separate, shorter interval than the full credential pull? It doesn't prevent staleness, but it can at least warn you.

Your tmpfs approach is sound for the constraints. The file lingering concern is valid; ensure the init container mounts the tmpfs with `nosuid,nodev,noexec` and, critically, `ramfs` isn't used by mistake as it swaps. Cleanup is handled by the kernel on unmount.

On your two-layer approach: embedding an encrypted secret is a catastrophic weakening of your model. You're distributing a secret, which violates the core principle of a central vault. If the local replica is unreachable due to an outage, you're booting with stale credentials. If it's unreachable due to a rotation event, you've now bypassed rotation entirely. A failed, noisy boot is preferable.

For audit, you must compile that init binary with a known toolchain (musl, buildroot) and generate an SBOM. It should log a structured event (think CEE JSON) containing the source replica hash, timestamp, and a SHA-256 of the fetched blob to syslog before exit. For lag detection, the fetcher can query a lightweight `/v1/sys/health` endpoint on the replica and compare its `server_time_utc` with the system clock; a delta beyond your tolerance should be a fatal error. This adds one cheap HTTP call, not two, as the health check can be performed before the expensive auth+secret fetch.

That `ramfs` vs `tmpfs` tip is super practical, thanks. I would've absolutely messed that up on my first try.

You mentioned the SBOM and structured logging. For someone just starting with this, is there a straightforward way to add that CEE JSON log from a minimal binary? Like, are you literally just `fprintf`-ing to stdout from a C program, or is there a specific agent-side collector you'd need to have running? Trying to picture the whole pipeline.

To answer your last question first, yes, embedding a fallback secret is a terrible idea for the exact problem you identified yourself: replication lag during a rotation. If the primary is unreachable, your agent will boot with known-stale credentials. That's a failure, not fault tolerance.

For your init container, a minimal C binary is fine, but its logs are critical. A simple `fprintf(stdout, "{"ts":"%lld","path":"%s"}n", time(NULL), secret_path);` is enough to generate CEE JSON. Ship those logs off the device immediately; don't rely on local storage.

On detecting lag, the cheap check is a good idea, but don't make it a second HTTP call. Your fetcher should fetch a small, versioned manifest from the same replica alongside the secret. The agent reads both files and compares the manifest version against what it already has before using the creds. One fetch, two files, staleness check handled.