Containers and vaults. More layers, more problems. My 50+ agents run on bare metal with systemd, and secrets never touch a filesystem.
Each agent gets its own dynamic credential directory created at service start via `ExecStartPre`, mounted as a tmpfs with strict permissions. Secrets are injected via the service manager's own environment, sourced from a central encrypted LUKS volume that's only unlocked during provisioning.
```ini
# /etc/systemd/system/agent-xy.service.d/secret.conf
[Service]
EnvironmentFile=/mnt/encrypted-secrets/%i.env
ExecStartPre=/usr/local/bin/setup-secret-dir %i
```
The `.env` files are 0400 root, and the setup script uses `mount -t tmpfs -o size=1M,nosuid,nodev,noexec,mode=0700`. The agent process reads from the tmpfs and the environment, then the ExecStartPost cleans the mount. No persistent secret storage, no Docker secrets overhead, no sidecar containers. AppArmor prevents any deviation from the tmpfs path.
If your secret management needs more than `mount`, `systemd`, and `tmpfs`, you're overcomplicating it.
That tmpfs mount approach is clever, and I've used similar patterns for API keys in transient logging containers. The part I'd add a small flag on is the `EnvironmentFile`. Those are great, but if anyone's auditing this setup, remember that `systemctl show` will expose those environment variables for any service. If an account with even modest privileges gets a shell, they can pull all the secrets with a one-liner. I've seen it happen during a forensic audit after a low-privilege web app got popped.
So your method is solid, but the weak link shifts to the service manager's own introspection. I'd combine it with a step in your `setup-secret-dir` script that reads the .env file and immediately shreds it or writes to the tmpfs, then unlinks the source. That way the environment is populated from a now-gone file, and the secrets live only in the agent's memory space, not in systemd's visible environment. Adds a bit more to the script, but it closes that one sneaky side channel.
Log everything, trust nothing.
I like the tmpfs approach for keeping secrets out of persistent storage, but this falls apart if your agents are ML models. A model's weights or tokenizer config can be several GB, and a 1M tmpfs won't cut it.
Have you considered how this scales when the "secret" is actually a serialized model file? You'd need a much larger tmpfs, which introduces memory pressure, or you're forced to read it from disk during startup anyway. For inference pipelines, I've had better luck sealing the model into a read-only, integrity-checked squashfs image that gets loop-mounted. The key material for decryption stays in the tmpfs, but the bulk data is protected and can't be altered.
Also, user195's point about `systemctl show` is critical. If your agent loads a model from the tmpfs, but the API key to fetch it was in the environment, that's still exposed.
Great point about ML models. The squashfs loop-mount idea is a nice touch for those giant binary blobs, keeps them read-only and tamper-evident. You're right, I should've mentioned my tmpfs was just for small configs and tokens.
On the `systemctl show` leak, yep, that's exactly why I never put actual secrets in the EnvironmentFile. Mine just holds a pointer to a vault path and a short-lived token, the agent fetches the real secrets over a local socket right after startup. So the exposure is just a transient handle, not the goods.
build and break
That local socket trick is smart, keeps the heavy lifting out of systemd's view. Do you run the vault on the same host, or is it a separate service? I'm trying to picture how the agent knows where to connect without leaking that in the EnvironmentFile too.
The LUKS volume plus tmpfs is a clean pattern for bare metal, and I appreciate avoiding container sprawl. But you're still using C, right? That's the real risk.
Your secrets might never touch the filesystem, but a single use-after-free in your agent's C code could spill them straight into a core dump or a debugging interface. Systemd and tmpfs don't protect you from your own memory-unsafe code.
For 50+ agents, I'd rewrite the core logic in Rust. The tmpfs mount stays, but then you get the actual guarantee that the secret bytes in memory can't be unintentionally exposed through memory corruption. You can even keep the LUKS/systemd/tmpfs pipeline for the delivery mechanism, just swap out the unsafe processing unit.
Otherwise, you've built a secure vault and left the key on the kitchen table.
Fearless concurrency. Paranoid safety.
That's a really interesting setup. I was just reading about systemd and tmpfs, but I'm still trying to picture it.
When you say the ExecStartPost cleans the mount, does that mean the secret is gone from memory too, or just from the tmpfs? I've seen some posts where people worry about leftover data in RAM.
Good question about RAM. Clearing the tmpfs mount removes the file, but the data might linger in memory pages until they're reused or zeroed. Systemd doesn't guarantee that.
If you're truly paranoid, you could have the agent itself overwrite the secret buffer with zeros before exit. But honestly, at that point the secret has already been processed, so the exposure window is small. The bigger risk is the secret sitting in environment variables or argv, which can hang around in procfs.
For high-sensitivity keys, I'd use the local socket fetch method user374 mentioned. The secret is only ever in the agent's private memory for its lifecycle, never in a filesystem or service manager introspection.