A persistent challenge in high-availability agent deployments is the rotation of foundational secrets—like Vault tokens or cloud IAM credentials—without incurring agent-wide downtime or triggering a thundering herd problem on the secret manager. The naive approach of simply issuing new credentials and restarting all agents is a non-starter for our workloads.

The core of the pattern lies in treating secret rotation as a stateful, phased deployment, not a monolithic event. We must design agents to hold dual credential sets and the runtime to manage the transition. This requires careful coordination between the secret management backend (e.g., Vault) and the agent orchestration layer.

Consider an agent that requires a Vault token to access application secrets. A blue/green rotation would proceed as follows:

1. **Phase - Blue Active, Green Issued:** The agent runtime (or a sidecar) requests a new, distinct set of credentials (the "green" set) from Vault, using the existing "blue" credentials for authentication. Both credential sets are now valid and held in memory.
```rust
// Pseudo-Rust for illustration
struct CredentialPair {
blue: VaultToken,
green: VaultToken,
active: CredentialColor, // Blue or Green
}

impl CredentialPair {
async fn rotate(&mut self, vault_client: &Client) -> Result {
// Use blue token to generate a new, green token with a fresh lease
let new_token = vault_client.create_token(&self.blue).await?;
self.green = new_token;
Ok(())
}
}
```

2. **Phase - Green Validation:** The agent begins a warm-up period using the green credentials for *new* requests or connections, while the blue credentials continue to service existing operations. This validates the new credentials' permissions and latency.

3. **Phase - Green Active, Blue Deprecated:** The agent runtime switches all new operations to the green credentials. The blue credentials are marked as deprecated but not yet revoked. A grace period allows for in-flight operations using blue to complete.

4. **Phase - Revocation & Cleanup:** After the grace period, the runtime explicitly revokes the blue credentials via the secret manager's revocation API. Only the green set remains active. The process can now repeat for the next rotation.

Critical to this pattern is the runtime's ability to handle credential lifecycle and fail gracefully. If green credential validation fails, the runtime must discard the green set and continue with blue, alerting operators. This also implies that the secret manager must support concurrent, revocable leases for the same entity.

The major technical hurdles are:
* Ensuring the secret manager's policy allows a credential to create a *different* credential for the same identity (often requiring `sudo` capability in Vault).
* Preventing secret leakage in memory; the dual credential state increases the attack surface slightly, necessitating secure in-memory storage (e.g., memory guards, mlock).
* Building idempotent revocation logic into the agent shutdown or crash-handling path.

Has anyone implemented this at scale, particularly with Rust-based runtimes? I'm interested in the specifics of handling lease renewal during the transition and how you structured the state machine. I've found that many client libraries are not designed for this multi-credential model, requiring a wrapper abstraction.

-- Oli