I have been conducting a performance audit of our attestation infrastructure for the new agent enclaves, with a particular focus on the consistency of the remote attestation handshake. As part of this, I built a monitoring dashboard that aggregates attestation latency from our global deployment points, calculating percentiles (P50, P90, P99) over rolling 24-hour windows.

The initial hypothesis was that network topography would be the primary variable, with some predictable baseline overhead from the attestation service itself. The observed data, however, reveals a problematic pattern of sporadic, significant latency spikes at the higher percentiles, which contradicts the assumption of predictable cryptographic overhead. For instance, in the us-east-2 region yesterday, the P99 latency exceeded 4.2 seconds, while the P50 remained at a stable 180 milliseconds. This degree of variance is operationally concerning for time-sensitive agent orchestration.

The dashboard is a simple Prometheus/Grafana setup. The collector, a sidecar to the agent launcher, records the duration from the initiation of the `sgx_init_quote` to the successful verification of the attestation evidence by our relying party. The key query for the percentile breakdown is as follows:

```promql
histogram_quantile(0.99, sum(rate(attestation_duration_seconds_bucket{region="$region"}[24h])) by (le))
histogram_quantile(0.90, sum(rate(attestation_duration_seconds_bucket{region="$region"}[24h])) by (le))
histogram_quantile(0.50, sum(rate(attestation_duration_seconds_bucket{region="$region"}[24h])) by (le))
```

This variance presents several risks from a threat modeling and operational security perspective:
* **Agent Launch Throttling:** Slow attestation directly impacts our ability to rapidly scale or replace agent instances, a key requirement for resilience.
* **Potential for Denial-of-Service:** An external dependency causing high P99 latency could be exploited to degrade our service availability, even if the mean latency appears healthy.
* **Obfuscated Failures:** Latency spikes may correlate with specific, non-fatal error conditions in the attestation service or the enclave platform itself, which could be early indicators of a systemic issue.

I am seeking to validate my analysis and gather data points from other deployments. My immediate questions for the forum are:

* Has anyone else instrumented and measured remote attestation latency with this granularity (percentiles, not averages)? If so, have you observed similar disparity between median and tail latencies?
* What are the most probable root causes? I am considering:
* Variable load or queuing within the Intel Attestation Service (IAS) or similar provider endpoints.
* Interactions with the enclave platform's quoting enclave, especially under concurrent launch conditions.
* Non-uniform performance of the trusted computing base (TCB) on different underlying hardware, despite identical CPU SKUs.
* From a zero-trust architecture standpoint, how should we architect around this unpredictability? Should we implement a failover to a different attestation service or region after a timeout, and if so, what are the security implications of switching attestation providers mid-workflow?