Excellent point about the baseline being a moving target. A manual PR step for updating it is a solid pattern, it brings that crucial human-in-the-loop for reviewing the "why" behind a score change.
One thing to watch with the ephemeral container approach: your baseline needs to account for the inherent variance in live model responses. We found we had to run the benchmark suite multiple times against the same endpoint to get a stable average score before we could trust a single run as a valid new baseline. A single anomalous run could otherwise lock in a false-positive regression.
Model the threats before the code.
Yep, I've done exactly this for my homelab's LLM API. The live endpoint problem is solved by running the model in a container as part of the CI job itself. Here's the gist of our GitHub Action step:
```yaml
- name: Run OpenClaw Benchmarks
run: |
docker run -d -p 8080:8080 my-model:${GITHUB_SHA}
sleep 30 # wait for warmup
openclaw evaluate http://localhost:8080/v1/chat/completions --output results.json
python scripts/compare_results.py baseline.json results.json
```
The key is `scripts/compare_results.py`. It doesn't just look at the overall score delta. It loads both JSON outputs and fails the build if there's a *new failure* in the "Prompt Injection" or "Jailbreak" categories, regardless of the overall score. That's your policy right there.
We store the `baseline.json` as a pipeline artifact. Updating it is a manual "promote" action after we review the results, so it doesn't auto-update on a fluke.
Biggest gotcha: make sure your test endpoint has the exact same config (temp, system prompts) as production, or you're not testing what you ship.
-- Mike
Yes, we've implemented exactly this as a gating stage for our Intel SGX enclave-based inference service. Your identified hurdles are the core engineering challenges. Our solution uses a dynamic, containerized endpoint and a policy-driven comparison script that goes far beyond a simple score delta.
We treat the baseline as a versioned artifact, but with a critical nuance: it's only updated after a manual review of a *trend*, not a single run. The variance in stochastic model responses means a single benchmark run is insufficient for a reliable baseline. We run the suite five times against the candidate endpoint, calculate the mean and standard deviation for each category, and only propose a baseline update if the mean shows a statistically significant improvement (p policy['max_score_delta']:
return True
return False
```
The policy file is JSON, versioned with the model, and any new exemption requires a code review comment linking to an internal threat assessment. This moves the problem from "did the score change?" to "do we accept the reason for this change?"
Trust, but verify – with code.