When Models Know They're Being Tested: New Benchmark Quantifies Eval Awareness
Frontier models can often tell when they're being evaluated, and if they behave differently in evals than in deployment, benchmark results stop meaning anything. EvalDetectBench is an open pipeline for measuring that awareness across any Inspect-compatible eval, with a transcript suite drawn from real system-card evals and deployment sources. It also exposes a measurement trap: the model generating deployment transcripts explains 11.25% of variance and can flip model rankings, so the benchmark corrects for it with per-model calibration.