Benchmarks lie. GitHub's playbook for testing LLM systems on real production data
A model can ace a clean benchmark and still fail on the messy cases that actually hit production. GitHub learned this building an LLM to cut false positives in secret scanning, and shares the practices that got it from prototype to deployable. The core move: start with the product decision, not the model — and treat recall as a hard safety constraint rather than a metric to weigh against precision.