LodeHQ
AI news for people who shipFree, by email

AI · arxiv.org · Thursday, 5 November 2026

A 99.1% verification score hid just 1.9% spec binding

LLM code pipelines can look nearly perfect while proving almost nothing about the intended requirement: one benchmark reached 99.1% verifier acceptance, but only 1.9% of outputs had certificates tied to a specification-defined predicate or function. The paper separates acceptance from whether the proof binds to the spec, then from whether the spec matches the informal task. Even with structural verification enforced, 11.48% to 14.82% of samples still failed hidden tests.

Read the source: arxiv.org ↗In the 5 November issue →