Text‑to‑SQL Benchmarks Miss the Mark: Real Warehouses Only 10% Accurate
Public benchmarks report about 80% accuracy for Text‑to‑SQL models, but Michael Stonebraker’s test on a 1,400‑table Oracle warehouse yielded only ~10% accuracy, rising to ~30% with explicit table hints. The gap stems from unseen proprietary data, far more complex queries, idiosyncratic naming, and tangled schemas. Teams risk over‑estimating AI tools if they rely on clean‑benchmark scores.