Your LLM judge is grading its own model's SQL, and it's too kind
A SQL-generation agent worked fine for weeks, until a query that should have been flagged ran and produced a confidently wrong number. Root cause: judge and generator shared the same model, and the judge proved measurably more lenient toward its own output than toward another model's. That self-preference bias is a known LLM-as-judge failure mode that feels academic until it costs you an analyst's afternoon. The fix is treating the judge itself as a component that needs testing and calibration against human reviewers.