Can AI benchmark subjective reasoning by predicting expert judgments?
A post argues that subjective conceptual tasks—like forecasting AI takeover risk—are poor benchmarks, but suggests turning them into “judgment prediction” tests where models aim to match specific experts’ answers under timed constraints. It outlines cheap and costly implementations and highlights the trade‑off between measuring capability gains and handling irreconcilable disagreements.