Fresh-on-demand benchmark exposes how badly LLMs fail at compositional graph reasoning
ClosureBench generates fresh graph-reasoning instances on demand with machine-verified ground truth, so it's contamination-resistant and can measure memorization directly. Across 26 task categories, frontier models degrade sharply from atomic to compositional queries — misreading the graph, then reasoning correctly over the wrong one — while a 4B model fine-tuned to emit programs hits near-frontier accuracy.