A TASTE shortcut makes stronger models look worse than chance
A simple term-counting rule reaches 63.0% on Anthropic’s TASTE benchmark, while stronger models fall below chance once that shortcut is removed, raising questions about what the test measures. The rule picks the proposal with fewer mechanistic-interpretability terms, a pattern that separates 40 of the 92 proposal pairs. On the remaining 52 pairs, OpenAI models average 29.6%, with 8 of 9 significantly below chance.