The cheap fix that beats linear attention: sliding windows win long-context tests
Sliding-window attention with attention sinks, one of the oldest and cheapest fixes for quadratic cost, matches or beats post-trained linear-attention models across several LLMs. On long-context reasoning (Needle-in-a-Haystack, BABILong) it scores 2 to 10 times higher, with zero post-training. The authors' blunt takeaway: labs spending compute on linearization may be solving a problem a rolling window already handles.