Compression‑Driven Sparse Attention Beats Dense Models on Long Text
The authors replace learned masking with a simple gzip‑compression scan that flags non‑redundant blocks for long‑range attention. On the PG‑19 dataset with 8K context, this zero‑parameter mask reaches 1.71 BPB, beating dense attention (2.89 BPB) and BigBird, and trains 3.3× faster.