Sparse attention cuts long-context work, but moves the bottleneck to indexing
For model developers handling 128K-token contexts, sparse attention reduces the number of keys each query processes, cutting attention’s compute and memory traffic, but it shifts work to finding those keys. An indexer scores all keys, Top-k keeps the most relevant few thousand, and core attention runs only on that subset. In NVIDIA’s DeepSeek V3.2 example, the indexer and Top-k still consume 47% of a prefill layer.