Wall Attention introduces persistent memory tokens for efficient long-context reasoning
Wall Attention is a new attention variant that adds per‑channel, per‑timestep decay via persistent 'wall' memory tokens, enabling better long‑context reasoning while keeping compute low. The open‑source implementation provides fused training kernels and a fast decode kernel, supporting GQA and scalable inference.