Parallax Introduces Parameterized Local Linear Attention Boosting LLM Efficiency
Parallax adds a scalable, parameterized Local Linear Attention mechanism that replaces the softmax constant estimate with a learned linear probe, improving the bias‑variance trade‑off in large language models. The new attention kernel matches or exceeds FlashAttention performance across batch sizes and context lengths, yielding consistent perplexity reductions at 0.6B‑1.7B scales and downstream gains.