Softmax‑free GPT‑2‑Medium model cuts VRAM, trades slight accuracy loss for efficiency
A new 354M‑parameter GPT‑2‑medium‑sized model replaces softmax with a structural‑sparsity attention mechanism, cutting VRAM usage by up to 55% on long contexts. Open weights and a custom Triton inference engine are released, showing a modest trade‑off: 0.02 lower CORE score than dense GPT‑2 medium but better on multiple‑choice reasoning.