LodeHQ
AI news for people who shipFree, by email

AI · github.com · Sunday, 6 September 2026

A shared CUDA arena keeps long-context KV cache from eating VRAM

This PR ports phase-arena KV streaming beyond its original Qwen-only implementation. It puts compute workspace, resident KV pages, and the async transfer ring in one bounded CUDA allocation, using adaptive residency and prefetching to keep long-context memory predictable. The port also fixes a scheduler path that reportedly caused 74 device splits per evaluation instead of 2, while restricting v1 to supported standard K/V types.

Read the source: github.com ↗In the 6 September issue →