A reproduction of Claude's tokenizer suggests Anthropic has been shrinking the vocab, roughly 50,000 entries in Claude 3 down to ~16,000 today, while rivals push into the hundreds of thousands. One plausible motive: the softmax head is a gradient bottleneck that compresses the backprop signal to rank 2D, so a huge vocab wastes information. It may also be why Claude sidesteps glitch tokens and big memory-hungry projection layers.
MoE serving has two bottlenecks with opposite fixes: prefill is compute-bound on per-token experts, decode is memory-bound on the batch's expert set. Prior training-free accelerators optimize only one, and throw away the skipped experts' output. ExFold treats both phases as one budgeted approximation problem, using calibrated scalar projectors to fold excluded experts' contributions into the retained ones. In a vLLM plugin it delivers up to 1.41x TTFT and 2.45x TPOT speedups while keeping ~99% of quality.
A systematic GPU-measurement study of four open-source LLMs shows that the attention mechanism is the dominant factor in how much energy decode consumes as context grows. MHA models scale steeply with length, GQA climbs more gently, and GQA with sliding-window attention holds energy nearly constant. Meanwhile batching slashes both energy per token and latency by up to 87%, offering clear guidance on energy-efficient inference setups.
Graph RAG normally taxes you three ways: keep a vector store, a graph DB and a document store in sync, and then trust a pipeline that never refuses output. post-graph-rag collapses everything into a single PostgreSQL instance, with extraction-time checks that reject vague predicates, pronominal names and bare quantities so the graph only asserts what the prose actually says. Against LightRAG it builds a denser, more queryable graph at lower latency, and adds a temporal layer that lets a later document supersede an earlier one.
Researchers anchored a single concept, "red," into a transformer's residual stream and watched it land exactly where they wanted. Nearby colors graded sensibly along it, real task accuracy held steady, and the result was estimated usable directions in the SCA framework. Steering is the next target.