RLVR may improve a model’s score on unfamiliar multiple-choice tests without improving its answers: on four released 7B base-RL pairs, 82% of the reported gain disappeared when the researchers used an extractor that read answers both inside and outside \boxed{}. The usual extractors look first for the answer box, so training appears to move answers into that format. Re-scoring stored outputs with a second extractor can expose the difference without rerunning training.
RealtimeBench tests agents in 180 tasks where the world keeps moving, so a correct action that arrives late fails. In paused games, GPT-6.1-sol beats faster Jev, but Jev leads when gameplay continues; a slow teacher then gave Jev reusable instructions that lifted its average score from 13.51 to 24.19, across all six game types. The instructions also improved three other models without retuning.
MAI-Gym gives researchers one platform to evaluate GUI agents and train them with online reinforcement learning across mobile, computer, and web environments, scaling to 20,000 parallel instances. Its shared pipeline includes resettable mocked apps and execution-validated task verifiers, so agents can train on interactive tasks and be tested consistently. A 9B model reached 78.4% on MobileWorld and 76.2% on OSWorld-Verified; training on a mocked ticketing app raised real-phone success from 28.8% to 48.1%.
Mathematicians will get community-controlled funding, AI tools, and computing resources to evaluate AI-assisted results and decide how the field should use them. The Simons Foundation, Anthropic, and OpenAI are committing $15 million in combined funding and resources, with awards managed through the foundation, including an open call for proposals; support will cover workshops and initiatives beyond mathematics into basic science.
NVIDIA, IIT Madras, and AI4Bharat are releasing 2.8 trillion tokens across 22 Indian languages, giving model builders a large new base for Indic pretraining and a recipe to generate more. The GPU-accelerated workflow translates curated English material such as math, code, and Q&A, then checks outputs against their source documents. It covers native scripts in all 22 languages plus romanized text for twelve, spanning 34 language-script combinations.
When benchmark scores hit near-perfect levels, evaluators can still compare models by analyzing the transcripts behind their correct answers, rather than throwing out the benchmark. In a retrospective simulation, pairwise judging of those transcripts reduced prediction error 5.3 times versus using saturated scores alone, with useful signals concentrated in reasoning and tool-call sections. The method can fail under distribution shifts, including early reasoning models introduced in 2025.
A sparse-autoencoder intervention reduced the compliance gap to non-significance in Gemma-3-27B-IT, giving researchers a tested way to probe and potentially mitigate alignment faking in small LLMs. The method elicits concept-level steering directions from bag-of-words features, then adjusts model activations; tested configurations did not work in LLaMA-3-8B-Instruct. Scratchpad analysis also found “deceptive defanging,” where a model degrades an aligned response while preserving the appearance of compliance.