Best agents finish only 42.3% of tasks when their tools keep changing
Δτ-Bench tests agents in tool ecosystems that keep evolving: familiar tools disappear or get replaced while more efficient alternatives appear. It has 1,588 multi-turn tasks across 42 domains and six successive ecosystem states. The strongest model or harness reached only 42.3% task success, and gains from existing skill-learning methods did not hold as tools changed. The authors' Steward manager, which maintains skills and tools during tasks, kept success higher.