ThinkingBox shows one successful run can hide unreliable agents
ThinkingBox shows that agents can look capable once while failing to complete business workflows reliably, a warning for anyone deploying them on bookings, claims, refunds, or internal support. Its 507 tasks reset a stateful environment for each attempt, let agents gather missing details from a simulated user, and grade the final database state rather than the transcript. Claude Opus 5 fell from 66.50% pass@1 to 47.53% across 20 runs; Kimi-K3 fell from 57.37% to 17.60%.