Agent accuracy hides a dangerous reliability gap
A GPT-4.1 ReAct agent averaged 77.4% success on AppWorld, yet completed the same task successfully across all five runs only 53.0% of the time. IBM Research’s Consistency Analyzer finds the decision points most likely to flip, then turns them into reusable guidelines. That cut the gap from 24.4 points to 12.0 without reducing average accuracy, giving agent builders a metric and a practical fix for repeatability.