Terminal-agent RL needs calibrated tasks, not just runnable tests
A Docker image and executable tests can still produce an unfaithful terminal-agent training pipeline, so teams need to audit task validity, harness brittleness, and reward alignment before trusting results. In the paper’s setup, prompt redesign and more context raise baseline solvability 5.6 times, but a 9B model levels off at 81.3% mean pass@2 within 20 steps. Adding harder tasks drops that score to 20.6% without changing training, showing the task difficulty band is model-specific.