AI agent long-horizon performance collapse

Multiple benchmarks converge on a consistent finding: AI agent reliability degrades sharply as tasks grow longer or more complex. Scale AI's SWE-Bench Pro records a drop from 70%+ on standard SWE-bench to roughly 23% on multi-file and cross-repository work. METR's July 2025 time-horizon analysis places the 50% success threshold at approximately 50 minutes on human expert tasks, with multi-hour autonomous work remaining out of reach. Some companies report agent failures on production workloads even when benchmark scores improve.

Microsoft's ToolQA benchmark tested nine models and found success rates consistently fell as dependent step counts rose, reaching 0-33% at 16 steps. Shortening context made the decline worse, ruling out context trimming as a fix.

The HORIZON benchmark, analyzing over 3,100 trajectories, characterizes the failure pattern as a structural shift in failure composition, not merely a drop in success rate. Performance gaps between frontier and smaller models collapse once agents enter the long-horizon failure regime. HORIZON distinguishes between environment disturbances, which require monitoring and recovery, and catastrophic forgetting, which requires improved memory and constraint tracking, as failure types needing distinct interventions.

On mitigation, Harness-of-Harness (HoH) carries current software state, test evidence, known problems, and a revised plan into each successive coding run. On GameCraft-Bench using Codex and GPT-5.5, HoH scored 71.52 after three passes versus 58.24 for naive agent continuation. Running HoH for 70 loops produced a playable first-person shooter from high-level requirements, though broader generalization remains unproven.