Microsoft and Tsinghua paper finds structured run traces raise GPT-4.1 failure localization from 3.63% to 31.35%
Concluded·Day 10·3 sources·updated 11 Sep 2026
AI agent reliability and task duration
The gist
Agent success rates fall exponentially with task length, benchmarks show
Long-horizon failure rates are exponential, not linear, so even modest increases in task duration produce disproportionate reliability losses. Zylos.ai projects enterprise deployment of long-running agents to grow 8x by end of 2026.
The full picture
OpenAI data shows agent success without human intervention dropping from 86% on sub-15-minute tasks to roughly 16% on tasks lasting 64 to 128 hours. Forethought.org fits an exponential decay model to Kwa et al. data, assigning each agent a half-life measured in human-equivalent task time. Zylos.ai finds that doubling task duration quadruples the failure rate and identifies 35 minutes of human time as the onset of performance degradation. Several architectural approaches show improvements: ContextPilot uses reinforcement learning for active context management, Amazon and Microsoft's SPACE reduces LLM decision calls by grouping sequential actions, and Stanford's Prefix Sliding discards intermediate chain-of-thought tokens for roughly 3x faster inference. Stale memory is a separate structural problem: Qwen3 models followed outdated memory values 92 to 100% of the time even when current evidence contradicted them.
How it developed
Microsoft ToolQA paper published: nine models collapse from near-perfect to 0-33% success by 16 steps
Paul argues time horizon is a more meaningful agent capability metric than tokens or benchmark scores, citing OpenAI data showing 86% to 16% success drop across task durations
On September 10, Forethought.org fit Kwa et al. benchmark data to an exponential decay model and assigned each agent a half-life in human-equivalent task time, while a Qwen3 study found agents followed stale stored memory 92 to 100% of the time despite contradicting evidence.
A joint Microsoft and Tsinghua paper reported structured run traces raised GPT-4.1 failure localization from 3.63% to 31.35%. A benchmark analysis also identified messiness factors as performance predictors beyond task length.
Amazon-Microsoft SPACE raises ScienceWorld success from 35.9% to 67.2% while cutting LLM rounds from 10.2 to 5.2
ContextPilot introduced, scoring 69.40 vs 45.93 for base Qwen3-8B across four long-context benchmarks
FM-Bench published: year-5 rankings correlated 0.19 with final; DeepSeek-V4-Pro led early but finished 12th overall
Sources
Rohan Paul Twitter
- I think "time horizon" is becoming one of the more useful ways to talk about agent capability.
- AI agents can trust stale memory over fresh evidence, and bigger models do not reliably fix this.
- New Amazon Microsoft paper shows long-horizon agents should not need an LLM decision after every tiny action; the hard p…
- New Stanford paper Prefix Sliding shows that long reasoning does not need the full chain of thought in memory: keep the …
- Long-running agents do not just need a bigger context window. They need to learn what deserves to stay in context at all…
forethought.org
The daily email
Want this in your inbox?
I send one email each morning with the stories that moved. If you would rather just read here, that works too.
Subscribe free