Long-horizon agent state and skill-file research

Harness-of-Harness scores 71.52 vs 58.24 for naive continuation on GameCraft-Bench

Amazon-Microsoft SPACE paper: grouping agent actions cuts LLM rounds from 10.2 to 5.2 while raising ScienceWorld success rate from 35.9% to 67.2%

Google and Purdue's SKILL.state, published September 3, replaced growing agent transcripts with a structured state, cutting tokens 94% on a 100-step task while holding accuracy at 0.94 versus a baseline's 0.91. ByteDance's Chain-of-Experience, published September 4, offered a direct contrast: keeping full attempt history in context raised accuracy from 66.8% to 79.3% across six benchmarks. Amazon and Microsoft's SPACE added a third angle, training agents to batch safe sequential actions, raising ScienceWorld success from 35.9% to 67.2% while cutting LLM rounds from 10.2 to 5.2.

ContextPilot paper described, using RL-trained context management

JIT-Agent paper: task-specific harnesses let smaller models outscore larger ones with 36% lower cost

Three benchmarks published September 2 document frontier model struggles on extended tasks: 17 models averaged a 6.4% pass rate on multi-step terminal tasks, 10 of 17 solving zero; the best agent scored 51.3/100 after 12 hours; and the top setup on year-long decisions reached 27.3% of human earnings. Investor Chamath Palihapitiya, at Stanford's AI Club, warned AI investment could reach a trough of disillusionment if these limitations are not addressed. A separate paper found agent traces compress into finite-state machines with 7 to 43 states.

Not Diamond publishes dynamic model routing methodology claiming Pareto-dominance over Opus xhigh at 20-80% lower cost

A head-to-head experiment published September 1 comparing OpenClaw 2.0 and Hermes agents on a video generation task found that self-verification depth, not model capability, drove token consumption: OpenClaw used approximately 2.1M tokens with 10 self-fixes, Hermes approximately 2.9M with 20, the extra roughly 800K tokens spent on Hermes's render-checking harness measuring frame timing, seam visibility, and spec compliance. The experiment frames a portion of agent token use as QA allocation, raising the question of how much compute agents should spend proving their work before returning results.

ByteDance Chain-of-Experience paper shows keeping full attempt history outperforms compression for test-time improvement

Multiple papers on how agents should manage state in long tasks, reported August 30-31, reached two conclusions: structured state cuts token use and raises accuracy, while skill file injection consistently lowers performance. Google and Purdue's SKILL.state reduced tokens 94% on a 100-step warehouse task; Meta's EvoHarness-RL raised ALFWorld accuracy from 47.9% to 96.9%; Nvidia's ACES found skill document scores correlate near-zero with runtime gains; and WebDev-Skills-Bench found skill injection lowered pass rates across all tested models.

Coverage confirms 94% token reduction and accuracy improvement from 0.91 to 0.94

SKILL.state paper published; 16.2x token reduction and accuracy gain reported on 100-step warehouse task with Gemini 3 Flash

Paper testing eight leading models on year-long decision tasks finds best setup achieves 27.3% of human earnings

WebDev-Skills-Bench finds skill file injection lowers pass rates and raises token costs across all four tested models

Nvidia ACES finds skill document scores correlate near-zero with measured runtime improvement

Paper argues on-policy distillation outperforms GRPO for long-horizon agent training