Long-horizon agent state and skill-file research

A paper identifies a systematic evaluation flaw in skill-enabled agents where overall gains can mask degraded performance on retrieval tasks; RAE proposed as a fix.

Data shows Google at 3.2 quadrillion tokens/month, agents 71% of OpenRouter traffic, Goldman projects 24x growth by 2030

Microsoft paper reported showing skill distillation from 35-50 trajectories recovers 55%-100%+ of reasoning model performance gap at 2.9-4.5x lower token cost

Harness-of-Harness scores 71.52 vs 58.24 for naive continuation on GameCraft-Bench

Google and Purdue's SKILL.state, published September 3, replaced growing agent transcripts with a structured state, cutting tokens 94% on a 100-step task while holding accuracy at 0.94 versus a baseline's 0.91.

ByteDance's Chain-of-Experience, published September 4, offered a direct contrast: keeping full attempt history in context raised accuracy from 66.8% to 79.3% across six benchmarks. Amazon and Microsoft's SPACE added a third angle, training agents to batch safe sequential actions, raising ScienceWorld success from 35.9% to 67.2% while cutting LLM rounds from 10.2 to 5.2.

ContextPilot paper described, using RL-trained context management

Three benchmarks published September 2 document frontier model struggles on extended tasks: 17 models averaged a 6.4% pass rate on multi-step terminal tasks, 10 of 17 solving zero.

The best agent scored 51.3/100 after 12 hours. And the top setup on year-long decisions reached 27.3% of human earnings. Investor Chamath Palihapitiya, at Stanford's AI Club, warned AI investment could reach a trough of disillusionment if these limitations are not addressed. A separate paper found agent traces compress into finite-state machines with 7 to 43 states.

A head-to-head experiment published September 1 comparing OpenClaw 2.0 and Hermes agents on a video generation task found that self-verification depth, not model capability, drove token consumption: OpenClaw used approximately 2.1M tokens with 10 self-fixes, Hermes approximately 2.9M with 20, the extra roughly 800K tokens spent on Hermes's render-checking harness measuring frame timing, seam visibility, and spec compliance.

The experiment frames a portion of agent token use as QA allocation, raising the question of how much compute agents should spend proving their work before returning results.

Memory-Augmented Compression method reported, recovering accuracy on GSM8K and MATH while faster than standard CoT

Multiple papers on how agents should manage state in long tasks, reported August 30-31, reached two conclusions: structured state cuts token use and raises accuracy, while skill file injection consistently lowers performance.

Google and Purdue's SKILL.state reduced tokens 94% on a 100-step warehouse task. Meta's EvoHarness-RL raised ALFWorld accuracy from 47.9% to 96.9%. Nvidia's ACES found skill document scores correlate near-zero with runtime gains. And WebDev-Skills-Bench found skill injection lowered pass rates across all tested models.

Paper testing eight leading models on year-long decision tasks finds best setup achieves 27.3% of human earnings

WebDev-Skills-Bench finds skill file injection lowers pass rates and raises token costs across all four tested models

Nvidia ACES finds skill document scores correlate near-zero with measured runtime improvement

Paper argues on-policy distillation outperforms GRPO for long-horizon agent training