SemiAnalysis launches GDPx; Claude Opus 5.5 leads at 12%, all models fail finance and law
ConcludedRan 5 daysin the edition since 29 Oct 20263 sourcesupdated 2 Nov 2026
Artificial Analysis v5 open-weights benchmark gap
The gist
Latest: Artificial Analysis v5: Open-Weights Gap Widens to 12 Points
The gap between open-weights and closed frontier models has grown fourfold in one benchmark cycle, suggesting the two tiers are diverging on the hardest tasks. The shift to 60% private datasets makes score inflation harder and puts more weight on held-out results.
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.
The full picture
Artificial Analysis v5 places Claude Opus 5.5 at 58 points, ahead of GPT-6 Astra at 57 and GPT-6.1 Sol at 55, with Anthropic and OpenAI holding the top five positions. The gap between closed and open-weights models widened from 3 points in v4.3 to 12 points in v5. Kimi K3.1 Pro leads the open-weights category at 46 but has not released its weights; among models with released weights, DeepSeek V4.1 Flash 1017 and GLM-5.3 top the field at 32. Claude Haiku 5.5 (max) outscores all open-weights models. The divergence is most pronounced on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34% against 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra. The v5 index is now 60% private datasets or held-out answers, and new components include FrontierCode 2.0, where the top score is 45%, and Terminal-Bench-Science. Cost per task varies by more than 13x among the top 10 models.
How it developed
Artificial Analysis releases AA-PersonalityBench covering 38 models and 30 traits
Claude Opus 5.5 leads the Artificial Analysis v5 index at 58 points, ahead of GPT-6 Astra (57) and GPT-6.1 Sol (55), with the index 60% private datasets and cost per task varying by more than 13x among the top 10.
A Google, UCLA, and CMU paper introduced GenXR-Bench, 112 tasks covering XR app coding, where Claude Opus 5 led at 50.9% but no model exceeded 29.2% on SDK features. A METR randomized study found developers using early-2025 AI coding tools took 19% longer on familiar codebases.
First citedZvi's AI Roundups
First in the edition · 29 October 2026
Artificial Analysis released v5 of its benchmark, showing the performance gap between open-weights models and frontier closed models growing from 3 points in v4.3 to 12 points in v5.
Kimi K3.1 Pro leads open-weights models at 46 but its weights are unreleased; among models with released weights, DeepSeek V4.1 Flash 1017 and GLM-5.3 top the category at 32, below Claude Haiku 5.5 (max). The gap is widest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34% against 68% for Claude Opus 5.5 (max). A separate commentary argues that identifying the right benchmarks to optimize against is significantly harder than it is generally credited for being.
University of Chicago paper published finding LLMs poorly predict what drives variation in their own answers
Meta and Michigan State University paper published on LLM reasoning graph quality variation
Sources
Dean Ball on X
Interconnects
- Moonshot AI's Kimi K3.1 Pro and cyber capability reduction
- Claude 5.5 family and GPT-6 Sol
- Claude Fable 5.1 on science benchmarks
- The open-weight versus closed AI model gap
BackgroundFrontier models, open models, and how they are compared
Get the daily email
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.