Atelier vs. vanilla Claude Code.
"Baseline" below is Claude Code with no plugin, no persona, no augmentation -- same model, same Docker image, same turn cap, same disabled-tools list as the Atelier arm in every suite. The only variable that changes is the runtime underneath the model.
Anthropic publishes model accuracy on SWE-bench Verified (Opus 4.8: 88.6%) -- but no cost, token, or turn-efficiency numbers for Claude Code the CLI against any baseline. That's a real, checked distinction: model benchmarks and runtime-efficiency benchmarks are different claims, and Anthropic only publishes the first. GitHub is the one vendor we found publishing the second kind -- a harness-vs-harness comparison of GitHub Copilot's agent against model-vendor harnesses, controlling for model and task. Atelier publishes the same kind of comparison for its own runtime against a vanilla Claude Code baseline, with every raw run committed to the repo below.
| Suite | Baseline correct | Atelier correct | Baseline $ | Atelier $ | Delta |
|---|---|---|---|---|---|
| SWE-bench Verified 50 tasks × 5 reps, claude-opus-4-8 | 202/250 (80.8%) | 232/250 (92.8%) | $234.84 | $165.45 | +12.0pp correct, 29.5% cheaper, 44.9% fewer tokens, 37.7% fewer turns |
| SWE-bench Lite 10 tasks × 3 reps | 28/30 (93.3%) | 30/30 (100%) | $12.38 | $10.79 | +6.7pp correct, 12.9% cheaper, 20.5% fewer tokens, 15.8% fewer turns |
| SWE-bench Pro 10 tasks × 5 reps -- Go/TS/Python, ScaleAI harness | 44/50 (88%) | 45/50 (90%) | $39.01 | $30.61 | +2.0pp correct, 21.5% cheaper, 35.4% fewer tokens, 28.1% fewer turns |
| Exploration tasks 7 large repos (Tokio, Django, VS Code...) × 5 reps | -- | -- | $19.11 | $6.29 | 67% cheaper, 91% fewer turns, 84% fewer output tokens |
| Telegraphic Q&A 20 engineering Q&A prompts × 5 reps, no code repo | -- | -- | $8.93 | $6.81 | 23.7% cheaper, 39.5% fewer output tokens (bigger system prompt, smaller replies -- wins anyway) |
| Terminal-Bench 2.1 89 tasks, 1 attempt vs. public tbench.ai leaderboard (5-rep avg) | 78.9% expected | 78.7% | $96.76 | $69.52† | -0.2pp (tie at n=1), 28.1% cheaper† -- floor, not ceiling: 5 Atelier runs timed out with real uncounted spend |
* Telegraphic Q&A turns: baseline's raw count includes a hidden Claude Code
session-title API call that fires only when no --agent is passed
(0/100 Atelier runs make it, since Atelier always pins --agent).
Corrected, both arms need ~1.4 real answering turns -- the cost and
output-token deltas above are unaffected. † 5 of 89 Terminal-Bench Atelier
runs hit a harness timeout with real, uncounted spend -- $69.52 is a floor.
Full caveats for every row → BENCHMARKS.md.
The true story
Atelier wins cost, tokens, and turns on every suite measured, and wins correctness on every suite with more than one rep. The one nominal loss (Terminal-Bench, -0.2pp) is a single-attempt run against a 5-rep public average, and it costs less anyway. Nothing here is cherry-picked -- read the honest caveats in BENCHMARKS.md before quoting a number out of context.