← All projects

Java

llm-framework-benchmark

Same spec, different stacks, graded by an external oracle

Can an AI coding agent build the same framework-neutral spec on different stacks? An external black-box oracle grades every run, so the only variable is the framework, and the agent.

Spring Boot 3.3.5 passes; 4.0.6 fails 19 of 20 one-shot runs · with a test loop, 12/12 recover

From the README

How well can an AI coding agent build the same specified system on different frameworks? A framework-neutral spec, an external black-box oracle that grades the result, and clean per-framework scaffolds — so the only variable is the framework (and the agent). The first spec is an event-driven Kafka → H2 → merged-notification service, built on Spring Boot (3.3.5, 4.0.6 and 4.1.1) and on Tiko DI, a framework the models were never trained on.

The benchmark grades each run on compliance (does it actually pass the acceptance scenarios, judged by an independent harness — not the agent’s own tests), with hooks for efficiency and code quality.

📝 Write-up: Your AI coding agent has a blind spot — and a bigger model doesn’t fix it — the story behind these numbers (LinkedIn). Feedback and your own results welcome.

Why Tiko is in the benchmark — it’s intentionally unknown to LLMs. Tiko DI is developed and used as an out-of-training-corpus framework. Its premise is that a framework can be AI-friendly without relying on the model’s prior familiarity, by shipping machine-readable, in-repo affordances instead: guidance (CLAUDE.md, .ai-skills/), compile-time wiring validation, and an MCP topology server. Here it is the test case for the question “can an agent succeed on a framework it was never trained on, given good in-repo affordances?” (Stage-1 answer: yes — 86% median on Tiko 0.2.2, then 100% on 0.5.0 and 0.6.0 across three model generations.)

Headline results — Stage 1

The sharpest result is version recency, isolated cleanly: same spec, same fixed scaffold, the only change is the Spring Boot version. Graded by the external oracle (compliance = scenarios passed / 7), fresh broker per trial, contestants run as isolated agents on the named model. Full write-up: results/RESULTS.md (see “Three-model extension” for the authoritative numbers below).

Spring Boot version Sonnet 4.6 Opus 4.8 Sonnet 5 Fable 5
3.3.5 — in the training corpus 100% (3/3) 100% (3/3) 100% (4/5) 100% (5/5)
4.0.6 — current major (~6 months old) 0% (0/5) 0% (0/5) 0% (0/5) 20% (1/5)

Four model generations, 20 independent trials on the current major — exactly one pass, and it’s the one trial that found the fix. Nineteen trials failed on the identical root cause (a missing KafkaTemplate bean from Boot 4’s Kafka autoconfig reorganization), even when they correctly adopted Jackson 3 and Spring Kafka’s newest serializer classes. The single pass is a Fable 5 trial that added spring-boot-starter-kafka — the actual Boot-4 fix — explicitly reasoning that Boot 4 moved Kafka autoconfig into its own module. Fable carries that knowledge but retrieved it 1 time in 5 in that run; no other model of that generation ever found it.

Today the gap is closed (2026-10-12/13). On the current Boot release, 4.1.1 (the Spring Initializr default; same major, same Kafka module split), the 5.5 models went 30/30 on both fixture sets, and every one of the 30 trials added spring-boot-starter-kafka and used Jackson 3:

Spring Boot version Opus 5.5 Sonnet 5.5 Haiku 5.5
4.1.1 — current release (~2 months old) 100% (10/10) 100% (10/10) 100% (10/10)
4.0.6 — the scaffold the older models scored 0–20% on 100% (10/10) 100% median (9/10) 100% (10/10)

The 4.0.6 control (2026-10-13) shows the patch release isn’t the difference: on the same 4.0.6 scaffold, all 30 builds added spring-boot-starter-kafka. The one failure, a Sonnet 5.5 trial, never started because Spring Data didn’t find its nested repository interfaces, which is unrelated to Boot 4. Both fixture sets gave the same scores.

One caveat on both cells: all trials share one Maven cache, which has held the Boot 4 Kafka starter since June. Five Haiku and Opus trials listed it and added the starter right after seeing it there. Counting those five as failures, the cells score 27/30 on 4.1.1 and 27/30 on 4.0.6. No July trial got the module name this way. Details: results/RESULTS.md → “Maven cache leak”.

But it isn’t cleanly a model-generation effect. Fable 5, the model that scored 1/5 on this scaffold in July, was rerun with the identical prompt and scaffold: 8/10 as a subagent, as in July, 9/10 as a top-level session, and 9/10 as a subagent on Claude Code 2.1.197, the version July ran on (8/10 without one cache-assisted pass). So neither the subagent form nor the Claude Code version explains July. What’s left is the July session itself (an interactive session with user settings, plugins and CLAUDE.md, and trials inside the repo), or a serving-side change under the same model ID. The July results measured the models under the July setup, and part of the 1/20 → 29/30 jump may be that setup rather than the models. Details: results/RESULTS.md → “Fable 5 on the 4.0.6 control”.

These Fable runs don’t represent the current Fable. Fable 5 is the previous model (the current one is Fable 5.1). Its October builds were a harness experiment on Boot 4.0.6 only, under three different harnesses. Don’t compare them with the 5.5 or OpenAI models as a generation benchmark. For the current Fable, see “Fable 5.1” below.

Fable 5.1 (2026-10-16/17). The current Fable ran on all three current cells as a sub-agent, with a per-trial home folder and Maven repository. N=10 per cell; v1 and v2 gave the same scores:

Boot 4.1.1 Boot 4.0.6 Tiko 0.6.0
Fable 5.1 9/10 (8/10 strict) 10/10 10/10
Median cost per build (exact) $2.70 $2.88 $4.50
  • The one failure is the July mistake: plain spring-kafka, so no KafkaTemplate bean.
  • One Boot 4.1.1 pass is cache-assisted. The trial globbed /c/Users/*/.m2, found the shared cache past its own home, and saw the Kafka starter there. Hence 8/10 strict.
  • Tiko costs about 1.6× Spring, through more tool calls.
  • Code quality matches Opus 5.5 on the full three-judge panel (Opus, Sonnet, gpt-6-astra): 22.0, 21.2 and 22.5 of 25 against Opus’s 21.8, 21.8 and 22.5. Fable 5.1 also tests its own work (test quality 4 in 7 of 12 trials, all four on Tiko).

Details: results/RESULTS.md → “Fable 5.1 on all three current cells”.

OpenAI’s models (2026-10-13/15). Three OpenAI models were run through Codex CLI 0.162.1 at medium reasoning. They got the same prompts and isolation as the Claude cells, plus their own Maven repository per trial. N=10 per cell, and v1 and v2 scores were identical in every cell:

Model Boot 4.1.1 Boot 4.0.6 Tiko 0.6.0
gpt-6-astra (frontier) 10/10 10/10 10/10
gpt-6.1-sol (workhorse) 10/10 10/10 10/10
gpt-6-luna (fast) 2/10 2/10 9/10

The frontier and workhorse models clear every cell. The fast model shows this benchmark’s core effect on another vendor: Luna went 4/20 on Spring Boot 4 and 9/10 on Tiko (p = 0.0004). Most of its Spring failures (13 of 16) are the July mistake: plain spring-kafka and no KafkaTemplate bean. Two more needed a Jackson 2 ObjectMapper that Boot 4 no longer provides. On Tiko, the in-repo docs carried it. Tiko builds took about twice the tool calls of Spring builds for astra and Sol, as with the Claude models. Their cost is measured in tokens rather than dollars, since the runs used a ChatGPT Plus plan. The 90 trials used 30.8M input tokens (88% cached) and 550K output tokens: about 340K in and 6K out per build. Per-trial counts are in results/openai-tokens.csv. Details: results/RESULTS.md → “Three OpenAI models on all three current cells”.

On the other axis — an out-of-training-corpus framework that ships in-repo agent docs, on the current release with the current model generation:

Framework Opus 5.5 Sonnet 5.5 Haiku 5.5
Tiko 0.6.0 — v1 fixtures 100% (10/10) 100% (10/10) 100% (15/15)
Tiko 0.6.0 — v2 fixtures 100% (10/10) 100% (10/10) 100% median (13/15)

35/35 on the original fixtures, 33/35 on the stricter v2 set (see fixtures/stage-1/README.md; N=10 for Opus and Sonnet, N=15 for Haiku). Both v2 misses are Haiku trials with the same bug: the bad-message handler looked for an exception Tiko never attaches, so it retried for ~15 s instead of skipping at once — a stall v1 couldn’t see. Tiko 0.6.0’s docs now steer agents toward blocking on bad messages (SEEK), which contradicts the spec’s log-and-skip; all 35 trials noticed and overrode it. Even the smallest current model builds this correctly on a framework it was never trained on, and cheaper models lose almost nothing: median input-side cost per build is $0.95 for Opus 5.5, $0.33 for Sonnet 5.5 and $0.05 for Haiku 5.5.

Tiko’s MCP server doesn’t change the result (2026-10-15). With the scaffold’s tiko-mcp topology server attached and the protocol’s “use it to validate” instruction, gpt-6-luna, Haiku 5.5 and gpt-6.1-sol scored 10/10, 10/10 (8/10 v2) and 10/10. Without MCP they scored 9/10, 8/10 (7/10 v2, same headless harness) and 10/10, and no difference is significant. The agents barely used the server: only 5 of 30 trials called it (Haiku none), and the rest validated against Tiko’s generated topology files. The instruction mainly added effort: Sol used 6× the input tokens and twice the output. Details: results/RESULTS.md → “Tiko 0.6.0 with the MCP topology server”.

Code quality: Tiko builds are better structured (2026-10-16). 72 passing builds were scored by three judges from two vendors (Opus 5.5, Sonnet 5.5, gpt-6-astra), each seeing only blinded code, against a six-dimension rubric (results/reviews/RUBRIC.md).

  • Tiko scores higher than Spring: 21.3 against 19.7 out of 25 (p = 0.002). The lead is in structure and idiomaticity. Spring is slightly better on error handling.
  • Tiko narrows the gaps between models. On Spring, Sonnet 5.5 and gpt-6-luna put everything into one class and score 15–18; on Tiko every model scores 20–22.5.
  • Opus 5.5 is first in every cell of the 72-trial sample. Fable 5.1, judged later by the same panel, is level with it (22.0, 21.2 and 22.5 against 21.8, 21.8 and 22.5).
  • The two vendors tie (20.1 vs 20.4).
  • Only Opus and gpt-6.1-sol test their own work.

Details: results/RESULTS.md → “Code quality of the passing builds”.

Both current releases on the current models: every model now builds the service on Spring Boot 4.1.1 and on Tiko 0.6.0. On the stricter v2 fixtures Spring went 30/30 and Tiko 33/35, and a Spring build costs about half as much, because it takes 1.7–1.9× fewer API calls:

Median input-side cost per build Opus 5.5 Sonnet 5.5 Haiku 5.5
Spring Boot 4.1.1 $0.44 · 12 calls $0.18 · 7 calls $0.013 · 10 calls
Tiko 0.6.0 $0.95 · 23 calls $0.33 · 12 calls $0.047 · 17 calls

(Average API calls per build. Input-side cost is exact; output tokens are only partly recorded in these transcripts, see results/RESULTS.md.)

The previous release, Tiko 0.5.0, was also a clean sweep (15/15) on the previous model generation — Opus 5, Sonnet 5, Fable 5. (Its Opus column was first published as “Opus 4.8”; the trial transcripts show the opus alias had already moved to Claude Opus 5 — corrected 2026-10-11.) It improved on Tiko 0.3.0, where Opus 4.8 and Sonnet 5 each had one failure (both independently reproducing a documentation gap, tiko-di#404, filed from static analysis before those runs). See results/RESULTS.md → “Tiko 0.6.0” and “Tiko 0.5.0”.

Tiko cost per build, release by release

Same spec, same oracle, Stage 1, one row per model. Each cell gives the average cost per build, average API calls per build, and median v1 compliance. API calls don’t depend on price, and call count is what drives cost (see finding 6). Costs are reconstructed from transcripts (conformance/token-accounting.py; read the erratum in results/pricing.md before comparing with older published figures).

Model Tiko 0.2.2 Tiko 0.3.0 Tiko 0.5.0 Tiko 0.6.0
Fable 5 $5.73 · 37 calls · 100% $4.68 · 29 · 100% $2.58 · 14 · 100% —
Sonnet 5 — $4.14 · 77 · 100% $2.34 · 50 · 100% —
Opus 4.8 $2.90 · 41 · 86% $3.11 · 50 · 100% — —
Sonnet 4.6 $2.42 · 93 · 86% — — —
Opus 5 — — $3.54 · 47 · 100% —
Opus 5.5 — — — $1.22¹ · 23 · 100%
Sonnet 5.5 — — — $0.36¹ · 12 · 100%
Haiku 5.5 — — — $0.06¹ · 17 · 100%

¹ Lower bound: the input side is exact, but only ~15% of 0.6.0’s API calls carry a final output-token record. N=10 per 0.6.0 model (Haiku N=15), N=5 elsewhere.

Fable 5 is the only model measured on three Tiko releases. With the model held fixed, cost per build fell 55% ($5.73 → $2.58) and API calls fell 61% (37 → 14) from 0.2.2 to 0.5.0. On the same model, Spring takes 14 calls on Boot 4.0.6 and 11 on Boot 3.3.5, so Tiko 0.5.0 roughly matches Spring on call count. In dollars Tiko is still 1.4–1.8× Spring ($1.90 / $1.47), down from 3.0–3.9× on 0.2.2. What remains is cost per call (~1.3× Spring’s), not number of calls. Sonnet 5 shows the same direction from 0.3.0 to 0.5.0: cost −43%, calls −35%.

Not every step improved: Opus 4.8 cost 7% more on 0.3.0 than on 0.2.2, with more calls (41 → 50). The drop that Fable 5 and Sonnet 5 show falls between 0.3.0 and 0.5.0, the window of Tiko 0.4.0’s agent-documentation overhaul. That is a timing match, not a measured cause. Compliance went from 86% median (Opus 4.8, Sonnet 4.6) on 0.2.2 to 100% on every model since.

What this can’t show: Tiko 0.6.0 was run only on the 5.5 generation, at new prices (Opus 5.5 bills cache reads at 0.05× input). Its builds are the cheapest of any Tiko cell and its call counts the lowest, but no model was run on both 0.5.0 and 0.6.0, so that step can’t be split between Tiko, the newer models and the pricing. The Tiko 0.3.0 and Spring token and cost tables, by model, are in results/RESULTS.md → “Erratum (2026-10-11)”.

What it found

  1. Version recency dominated until the training data caught up — neither size nor tokens fixed it, a newer model generation did. Across four model generations and 20 independent trials, Spring Boot 4.0.6 produced one working app, while the same models scored ~100% on Boot 3.3.5. The failure is silent and idiomatic: every Boot-4 build compiled, but the models reflexively wrote the Boot-3 Kafka idiom (bare spring-kafka + trust auto-configuration) that Boot 4 reorganized away — so apps either failed to start (no KafkaTemplate bean) or booted and exited consuming nothing. Recency bites at version choice (given freedom, the models downgrade — Sonnet 5/5) and at execution (forced onto the new major, they use the old major’s muscle memory). The newest model (Fable 5) is the only one that ever produced the fix (spring-boot-starter-kafka) — once in five tries. The next generation closed the gap: on Boot 4.1.1 (same Kafka module split), Opus, Sonnet and Haiku 5.5 went 30/30, every trial using that starter. The failures were measured 7–9 months after Boot 4.0.0 shipped (2025-11-20); the 5.5 run, about 11 months after, so the blind spot outlasted at least one model generation and closed by the next.

  2. Token spend and correctness are uncorrelated here. Sonnet 5 spent 1.3×–1.8× the real output tokens of Opus 4.8 on identical tasks for the same compliance (deeper bytecode-level exploration that didn’t change outcomes), while Fable 5 got the best compliance results of any model at the highest dollar cost per build. What separated models on compliance was what they knew, not how hard they worked.

  3. An out-of-training-corpus framework with in-repo docs holds up well — and keeps improving. Tiko 0.3.0 reached 100% median for both Opus 4.8 and Sonnet 5, with the sole failures being already-filed documentation gaps (kebab-case vs. camelCase config keys), not framework defects. On the most recent benchmarked release, Tiko 0.5.0, all three models go 15/15 (100%) — that doc gap doesn’t recur anywhere in the newer run. The 0.5.0 docs are also a cost win for the models measured on both versions (Sonnet 5 −43%, Fable 5 −45% per build). The Opus trials on 0.5.0 self-authored an integration test 5/5 times and cost more — but they ran on Opus 5, not the Opus 4.8 measured on 0.3.0, so that change can’t be pinned on the docs — see results/RESULTS.md → “Erratum (2026-10-11)”.

  4. The MCP topology server showed no compliance lift in the original Tiko 0.2.2 run — tiko and tiko-mcp were identical (0 0 86 86 100). The validation gate targeted wiring, which the model already got right; it did not target config, where Tiko’s failures actually concentrated.

  5. Tiko’s dominant failure mode is strict config validation — kebab-case / Spring-style keys that Tiko’s @Configuration records reject. Filed and tracked as tiko-di#404, among 8 doc-friction issues this benchmark surfaced and filed directly against Tiko.

  6. Tiko’s setup-cost gap is a call-count problem, and it has been shrinking. Through 0.3.0, Tiko cost more than Spring on every model, and only ~13% of that gap was output tokens (Tiko has no first-party DB/HTTP module, so the agent hand-writes more code). What dominated was ~2× more turns (API calls) per build, which compounds into ~4× more re-read conversation context (~86% of the gap with cache writes). The agent needed more rounds to discover Tiko’s API surface and satisfy its strict compile-time DI validation, not more lines. That is the axis later releases moved: on Fable 5, Tiko 0.5.0 needs about as many calls as Spring Boot 4.0.6 (14 vs 14) and costs 1.4× instead of 3.0× (see “Tiko cost per build, release by release” above). Full component breakdown: results/RESULTS.md → “Cross-cell cost analysis”.

These are directional results: small N and a few points in time per cell. See results/RESULTS.md → “Three-model extension” for the full breakdown, the grading erratum (a harness bug that produced false negatives, now corrected), and “Caveats / validity”.

Headline results — Stage 2

Stage 1 is one-shot: build, grade, done — the agent never runs its own app. Stage 2 keeps the identical task but lets the agent iterate against a CI-gate (pass/fail per scenario, no expected values — mirrors a black-box integration-test suite) until it believes it complies. Only the Boot 4.0.6 cell is informative (everything else already complies one-shot). Four batches of N=3 on three models. Full write-up, methodology, and the token-accounting errata worth reading: results/RESULTS.md.

Model Converged Iterations to pass Cost to compliance ($, real — input + cache write/read + output)
Fable 5 3/3 2, 2, 1 avg $1.84
Opus 4.8 3/3 2, 2, 2 avg $1.23 — cheapest
Sonnet 5 (batch B) 3/3 2, 1, 2 avg $1.25
Sonnet 5 (batch A) 3/3 3, 2, 3 avg $2.89 — most expensive, by ~2.3×

(Corrected 2026-10-11: costs re-derived from final usage records, and batch A — originally published as “Sonnet 4.6” — ran on Claude Sonnet 5 per its transcripts, so Stage 2 covers three models, with Sonnet 5 sampled twice.)

The version-recency wall is fully iteration-proof: 12/12 trials converge to 100% compliance, across every model tried, in 1–3 gate runs — the same cell that scored 0/5 one-shot for Opus 4.8 and Sonnet 5. The one-shot 0% headline above is a feedback problem, not a knowledge wall: an agent that never runs its own app ships a silently-broken Boot-4 service every time; an agent that runs-and-checks recovers every time, cheaply. Cost tracks iteration count — the single most expensive trial (Sonnet 5 batch A, 3 iterations, $5.22) costs ~2.4× any trial outside its batch, because every extra gate iteration means re-reading a larger accumulated conversation, and that cache-read volume dominates real cost. The two Sonnet 5 batches — same model, same task — differ 2.3× in average cost purely through iteration count.

Haiku 5.5 on Tiko 0.6.0 (2026-10-17): 10/10. The cheapest current model got the same loop on the out-of-training-corpus framework: the Stage 1 Tiko prompt, the CI gate in place of the “don’t run the app” rule, v2 fixtures, at most 8 gate runs. Every trial passed on its first gate run (one ran it again to confirm), and the final oracle grade was 100% on v1 and v2 in all ten. Median cost to compliance was $0.03 per build ($0.02–$0.10, exact sub-agent cost). Since Haiku already builds Tiko almost perfectly in one shot, this confirms rather than rescues; the loop costs it little. The planned Spring counterpart, gpt-6-luna on Boot 4 (2/10 one-shot), is still to run.

Headline results — Stage 3

Stage 3 is a brownfield task: the agent gets a working build of the Stage 1 service and adds full-text product search over HTTP with Apache Lucene 9.11.1 (GET /search?q&category&limit; spec: docs/specs/stage-3-spec.md). It is one-shot, like Stage 1: the agent may run the tests but never the app.

  • Same starting code for every trial. One fixed passing Stage 1 build per framework, both by gpt-6-astra, with the Stage 1 spec renamed SERVICE-SPEC.md and the new feature as TASK-SPEC.md (scaffolds/stage3/). The untouched starting code scores 7/7 on regression and 0/12 on search.
  • Graded in two parts on one app instance: the Stage 1 v2 scenarios (regression, 7) and 12 HTTP search checks (S1–S9 in the spec: whole-word, case-insensitive matching, no stemming, a category filter, a limit, upserts reflected within 5 s, prices read at query time, 400 errors). Grader: conformance/stage-3/grade-stage3.ps1.
  • Harness: sub-agents, as in Stage 1, with a per-trial home folder and Maven repository and a git baseline commit per trial. Trials cut off by harness problems (a network outage, sub-agents launched in the background, orchestrators that didn’t hand off) were archived and rerun; every graded trial ends with the contestant’s own report.

Compliance: 80/80 (5.5 models 2026-10-17/18, Fable 5.1 2026-10-30/31). Every build passes all 7 regression scenarios and all 12 search checks. A 4-trial Sonnet pilot, kept separately in results/metrics-stage3-pilot.csv, was also perfect.

Median per build Boot 4.1.1 Tiko 0.6.0 Tiko ÷ Spring
Fable 5.1 $2.28 · 28 tool calls $3.62 · 41 1.59×
Opus 5.5 $0.66 · 17 $0.84 · 22 1.26×
Sonnet 5.5 $0.26 · 22 $0.28 · 21 1.09×
Haiku 5.5 $0.030 · 31 $0.044 · 36 1.49×

(N=10 per cell, all 10/10. Exact sub-agent cost: the CLI total minus the orchestrator’s share, output included. Per-trial rows: results/metrics-stage3.csv.)

What it found:

  1. Adding a feature to existing code is easy for every current model, on both frameworks. No regressions anywhere, and no search check failed, including Tiko, where the agent hand-builds the HTTP server on the JDK’s com.sun.net.httpserver. Fable 5.1 is no exception: 20/20, at 3.4× (Spring) and 4.3× (Tiko) the cost of Opus 5.5 and with more tool calls; it is also the model with the widest Tiko ÷ Spring gap.
  2. Tiko’s cost penalty almost disappears on existing code: 1.1–1.6× Spring, against 1.8–3.6× (input-side) for building from scratch in Stage 1. A likely reason: the existing build already shows how Tiko wires a component, so the agent doesn’t have to discover it.
  3. The Lucene lifecycle is mostly right. All 80 builds keep one long-lived IndexWriter, and 77 refresh a SearcherManager after each write (one Fable 5.1 Spring build reopens a near-real-time reader with openIfChanged instead, which is equivalent). Two Haiku Spring builds open a new reader per query, and four Spring builds (Opus, Sonnet, two Fable 5.1) commit on every update, which buys nothing on an in-memory index. All are correct, only wasteful.
  4. No build persists the index: 0/80. Every build uses an in-memory ByteBuffersDirectory and rebuilds it from H2 at startup. The spec allows either, and the rebuild is the simpler correct choice at this size.

Code quality of the Stage 3 builds is being scored by the same three-judge panel as Stage 1. It will be published once the remaining Stage 3 runs (OpenAI) and the judging are done. Run notes for Fable 5.1 (usage-limit cuts, pacing): results/RESULTS.md.

Headline results — Stage 3a

Stage 3a gives the agent no spec of the change, only a vague ticket from a junior analyst: “someone mentioned that our search should really have a cache, something like 60 seconds, so it’s faster” (spec and ticket: docs/specs/stage-3a-spec.md). The agent decides what to build and writes a pull-request note to the ticket’s author.

The ticket gives no measurement, and it doesn’t mention the conflict: SEARCH-SPEC.md requires updates to be searchable within 5 s and prices to be read at query time. A plain 60-second cache breaks both rules, so the test is whether the agent notices.

  • Each trial starts from the same model’s own Stage 3 build, with a fresh git baseline. Caffeine 3.1.8 is allowed on both frameworks; nothing else is new.
  • Graded on one app instance in three parts:
    • the 7 Stage 1 regression scenarios;
    • the 12 Stage 3 search checks;
    • 9 checks that fill the cache and then change the data: a price update, a rename, a new product, the cache key, the category and limit filters.
  • Calibration of the cache checks: a deliberately naive 60-second cache passes only 4 of the 9, and also fails one search check. The unchanged Stage 3 builds pass all 9.
  • The change and the note are judged from blinded bundles: the ticket, both specs, the diff, the final code and the agent’s own report.
    • Ticket handling and note quality are each scored 1–5; the decision score is 2 × ticket
      • note, out of 15.
    • Cache code quality (1–5) is scored only for builds that added a cache.
    • A correct cache and a well-argued refusal can both score 5.
    • The judges are Opus 5.5, Sonnet 5.5 and gpt-6-astra, on all 60 trials of the 5.5 models, median per score; the 20 Fable 5.1 trials have the two Claude judges so far. The two Claude judges were within 1 on every score and identical on 118 of 162. Astra was within 1 of Opus on 130 and of Sonnet on 128 of 162; its disagreements sit almost entirely on Sonnet’s cache builds (finding 4).

Compliance: 80/80 (5.5 models 2026-10-18, Fable 5.1 2026-10-31/11-01). Every trial passes all 28 checks. The agents differ in what they chose:

Built a cache · made no code change Boot 4.1.1 Tiko 0.6.0 Median cost, Spring / Tiko
Fable 5.1 10 · 0 10 · 0 $2.23 / $2.43
Opus 5.5 1 · 9 3 · 7 $0.21 / $0.29
Sonnet 5.5 9 · 1 9 · 1 $0.18 / $0.09
Haiku 5.5 10 · 0 10 · 0 $0.017 / $0.018
Median of the judges Asked if a cache is needed Decision, of 15 Note, of 5 Cache code, Spring / Tiko
Opus 5.5 20/20 14 4 5 (1 build) / 4 (3)
Sonnet 5.5 5/20 12 4 4 / 3
Fable 5.1 (Opus + Sonnet judges) 9/20 11.5 / 11.25 3.5 / 3.25 4 / 4
Haiku 5.5 0/20 11 3 4.5 / 5

(N=10 per cell. Cost is the exact sub-agent cost, the CLI total minus the orchestrator’s share; the 5.5 rows were first published as CLI totals ($0.42 / $0.40, $0.23 / $0.14, $0.023 / $0.021) and corrected on 2026-11-01. For the 5.5 models the decision and note medians are the same on both frameworks, and Astra’s judgments, added 2026-10-29, moved the Opus + Sonnet medians by at most 0.5 and changed no ranking. Fable 5.1’s two-judge medians are given as Spring / Tiko; Astra’s judgments are pending.)

What it found:

  1. Every agent saw the conflict, and none shipped a cache that breaks it. All 80 notes name the 5-second and live-price rules. All 62 caches are cleared or versioned on every product and price update, keep 60 s only as an upper limit, and have a size bound. None sends a Cache-Control header.
  2. What separates the models is asking whether a cache is needed. Opus asked in all 20 trials and declined in 16. Its notes say nothing is slow yet (the index and the database are both in memory) and suggest measuring under real load. Several name the likely fix if search gets slow: one price query per page of results instead of one per result. Its 4 cache builds asked the same question, and they score 5 on ticket handling too.
  3. Haiku always builds, cleanly, but explains poorly. Its caches score 4.25–4.5. Its notes score 3: they are written for engineers rather than the analyst and end without a next step. Two state a speed-up nobody measured.
  4. Sonnet’s caches are the weakest, especially on Tiko, and the judges split on how weak.
    • A race the oracle can’t trigger: in 4 of its 18 caches, a search that read old data can store its result just after the cache is cleared, and serve it for up to 60 s. One note admits this as a “known gap”. The Claude judges named 3 of the 4, Astra all 4.
    • Raw NUL bytes: in 7 builds Sonnet wrote them into Java source as cache-key separators, so git and code review treat the file as binary.
    • “No category” and the text null share a key: 6 builds glue the category into the key as a string, so a request with category=null gets the unfiltered result. Only Astra caught it. It is reachable, but only by a client sending that literal value.
    • Astra classed 13 of Sonnet’s 18 caches as “partial invalidation” and scored them 2 or 3 where the Claude judges gave 4, letting the code defect pull the ticket score down too. Besides the defects above, it counts 3 builds that re-check the generation right after the cache write as racy, although their stale window is microseconds, not 60 s. On Astra’s scores alone Sonnet’s decision median is 6 and it falls below Haiku; the three-judge median absorbs the split, keeps Sonnet at 12 and lowers its Tiko cache code from 3.5 to 3.
  5. Cost follows the decision, among the 5.5 models. Opus spends the most of the three and changes the least code: its sub-agent cost is mostly the reading and the note. On Tiko the median ticket costs 0.5–1.4× the Spring one (Sonnet 0.5×, Haiku 1.1×, Opus 1.4×; the first write-up said 0.6–1.0× from the CLI totals).
  6. Fable 5.1 always builds, and builds correctly, but decides like Haiku. 20 of 20 trials add a Caffeine cache; every one is invalidated on product and price updates through a generation counter in the key, and every judge check passes (cache code 4 on both frameworks, with the deductions for extra surface: disable flags, hit counters, an X-Cache header nobody asked for). Every note names the conflict, and 9 of 20 also say the search is already fast and the benefit unmeasured, yet all 20 ship the cache; the two notes that argue the need hardest score 5 on ticket handling. Its decision median (11.5 Spring, 11.25 Tiko) sits between Haiku and Sonnet, far below Opus’s 14, and its notes score 3–3.5: long, written for engineers, with 3 of 20 claiming a speed-up nobody measured. At $2.23 / $2.43 the vague ticket costs Fable 5.1 about ten times what it costs Opus, and on Spring as much as the whole Stage 3 feature ($2.28).

Still to come:

  • gpt-6-astra’s judgments of the 20 Fable 5.1 trials (three-judge medians, as for the 5.5 models).
  • Stage 3a for the OpenAI models, once they have Stage 3 builds.

The cache checks, rubric, judge prompt and per-trial data stay private until those runs are done.

How it works

  • Framework-neutral spec (docs/specs/stage-1-spec.md) — describes what to build, with zero framework words.
  • External conformance oracle (conformance/oracle/, Java) — brings up Kafka, publishes the shared fixtures (fixtures/stage-1/scenarios.json), drains the output topic, and asserts the result. The agent’s own tests don’t count.
  • Asymmetric-native fairness — each framework uses its own first-party stack; neither may copy from example repos. Tiko, having no first-party DB/HTTP module, must hand-build those. Full rules in docs/benchmark-protocol.md.
  • Pinned substrate — apache/kafka 3.8.0, H2 2.2.224, Lucene 9.11.1, JDK 21, identical for everyone, so the only variable is the framework + the agent.

Run it — including with other agents

See RUNNING.md for the canonical, neutral per-trial procedure: copy a golden scaffold, hand the agent only the spec + dependency rules (no hints), let it build, then grade with the oracle. The procedure is agent-agnostic — point Claude, GPT, Gemini, Cursor, etc. at the same scaffold + spec and compare. Running it across agents is the natural next step.

Layout

docs/   benchmark-protocol.md (rules) + specs/ (framework-neutral WHAT)
fixtures/   shared acceptance vectors (input -> expected output)
conformance/   oracle (Java) + docker-compose Kafka + run scripts
scaffolds/   golden per-framework starting points (+ GENERATE.md)
results/   metrics.csv + per-trial oracle JSON + RESULTS.md
runs/   per-trial workspaces (git-ignored; see runs/README.md)

Status

  • Protocol, Stage-1 spec, fixtures, conformance oracle (tests green)
  • Golden scaffolds (Spring Boot 4.0.6 + 3.3.5 reference; Tiko 0.2.2 → 0.3.0)
  • Stage-1 run: spring, tiko, tiko-mcp (5x) + spring3 reference — results/RESULTS.md
  • Clean Spring re-run, fixed scaffold, Sonnet 4.6 + Opus 4.8 (version-recency isolated)
  • Multi-model extension: Sonnet 5 + Fable 5 added, Tiko bumped to 0.3.0 (1/20 pass on Boot 4.0.6 across four models — the one trial that found the new starter; per-model token costs measured)
  • Stage 2 — loop-until-complies harness on the Boot 4.0.6 cell, four batches of N=3 on three models (Opus 4.8, Fable 5, and Sonnet 5 twice — one batch was first published as Sonnet 4.6): 12/12 converged to 100% compliance in 1–3 gate iterations — the one-shot 0% is a feedback problem, not a knowledge wall. Real (transcript-reconstructed, corrected 2026-10-11) cost: Opus 4.8 cheapest (avg $1.23), Sonnet 5 batch A most expensive by ~2.3× (avg $2.89, driven by 3-iteration trials). See results/RESULTS.md → “Stage 2 — loop until complies” and the token-accounting errata in results/pricing.md.
  • Current releases on the current models (2026-10-11/12): Tiko 0.6.0 and Spring Boot 4.1.1 on Opus, Sonnet and Haiku 5.5, graded on v1 and v2 fixtures. Spring 30/30 on both; Tiko 35/35 on v1, 33/35 on v2; Spring about half the cost per build.
  • Spring Boot 4.0.6 control on the 5.5 models (2026-10-13): 29/30 on v1 and v2, every build added spring-boot-starter-kafka, so the 4.0.6 → 4.1.1 change isn’t the difference. Cost and calls match 4.1.1.
  • Fable 5 on the 4.0.6 control (2026-10-13): 9/10 headless and 8/10 as a subagent, against July’s 1/5 with the same prompt, so the subagent form isn’t the cause. Cost $2.14 and $2.24 per build (exact).
  • Fable 5 on Claude Code 2.1.197 (2026-10-13): 9/10 (8/10 without a cache-assisted pass), so the Claude Code version doesn’t explain July’s 1/5. Cost $2.04 per build.
  • Maven cache check (2026-10-13): five 5.5 trials and one Fable 5 trial read the Kafka module name from a listing of the shared ~/.m2; lower bounds 27/30 on both 5.5 Spring cells. No July trial did.
  • Per-trial -Dmaven.repo.local: used for every cell since 2026-10-13 and required by the protocol
  • First non-Claude agent (2026-10-13): gpt-6-astra via Codex CLI 0.162.1 on the 4.0.6 control, 10/10 on v1 and v2, per-trial Maven repositories
  • OpenAI lineup on all three current cells (2026-10-13/15): gpt-6-astra and gpt-6.1-sol 10/10 everywhere; gpt-6-luna 2/10, 2/10 on Boot 4.1.1/4.0.6 and 9/10 on Tiko 0.6.0
  • Tiko 0.6.0 with the MCP topology server (2026-10-15): no significant change for gpt-6-luna, Haiku 5.5 or gpt-6.1-sol; only 5 of 30 trials called the server
  • Optional: an MCP arm that requires the tools, to test whether MCP helps when used
  • Other vendors’ agents (e.g. Gemini), and OpenAI models on the older cells
  • RUNNING.md carries the exact 2026-10 contestant prompts (Spring, Tiko, Tiko MCP arm), checked byte for byte against the trials; the earlier prompt is kept as a dated note
  • Code-quality rubric (2026-10-15): six anchored dimensions with spec-specific checks (results/reviews/RUBRIC.md), a blinded judge prompt (results/reviews/JUDGE.md) and a measured-signals script (conformance/quality/code_metrics.py)
  • Code-quality scoring (2026-10-16): a 72-trial stratified sample of spring-411, spring-406 and tiko-060 passes, judged by Opus 5.5, Sonnet 5.5 and gpt-6-astra; Tiko 21.3 vs Spring 19.7 of 25 (p = 0.002), vendors tie, Opus first in every cell
  • Fable 5.1, the current Fable, on spring-411, spring-406 and tiko-060 (2026-10-16/17): 9/10 (8/10 strict), 10/10 and 10/10 on v1 and v2; code quality level with Opus 5.5 on the three-judge panel (Astra added 2026-10-27: 22.0, 21.2 and 22.5 of 25)
  • Stronger cache isolation for Spring cells: a per-trial home doesn’t stop a /c/Users/*/.m2 glob (e.g. move the shared cache aside during builds)
  • Stage 2 on Tiko 0.6.0 (2026-10-17): Haiku 5.5 10/10, every trial passing on its first gate run, median $0.03 per build
  • Stage 2 — gpt-6-luna on Spring Boot 4 (2/10 one-shot), and a held-out acceptance set to detect gate overfitting
  • Stage 3 spec, starting code and grader (2026-10-17): brownfield Lucene search over HTTP on a fixed Stage 1 build, 7 regression scenarios + 12 search checks
  • Stage 3 runs (2026-10-17/18): Opus, Sonnet and Haiku 5.5 on spring-411 and tiko-060, 60/60 perfect; Tiko costs 1.1–1.5× Spring
  • Stage 3 for Fable 5.1 (2026-10-30/31): 20/20 perfect on spring-411 and tiko-060; median $2.28 and $3.62 per build, the most expensive model and the widest Tiko ÷ Spring gap (1.59×)
  • Stage 3 for the OpenAI models, then the Stage 3 code-quality results
  • Stage 3a spec and runs (2026-10-18): a vague “add a 60 s cache” ticket on each 5.5 model’s own Stage 3 builds; 60/60 pass all 28 checks; Opus declines 16/20 and leads the judging (14 of 15 on the three-judge median, against 12 for Sonnet and 11 for Haiku)
  • Stage 3a: gpt-6-astra judgments (2026-10-27/29): the three-judge medians move the Opus + Sonnet numbers by at most 0.5; Astra alone ranks Sonnet below Haiku
  • Stage 3a for Fable 5.1 (2026-10-31/11-01): 20/20 pass all 28 checks; it builds a cache in every trial and scores 11.5 / 11.25 of 15 on the decision (Opus + Sonnet judges; Astra pending). The 5.5 cost row corrected to exact sub-agent cost
  • Stage 3a: Astra on the Fable 5.1 cases; the OpenAI models; then publish the cache checks and rubric
  • Stage 4 — an internal “corporate” library the models were never trained on, offered documented, with stale docs, or undocumented
  • Optional, last: Fable 5 with the July session context restored (user settings, plugins, CLAUDE.md, trials inside the repo), the remaining candidate for July’s 1/5

License

MIT — see LICENSE.

Continue on GitHub →