Agendo robot logo

Agendo benchmark · one-shot-shop-light-v1

OneShotShopLight

Six proven coding models received the same protected Next.js storefront, the same short prompt, isolated workspaces, and at most two hours. Completeness comes from 42 headless acceptance criteria; code quality comes from SonarQube.

Models6
Actual spend$7.085296
Theoretical spend$22.11
Total model time5h 20m
42 scored criteria6 protected API checksHeadless ChromiumTerra medium independent QAOfficial inference preferredActual + theoretical costSonarQube Community Build
How to read this benchmark

Actual cost is the incremental charge for this run. GPT-5.6 Sol and MiniMax M3 show $0 because they used existing Codex and OpenCode Go subscriptions. Theoretical cost reprices their measured native input, cached-input, and output tokens at the frozen public API rates, so included models remain economically comparable.

Cached input is a subset of input, not an additional token bucket. Direct DeepSeek runs use official request-level pricing; OpenRouter runs use provider-native response usage plus generation cost metadata from the pinned official inference provider. Per-request long-context overrides are applied when applicable. A model with higher list prices can therefore cost less when it makes fewer requests, produces fewer output tokens, or benefits more from cache reads.

Time is Agendo wall-clock time from contestant start to exit and excludes controller QA, screenshots, and SonarQube. “Total model time” sums the six individual durations; it is not elapsed experiment time. Prepared API checks are mandatory but never score. The protected backend is excluded from SonarQube, and confirmed bugs are grouped by independent Terra review rather than equated one-for-one with failed criteria.

Leaderboard

RankModel · providerPassPartialFailBugsTimeActualTheoreticalLOCSmellsDebtDup.Team
#1GLM 5.2Z.AI · high41/420111h 9m$3.664656$3.6646561,843171.4 h0%0
#2Grok 4.5xAI · high39/4203253m 7s$3.198722$3.1987223,855363.4 h0%0
#3GPT-5.6 SolOpenAI · medium38/420441h 16m$0.000000$12.99937323 h0%5
#4DeepSeek V4 FlashDeepSeek · high25/42017553m 4s$0.112567$0.1125671,855252.2 h1.3%0
#5MiniMax M3OpenCode Go · off16/42026440m 59s$0.000000$2.0369642,461323 h0%0
#6DeepSeek V4 ProDeepSeek · high0/42042127m 32s$0.109351$0.1093512,496282.7 h0%0

Full results

#1

GLM 5.2

97.6%41/42 weighted

Z.AI · z-ai/fp8 · openrouter-official · high reasoning

41 pass0 partial1 fail1 bugs

Efficiency shorter is better

$3.664656
$3.664656 · metered
1h 9m

Code quality SonarQube

1,843
17
1.4 h
0%
Tokens in / out12.53M / 50.3K
Cache read / write12.37M / 0
Reasoning tokens6.6K
Requests / errors164 / 0
Teammates0
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://openrouter.ai/api/v1/models/z-ai/glm-5.2/endpoints

Sonar findings: 0 static bugs · 0 vulnerabilities · 0 security hotspots.

  • BUG-01 minor The cart omits the Discount summary line until a discount has been applied. K06
#2

Grok 4.5

92.9%39/42 weighted

xAI · xai · openrouter-official · high reasoning

39 pass0 partial3 fail2 bugs

Efficiency shorter is better

$3.198722
$3.198722 · metered
53m 7s

Code quality SonarQube

3,855
36
3.4 h
0%
Tokens in / out7.73M / 98.9K
Cache read / write7.56M / 0
Reasoning tokens13.7K
Requests / errors72 / 0
Teammates0
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://openrouter.ai/api/v1/models/x-ai/grok-4.5/endpoints

Sonar findings: 0 static bugs · 1 vulnerabilities · 0 security hotspots.

  • BUG-01 major The skip link scrolls to the main region but does not move keyboard focus there. H06
  • BUG-02 minor The home page title is ‘Home’ and omits the Northstar Supply site identity. Q03
#3

GPT-5.6 Sol

90.5%38/42 weighted

OpenAI · codex-subscription · medium reasoning

38 pass0 partial4 fail4 bugs

Efficiency shorter is better

$12.99
$0.000000 · included subscription
1h 16m

Code quality SonarQube

937
32
3 h
0%
Tokens in / out10.43M / 84.7K
Cache read / write9.27M / 0
Reasoning tokensn/a
Requests / errors187 / 0
Teammates5
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://developers.openai.com/api/docs/models/gpt-5.6-sol

Sonar findings: 0 static bugs · 0 vulnerabilities · 0 security hotspots.

  • BUG-01 major Skip link navigates to main but does not move keyboard focus there. H06
  • BUG-02 major Catalog reset control does not restore product results after a no-results search. C07
  • BUG-03 minor Product gallery exposes only one meaningful image alternative; alternate media is inaccessible by name. P02
  • BUG-04 major Checkout client-side validation errors are not announced through an alert or live region. O06
#4

DeepSeek V4 Flash

59.5%25/42 weighted

DeepSeek · deepseek-official · deepseek-official · high reasoning

25 pass0 partial17 fail5 bugs

Efficiency shorter is better

$0.112567
$0.112567 · metered
53m 4s

Code quality SonarQube

1,855
25
2.2 h
1.3%
Tokens in / out26.66M / 79.9K
Cache read / write26.55M / 0
Reasoning tokens23.8K
Requests / errors256 / 0
Teammates0
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://api-docs.deepseek.com/quick_start/pricing/

Sonar findings: 0 static bugs · 0 vulnerabilities · 0 security hotspots.

  • BUG-01 critical Server-side API helper hard-codes http://127.0.0.1:3000, so SSR home and product requests fail when QA runs on port 48305. H02, H05, P01, P02, P03, P04, P05, P06, P07, P08
  • BUG-02 minor Cart control has accessible name 'Shopping cart' rather than a Cart-named control required by the acceptance flow, preventing shell and touch-target verification. H01, H07
  • BUG-03 minor Skip link targets a main element that is not programmatically focused after activation. H06
  • BUG-04 minor Cart hides the Discount row when discount is zero instead of always displaying the required totals breakdown. K06
  • BUG-05 major Checkout offers a Credit Card radio instead of an explicitly named Mock Card option, blocking the mock-card checkout and confirmation acceptance flow. O05, O07, O08
#5

MiniMax M3

38.1%16/42 weighted

OpenCode Go · opencode-go · opencode-go · off reasoning

16 pass0 partial26 fail4 bugs

Efficiency shorter is better

$2.036964
$0.000000 · included subscription
40m 59s

Code quality SonarQube

2,461
32
3 h
0%
Tokens in / out31.23M / 78.3K
Cache read / write30.94M / 0
Reasoning tokensn/a
Requests / errors263 / 0
Teammates0
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://opencode.ai/docs/go/

Sonar findings: 0 static bugs · 0 vulnerabilities · 0 security hotspots.

  • BUG-01 major Server-rendered catalog and product requests use a fixed http://127.0.0.1:3000 API base instead of the active application origin, so the QA server on its assigned port renders empty catalog/home data and product not-found pages. H02, H05, C01, C02, C03, C04, C05, C06, C07, P01, P02, P03, P04, P05, P06, P07, P08
  • BUG-02 minor The coupon form itself has an aria-label matching discount/coupon while its input is labeled only 'Have a code?', preventing an unambiguous programmatic coupon input target. K04, K05
  • BUG-03 minor The draft-product not-found screen omits the expected 'not found' or '404' wording despite correctly avoiding a product page. C08
  • BUG-04 minor The home-page title overrides the layout template and omits the Northstar Supply brand. Q03
#6

DeepSeek V4 Pro

0.0%0/42 weighted

DeepSeek · deepseek-official · deepseek-official · high reasoning

0 pass0 partial42 fail1 bugs

Efficiency shorter is better

$0.109351
$0.109351 · metered
27m 32s

Code quality SonarQube

2,496
28
2.7 h
0%
Tokens in / out8.32M / 59.2K
Cache read / write8.25M / 0
Reasoning tokens8.7K
Requests / errors112 / 0
Teammates0
Hard timeoutnot reached
Tech stack, cost basis, and confirmed bugs

Prepared: Next.js 16.2.12 · React 19.2.8 · Tailwind CSS 4.3.3 · Prepared SQLite HTTP API

Added by contestant: No additional detected libraries

Pricing source: https://api-docs.deepseek.com/quick_start/pricing/

Sonar findings: 0 static bugs · 0 vulnerabilities · 0 security hotspots.

  • BUG-01 critical The production build cannot prerender /products because CatalogClient calls useSearchParams() without a Suspense boundary, so Next.js aborts next build and the storefront cannot be deployed or browser-tested under the frozen protocol. H01, H02, H03, H04, H05, H06, H07, C01, C02, C03, C04, C05, C06, C07, C08, P01, P02, P03, P04, P05, P06, P07, P08, K01, K02, K03, K04, K05, K06, K07, K08, O01, O02, O03, O04, O05, O06, O07, O08, Q01, Q02, Q03