ARBITER

Model evidence

Same 6 synthetic proposal cases × 3 repetitions on Nebius Token Factory: Nemotron-3 Super passed 17/18 with 18/18 schema-valid answers and a 5.8 s median; Nemotron-3.5 Lightning passed 16/18 (2 schema failures), was 22.5% cheaper but 4.18× slower (median 24.0 s; 4/18 answers over our 45 s timeout). The room agent runs on Super. Closed-model baseline Claude Sonnet 5 (run through the Claude Code CLI): 16/18 with 2 false blocks, median 10.1 s, $0.0185 per request at API list price, 13.2× Super's cost.

Corpus dfc5a112a711 (S1–S7, D1–D5, document v3); cases T1–T6 frozen after DEV1–DEV2; prompt agent-prompt-v1.1 identical for both models.
response_format json_object; temperature 0.2; max_tokens 8000; timeout 90 s; at most one retry on network/429/5xx (0 retries happened).
Sequential calls with alternating model order; run started 2026-09-23T09:13:21.129Z (UTC) from the venue network. No streaming, so time to first token is N/A; latency = full validated answer including retries.
Scoring: heuristic auto-check against the pre-registered gold rubric (eval/rubric.ts, auto-v3). Answers that fail the server schema count as failures because the product rejects them. A blind A/B sheet is published for human grading (not graded yet).
Prices from GET /v1/models?verbose=true, converted once from USD/token to USD per 1M tokens: Super $0.30 in / $0.90 out, Lightning $0.06 / $0.24. Reasoning tokens are part of output tokens. Cached tokens: Super reported 0, Lightning N/A.

nvidia/nemotron-3-super-120b-a12b

Pass: 17/18
By case: T1 3/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 2/3 · T6 3/3
Schema-valid: 18/18
Critical: 0
False blocks: 0
Latency median / max: 5.8 s / 11.4 s
Tokens in / out (reasoning): 33486 / 16793 (12394)
Total USD: $0.0252
USD per pass: $0.00148

nvidia/Nemotron-3_5-Lightning

Pass: 16/18
By case: T1 3/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 2/3 · T6 2/3
Schema-valid: 16/18
Critical: 0
False blocks: 0
Latency median / max: 24.0 s / 60.0 s
Tokens in / out (reasoning): 33486 / 72860 (67557)
Total USD: $0.0195
USD per pass: $0.00122

claude-sonnet-5 (Claude Code CLI, subscription)

Pass: 16/18
By case: T1 1/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 3/3 · T6 3/3
Schema-valid: 18/18
Critical: 2
False blocks: 2
Latency median / max: 10.1 s / 24.1 s
Tokens in / out (reasoning): 58352 / 17643 (6126)
Total USD: $0.3330
USD per pass: $0.02081

Model (Token Factory)	Pass	By case	Schema-valid	Critical	False blocks	Latency median / max	Tokens in / out (reasoning)	Total USD	USD per pass
nvidia/nemotron-3-super-120b-a12b	17/18	T1 3/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 2/3 · T6 3/3	18/18	0	0	5.8 s / 11.4 s	33486 / 16793 (12394)	$0.0252	$0.00148
nvidia/Nemotron-3_5-Lightning	16/18	T1 3/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 2/3 · T6 2/3	16/18	0	0	24.0 s / 60.0 s	33486 / 72860 (67557)	$0.0195	$0.00122
claude-sonnet-5 (Claude Code CLI, subscription)	16/18	T1 1/3 · T2 3/3 · T3 3/3 · T4 3/3 · T5 3/3 · T6 3/3	18/18	2	2	10.1 s / 24.1 s	58352 / 17643 (6126)	$0.3330	$0.02081

Context ablation (same model Super, same cases T1–T5 × 3, same settings): what the shared room context adds

Full room context (decisions, sources, questions)	14/15 pass · 0 critical errors	T1 3/3, T2 3/3, T3 3/3, T4 3/3, T5 2/3
Document only (like a newcomer's private chat)	3/15 pass · 4 critical errors	T4: changes the date without an approved decision ×3; T3: restores archived budget/filming ×1. T2: both conflicts still found from the document text (6/6); auto-fail there is only the missing S2/S3 citation.

Limits

N = 6 scenarios × 3 repetitions per model in one synthetic domain: not a statistical or general claim.
Both models run on Token Factory; no closed-model baseline was run.
Pass is a heuristic auto-check; blind human grading is pending. Rubric changes after seeing results, applied to all runs: auto-v2 (T4 request for the operations check may be in suggested_safe_revision) and auto-v3 (T1 false block counted once; T4 questions/negations are not availability claims). Nemotron numbers did not change under v3.
Latency was measured over the venue Wi-Fi; Lightning latency varied widely during the run.
Claude baseline ran through the Claude Code CLI on a subscription, not the raw API: The CLI adds ~0.7k tokens of its own context to the input (measured on a probe call). Temperature and JSON mode cannot be set through the CLI; the JSON instruction is only in the prompt. Latency is wall time including CLI start-up; duration_api_ms is recorded separately. Cost is the CLI's API list-price equivalent for the model (costBasis list); subscription usage is not billed per token. A small background Haiku call made by the CLI is excluded.
Model-side ablation with Super, not the human multiplayer experiment (time-to-contribution with volunteers is not measured yet).

Raw files

Generated 2026-09-23T09:13:21.129Z. Synthetic corpus; no personal data.