# Scored with auto-v3: 2026-09-23T09-30-59_ablation_doc-only_super

Context mode: doc-only; prompt agent-prompt-v1.1; json_mode true; max_tokens 8000; temperature 0.2; timeout 90000 ms; reps 3.
Pass = heuristic auto-check against the frozen gold rubric (see eval/rubric.ts); schema-invalid answers fail because the server rejects them. Blind human grading: blind_sheet.md.

| Model | Runs | Pass | Pass by case | Schema-valid | Critical | False blocks | Conflict recall | Latency median / max | Tokens in / out (reasoning) | Total USD | USD per pass | Provider errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| nvidia/nemotron-3-super-120b-a12b | 15 | 3/15 | T1 3/3, T2 0/3, T3 0/3, T4 0/3, T5 0/3 | 15/15 | 4 | 0 | 10/18 | 6.2 s / 16.6 s | 16770 / 13604 (10735) | $0.0173 | $0.00576 | 0 |

| Order | Case | Rep | Model | JSON | Schema | Pass | Critical | Latency ms | Tokens in/out | Notes |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | T1 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 6019 | 1121/672 | - |
| 2 | T2 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 6816 | 1120/899 | Unknown source/decision ids: format, budget; server validation: Unknown source/decision ids: format, budget |
| 3 | T3 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 7532 | 1115/1300 | - |
| 4 | T4 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | n | changes the date without an approved decision | 3589 | 1121/560 | - |
| 5 | T5 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 5383 | 1113/680 | - |
| 6 | T1 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 3198 | 1121/468 | - |
| 7 | T2 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 16568 | 1120/2399 | Unknown source/decision ids: format, budget; server validation: Unknown source/decision ids: format, budget |
| 8 | T3 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 10173 | 1115/1224 | Unknown source/decision ids: budget, format; server validation: Unknown source/decision ids: budget, format |
| 9 | T4 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | n | changes the date without an approved decision | 5974 | 1121/655 | - |
| 10 | T5 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 4570 | 1113/626 | - |
| 11 | T1 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 6443 | 1121/611 | - |
| 12 | T2 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 10016 | 1120/1184 | Unknown source/decision ids: format, budget; server validation: Unknown source/decision ids: format, budget |
| 13 | T3 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | n | restores archived budget/filming | 15407 | 1115/1272 | does not mention a new approval |
| 14 | T4 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | n | changes the date without an approved decision | 6164 | 1121/532 | - |
| 15 | T5 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 4064 | 1113/522 | - |