# Scored with auto-v3: 2026-09-23T09-13-21_final_T1-T6_x3

Context mode: room; prompt agent-prompt-v1.1; json_mode true; max_tokens 8000; temperature 0.2; timeout 90000 ms; reps 3.
Pass = heuristic auto-check against the frozen gold rubric (see eval/rubric.ts); schema-invalid answers fail because the server rejects them. Blind human grading: blind_sheet.md.

| Model | Runs | Pass | Pass by case | Schema-valid | Critical | False blocks | Conflict recall | Latency median / max | Tokens in / out (reasoning) | Total USD | USD per pass | Provider errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| nvidia/nemotron-3-super-120b-a12b | 18 | 17/18 | T1 3/3, T2 3/3, T3 3/3, T4 3/3, T5 2/3, T6 3/3 | 18/18 | 0 | 0 | 21/21 | 5.8 s / 11.4 s | 33486 / 16793 (12394) | $0.0252 | $0.00148 | 0 |
| nvidia/Nemotron-3_5-Lightning | 18 | 16/18 | T1 3/3, T2 3/3, T3 3/3, T4 3/3, T5 2/3, T6 2/3 | 16/18 | 0 | 0 | 19/21 | 24.0 s / 60.0 s | 33486 / 72860 (67557) | $0.0195 | $0.00122 | 0 |

| Order | Case | Rep | Model | JSON | Schema | Pass | Critical | Latency ms | Tokens in/out | Notes |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | T1 | 1 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 25904 | 1858/5653 | - |
| 2 | T1 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 5499 | 1858/756 | - |
| 3 | T2 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 8265 | 1857/1506 | - |
| 4 | T2 | 1 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 17452 | 1857/3486 | - |
| 5 | T3 | 1 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 33480 | 1852/5393 | - |
| 6 | T3 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 8715 | 1852/791 | - |
| 7 | T4 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 5096 | 1858/768 | - |
| 8 | T4 | 1 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 11222 | 1858/2488 | - |
| 9 | T5 | 1 | nvidia/Nemotron-3_5-Lightning | y | n | n | - | 25001 | 1850/5169 | Model reply does not match the schema: base_version Invalid input: expected number, received undefined; no parseable output |
| 10 | T5 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 5893 | 1850/899 | - |
| 11 | T6 | 1 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 7696 | 1887/805 | - |
| 12 | T6 | 1 | nvidia/Nemotron-3_5-Lightning | y | n | n | - | 23097 | 1887/4166 | Model reply does not match the schema: conflicts.0.explanation Invalid input: expected string, received undefined; no parseable output |
| 13 | T1 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 8485 | 1858/916 | - |
| 14 | T1 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 50512 | 1858/5491 | - |
| 15 | T2 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 46987 | 1857/3793 | - |
| 16 | T2 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 4636 | 1857/829 | - |
| 17 | T3 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 5539 | 1852/920 | - |
| 18 | T3 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 21284 | 1852/3363 | - |
| 19 | T4 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 15964 | 1858/3080 | - |
| 20 | T4 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 4409 | 1858/681 | - |
| 21 | T5 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | n | - | 8460 | 1850/1448 | - |
| 22 | T5 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 13138 | 1850/3350 | - |
| 23 | T6 | 2 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 10908 | 1887/2595 | - |
| 24 | T6 | 2 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 11356 | 1887/1209 | - |
| 25 | T1 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 25100 | 1858/5462 | - |
| 26 | T1 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 8938 | 1858/1252 | - |
| 27 | T2 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 6745 | 1857/955 | - |
| 28 | T2 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 59990 | 1857/4699 | - |
| 29 | T3 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 19691 | 1852/3269 | - |
| 30 | T3 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 5626 | 1852/965 | - |
| 31 | T4 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 4821 | 1858/853 | - |
| 32 | T4 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 42304 | 1858/3281 | - |
| 33 | T5 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 57196 | 1850/4283 | - |
| 34 | T5 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 3689 | 1850/586 | - |
| 35 | T6 | 3 | nvidia/nemotron-3-super-120b-a12b | y | y | y | - | 4151 | 1887/654 | - |
| 36 | T6 | 3 | nvidia/Nemotron-3_5-Lightning | y | y | y | - | 17808 | 1887/3839 | - |