Unrote bench, season 1 - methods, thresholds, data hashes, runs
Published 2026-09-22 with the article https://unrote.ai/articles/bench-season-1/

Scores are per-field or weighted percentages as defined below, not counts of passed items. A test passes when its score meets the bar written in the config before the first run.
Threshold config: unrote_bench/config.py, committed 2026-09-11 (git b6fdfbd), unchanged since; sha256 14cfbbe7d9122bc4... (first 16 hex).


== Scorers (what the score means per test) ==
1 Invoices: % of fields right, amounts within 0.01, dates normalised. Bar 90.
2 Contracts: % of fields right, null counts as a field; parties: each gold party found in the model's list with at most two extras; free text (governing law, liability cap) token-F1 >= 0.6. Bar 85.
3 No answer: number of invented answers out of 20. Bar at most 1.
4 Citation: 70% answer accuracy (token-F1 >= 0.6 or the gold answer contained in a short model answer) + 30% right passage id. Bar 80.
5 Long document: the change is found when the model names the old and the new value; 5 of 5 at the 32k tier is the bar; 64k and 128k reported separately.
6 Inbox routing: 60% macro-F1 on category + 20% urgency + 20% department. Bar 85. (3 wrong categories of 100 read as 97.1, not 97.0.)
8 Languages: no language more than 10 points below English. Layer run: pooled per-item comparison across all multilingual items (n = 3 languages). Bare and Gemini and Claude runs: the earlier per-test comparison (n = 5 tests); the layer's value under that earlier rule was not computed.
9 Format: % of valid JSON answers across all calls. Bar 98.
10 Throughput: output tokens per second at 16 concurrent requests on the card. Bar 300.
V1 Scanned invoice: % of fields right; the ten trap scans reported separately. Bar 85.
V2 Handwriting: % of lines right. Bar 70.
V3 Table: % of cells right. Bar 85.
V4 Moderation: macro-F1 x 100. Bar 85.
V5 Screenshot: % of answers right. Bar 80.
V6 Difference: % found, bar 12 of 15.
V7 No fabrication: 20 images without the asked object, 10 control images with it. Counted: invented answers on the 20, refusals on the 10 controls, unparsable replies. Bar: at most 1 invented and at most 1 refusal.
V8 Speed: seconds per document, 100 receipt photos in a row on one card. Bar 3.0.


== TEXT / Qwen 3.5 35B-A3B FP8, as it ships ==
run file: 2026-09-12-qwen35-35b-a3b-fp8-text.json
started: 2026-09-12T14:33:46+00:00
model: qwen35-35b-a3b-fp8  hosting: self  prompt_version: s1.1  pipeline_version: None
note: RunPod 1x L40S 48GB, vLLM, FP8, tp=1, max-model-len 32k; test 1 v3.1, v2 scorers
datasets (file, sha256 first 12, n): 1: invoice.jsonl c3809722fb44 n=66; 2: contract.jsonl 52757d82578d n=50; 3: rag.jsonl 887e2582043a n=20; 4: rag.jsonl 887e2582043a n=30; 5: long.jsonl 42d66676a33d n=15; 6: triage.jsonl 19f404e15258 n=100; 10: ops.jsonl fcd5199683fa n=1000
  test 1 Invoices from OCR: n=66 score=95.3 bar=>= 90 % fields pass=True calls_per_item=None 
  test 2 Contracts with missing fields: n=50 score=86.7 bar=>= 85 % fields pass=True calls_per_item=None 
  test 3 Questions with no answer: n=20 score=3 bar=<= 1 invented answers pass=False calls_per_item=None {'invented': 3, 'unparsed': 0}
  test 4 Answer with a citation: n=30 score=87.3 bar=>= 80 points pass=True calls_per_item=None 
  test 5 A 128k document in a 32k window: n=15 score=100.0 bar=>= 100 % of 32k band pass=True calls_per_item=None 
  test 6 Inbox routing, short requests: n=100 score=57.1 bar=>= 85 points pass=False calls_per_item=None 
  test 8 Drop across DE/ES/FR vs EN: n=5 score=29.9 bar=<= 10 points below en pass=False calls_per_item=None 
  test 9 Strict output format: n=281 score=96.4 bar=>= 98 % valid JSON pass=False calls_per_item=None 
  test 10 Throughput on the card: n=192 score=368.6 bar=>= 300 output tok/s at concurrency 16 pass=True calls_per_item=None 
  throughput sweep (concurrency: output tok/s, total tok/s, $ per 1M total tokens, first token p50): 1: 54.0, 469.7, 0.6446, 0.226 s; 16: 368.6, 3181.2, 0.0952, 0.388 s; 64: 663.0, 5754.9, 0.0526, 1.078 s

== TEXT / Qwen 3.5 35B-A3B FP8 + Unrote layer p1.2 ==
run file: 2026-09-13-qwen35-35b-a3b-fp8-text-pipeline-5.json
started: 2026-09-13T00:45:38+00:00
model: qwen35-35b-a3b-fp8  hosting: self  prompt_version: s1.1  pipeline_version: p1.2
note: FROZEN season run, pipeline p1.2: deterministic triage votes, sharper label definitions, pooled multilingual rule; RunPod 1x RTX PRO 6000 MIG 48GB, vLLM FP8, 32k
datasets (file, sha256 first 12, n): 1: invoice.jsonl c3809722fb44 n=66; 2: contract.jsonl 52757d82578d n=50; 3: rag.jsonl 887e2582043a n=20; 4: rag.jsonl 887e2582043a n=30; 5: long.jsonl 42d66676a33d n=15; 6: triage.jsonl 19f404e15258 n=100; 10: ops.jsonl fcd5199683fa n=1000
  test 1 Invoices from OCR: n=66 score=99.6 bar=>= 90 % fields pass=True calls_per_item=1.08 
  test 2 Contracts with missing fields: n=50 score=90.2 bar=>= 85 % fields pass=True calls_per_item=7.08 
  test 3 Questions with no answer: n=20 score=0 bar=<= 1 invented answers pass=True calls_per_item=1.7 {'invented': 0, 'unparsed': 0}
  test 4 Answer with a citation: n=30 score=81.7 bar=>= 80 points pass=True calls_per_item=2.47 
  test 5 A 128k document in a 32k window: n=15 score=100.0 bar=>= 100 % of 32k band pass=True calls_per_item=8.4 
  test 6 Inbox routing, short requests: n=100 score=97.1 bar=>= 85 points pass=True calls_per_item=3.0 
  test 8 Drop across DE/ES/FR vs EN: n=3 score=6.2 bar=<= 10 points below en pass=True calls_per_item=None 
  test 9 Strict output format: n=281 score=100.0 bar=>= 98 % valid JSON pass=True calls_per_item=None 
  test 10 Throughput on the card: n=192 score=410.7 bar=>= 300 output tok/s at concurrency 16 pass=True calls_per_item=None 
  throughput sweep (concurrency: output tok/s, total tok/s, $ per 1M total tokens, first token p50): 1: 60.0, 524.2, 0.9486, 0.192 s; 16: 410.7, 3545.0, 0.1403, 0.354 s; 64: 783.9, 6815.1, 0.073, 1.014 s

== TEXT / Gemini 3.8 Flash, as it ships ==
run file: 2026-09-12-gemini-3.8-flash-text-3.json
started: 2026-09-12T23:05:50+00:00
model: gemini-3.8-flash  hosting: cloud  prompt_version: s1.1  pipeline_version: None
note: FROZEN season run, raw lane, current data and scorers; Google AI Studio free tier
datasets (file, sha256 first 12, n): 1: invoice.jsonl c3809722fb44 n=66; 2: contract.jsonl 52757d82578d n=50; 3: rag.jsonl 887e2582043a n=20; 4: rag.jsonl 887e2582043a n=30; 5: long.jsonl 42d66676a33d n=15; 6: triage.jsonl 19f404e15258 n=100
  test 1 Invoices from OCR: n=66 score=99.2 bar=>= 90 % fields pass=True calls_per_item=None 
  test 2 Contracts with missing fields: n=50 score=79.5 bar=>= 85 % fields pass=False calls_per_item=None 
  test 3 Questions with no answer: n=20 score=1 bar=<= 1 invented answers pass=True calls_per_item=None {'invented': 1, 'unparsed': 0}
  test 4 Answer with a citation: n=30 score=80.3 bar=>= 80 points pass=True calls_per_item=None 
  test 5 A 128k document in a 32k window: n=15 score=100.0 bar=>= 100 % of 32k band pass=True calls_per_item=None 
  test 6 Inbox routing, short requests: n=100 score=91.2 bar=>= 85 points pass=True calls_per_item=None 
  test 8 Drop across DE/ES/FR vs EN: n=5 score=25.0 bar=<= 10 points below en pass=False calls_per_item=None 
  test 9 Strict output format: n=281 score=100.0 bar=>= 98 % valid JSON pass=True calls_per_item=None 
  test 10 Throughput on the card: n=None score=None bar=>= 300 output tok/s at concurrency 16 pass=None calls_per_item=None 

== TEXT / Claude Fable 5.1, as it ships (agent harness, no API) ==
run file: 2026-09-12-claude-fable-5.1-text.json
started: 2026-09-12T08:30:30+00:00
model: claude-fable-5.1  hosting: cloud  prompt_version: s1.0  pipeline_version: None
note: answered by Claude Fable 5.1 agents from the exported prompts, no API
datasets (file, sha256 first 12, n): 1: invoice.jsonl 9f661458b623 n=56; 2: contract.jsonl 52757d82578d n=50; 3: rag.jsonl 887e2582043a n=20; 4: rag.jsonl 887e2582043a n=30; 5: long.jsonl 42d66676a33d n=15; 6: triage.jsonl 19f404e15258 n=100
  test 1 Invoices from OCR: n=56 score=100.0 bar=>= 90 % fields pass=True calls_per_item=None 
  test 2 Contracts with missing fields: n=50 score=88.4 bar=>= 85 % fields pass=True calls_per_item=None 
  test 3 Questions with no answer: n=20 score=0 bar=<= 1 invented answers pass=True calls_per_item=None {'invented': 0, 'unparsed': 0}
  test 4 Answer with a citation: n=30 score=82.7 bar=>= 80 points pass=True calls_per_item=None 
  test 5 A 128k document in a 32k window: n=15 score=100.0 bar=>= 100 % of 32k band pass=True calls_per_item=None 
  test 6 Inbox routing, short requests: n=100 score=97.9 bar=>= 85 points pass=True calls_per_item=None 
  test 8 Drop across DE/ES/FR vs EN: n=5 score=5.0 bar=<= 10 points below en pass=True calls_per_item=None 
  test 9 Strict output format: n=271 score=100.0 bar=>= 98 % valid JSON pass=True calls_per_item=None 
  test 10 Throughput on the card: n=None score=None bar=>= 300 output tok/s at concurrency 16 pass=None calls_per_item=None 

== IMAGE / Qwen 3.5 35B-A3B FP8, as it ships ==
run file: 2026-09-14-qwen35-35b-a3b-fp8-vision-2.json
started: 2026-09-14T07:07:51+00:00
model: qwen35-35b-a3b-fp8  hosting: self  prompt_version: s1.1  pipeline_version: None
note: RTX PRO 6000 MIG 48GB, vLLM, 32k, картинки на той же текстовой сборке
datasets (file, sha256 first 12, n): V1: vision\v1_invoice_scan.jsonl 6be758f28d44 n=50; V2: vision\v2_handwriting.jsonl 621236c82224 n=20; V3: vision\v3_tables.jsonl 8cc97bb98d40 n=20; V4: vision\v4_moderation.jsonl 6e8e91dfc12b n=100; V5: vision\v5_screens.jsonl 9d54c9944478 n=20; V6: vision\v6_diff.jsonl 7791e2eaa4be n=15; V7: vision\v7_null.jsonl 558093cfded5 n=30; V8: vision\v8_speed.jsonl 0f819c9d1e75 n=100
  test V1 Scanned invoice: n=50 score=98.4 bar=>= 85 % fields pass=True calls_per_item=None 
  test V2 Handwritten note: n=20 score=15.0 bar=>= 70 % lines pass=False calls_per_item=None 
  test V3 Table as an image: n=20 score=85.4 bar=>= 85 % cells pass=True calls_per_item=None 
  test V4 Photo moderation: n=100 score=100.0 bar=>= 85 macro-F1 x100 pass=True calls_per_item=None 
  test V5 UI screenshot: n=20 score=90.0 bar=>= 80 % answers pass=True calls_per_item=None 
  test V6 Spot the difference: n=15 score=100.0 bar=>= 80 % found (12 of 15) pass=True calls_per_item=None 
  test V7 No fabrication from an image: n=30 score=2 bar=<= 1 invented answers pass=False calls_per_item=None {'invented': 2, 'unparsed': 21}
  test V8 Speed, seconds per document: n=100 score=1.18 bar=<= 3.0 s per document pass=True calls_per_item=None 

== IMAGE / Qwen 3.5 35B-A3B FP8 + Unrote layer p1.4 ==
run file: 2026-09-14-qwen35-35b-a3b-fp8-vision-pipeline-19.json
started: 2026-09-14T08:21:59+00:00
model: qwen35-35b-a3b-fp8  hosting: self  prompt_version: s1.1  pipeline_version: p1.4
note: RTX PRO 6000 MIG 48GB, vLLM, 32k, конвейер по картинкам p1.4 (увеличение страницы + голосование в V3)
datasets (file, sha256 first 12, n): V1: vision\v1_invoice_scan.jsonl 6be758f28d44 n=50; V2: vision\v2_handwriting.jsonl 621236c82224 n=20; V3: vision\v3_tables.jsonl 8cc97bb98d40 n=20; V4: vision\v4_moderation.jsonl 6e8e91dfc12b n=100; V5: vision\v5_screens.jsonl 9d54c9944478 n=20; V6: vision\v6_diff.jsonl 7791e2eaa4be n=15; V7: vision\v7_null.jsonl 558093cfded5 n=30; V8: vision\v8_speed.jsonl 0f819c9d1e75 n=100
  test V1 Scanned invoice: n=50 score=100.0 bar=>= 85 % fields pass=True calls_per_item=1.28 
  test V2 Handwritten note: n=20 score=15.0 bar=>= 70 % lines pass=False calls_per_item=None 
  test V3 Table as an image: n=20 score=89.4 bar=>= 85 % cells pass=True calls_per_item=3.05 
  test V4 Photo moderation: n=100 score=99.0 bar=>= 85 macro-F1 x100 pass=True calls_per_item=1.0 
  test V5 UI screenshot: n=20 score=95.0 bar=>= 80 % answers pass=True calls_per_item=1.0 
  test V6 Spot the difference: n=15 score=100.0 bar=>= 80 % found (12 of 15) pass=True calls_per_item=1.0 
  test V7 No fabrication from an image: n=30 score=1 bar=<= 1 invented answers pass=False calls_per_item=2.0 {'invented': 1, 'unparsed': 0}
  test V8 Speed, seconds per document: n=100 score=1.2 bar=<= 3.0 s per document pass=True calls_per_item=1.0 

== IMAGE / Gemini 3.8 Flash, as it ships ==
run file: 2026-09-12-gemini-3.8-flash-vision.json
started: 2026-09-12T09:36:21+00:00
model: gemini-3.8-flash  hosting: cloud  prompt_version: s1.0  pipeline_version: None
note: Google AI Studio free tier; reasoning low, max_tokens 4096
datasets (file, sha256 first 12, n): V1: vision\v1_invoice_scan.jsonl 82810dd85f1e n=40; V2: vision\v2_handwriting.jsonl 621236c82224 n=20; V3: vision\v3_tables.jsonl 8cc97bb98d40 n=20; V4: vision\v4_moderation.jsonl 6e8e91dfc12b n=100; V5: vision\v5_screens.jsonl 9d54c9944478 n=20; V6: vision\v6_diff.jsonl 7791e2eaa4be n=15; V7: vision\v7_null.jsonl 558093cfded5 n=30
  test V1 Scanned invoice: n=40 score=100.0 bar=>= 85 % fields pass=True calls_per_item=None 
  test V2 Handwritten note: n=20 score=40.0 bar=>= 70 % lines pass=False calls_per_item=None 
  test V3 Table as an image: n=20 score=78.6 bar=>= 85 % cells pass=False calls_per_item=None 
  test V4 Photo moderation: n=100 score=100.0 bar=>= 85 macro-F1 x100 pass=True calls_per_item=None 
  test V5 UI screenshot: n=20 score=90.0 bar=>= 80 % answers pass=True calls_per_item=None 
  test V6 Spot the difference: n=15 score=100.0 bar=>= 80 % found (12 of 15) pass=True calls_per_item=None 
  test V7 No fabrication from an image: n=30 score=4 bar=<= 1 invented answers pass=False calls_per_item=None {'invented': 4, 'unparsed': 0}
  test V8 Speed, seconds per document: n=None score=None bar=<= 3.0 s per document pass=None calls_per_item=None 

== IMAGE / Claude Fable 5.1, as it ships (agent harness, no API) ==
run file: 2026-09-12-claude-fable-5.1-vision.json
started: 2026-09-12T08:30:31+00:00
model: claude-fable-5.1  hosting: cloud  prompt_version: s1.0  pipeline_version: None
note: answered by Claude Fable 5.1 agents from the exported prompts and images, no API; V8 speed not applicable
datasets (file, sha256 first 12, n): V1: vision\v1_invoice_scan.jsonl 82810dd85f1e n=40; V2: vision\v2_handwriting.jsonl 621236c82224 n=20; V3: vision\v3_tables.jsonl 8cc97bb98d40 n=20; V4: vision\v4_moderation.jsonl 6e8e91dfc12b n=100; V5: vision\v5_screens.jsonl 9d54c9944478 n=20; V6: vision\v6_diff.jsonl 7791e2eaa4be n=15; V7: vision\v7_null.jsonl 558093cfded5 n=30
  test V1 Scanned invoice: n=40 score=87.5 bar=>= 85 % fields pass=True calls_per_item=None 
  test V2 Handwritten note: n=20 score=55.0 bar=>= 70 % lines pass=False calls_per_item=None 
  test V3 Table as an image: n=20 score=93.8 bar=>= 85 % cells pass=True calls_per_item=None 
  test V4 Photo moderation: n=100 score=100.0 bar=>= 85 macro-F1 x100 pass=True calls_per_item=None 
  test V5 UI screenshot: n=20 score=95.0 bar=>= 80 % answers pass=True calls_per_item=None 
  test V6 Spot the difference: n=15 score=100.0 bar=>= 80 % found (12 of 15) pass=True calls_per_item=None 
  test V7 No fabrication from an image: n=30 score=1 bar=<= 1 invented answers pass=True calls_per_item=None {'invented': 1, 'unparsed': 0}
  test V8 Speed, seconds per document: n=None score=None bar=<= 3.0 s per document pass=None calls_per_item=None 

== IMAGE / table test V3, bare lane, all runs ==
  2026-09-14-qwen35-35b-a3b-fp8-vision-12.json: 82.8 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-13.json: 83.9 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-14.json: 83.2 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-15.json: 82.0 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-16.json: 84.3 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-2.json: 85.4 pass=True
  2026-09-14-qwen35-35b-a3b-fp8-vision-3.json: 82.0 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-4.json: 84.3 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-5.json: 82.8 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-6.json: 82.9 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-7.json: 82.3 pass=False
  2026-09-14-qwen35-35b-a3b-fp8-vision-8.json: 83.7 pass=False
  12 runs, median 83.1, range 82.0-85.4, 11 below the bar of 85. The article reports the median. No other cell has a published series.

== Notes ==
Gemini 3.8 Flash: called on 2026-09-12 through Google AI Studio API keys using the OpenAI-compatible endpoint, reasoning at minimum, max output 4096 tokens. The run files say "free tier"; by the key statistics of the runs most requests were served by a paid key from the same project. Text run: the same prompts (s1.1) and the same frozen files as the open model (hashes above); the long-document test was given each document whole in its native window. Image run: prompt version s1.0, which differs from s1.1 only in the invoice test's output schema, and it predates the ten trap scans (40 items instead of 50). Gemini was not run through the Unrote layer.
Claude Fable 5.1: answered the exported prompts (version s1.0) through an agent harness on 2026-09-12, not through the API; the harness is a layer of its own kind, so this lane is not strictly "bare". Invoice test on the earlier 56-item set (hash 9f661458b623) before the ten traps; image sets are the earlier versions (hashes above). Speed and price were not measured; list price $10.40 per 1M tokens blended at the document mix.
Layer prompts: the layer run (p1.2) uses sharper category definitions and examples in the routing prompt and per-field definitions in the contract prompt; the cloud lanes ran the plain s1.1 / s1.0 prompts. Part of the routing gain is therefore prompt work the cloud models did not receive.
Token mix: the throughput sweep prompts are about 88% input tokens (output/total 11.5% at every concurrency). The document tests are about 98% input (Gemini text run: 1,149,276 input, 13,520 output tokens). Cloud list prices in the article are blended at the document mix.
Text lanes: the bare run was on a RunPod L40S 48 GB (2026-09-12), the layer run on a RunPod RTX PRO 6000 48 GB MIG partition (2026-09-13); quality tests do not depend on the card, the throughput row does. Image lanes: both Qwen lanes on the RTX PRO 6000 partition (2026-09-14). GPU price used in the article: $1.79 per hour for the partition. The L40S sweep is listed above for comparison.
Invoice data: FATURA's printed totals are random in the source; the text was rebuilt so that net + VAT = total to the cent before OCR. The arithmetic check is therefore scored on data that adds up by construction; the ten trap invoices (two currencies, credit note, several VAT rates, Net 30, a foreign total in the footer), derived by code with gold recomputed, are what tests the rules.
Test 5, bare lane: the 64k and 128k items failed with HTTP 400 from the server (request larger than the 32k window); the bar for the test is 5 of 5 at 32k, which the bare lane meets.
Test 7 (billable hours) was cut from the season before the first run.
Article tables (from 2026-09-23): the season-1 tables in the article list 9 text and 6 image tests. V2 Handwritten note and V7 No fabrication from an image are reported in the article's update section instead, with their season-1 numbers, because their season-1 sets were found flawed and rebuilt (SEASON 2 ADDENDUM below). The full season-1 runs of both, every lane, remain in this file unchanged.
Failure log entries are exactly as recorded, including items that models got wrong in the same way (ct-004, rag-en-022, v7-008) and items the layer did not fix (ct-004 term, v7-008).
Contact for the full run files: contact@unrote.ai


== SEASON 2 ADDENDUM (2026-09-23): two tests rebuilt ==
Season-1 numbers above are unchanged. The two sets below are new files; their season-1 versions are kept as v2_handwriting_s1.jsonl (621236c82224) and v7_null_s1.jsonl (558093cfded5, image paths changed only). Bars unchanged (same config hash).

V2 Handwritten note, v2: source RA-Data-Science/modern-danish-handwriting (HF, CC BY 4.0; pages written 2025 by volunteers of the Danish National Archives, transcriptions proofread, line polygons in PAGE XML). One line per page, 10 lines with a digit + 10 without, lengths 8-70 characters, crops with a 10 px margin; 32 candidates reviewed by eye against the transcription, 6 rejected for a mismatch (extra word, missing word, polygon over three lines, wrong line), 20 kept. Build: scripts/build_v2_v2.py (seed 20260923). File vision/v2_handwriting.jsonl d8f27ae2094f n=20, images f2745b07142f. Tuning split: 40 lines from other pages (seed 20260913), not reviewed by eye, never scored.
V7 No fabrication, v2: source POPE adversarial as before; categories the COCO taxonomy confuses removed (car/truck/bus, skis/snowboard, hot dog, cup/bowl, chair/couch, ...), at most 2 questions per photo, 20 absent + 10 present, 54 candidates reviewed by eye, 30 kept (28 photos). Build: scripts/build_v7_v2.py (seed 20260923). File vision/v7_null.jsonl 88e438359032 n=30, images c057fa3853e8. Tuning split: 60 questions on other photos (seed 20260913); three of its POPE labels were corrected by eye after a run disputed them (flower labelled bird, toilet out of frame, a real dog labelled absent); never scored.
Known label issues found after freezing, kept until the season ends: v2-003 gold "på 1, 3 mia. kr.." (space inside the number, double stop; the ink reads "1,3"); v7-023 "carrot: absent" while the coleslaw on the plate shows shredded carrot.

Hardware: RunPod 1x L40S 48 GB, vLLM, Qwen 3.5 35B-A3B FP8, max-model-len 32000, temperature 0, prompts s1.1. One run per lane on each frozen set; the layer was tuned on the tuning splits only.

== IMAGE season 2 / Qwen 3.5 35B-A3B FP8, as it ships ==
  test V2 Handwritten note: n=20 score=80.0 bar=>= 70 % lines pass=True mean_token_f1=0.771 (results/2026-09-23-qwen35-35b-a3b-fp8-vision-2.json)
  test V7 No fabrication from an image: n=30 score=1 bar=<= 1 invented answers pass=False {'invented': 1, 'controls_unanswered': 2, 'unparsed': 21} (results/2026-09-23-qwen35-35b-a3b-fp8-vision.json)

== IMAGE season 2 / Qwen 3.5 35B-A3B FP8 + Unrote layer ==
  test V2 Handwritten note (layer p1.6): n=20 score=80.0 bar=>= 70 % lines pass=True mean_token_f1=0.805 calls_per_item=1.45 tokens/item in 1714 out 31; trace numbers:checked 9, numbers:count_differs 3 (results/2026-09-23-qwen35-35b-a3b-fp8-vision-pipeline-2.json)
  test V7 No fabrication from an image (layer p1.5): n=30 score=1 bar=<= 1 invented answers pass=True {'invented': 1, 'controls_unanswered': 0, 'unparsed': 0} calls_per_item=2.07 tokens/item in 1384 out 74; trace agree:absent 19, agree:visible 9, look:null 21, ground:found_after_refusal 2, recheck:found 2 (results/2026-09-23-qwen35-35b-a3b-fp8-vision-pipeline.json)

Layer changes: V7 p1.5 = look pass (name the object, say if visible, then answer) + a bounding-box pass (box the object or found: no); the two agreeing settle the item, a third call (verify or recheck, as in p1.4) breaks a tie. p1.4 (look + verify on every answer) lost controls to the verify pass alone. V2 p1.6 = crop enlarged to 2400 px long side, the language of the client's documents stated in the prompt with an instruction not to translate or change alphabet, answer as a JSON field under guided decoding; a second call re-reads the numbers of the line when the first read contains a digit and replaces digits only when sure and the count matches. Tried on the tuning split and dropped: the digit pass on every line with a hinted re-read (92.5 vs 95.0 on the split).

Failures, season 2 (id | gold | reply):
  V2 bare: v2-001 En 28-årig restauratør fik for 21. | En 28-årig restauratör fick för 21.; v2-003 på 1, 3 mia. kr.. | на 1,3 т.кв.; v2-008 med en 43-årig repræsentant. | ты м из ас пером был.; v2-009 - uge 47" og "DUPLO legetøj". UGENS | - Lge 47" og DUPLO Legetøj" UGENS
  V2 layer: v2-003 på 1, 3 mia. kr.. | på 1,3 mia. kr.; v2-008 med en 43-årig repræsentant. | til m 43- årg Purla m bid.; v2-009 - uge 47" og "DUPLO legetøj". UGENS | - Uge 47" og DUPLO Legetøj" UGENS; v2-013 meget godt i dag. | meset godt i dag,
  V7 bare: v7-023 carrot, absent | orange (invented); v7-001 person, present | unparsed; v7-002 bed, present | unparsed; plus 19 unparsed replies on absent items (counted as not invented)
  V7 layer: v7-023 carrot, absent | orange (invented; see label note above)