We measured an open model on 15 document tests. It failed five. Here is what fixed all five, and what it costs per thousand documents

This is for companies with a steady stream of documents, hundreds a day and up, or documents that cannot leave the building. For a few requests an hour the card is started for the batch and stopped after it, or a cloud API takes the single requests; the price section has the numbers.

Few decisions about which model to run on company documents are made by measuring on the company's own tasks. The mistake surfaces six months later, in the reconciliation queue and on the cloud invoice. A business pays for three things: quality, meaning the answer can go downstream without a human re-checking it; speed, meaning the system holds a real stream and not one demo; and price, meaning what that costs at volume. If any of the three drops, the rollout does not happen.

The same mistakes

A contract between HERC Rentals and Hertz System. The document has no term. The open model wrote "48 months". Gemini 3.8 Flash wrote "48 months" too. So did Claude Fable 5.1 in its own run. All three put the same third party, The Hertz Corporation, at the head of a list that the contract fills with two. With our layer the invented notice period went away; the 48 months and the extra party did not. That item is in the failure log.

A question about what biogeochemical pathways are used to decipher. The passage does not say. The open model answered "Earth history and the processes that occur on and inside the Earth". Gemini answered "Earth history". Neither could say "not in the document". The open model also answered "west side" to which side State Route 99 is built on, which the text does not say either; Gemini did not take that one.

These are how models behave when nothing checks them, whichever model it is. What we sell is the checking: the tests, the rules a document obeys, and the layer that enforces them around whatever model runs underneath.

Nine tests, three lanes

The layer is what we sell. It works around any model; the open one is what makes it seven cents per million tokens and lets it run on your own hardware.

One open model, Qwen 3.5 35B-A3B (weights stored at 8 bits, FP8), served by vLLM with a 32k window, which is about 24,000 words per read. Two lanes: the model as it ships, and the same model with our layer around the call. Same weights, same frozen items; the layer adds calls, checks and sharper task definitions, and those are part of what is being measured. Gemini 3.8 Flash ran the same frozen text items with the same prompts, as it ships, through its API; it was not run through the layer.

On the eight quality tests alone, leaving throughput aside: 4 of 8 as it ships, 8 of 8 with the layer, 6 of 8 for Gemini. Test 7, billable hours, was cut from the season before the first run, which is why the numbering skips. Scores are per-field or weighted percentages as defined per test in methods.txt, not counts of items; item counts are in the test names. On sets of 15 to 30 items one item moves the score by 3 to 7 points, so a pass by two points is a pass by one document; the bars are pass-or-fail screens set in advance, not measurements of a difference.

Out of the box: five of nine. Three fabrications out of twenty questions with no answer. Inbox routing at 57 against a bar of 85. Spanish 30 points behind English. Output format valid 96.4% of the time, and the malformed 4% is enough to stall a document pipeline. The bare model cannot take a document past its window: the server rejects the 64k and 128k requests outright; the bar for that test only asks for the 32k tier, and the 64k and 128k results are the layer's extra. Speed and price were fine from the start; the whole problem was quality.

Of the four flips, one is cheap: strict output format is guided decoding, where the server forces the output shape, a setting. The three that cost weeks are fabrications, inbox routing and the language drop; the routing gain includes sharper category definitions in the prompt, which Gemini did not get, and the language row for the layer is scored under the pooled rule described in methods.txt. One test got worse with the layer, which is covered below. The throughput row is the server, not the layer: the bare text run was on a rented L40S, the layer run on a rented RTX PRO 6000 partition. The layer makes more than one call: 1.1 per invoice, 7.1 per contract, 1.7 and 2.5 per question, 8.4 per long document, 3 per routed request. The per-document prices below include those calls.

Claude Fable 5.1, for scale: it answered the exported prompts through an agent harness rather than the API, on the previous prompt version and on the invoice set before the ten traps, so it is not the same exam and it is not in the tables. Under those conditions it passed all eight quality tests on text and five of five on images. It lists at about $10.40 per million tokens and cannot run inside your perimeter. The count the layer reached on the open model is that count, at a different price and in a different place.

How it was measured

Two rented cards: the bare text run on an L40S, everything else on a 48 GB partition of an RTX PRO 6000 at $1.79 an hour for the partition; every price below is from the partition. Temperature zero, reasoning off, prompts not tuned per model. Datasets are frozen and hashed on every run; tuning happened on a separate practice set from the same public sources, and a test checks that no item overlaps the exam. Thresholds sit in a config file committed on 11 September 2026, the day before the first run, and unchanged since. The frozen sets were run through the layer repeatedly as it was built, five times on text and nineteen on images, most of the image runs being reruns of the table test alone; every run is saved and no change was made against a specific frozen item. Season two adds a held-out set that no run touches until the season ends.

The layer is what we sell. It works around any model; the open one is what makes it seven cents per million tokens and lets it run on your own hardware.

The sources are public: FATURA and CORD for invoices, rendered to A4, 30% degraded, and read by a real OCR engine; CUAD for contracts and the long document; SQuAD 2.0 and XQuAD for questions; Amazon MASSIVE for inbox requests, because no real mail set in four languages has a usable licence; PubTabNet and POPE for images. FATURA's own totals are random, so the text was rebuilt to add up before OCR; the arithmetic check is therefore scored on data that adds up by construction, and the ten trap invoices are what tests the rules. How Gemini and Claude were called, the scorer of every test, the two scorer changes made mid-season, the dataset hashes and run dates, and the failure log, up to twenty items per test and lane with the expected and the actual answer, are in methods.txt and failures.txt next to this article.

Five places where a rollout breaks

1. The model has to admit the answer is not there. It is trained to answer, so it answers. Between retrieval and output there has to be a pass that checks whether the retrieved passage is actually about the question, and withdraws the answer when it is not. The judgment is in not overdoing it: a filter that is too strict cuts correct answers, and you end up with a system that always says no. Three confident fabrications out of twenty became zero.

This is not free. On the neighbouring test, answer with a citation, the same pass cost some correct answers: 87.3 became 81.7 against a bar of 80. A margin of 1.7 points, and the thinnest spot of the season.

2. The document is bigger than the window. The usual vendor answer is "get a model with a bigger window", which means paying for a window that idles most of the time. Cut the document into overlapping pieces, compare, reduce: from "nothing past 32k" to 128k on the same card, with the one altered clause found.

3. Instability. Same question, same document, slightly different answer each time. Invisible in a demo. In a system that routes incoming mail it means the same letter goes to one team today and another tomorrow. Several independent reads with a majority vote, and category boundaries defined with examples rather than adjectives: 57.1 became 97.1.

4. Arithmetic. The numbers on a document have to add up, and the thing checking that must not be a neural network. A model computes plausibly, not correctly, and a plausible total passes a human eye and surfaces a quarter later. Ordinary code does the sums, and when net plus VAT does not equal the total to the cent, the model gets the exact address of the error: 95.3 became 99.6.

5. Knowing how documents work. A credit note has no due date and its amounts are negative. If an invoice carries two currencies, take the one it is paid in. Several VAT lines at different rates are summed. "Net 30" is thirty days from the invoice date, not from today. None of this is about neural networks; it is how bookkeeping works, and the model does not know it. On the ten trap scans the bare model failed two; with those rules, ten of ten.

Images: one model for everything

We had planned to bring up a separate vision model for the scans. The same Qwen 3.5 35B-A3B accepts images, so one network covers text and scans.

Bare and layer image lanes ran on the same card on the same day. The bare table-test figure is the median of twelve runs, 82.0 to 85.4, eleven of them below the bar; the one full run of the bare lane landed at 85.4 and would have passed on its own, which is why a single run of that test proves nothing. That is the only cell with a published series; every other cell is one run, and the ones within two points of a bar should be read with the run-to-run spread in mind, about half a point on text. Spot the difference is 15 of 15 for every lane and separates nothing; it stays in the count because the bar was set before the run.

Scanned invoices: 100% on fields across 50 documents, including all ten traps; the bare model tripped on the credit note and on two VAT rates. Speed: 1.2 seconds per document on one card, median 0.86.

The most useful finding was a cause, not a number. The table test kept failing and we blamed the prompt. The problem was resolution: the crops were about 500 pixels wide, and the image encoder turns a page into a fixed number of patches per pixel, so six columns of small print reached the model as a couple of rows of tokens. Upscaling before the call adds no information; it gives the encoder more tokens to spend on the same print. On the frozen set the result is 89.4, tuned on adjacent tables from the same source: the practice set happened to contain no two-line table rows, the type that eats points on the exam. Voting three reads added nothing measurable to accuracy but took the run-to-run spread from 2.3 points to zero, and in production stability is worth more than an extra percent.

Two image tests ran in this season and are not in the table: a handwritten line and "do not fabricate from an image". Every lane failed both, and on review both exams were flawed; they were rebuilt, and the update at the end reports them with their season-1 numbers. Photo moderation lost a point with the layer, 100 to 99.

Price, quality, speed

Quality: 5 of 9 became 9 of 9 on text, 8 of 8 on the quality tests alone, two more than Gemini 3.8 Flash as it ships. 5 of 6 became 6 of 6 on images, against 4 of 5 for Gemini.

Speed: a little over a second per scanned document on one card; 410 output tokens per second with 16 requests in flight.

Price, in the unit a business pays in. Measured on the card at full load, counting every extra call the layer makes:

The scan row is two numbers because the throughput sweep ran on text; the batched figure assumes image tokens flow at the text rate. A thousand invoices a day is about five minutes of card time; a thousand contracts, about an hour and a half. The card does not stay on: it is brought up for the queue and shut down when it empties, so you pay for hours worked, not for a month of idle hardware. Where documents arrive in batches, or where they cannot leave the building at any price, this is the economics. Single requests do not need a card kept warm either: they queue, the card starts for the queue, about two minutes to load the model on a rented card, and stops after. Only when a handful of requests a day each need an answer within seconds is a cloud API the simpler buy.

Behind the per-document figures is a sweep of the same batch at concurrency 1, 16 and 64, meaning that many requests in flight at once; the price blends input and output tokens, and a token is about three quarters of a word:

The card does not stay on: it is brought up for the queue and shut down when it empties, so you pay for hours worked, not for a month of idle hardware.

Between a single request and a fully loaded card the price differs thirteenfold on the same card. The bare run's card, the rented L40S, came out cheaper still on the same sweep, $0.053 per million at concurrency 64. Against cloud list prices on 12 September 2026, blended at the document tests' mix of about 98% input tokens: $0.073 against $0.78 for Gemini 3.8 Flash per token, 10.7x at full load and 5.6x at moderate load; per document the layer's extra calls narrow that to about 1.5x on contracts, where it makes seven calls, and about 10x on invoices, where it makes one. Not in that comparison: batch discounts and context caching on the cloud side, and the OCR engine, the card's cold start and the weeks of engineering on ours.

The card is the running cost. The tests, the rules and the layer are the project, at a fixed price, and the client version runs on the client's own card; the rented one here is the bench. The rules are set per client and per document class, not once for everyone: a new counterparty's format or another country's convention breaks a rule before it breaks anything downstream, because the bench reruns and the break shows up as a failure. Keeping those rules current is the support contract, priced as such, not hidden in the card.

What is weak

• The citation test passed 1.7 points above the bar, the one text test where the layer made things worse.
• Runs are not identical run to run: about half a point of spread on text at temperature zero, up to three points on images. Only the table test has a published series of runs.
• Two image tests are outside the tables, handwriting and image fabrication: every lane failed them in season 1 and both exams turned out flawed. They were rebuilt on 23 September and are reported in the update below, season-1 numbers included.

The stack is ordinary and anyone can copy it: vLLM, schema-guided decoding, our own orchestrator in Python. What took weeks is fifteen tests with traps and the rules a credit note obeys, and that is what we bring to your documents.

Update, 23 September: the two tests outside the tables

Two tests ran in season 1 and are not in the tables above: a handwritten line and "do not fabricate from an image". Every lane failed both, and both exams turned out to be flawed, so we rebuilt them rather than tune against their noise. The season-1 numbers are here, not hidden. Handwriting, bar 70: 15 bare, 15 with the layer, 40 Gemini, 55 Claude. Image fabrication, bar at most one invented answer and at most one refusal on the control images: bare 2 invented with 5 refusals and 21 unparsable replies, with the layer 1 invented with 2 refusals, Gemini 4 with 5, Claude 1 with 1. The full runs are in the methods file. The rebuilt sets open a new season and are not mixed into the tables.

Handwriting. The season-1 lines were Paris census registers from 1926: column fragments with abbreviations that no bookkeeper receives and no model read. The rebuilt test is 20 lines written in 2025 by volunteers of the Danish National Archives, transcriptions proofread, half the lines with a number in them, every crop checked against its transcription by eye, and the same bar. Six of 32 candidate lines were thrown out because the crop and the transcription disagreed. The bare model reads 80% of the lines. With the layer, 80% as well, with a closer average match: the layer states the language of the client's documents, which turned two lines the bare model had returned in Cyrillic back into Danish, and it re-reads every number in a second pass. Both lanes pass. The layer did not add a line, and we say so. One line is lost by both lanes because the proofread transcription has a space inside "1,3"; the model read the ink, the set is frozen, the number stays.

Do not fabricate from an image. The season-1 set carried labels the layer was failing on for the wrong reason: a pickup counted as "no car", a snowboard counted as "skis present", a question pasted into the object field. Every remaining layer failure sat on those items. The rebuilt set drops the categories the source dataset itself confuses, spreads 30 questions over 28 photos and was checked by eye. The layer changed too. The second pass no longer judges the first answer; it has to point at the object with a bounding box or say it found none, and a third call breaks ties. That was tuned on 60 questions from other photos, never on the frozen set, and each lane ran once. Bare: 1 fabrication, 2 refusals on the control images, 21 unparsable replies, fail. With the layer: 1 fabrication, 0 refusals, pass, at 2.07 calls per item. The one fabrication is a carrot: the photo shows coleslaw with shredded carrot, labelled "no carrot" by the dataset and, on review, by us. The model was right; the number stays, because the set is frozen.

What this changes in the count: on the rebuilt sets the layer passes both. On the fifteen tests above nothing changes. The failure log for these runs is in the methods file, in the same format.

What happens next

Every week a new model goes onto the same bench, both lanes, same frozen tests. The leaderboard grows; the bar does not move until the season ends. If a fresh model beats the one we recommend by default, we change the default and say so.

To see this on your own documents: 20-50 documents of one type, run on your machine if they cannot leave it, or sent to contact@unrote.ai; redacted copies are fine, an NDA is available on request, and files sent to us are deleted after the run. You get the pass-and-fail table on those documents and the failed items within five working days, at no charge, before any proposal. That run is step zero: if it passes, the two-week check of your process and the pilot follow, at a fixed price.