A 9B decision model matched our tuned pipeline on inbox routing, in one pass
On 1 October Cloudflare and Amazon released open decision models: Clef and Clef-flash from Cloudflare, Strands Decider 2B from Amazon, all under Apache 2.0. A decision model does not write text. You give it a message and a question with the allowed answers, and in one pass it returns a probability for every answer. For routing mail, sorting tickets or checking a rule, that is all a business needs from a model, and usually what it pays a large language model to imitate.
Disclosure: the author leads engineering at Unrote, which builds process automation on open models. The test, its scorer and the season 1 results are published on our test bench.
We ran all three on the inbox routing test from our own bench, the one we spent the most engineering on last month.
How we tested
Same exam as season 1: a hundred short requests in four languages, twenty-five each in English, German, Spanish and French, from the public Amazon MASSIVE set. Each request has seven possible categories, such as calendar, email, transport or news. Same scorer, same pass mark of 85, written down on 11 September. Nothing was tuned on the exam.
Each model got the request and one question: which category names what the sender wants done. We ran it twice: once with the category names only, the way an off-the-shelf model gets them, and once with the category definitions our own pipeline uses.
One rented RTX PRO 6000 with 96 GB, one model at a time. The whole run, setup included, cost $3.39.
Results
| Setup | Score | Calls per request |
|---|---|---|
| Qwen 3.5 35B-A3B, as it ships (season 1) | 57.1 fail | 1 |
| Gemini 3.8 Flash, as it ships (season 1) | 91.2 | 1 |
| Qwen 3.5 35B-A3B with our pipeline (season 1) | 97.1 | 3 |
| Clef-flash 9B, category names only | 97.2 | 1 |
| Clef-flash 9B, with our definitions | 97.2 | 1 |
| Clef 27B, category names only | 96.1 | 1 |
| Clef 27B, with our definitions | 95.1 | 1 |
| Strands Decider 2B, category names only | 77.7 fail | 1 |
| Strands Decider 2B, with our definitions | 81.5 fail | 1 |
Three things stand out.
The 9B model matched our pipeline with no engineering at all. Last month it took us three reads of every message, category definitions, examples from a practice set and a majority vote to lift a general model from 57 to 97. Clef-flash reached 97.2 with the category names alone, in one pass. On a hundred requests one request is one point, so 97.2 and 97.1 are the same result. English scored 100; German, Spanish and French between 95.9 and 96.4.
Bigger was not better. Clef 27B is three times the size, scored a point lower and answered two and a half times slower.
The smallest one failed. Strands Decider 2B scored 77.7, under the pass mark of 85. Our definitions lifted it to 81.5 but widened the gap between languages: English 92.1, Spanish 73.3. Definitions did nothing for Clef-flash and cost it some speed, since the input got longer.
Speed and cost
| Model | Median answer | Requests per hour | Per thousand requests |
|---|---|---|---|
| Clef-flash 9B | 31 ms | about 105,000 | about $0.02 |
| Clef 27B | 77 ms | about 47,500 | about $0.04 |
| Strands Decider 2B | 32 ms | about 56,000 | about $0.04 |
| Our LLM pipeline (season 1) | about 11,000 | $0.16 |
Answer times are measured inside the server, without the network. Requests per hour come from sixteen callers at once on the same hundred requests; the cost is the card's hourly price divided by that number. The season 1 figure ran on half of the same card type at $1.79 an hour, this run on a whole card at $2.09, so the comparison is per request, not per card.
Two caveats on our own numbers. Our server for the Clef models answered one request at a time; a batched path we tried was slower, and we did not tune it further, so the throughput is a floor. And Clef-flash used 18 GB of the card's memory, so it would fit on a much smaller card than the one we rented.
Where it went wrong
Clef-flash got three requests wrong. Two of them were the same question in two languages, "what is the next event at the library", which the dataset files under recommendation and the model under calendar. The third was "give me an update about the elections in an hour": the dataset says calendar, a reminder; every model in the table answered news, our own pipeline included. Public labels are not always the ones a business would choose, which is one reason to measure on real mail.
Every answer comes with a probability. On the three wrong ones Clef-flash was 74%, 80% and 93% sure; on the right ones, 97% on average. That is what makes these models usable in a process: the uncertain answers can go to a person instead of the queue.
What we cannot claim
- The set is public. Amazon lists MASSIVE among the training data of Strands Decider. Cloudflare does not publish Clef's training data, so it may have seen these requests too. A model that has seen the exam scores well on the exam.
- Only the category was scored. The public set has no urgency or department labels, so this is one decision per message, not a full routing.
- A hundred requests is a small exam. It is enough to separate 97 from 78, not 97 from 96.
What this means for a company
Much of what companies use large language models for is not writing. It is deciding: which team gets this message, is this invoice a duplicate, does this ad break a rule, can this request be approved. For those steps a decision model is the cheaper tool. It answers in one pass, it cannot make up an answer outside the list, and it says how sure it is.
It also runs where the data is. All three models are open and run on a company's own servers, and the one that did best needs a single card.
It does not replace the rest. Pulling fields out of a contract, summarising a thread or drafting a reply still need a model that writes. And the score on a public set says nothing about your mail. Before we would put a decision model into a client's process, we would run it on a sample of that client's own messages against a pass mark agreed in advance, the same way we ran this test.
That is how we work. The work is priced as a fixed project, with no meter on usage. To start, write to contact@unrote.ai with the process you have in mind.
For deciding, not writing, a 9B open model now does in one pass what took us three calls and weeks of tuning.