Measurements and failures from our own bench and from production work, written for engineers and for the people who sign off on what they build.
On 6 October Musubi released PolicyLM-1.7B, an open model for content moderation under Apache 2.0. It is a decision model: it does not write text. You give it a message and your rules in plain English, and it returns a score from 0 to 1 for every rule. Musubi says it answers in about 35 ms on an NVIDIA L4 and that you can change the rules without retraining.
On 1 October Cloudflare and Amazon released open decision models: Clef and Clef-flash from Cloudflare, Strands Decider 2B from Amazon, all under Apache 2.0. A decision model does not write text. You give it a message and a question with the allowed answers, and in one pass it returns a probability for every answer. For routing mail, sorting tickets or checking a rule, that is all a business needs from a model, and usually what it pays a large language model to imitate.
On 29 September OpenAI released dots: agents inside ChatGPT that keep working when nobody is talking to them. A dot has its own computer in OpenAI's cloud, connects to more than 4,000 apps, reads your mail and messages, and acts on them: drafts, sends, books and, with your approval, buys.
We gave three AI models a contract and asked how long its initial term was: the open model Qwen 3.5, Google's Gemini 3.8 Flash and Anthropic's Claude Fable 5.1. The contract sets no fixed term. The instructions said to answer "null" when a value is not in the document, and never to invent one. All three answered "48 months". The number 48 appears nowhere in the document.
Before a rollout we run the model on the client's class of documents against bars fixed in advance. This is that run on public data: nine text tests and six image tests on invoices, contracts, inbox requests and scans, one rented GPU, one open model. As it ships it failed five of the fifteen. With our layer around the call, all five passed. Up to twenty failures per test are published, including the items the layer did not fix. Two more image tests ran and are reported at the end: every lane failed them, and on review both exams were flawed, so they were rebuilt.
20-50 documents of one type, the table and the failed items back within five working days, at no charge.
Get a free document test