If more than one person does it, it can be automated. If one person does it, build them a tool

Three models from three companies invented the same answer, with the same confidence as a correct one, against a direct instruction not to. Nothing in the output said "I made this up". Models fail in ways that look exactly like success.

That is why "can AI do this?" is the wrong first question. The one that decides how far a process can be automated is less exciting: when the machine gets it wrong, will anyone notice before it costs money?

Two kinds of automation

Most office work has two parts. On top: decisions, positions, conversations with clients, the things people are paid for. Underneath: moving data from one place to another, sorting, searching, retyping, logging hours.

So there are two kinds of automation, and only one of them takes work away.

Taking the work off people's hands. The system does the task from start to finish, and people only see the exceptions. This is where the work underneath goes.

Making the work easier. The person still decides and still signs. The system does everything around the decision: the reading, the searching, the first draft, the retyping. The person gets through more in the same day.

That gives us our rule. If more than one person in the company does the same work the same way, again and again, it has already become a routine: there is a way it is done, new staff are taught it, colleagues spot each other's mistakes. A routine can be taken off people's hands. If only one person does it, because it takes their judgment or their signature, we do not replace them. We make their work easier.

Contracts show both at once. Many lawyers in a firm read contracts and pull out the same fields: parties, dates, terms, governing law. That reading is a routine, and it can be taken off their hands. Each contract still ends with one signature, and that part we only make easier.

How far any step goes is decided by whether its result can be checked. That is what the four questions below are for.

Four questions we ask about a process

1. Is there a right answer that can be checked? Net plus VAT has to equal the total. A due date cannot come before the issue date. An answer has to quote the passage it came from. When the correct result obeys a rule, the rule can catch the machine. When the correct result is a matter of judgment, nothing catches it.

2. What does a mistake cost, and who notices it? A misrouted email costs a few minutes and the recipient sees it immediately. A wrong figure in a signed contract can cost the deal and nobody sees it for months. Same model, same accuracy, completely different decision.

3. Can someone write the rule down? If your best person cannot explain how they decide, we cannot check whether the system decided the same way. "It depends on the client" is not a rule. "Credit notes have no payment term" is.

4. How often does it happen? Automation is a fixed cost: rules, tests, upkeep. A task that happens twice a month rarely pays that back. A task that happens two hundred times a day almost always does.

One more thing decides not whether, but where. Law firms, accountants and anyone holding personal data are often bound not to send documents to an outside service. That does not rule automation out. It means the models run on servers the company controls, and the documents never leave them.

How we measured

The numbers below come from our own test bench, published in full at unrote.ai/articles/bench-season-1. The document sets are public, so anyone can rerun them. Public sets can also have been seen by a model during training, which is one more reason we always measure on a client's own documents before building anything.

The pass mark for each test was written down before the first run and has not moved since. We tuned our checks on a separate practice set drawn from the same sources, never on the exam itself. This article uses nine of the fifteen tests. Unless a line says otherwise, the results are the open model with our checks, and each line says what was counted and on how many documents.

By "our checks" we mean what we build around the model: written rules the answer has to pass, a second pass that looks for each value in the source text, and a fixed format the answer must fit.

What we take off people's hands

These passed with room to spare. The answer can be checked by a rule, or a mistake is cheap and visible.

• Pulling data from invoices. 99.6% of fields correct on 66 invoices read by OCR, against a pass mark of 90. Ten of the invoices were built on purpose to be awkward: two currencies, several VAT rates, a credit note, a foreign total in the footer. Out of the box the model got 95.3%, and on one of those ten it put a due date on the credit note. A credit note is a refund and has no due date. That is exactly the kind of mistake a written rule removes.
• Sorting incoming email. 97 out of 100 on a combined score for the right team, urgency and department, against a pass mark of 85, on a hundred short requests in four languages, twenty-five per language. Out of the box the model scored 57, and Spanish trailed English by 30 points. Part of the gain came from writing sharper definitions of each category. With our checks the gap between languages fell to about 6 points, which on twenty-five requests per language is a couple of emails. If one lands in the wrong queue, whoever opens it sees at once.
• Long documents. 15 of 15 altered clauses found, five contracts at each length, up to around 200 pages. In one of the longest, halfway through an exhibit on dispute resolution, the time the arbitrators have to agree had been changed from thirty days to forty-five. The system named the clause and both values. In our setup the model reads about 50 pages at once, so our checks cut the document into overlapping parts, compare each part with the original, and bring the answer back together.

The cost follows. On one rented GPU at $1.79 an hour, run at full load, the compute for a thousand invoices comes to about 15 cents, for a thousand contracts about $2.50. That is compute only: no setup, no upkeep, no people. On a company's own hardware the running cost is the hardware itself.

Where the money usually is

Everyone expects the case for speed: the same work, done faster. In our conversations the larger case is usually something else, the things that stop getting lost. Hours worked but never billed. Extras delivered but never invoiced. Emails that sat in the wrong inbox for a week. A law firm we spoke with named exactly this as its first problem, before anything about speed.

Speed saves hours. Things that stop getting lost bring back money the company already earned.

Where automation makes the work easier

Some steps reach the pass mark but belong to one person who answers for the result. There the system does not decide. It prepares, so that person handles more cases in the same day.

• Contract fields. 90.2% of fields correct on 50 contracts, against a pass mark of 85. Good enough to prepare the summary, not good enough to sign it. The invented "48 months" from the start of this article got through our checks as well, and we know exactly how: a contract term is often written as a date rather than a number of months, so our check accepts a supporting sentence that contains a date, and the made-up number slipped through that door. A lawyer reviews the terms. The system saves them the reading, not the responsibility.
• Answers from your own files. 81.7 out of 100 on a score that combines the right answer with the right source passage, on 30 questions, against a pass mark of 80. That is at the pass mark, our thinnest margin, and the check that stopped invented answers also cost some correct ones. The citation is what makes each answer checkable: whoever reads it can open the source.
• Tables in scanned documents. 89.4% of table cells correct on 20 tables, against a pass mark of 85. On the practice set we found why they kept failing: the scans were too small to read, not the instructions too vague. Enlarging them before reading fixed most of it. Good for filling a draft. A person confirms the totals.

What stays with people

• Old or unusual handwriting. On twenty lines from Paris census registers of the 1920s, all three models failed a pass mark of 70% of lines read correctly: 15, 40 and 55 for Qwen, Gemini and Claude. On twenty lines written this year, the open model reads 80%. Both samples are small, and the point is narrow: handwriting is not one task, and a demo on neat notes proves nothing about your archive.
• Decisions that carry a signature. Legal judgment, credit decisions, anything where the law expects a named person to answer for it. The law firm above drew this line for us in the first meeting: billing and time records, yes; transaction documents, no. The risk in a transaction sits in a partner's signature, and no check we can write moves it.
• Rare tasks and undescribed processes. A task that comes up a few times a month does not pay back the setup, and a process nobody can describe cannot be turned into rules we can check. A good template and a person are cheaper there.

Before anything is built

Whoever builds your system, ask for one thing first: a measurement on your own documents. A public test set says nothing about your suppliers and their layouts. Take a sample of one document type, agree the pass mark in advance, and look at every failure, not just the score. If a step cannot be checked, it stays with your people.

That is how we work. The work is priced as a fixed project, with no meter on usage. Pricing and past cases are on unrote.ai. To start, write to contact@unrote.ai with the process you have in mind.

What several people do, we take off their hands and check. What one person answers for, we make easier, so they get more done in less time.