A classifier is only as good as its rulebook

Most classification systems stall because their rules never change

The common way to build an automated sorter is to write definitions on day one, connect the model, and move on. Those definitions come from what you imagine the inputs look like. Real inputs are stranger, and a system running on a first draft either guesses confidently or has no way to say it doesn't know.

A sorter is a judgment repeated thousands of times. You improve a judgment by studying the cases where it struggled, so the design question is how to surface those cases automatically.

A cascade acts alone only when it is sure

The pattern is called a cascade. A fast, inexpensive model reads each item and returns a label with a confidence score. Above a threshold, the label is used directly. Below it, the item escalates to a stronger model or a human reviewer.

The threshold is a policy choice. Ninety percent is a sensible starting point, and you move it as you learn what errors cost in your setting. A support inbox routing to billing, tech and sales is a clean first test, because a wrong route is cheap to detect and fix.

Escalated cases are the training signal

Every item that falls under the bar is a case where the fast model's current instructions were not enough. The stronger model or the human resolves it, and that resolution is the most informative data you will collect.

The usual instinct is to retrain the model on those cases. For most business workflows that is the wrong tool. Retraining is slow, hard to inspect, and changes behavior in ways you cannot easily predict.

Edit the rubric, not the model

The fast model works from a written rubric: definitions of each category and the rules for the boundaries between them. When an escalated case is resolved, you ask what line in the rubric would have let the fast model handle it, and you add that line. A boundary rule might say that a message mentioning the payment page and carrying no order number is a technical issue until shown otherwise.

Next week, that kind of case clears the threshold without help. The share that needs escalation shrinks, and the model itself is unchanged.

This has a property that matters for anyone accountable for the system. The improvement lives in a plain-language document. You can read it, compare versions, show it to a compliance reviewer, and revert a bad edit in seconds. A retrained model offers none of that.

A none of these option prevents forced picks

A classifier asked to choose among three labels will choose one, even when the input fits none of them. The forced pick produces inflated confidence in wrong answers, which is the failure that most damages trust.

Add a final option that says none of these. It gives the model an honest exit, and the pile it fills tells you where the rubric has a gap. It does not suit every task, but where it applies it is close to free.

Shadow mode comes before switching on

Before a new sorter takes over, run it in shadow mode. It processes every item and records its decision while the existing process, human or otherwise, keeps making the real ones. After a period you compare the two.

Where they disagree, you learn whether the new system was wrong or the old habit was. Nothing in production is at risk, and your first rubric revision is already drafted from the disagreements.

Model choice matters less than the loop

Very fast classifier models make this pattern cheap enough to run on every item. Jev is one that people are discussing. I haven't tested the speed or cost figures circulating about it, and you don't need them to start. The loop runs today with Claude.

Try it tomorrow

Paste fifteen real messages from your own inbox into Claude. Ask it to sort them into named piles, including one called not sure, and to place anything under ninety percent confidence there. Then ask for one rubric edit based on what landed in that pile.

Read the edit. If it is a rule you would have written yourself, you have completed the first turn of the loop.