Add a camera and a referee before you add a second agent

Most agents are judged by feel

A team runs an agent a few times, the answers look reasonable, and it goes live. Two weeks later someone notices it has been skipping a step. Nobody can say since when, because nothing recorded what it did.

That is the usual situation, and it hides two separate gaps. You can't see what the agent did, and you can't say whether what it did was good enough. They need different fixes.

The camera: a trace of every run

A trace is the record of one run, step by step. Which model was called, which tools, in what order, what came back, how long each step took, what it cost. Nothing exotic is needed. Every call to a model or a tool passes through some point that can log it, the way a meter sits on a water line.

From the raw trace you derive things a person can read: cost per run, time per step, error count, a timeline. They answer practical questions. Why was this run slow? Which step spends the money? Did a tool fail, and did the agent say so, or did it quietly work around it? Without a trace, each of those is a guess about a black box.

You can start without special software. A written receipt at the end of each run (steps taken, files and sources used, assumptions made, what could not be verified) is a trace a human reads in thirty seconds.

The referee: deciding what ships

Observability tells you what happened. Evaluation decides whether it was good enough to release. There are three kinds of referee, and each fits a different job.

Code checks are exact and cheap. Use them for anything you can state as a rule: the total in the summary matches the source, the report was saved once, every figure has a source. If a rule can be code, make it code.

An AI judge is a second model that reads the output against your criteria. It handles what rules can't, such as whether an answer is complete or the tone fits. It is still a model and can be wrong, so you spot-check it.

A human reviews a sample and rules on the cases where the other two disagree. That is the expensive one, so spend it where judgment matters.

The part everyone skips: a fixed test set

A referee needs something to referee. Take five to twenty real cases from your own work, each with the answer you would accept, and keep them unchanged. After every edit to the prompt, the instructions or the model, rerun all of them and compare.

This is the only way to catch the change that improves one case and quietly breaks three others. Trying the new prompt on whatever example is on your screen will not show you that. Add a case every time something goes wrong in real use, so the set grows out of your actual failures.

Where it breaks

A test set covers only what you put in it, and an AI judge shares blind spots with the agent it grades. Neither replaces reading real outputs now and then. They make that reading cheaper and better aimed.

What to do tomorrow

Before building anything new, add a receipt instruction to the agent you already have, and write down five past cases with the answers you would accept. Then paste this into the same chat:

Run the five cases in the document using the current instructions. Show a table: case, your answer, my accepted answer, pass or fail, and the reason.

Read the failures first. They tell you what to fix before you add anything.