How to test AI agents: evals, golden traces and CI gates

You test an AI agent the way you would test a new employee, not the way you test software. Before it touches anything real, it sits a probation exam built from real past work (a golden-trace eval suite). Its answers are marked against a written scheme (code scorers plus LLM-as-judge). It does not get the keys until it passes (a CI gate). And once it is live, the spot-checks never stop — every incident becomes a new exam question.

Why do AI agents fail differently to normal software?

Normal software fails loudly. A bug throws an exception, a service returns a 500, a dashboard goes red — and the same input produces the same failure every time, which is why a tester can find it and a developer can reproduce it. The entire discipline of software QA rests on that determinism.

An AI agent breaks both assumptions at once. It is non-deterministic — the same prompt can produce a different plan on Tuesday than it did on Monday — and its failures are usually silent. The chaser email reads beautifully but went to the wrong debtor. The compliance summary is fluent, confident, and has quietly dropped the one certificate that expires next month. The agent reports the invoice as filed when it never called the filing tool at all. Nothing crashed. No dashboard went red. The output was plausible; the action was wrong.

Traditional software fails loudly; agents fail politely — plausible output, wrong action, and no error message anywhere.

Notice what kind of failure that is. In almost every case the model itself is fine — the same model writes a flawless email in the next run. What failed is behaviour in context: given this inbox, this ledger history, this half-ambiguous thread, the agent chose the wrong action. You do not fix that by swapping models, and you do not catch it by watching a demo, because a demo is a job interview and every candidate performs at interview. "It looked fine in the demo" tells you an agent can be impressive for twenty minutes. It tells you nothing about the Tuesday in month three when a tenant replies with a forwarded chain, two PDFs and a partial payment.

The useful news is that you already know how to manage exactly this class of problem, because it is the problem of hiring people. Nobody unit-tests an employee. You set them a probation exam, mark their work, withhold permissions until they are earned, and keep appraising forever. Every eval concept in this article maps onto a management concept you already trust:

The rest of this article walks that table one row at a time.

What is a golden trace?

A golden trace is a recorded example of the job done right, end to end: the input exactly as it arrived (the tenant email with its typos and reply-all baggage, the overdue invoice with the disputed line), the steps a competent operator took (looked up the account, checked the payment history, drafted the chaser, attached the statement, logged the call), and the outcome that counts as correct. A set of golden traces is the exam an agent sits before it is allowed anywhere near your systems — and, like any serious probation exam, it is set from real past work, not hypotheticals.

That last clause is the most important sentence in this article. Tests invented from imagination can only probe the failures you already anticipated, and anticipated failures are precisely the ones your prompts and guardrails already cover. Real traces carry the mess nobody thinks to invent: the supplier who invoices in two currencies, the leaseholder who pays irregular lump sums against the oldest balance, the email where the actual request is buried in the third message of a forwarded chain. As of July 2026, the rule of thumb that has emerged across the field is to start with a golden set of roughly 20–30 real examples and grow it from production traces rather than imagination.

Twenty to thirty examples sound thin beside the thousands of unit tests behind ordinary software, but the comparison misleads. A unit test checks one assertion. A golden trace compresses a whole shift's judgement into one case: did the agent read the situation, choose the right action, execute it exactly, and stop where it should have stopped? Failing one golden trace usually tells you more than failing a hundred unit tests.

The exam also gets easier to set the longer the system runs. Every Usermode agent run lands in an append-only audit ledger, so a golden trace is never an artefact someone has to manufacture — it is a by-product of operating. When we take on a workflow, the first fortnight (how we work) is partly archaeology: pulling the real emails, real invoices and real edge cases that become the agent's entrance exam.

What are code scorers and LLM-as-judge, in plain English?

An exam needs a marking scheme, and agent work needs two kinds of marking for the same reason employee work does: some of it is objectively checkable, and some of it is a judgement call. The field has settled on one tool for each.

Code scorers are the objective checklist — deterministic checks a script verifies with no opinion involved. For a credit-control agent like our Sarah, the checklist reads:

• •Did the email go to a recipient on the allow-list, and only to them?
• •Is the amount in the chaser exactly the amount in the ledger, to the penny?
• •Is the statement actually attached — not mentioned, attached?
• •Did the run stop at the approval gate for anything over the spend threshold?
• •Was the action logged, with the right reference, in the right system?

Each check is pass or fail, runs in milliseconds, costs nothing and cannot be charmed. When "recipient is on the allow-list" fails, that is not a quality debate — it is a hard failure, however lovely the prose around it.

LLM-as-judge covers everything a checklist cannot reach: a second model is given the agent's work plus a written rubric, and marks the qualities that require judgement. Was the tone right for a leaseholder three months in arrears — firm without being threatening? Is the summary faithful to the source documents, or has it smoothed over the awkward part? Should this case have been escalated to a human instead of handled? It is a manager marking coursework against criteria: subjective in nature, consistent in practice, because the rubric is written down, versioned, and applied identically to every case.

The order of operations is the discipline. Code scorers run first, because hard failures are non-negotiable and cheap to detect. The judge marks second, grading quality within the space of actions that were at least safe and correct. And the cardinal sin is using the judge where a script would do — never ask a model whether the amount is right. Ask a script; scripts are never eloquent and never wrong.

What is a CI gate, and why does no pass, no deploy matter?

CI — continuous integration — is the pipeline every software change passes through before it ships, and a CI gate on evals means one specific thing: every time anything about the agent changes, it re-sits the entire exam, and if its score drops below the bar, the change does not go live. Not "should not". Does not — the pipeline refuses mechanically, the way a turnstile refuses a dud ticket. No pass, no deploy.

The management translation is the probation principle: nobody gets the keys until they pass. The part buyers underestimate is the re-sitting. Agents regress in ways no intuition predicts: a prompt tweak that improves the tone of bid answers can quietly weaken the arrears-escalation logic; a model upgrade can change how one supplier's date format gets parsed. In ordinary software you would notice, because the failure would be loud. In an agent the failure is polite — so without a gate you discover the regression from a customer, and with a gate you discover it from a red build, ten minutes after the change was proposed and days before it could touch a real inbox.

A demo tells you an agent could do the job once. A gated eval suite tells you it still can, re-verified on every change. That is the honest difference between "we tested it before launch", which describes one afternoon months ago, and "it cannot ship without passing", which describes a permanent condition. When a vendor tells you their agent is tested, one question separates the two: when did it last sit the exam? The answer you want is "the last time anything changed — probably this week."

Why do evals never stop once the agent is live?

Evals never stop because the world the agent works in never stops changing — and because production is where the real exam questions come from. New suppliers arrive with new invoice formats. A tenant behaves in a way no golden trace anticipated. An upstream system renames a field. The model provider ships an update. None of this announces itself, and the only way to notice quiet drift is the way you notice it in a team: spot-checks and appraisals, on a schedule, forever.

Live evaluation in practice has three layers:

• •Sampled marking. A slice of real production runs is re-marked with the same code scorers and judge rubrics the agent faced before launch, so quality has a number and the number has a trend.
• •Hard signals. Escalation rates, declined approvals, failed sends and tool errors are counted continuously, with a human paged when they move.
• •Failure-driven test growth. Every incident, however small, becomes a new golden trace. The exam grows the way a good training manual grows — one real mistake at a time — so the same failure can never again happen silently.

This is where platform design does most of the work. Usermode's delivery contract — a run cannot end without a logged outbound action or an explicit escalation — exists precisely so that failure is visible and countable. An agent that is not allowed to go quiet cannot fail quietly: every run terminates in something you can count, and the counts are what you monitor. Because every action also lands in the tamper-evident, append-only ledger (the spine of our governance stack), each incident arrives with its full trace attached — what the agent saw, which tools it called, what it sent, who approved it. The postmortem writes its own test case. The same ledger, not coincidentally, is what makes measuring AI agent ROI tractable: the trace you audit is the work you count.

Approval gates earn a specific mention here, because they are both a control and a sensor. Every time a human declines an agent's proposed action, that is a save and a signal in the same moment — a live-fire test case, handed to you free, marking exactly where the agent's judgement and yours diverge. Feeding declined approvals back into the golden set is the highest-value habit in live evaluation; human-in-the-loop approvals covers the gates themselves.

What eval questions should you ask any AI vendor?

You do not need to run evals yourself to benefit from them — you need to ask vendors the questions that only a real evaluation practice can answer. Five are enough:

• •Show me your eval suite for my workflow. Not model benchmarks, not a polished demo — the exam this agent will sit for my invoices, my tenants, my tenders. A good answer includes real cases from your domain built during onboarding, and a willingness to show failures as well as passes.
• •What score gates a deploy? Listen for a specific bar enforced by the pipeline rather than by promise, and for what happens when it is missed. "The change doesn't ship" is the only right answer.
• •How do production failures become new tests? There should be a named, routine process — incident to trace to new golden case, within days. If failures and tests live in different systems owned by different people, the same failure will repeat.
• •What do you monitor live, and who gets paged? Expect specifics — escalation rates, declined approvals, hard failures — and a human on the end of them. "We review the logs" is not monitoring; it is archaeology.
• •How do you test the approval gates themselves? The sharpest of the five. Ask whether the suite contains negative cases — situations where the correct behaviour is to do nothing — and whether a declined action is verified dead rather than merely hidden. Can a declined send still leak out through a retry, a queue, a second tool path? A vendor who has never tested that has never thought about it.

None of this requires you to know what a regression suite is. These are the questions you would ask about any new hire: show me the probation results, tell me who signs off, show me what happens when they get it wrong — and prove that no means no.

If you would like to watch an eval suite mark a live workflow — golden traces, gates, and the ledger trace behind a real escalation — you can book a demo.