What a good eval suite looks like

Evals are having a moment. Every agent framework pitch, every AI job listing, every reliability thread now says the same thing: write evals. The word is doing the work that "tests" did in 2010 — a box to tick, a skill to demand, a thing that makes the problem go away.

It doesn't. An eval is not a solution. It is an executable claim about how your agent behaves, and it is exactly as good as the person who wrote the claim. It is also a measurement system, and measurement has an old rule that testing never had: validate the gauge before you trust its data, because an instrument noisier than the process it measures produces pure false signal. A vague eval graded by a language model passes without measuring anything. That is worse than no eval at all, because it reports success while failing — the precise disease it was meant to cure.

We run a fleet of production agents and MCP servers, and most of what we know about evals we learned from incidents, not blog posts. Our judge model once started returning scores our harness couldn't parse. A behavioural eval kept flaking until we accepted the failure was model mood, not wiring, and stopped letting it block builds. Those lessons condensed into a checklist — and the interesting thing is that the checklist is really a list of features your eval tooling either has or makes you build yourself.

Here is what good looks like.

1. Evals hit the boundary your users hit. An eval that exercises a mocked-out core proves the mock works. A passing eval should mean the agent booted, accepted a real request over the real protocol, and produced the result you asserted. Anything less is a unit test wearing an eval costume.

2. Severity lives on the assertion. Deterministic checks — the run succeeded, the right tool was called, the reply contains the account number — should fail the build. Model-graded checks should not, at least not at first. Good tooling makes this distinction native: hard gates and tracked-only scores are different assertion types, and a judge score starts as data you watch, earning a threshold only after its distribution proves stable. The distinction is the one Deming drew: judge wobble is common-cause variation, a tool falling off the registry is special cause, and reacting to the first as if it were the second is what he called tampering — it makes the process worse. A harness with one severity for everything forces you to tamper: block builds on noise, or train your team to ignore red.

3. A model-free lane exists. Some regressions are pure wiring: a tool that fell off the registry, a subagent that stopped being called. You want to catch those with a deterministic mock model, no provider in the loop, every commit. Model-in-the-loop evals sit in a second lane with looser cadence. When both lanes share one runner, the fast lane actually gets run.

4. The judge is never the agent under test. The model grading the output must be resolved separately from the model producing it, pinned in config, and swappable without touching the agent. And the judge gets judged: when it drifts or flakes — ours did — that is a tracked event, not a shrug.

5. Assertions and observability read the same events. This is the one that decides whether your setup survives contact with production. When an eval fails, the next question is always what did the agent actually do — which tools, in what order, with what payloads. If your eval harness and your tracing are separate systems, you now have two sources of truth about the same run, and they will disagree. In a good setup, assertions are written over the same typed event stream the traces are built from, and a failed eval ships with its full event log attached. The debugging is in the artifact, not in a second investigation.

6. CI semantics are built in. Exit codes that mean something. A strict mode that promotes score regressions to failures once you're ready. Machine-readable results and per-eval annotations your CI provider understands. None of this is hard to build; all of it is tedious, and tedious things get skipped.

If you are hand-rolling, notice the trajectory. The first version is an afternoon: a script that sends prompts and greps responses. Then you need severity, so you add a config format. Then a judge, so you add model management. Then the judge flakes, so you add quarantine logic. Then a failure needs debugging, so you bolt on logging — and discover your harness and your observability don't share a data model. Each step was reasonable on its own — and the sum of local optima is not a global optimum, which is Goldratt's oldest warning. The sum here is an unowned internal product that does badly what a framework should do well, maintained by whoever broke it last.

We get all six properties from eve, the framework our agents run on. The point of this post is not the name — it's that the list above is a requirements list, not a wish list. Whichever framework you build on, hold it to these six — together, not à la carte. Plenty of tools offer a judge or a dataset runner; the pair that's rare is severity on the assertion plus one event stream shared with your traces. Frameworks that lack them will need to catch up, because teams that run agents in production discover these requirements the hard way, in order, every time.

Two honest limits, because an eval suite that doesn't name its blind spots is lying by omission. Evals drive the agent through its API — they don't see your channel adapters, your OAuth consent screens, or your webhook delivery, so those need their own checks, sometimes a thirty-second human one. And no harness fixes judge variance; it can only contain it. A good suite knows what it doesn't cover and says so.

The deeper reason any of this matters: AI has made building cheap, and the binding constraint has moved to confidence — knowing that what you built works. That's why "can you write evals" is suddenly in job listings written by people who couldn't tell a good eval from a vague one. A good eval suite is verification capacity bought at exactly that constraint. A bad one is a dashboard that says healthy.

And constraints, once elevated, move. Automate the checking well enough and the scarce thing stops being verification and becomes judgment: knowing which claims are worth asserting in the first place. No framework ships that.