Agent Evals
A frozen test set that runs live against three of my own agents — and leaves the failures on the page.
The problem
Anyone can ship an agent demo in a weekend. Almost nobody can show evidence that one works — so every buyer, every legal review, and every skeptical VP asks the same question ("how do you know?") and gets a shrug. Demos are cheap. Proof is scarce.
What I built
Freeze the test set
Six cases, versioned golden-v1-2026-08-11, so a score today is comparable to a score next month. One score is a screenshot; a series is evidence.
Check what can be checked deterministically
Contract shape, subtask count, subtask distinctness (Jaccard < 0.6), source discrimination, context grounding, and latency budget — measured, not judged.
Anchor the judge to real output
Everything left over goes to a judge that has to quote the actual output to score it. If it cannot quote, the score is thrown out rather than kept.
Plant traps and publish the misses
One case plants a fabricated source to see if the agent refutes it; one starves the agent of context to see if it invents. Whatever fails stays visible on the page.
The judge is not fully independent of the system it grades — that limitation is written into the config rather than hidden. Results are what the harness measured, never what it hoped for.