Hisham AlhussainFor a long time, evals in my projects were something we ran occasionally. Usually before a big demo, or after someone complained. They lived in a notebook, and running them was a bit of an event.
Earlier this year I flipped that around, and I wish I'd done it sooner. Evals are now part of CI, just like unit tests. If they don't pass, the change doesn't deploy. No exceptions.
Why it took me so long
Honestly, I think I was treating evals as a research problem. I wanted the perfect metric, the perfect dataset, the perfect judge. So I didn't build anything, because nothing was perfect.
What finally clicked is that unit tests aren't perfect either. They don't prove your code is correct. They just catch the regressions you know to look for. Evals can do exactly the same job.
What my setup looks like
Nothing fancy:
- A versioned eval set in the repo. Real examples, with expected behavior written down. It grows every time we find a new bug.
- Different checks for different things. Exact-match or code checks for structured output and tool calls. An LLM judge with a tight rubric for open-ended answers. A few hard rules, like "never confirms a booking without a time."
- Thresholds, not vibes. Each category has a minimum pass rate. Safety-critical ones need 100%.
- Runs on every PR that touches prompts, models, tools, or retrieval. The results show up right in the PR.
Every bug becomes a test
This is the habit that made the biggest difference. When something goes wrong in production, the fix isn't done until there's an eval case that would have caught it. Over time, your eval set turns into a record of every mistake you've learned from, and the same mistake never ships twice.
Keep it fast and cheap
If evals take an hour and cost a fortune, people will find ways to skip them. I keep a fast core set that runs on every PR in a few minutes, and a bigger set that runs nightly or before releases. Caching responses for unchanged cases helps a lot too.
Watch out for judge drift
If you use an LLM as a judge, the judge is part of your test setup. Pin its model and prompt, and spot-check it against human grades every so often. A judge that quietly gets more lenient is worse than no judge, because it gives you false confidence.
The real benefit: speed
The funny thing is, adding this gate made us faster, not slower. When there's a safety net, you can change a prompt or try a new model without that nervous feeling. You make the change, look at the numbers, and ship.
We've had another round of big model releases recently, and the eval suite made deciding whether to upgrade a one-afternoon job instead of a one-week debate. That alone was worth it.
No green, no deploy. It's a simple rule and it's changed how I build.