evals
An evaluation measures an AI system on a defined set of tasks and criteria.
Use this when A prompt improvement helps a demo but may regress other cases.
without the term
“I tweaked the prompt and it works great on my example, but I'm worried it got worse on other stuff. How do I check it properly across a bunch of cases?”
with the term · How do you ask a model for evals?
Add an evaluation for this failure and run the representative suite before and after the change. Report regressions as well as improvements.
What is evals?
Evals are a fixed set of tasks, with criteria, that you run before and after a prompt, model, or tool change to see what improved and what regressed.
Why does evals matter when vibe coding?
One convincing answer is weak evidence of repeatable performance.
How do you do evals?
Keep representative cases, failure cases, and held-out examples; compare under the same conditions.
What goes wrong with evals?
Tuning repeatedly to the same examples while calling them unseen evaluation.