22
evals
Score the prompt on a fixed task set. A vibe is not a regression suite.
What is evals?
Evals are tests for model behaviour: the same inputs, the same scoring — asserts, diffs, spot checks. Changing a prompt is like changing code. You watch the eval bar.
Why does evals matter when vibe coding?
One prompt edit can bend the whole agent chain. Without evals you remember “it felt better.” With evals you see whether that sentence broke reverse-engineering.
How do you do evals?
Collect ten real tasks, including failures. Make them a script. Prompt edits run those ten. One drop, roll that sentence back.
How do you ask a model for evals?
Do not edit the prompt by feel. Add one eval: (input) → (required result). Run the whole eval set before the next sentence.
What goes wrong with evals?
Evals that are all happy paths. The model will study for the test. Put failures in, or the suite is theatre.