53
flaky test
A test that fails without a product bug: time, order, network, leftover state. Delete or fix. Never retry to green.
What is flaky test?
Flakes are non-determinism in the suite. Retrying them in CI launders the failure. The suite stops being a judge.
Why does flaky test matter when vibe coding?
Models “fix” flakes by raising timeouts or catching all errors. That hides the race. You wanted the race named and killed.
How do you do flaky test?
Quarantine a flake in the open. Next turn: make it deterministic or delete it. Do not add `retries: 3` as the solution.
How do you ask a model for flaky test?
This test is flaky. Do not raise timeouts and do not retry. Find the race or the shared state. Make it fail the same way every time, then fix the cause.
What goes wrong with flaky test?
Skipping the test to ship. That is firing the judge for being honest.