167
holdout set
Data the model never trains on and you rarely look at. It is the only honest number.
What is holdout set?
A holdout (test) set is separated before training and tuning. Validation is what you look at while tuning. The test set is touched once at the end. Every time you change the model because of the test score, the test set becomes training.
Why does holdout set matter when vibe coding?
Models split randomly each run, or they peek, or they report training accuracy. Name a frozen split written to disk.
How do you do holdout set?
Write the split once, by user or by time, and commit the ids. Tune on validation. Report test once. If you must change after seeing test, say the number is now optimistic.
How do you ask a model for holdout set?
Create a frozen holdout split by (user or time) and save the ids. Do not train or tune on it. Report that number once. Do not reshuffle every run.
What goes wrong with holdout set?
A holdout so small that one row swings the score. Size it or stop quoting a percentage.