171
inference
Running the trained model on new inputs. Training is the other, heavier loop. Do not do training inside the request.
What is inference?
Inference is a forward pass: input to prediction. It has a latency budget, a batch option, and a failure mode. Training updates weights and does not belong on the user request path. A GPU is not implied by the word model.
Why does inference matter when vibe coding?
Models load a huge checkpoint on every request and call that an API. Cold start and tail latency fall over. Name load-once, timeout, and what you return if it fails.
How do you do inference?
Load weights at process start. Bound the time. If you can batch, batch. Degrade if the model is down. Do not train online unless that is an explicit product.
How do you ask a model for inference?
Serve inference for (model) with weights loaded at startup. Timeout inside the request budget. If the model is down, (fallback). Do not train or download weights per request.
What goes wrong with inference?
A synchronous inference call on the only web thread. One slow model stalls unrelated pages. Give it its own pool.