159
idempotent pipeline
Running yesterday’s job again replaces yesterday. It does not add yesterday twice.
What is idempotent pipeline?
An idempotent data job produces the same table whether it runs once or twice for the same window. The usual mechanism is overwrite-a-partition or merge-on-key, not append.
Why does idempotent pipeline matter when vibe coding?
Models INSERT the day’s rows and let the scheduler retry. The job runs twice and yesterday’s revenue doubles. Name overwrite or merge, or the retry is a second copy.
How do you do idempotent pipeline?
Write the partition for that window from scratch, or MERGE on a primary key. Do not INSERT and hope. A retry is a normal path.
How do you ask a model for idempotent pipeline?
Make the (daily job) idempotent. Re-running a date overwrites that partition or merges on (key). Do not append. A retry must not double counts.
What goes wrong with idempotent pipeline?
Overwrite that deletes the partition first and then fails. You replaced data with nothing. Write the new partition, then swap.