A load job dies halfway through at 3 a.m., and someone reruns it. What, specifically, stops the rerun from double-counting every row that already landed? If the honest answer is "whoever reruns it knows to clean up first", the pipeline is safe by discipline, and discipline fails at the worst moments. This guide covers how to make reruns safe by default on Redshift and Spark, against three simple criteria: A rerun must not need a human to remember a cleanup step. It must not depend on the database to reject duplicates. The input set must be the same on the second attempt as on the first. Redshift will accept your duplicate The first trap is assuming a primary key protects you. In Redshift, uniqueness, primary key and foreign key constraints are informational . An insert can succeed even when it violates one, and the query planner still treats the constraint as true when building plans. If your data breaks that assumption, some query results can come back wrong. The be...