## TL;DR
The incremental job trusts its watermark blindly, so a missing or reset watermark silently becomes "load everything". Validate the watermark before each load (it must exist, parse, and fall in sane bounds), require an explicit flag for full reloads, and only advance the watermark after the load succeeds. A bad watermark should abort the run, never widen it.

## Symptom
The incremental job that should load today's rows reloads the whole table. Runtimes balloon, downstream dedupe breaks, or warehouse costs spike, and the stored watermark looks wrong, empty, or suspiciously old. The agent reports success because the load itself worked; it just loaded far too much.

## Steps

1. Read the stored watermark before the load and sanity-check it: it must exist, parse as the expected type, and fall within plausible bounds (not null, not the epoch, not in the future).
Expected output: a valid watermark, or a loud refusal to run. A missing watermark must never silently mean "load everything".

2. If the watermark is invalid, require an explicit full-load flag to proceed; otherwise abort with the reason.
Expected output: full reloads only happen when a human or config asked for one. The default for a broken watermark is "do nothing", not "load it all".

3. Filter the source on the watermark and record the new high-watermark from the loaded batch (the max of the watermark column), not from the clock.
Expected output: the next run starts where this one ended, with no gaps and no overlaps.

4. Commit the new watermark only after the load succeeds end to end. On failure, keep the old watermark so the retry reprocesses the same batch, not a partial one.
Expected output: failures are retry-safe; the watermark never advances past uncommitted data.

## When to use
- An incremental job reloads the full table unexpectedly
- Watermarks are stored somewhere the agent can overwrite or lose (a file, a table it also writes)
- You are handing incremental logic to an agent for the first time

## When not to use
- The job is intentionally a full refresh (then there is no watermark to validate)
- The source has no reliable watermark column (that is a CDC or snapshot problem, not validation)
- Row counts are wrong but the watermark is fine (check the filter logic)

## Variant phrasings

### incremental load keeps reprocessing the same batch
The watermark never advances, usually because the commit in step 4 runs before the load finishes, or the new watermark is computed from an empty batch. Check the commit order.

### watermark is fine but rows are duplicated downstream
The load window overlaps (watermark inclusive on both ends). Make the lower bound exclusive or dedupe on the watermark column downstream.

## Why it happens
Incremental loading has an implicit contract: the watermark marks "everything up to here is done". Code that treats a missing watermark as "start from the beginning" turns any storage hiccup, first run, or agent mistake into a full reload. Validation makes the contract explicit: no valid watermark, no run.

## Edge cases
- Late-arriving data older than the watermark is skipped by design; handle it with a separate lookback window, not by widening the watermark.
- Timezone-naive vs timezone-aware watermarks compare wrong across systems; store and compare in UTC.
- Multiple writers to the same watermark key race; use one writer or atomic compare-and-set.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_w4HxeNfDDPeC0mf0Tul8zA