agent looped re-running a failed glue job bookmark commit, duplicates piling up

TL;DR

Glue job bookmarks track what was processed so the next run starts where the last committed. When a run writes output but the bookmark commit fails, or an agent reruns with bookmark handling that resets or ignores the state, the same source data is read again and written again, and an append style target accumulates duplicates with every retry. Stop the loop, establish what the bookmark last committed and what the failed runs actually wrote, clean the duplicated output window, then fix the rerun to resume from the committed bookmark or to write idempotently per partition.

agent looped re-running a failed glue job bookmark commit, duplicates piling up

Use this when

• A Glue job's output table or prefix shows duplicate records growing with each rerun.
• Job run history shows repeated failures mentioning bookmark or commit, after output was written.
• An agent or scheduler keeps relaunching the job on failure with no change between attempts.

Not for this skill when

• The job uses bookmark disable by design and its target is overwritten per run. Duplicates there have another cause.
• Duplicates come from the source files themselves being delivered twice into the landing zone. That is an upstream delivery problem.

Steps

1. Stop the relaunching first: pause the agent or schedule driving the reruns and note the run ids and times of every attempt in the loop. Note the count of attempts. Verify: no further copies are written while you assess the damage
2. Read the Glue job run details for a failed attempt: where it failed, whether the error is in the bookmark commit phase, and what the job's bookmark option is set to, enable, pause, or disable. Note the setting and the failure phase. Verify: the bookmark's role in the failure is established from the run record
3. Inventory the output: for the window the loop covered, count records per source key or per partition and compare with the source counts. Note the excess copies and the partitions affected. Verify: the duplicate damage is bounded to named partitions with numbers
4. Clean the affected output by rebuilding those partitions once from the source, or by deduplicating on the job's natural key keeping one copy, and verify counts against the source. Note the method and the verified counts. Verify: the output holds one copy per source record again
5. Fix the rerun design before resuming: let bookmarks resume from the last commit rather than resetting them, make the write replace per partition or merge on the natural key so a reprocessed window cannot duplicate, and cap agent retries with a stop on bookmark errors specifically. Run the job once and then deliberately rerun it and note the second run adds no rows. Note the proof. Verify: a failed commit followed by a rerun is now harmless by construction

Variant phrasings

aws glue job bookmark commit failed rerun duplicate records s3

The bookmark commit phrasing.

glue job reprocesses same files bookmark not advancing duplicates

The bookmark not advancing symptom search.

Compatibility: AWS Glue ETL jobs using job bookmarks, writing to S3 based tables in Parquet or similar formats. Bookmark options and run detail fields are Glue console and API standard.

Why it happens

Bookmarks are Glue's memory, and like any memory they only help if the commit lands. The dangerous sequence is a split outcome: the data write, which is visible and durable, succeeds, while the bookmark commit, which records that the work is done, fails, so the system's official position is that nothing happened. A retry driven by that official position is behaving logically and destructively at once, reprocessing data the target already holds. Agent loops multiply it because each attempt is justified by the same unadvanced bookmark. Idempotent writes are the seatbelt that makes bookmark failures survivable.

Edge cases / pitfalls

• Resetting bookmarks to recover from an unrelated failure reprocesses history by design. Treat a bookmark reset as a planned full reload with a partition replacement strategy, never as a casual retry step.
• Bookmark state is per job name and can be shared or clobbered when jobs are copied for testing. Test copies writing the same target with their own bookmark are a duplicate factory.
• Some failures labelled commit are really permission or state store problems on the Glue side. Read the run's error message in full before assuming the job code is at fault.
• If the target is a table format with merge support, prefer its upsert semantics for bookmarked jobs so reprocessing is safe even when bookmarks misbehave.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lOhgNwmnzJdbMoAI2LHHFA