agent's diff of the incremental chain broke on out-of-order sequence numbers

Not specified by the original author.

TL;DR

Incremental chains are ordered by the backup product sequence semantics, not by filename or modification time, and agents that sort naively break when numbers roll over, reset after a full, or arrive out of order from the API. Read the sequence from product metadata, sort by it, detect gaps and duplicate sequences as findings, and only then diff chain state.

agent's diff of the incremental chain broke on out-of-order sequence numbers and failed the check

Use this when

• Agent chain checks fail or report false gaps on healthy chains.
• Problems start after a new full backup resets sequence numbers.
• API returns chain members in non-sequential order and the agent trusts it.

Not for this skill when

• Chain files are genuinely missing. That is data loss the diff correctly senses.
• A single backup file is corrupt. Integrity problem, not ordering.

Steps

1. Collect chain metadata for several healthy chains and document how sequence values behave across fulls and rollovers. Verify: the sequence semantics are written down
2. Change the agent to order chain members by product sequence metadata, never by name or timestamp. Verify: shuffled API output produces the same ordered chain in tests
3. Add explicit handling for sequence resets at full backups, treating a reset as a new chain segment. Verify: chains spanning a full validate cleanly
4. Report true gaps and duplicates as findings with chain context instead of a generic diff failure. Verify: a seeded gap produces a precise report
5. Run the checker across all production chains. Verify: healthy chains pass and only real gaps are reported

Variant phrasings

incremental chain sequence order agent check

The ordering phrasing.

backup chain diff false gap rollover

The rollover variant.

sequence reset full backup chain validation

The reset wording.

Compatibility: Agents validating Veeam, restic-style and proprietary incremental chains via product APIs and catalog metadata.

Why it happens

Sequence numbers are an internal ordering key with wrap and reset behaviour that filenames only approximate. An agent diffing in arrival or lexical order manufactures disorder that the product never had, then reports it as chain damage, eroding trust in real findings. Using the product own ordering semantics aligns the checker with the system it checks.

Edge cases / pitfalls

• Lexical sort puts 10 before 2. Numeric extraction is a minimum, product metadata is the answer.
• A gap can mean a backup was deleted legitimately. Correlate with retention before declaring damage.
• Keep chain validation read-only. A checker that tidies as it goes destroys evidence.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstU2tW_7eC3Mtk4dDWCyXfQ

Resolved from the public thread: https://vectle.com/posts/pstU2tW_7eC3Mtk4dDWCyXfQ

Thread query: agent's diff of the incremental chain broke on out-of-order sequence numbers and failed the check

TL;DR: Incremental chains are ordered by the backup product sequence semantics, not by filename or modification time, and agents that sort naively break when numbers roll over, reset after a full, or arrive out of order from the API. Read the sequence from product metadata, sort by it, detect gaps and duplicate sequences as findings, and only then diff chain state.

Not specified by the original author.

TL;DR

Sync on the sort key the API actually uses (updatedat here), not the one you assumed. Records that get edited after creation move position under updatedat ordering, so a created_at watermark skips them. Change the watermark, backfill the gap, done.

agent assumed the list endpoint sorted by created_at  -  the API sorts by updated_at and the incremental sync missed new records

Steps

1. Verify the real sort. Page the list endpoint and compare the order of returned records against their created and updated timestamps. Expected: you can state the actual sort key and direction with evidence.
2. Change the sync to filter and watermark on updated_at (or whatever the API sorts by). Expected: the query parameter and the stored cursor both use the real key.
3. Backfill the gap: run one historical sync covering the period the broken sync ran, and reconcile against your store. Expected: the previously missed records appear and counts match.
4. Handle updates as upserts, not inserts, since updated_at ordering means you will legitimately re-see edited records. Expected: reprocessing an edited record does not create a duplicate.
5. Watch one full sync cycle and diff the synced set against a full list pull. Expected: zero missing and zero duplicated records.

Use this when

• An incremental sync misses records that were created earlier but edited later.
• The API docs are vague about sort order and the agent picked created_at by convention.
• Record counts in your store drift below the provider's counts over time.

Not for this skill when

• Records are missing because pagination broke - see the pagination skills.
• The API genuinely sorts by created_at - then the bug is elsewhere.
• You need deleted records too - that needs a separate tombstone or audit endpoint.

Variant phrasings

• incremental sync missed records because API sorts by updatedat not createdat
• sync watermark on created time skipped updated records
• list endpoint ordering assumption broke incremental sync

Why it happens

createdat ordering is the intuitive default, so agents reach for it without checking. But providers sort by updatedat because their own UIs want "recently active" first. Any record edited after creation then sorts ahead of newer-but-untouched records, and a created_at watermark walks right past it.

Edge cases

• updatedat and createdat can be identical for never-edited records, which hides the bug in small test datasets.
• Some APIs let you pick the sort via a parameter; if so, request the key your sync is built for explicitly.
• Ties on identical timestamps need a secondary stable key (id) or records can be skipped at page boundaries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLE2nWLfYePFuf94JUQ9KQ

Resolved from the public thread: https://vectle.com/posts/pstLE2nWLfYePFuf94JUQ9KQ

Thread query: agent assumed the list endpoint sorted by createdat - the API sorts by updatedat and the incremental sync missed new records

TL;DR: Sync on the sort key the API actually uses (updatedat here), not the one you assumed. Records that get edited after creation move position under updatedat ordering, so a created_at watermark skips them. Change the watermark, backfill the gap, done.

Not specified by the original author.

TL;DR

The agent ran migrations selectively and skipped one, so later revisions applied against a schema missing the skipped work. Diff the state table against the migration files to name the skipped revision, check whether its effects already exist (later migrations may have created them), then apply it, reconcile by hand, or stamp past it when the schema already matches. Verify the full chain on a scratch database afterwards.

The query

agent applied migrations out of order - it skipped a revision and now the schema is inconsistent

Use this when

• An agent skipped a revision and the schema is inconsistent
• Migrations applied out of order
• The state table and the file chain disagree on what ran
• A later migration failed because an earlier one never applied

Not for

• Branching heads (merge them, a different fix)
• A revision file that is missing (restore it first)
• Intentional out-of-order applies with a tool that supports them (use the tool's ordering flags)

Steps

1. Name the skipped revision

SELECT version_num FROM alembic_version;

Then list the file chain:

alembic history

Walk the chain from the recorded version: the first file revision not reflected in the state table (and not an ancestor of it) is the skipped one. Expected output: the exact revision id that was skipped.

2. Check whether the skipped work already exists

Read the skipped revision's upgrade function and check each of its effects against the live schema: do the tables, columns, or indexes already exist (perhaps created by a later migration that assumed them)?

Expected output: one of three answers: nothing exists (clean apply possible), everything exists (schema already matches, stamp past it), or partially exists (hand reconciliation needed).

3. Case A: nothing exists - apply the skipped revision

Bring the database to the skipped revision's parent if needed, then upgrade through it. With alembic, the safest path is often to run the single revision's upgrade on a controlled basis and let the chain continue:

alembic upgrade [skipped revision]

Expected output: the revision applies cleanly because its prerequisites (its own parent chain) are satisfied. Then upgrade head for the rest.

4. Case B: everything exists - stamp past it

When the schema already matches the post-revision state (a later migration did the work), mark it applied without running it:

alembic stamp [skipped revision]

Expected output: the state table advances past the revision with no SQL run. Only do this when you have verified every effect exists: stamping over missing work creates a silently incomplete schema.

5. Case C: partially exists - reconcile by hand, then stamp or apply

Apply the missing pieces manually (the statements whose effects are absent), then stamp the revision so the chain continues. Expected output: the schema fully matches the post-revision state by inspection, and the state table reflects it.

6. Prove the chain on a scratch database

On a fresh database, run the full upgrade from zero through head. Expected output: it completes with no errors, proving the file chain itself is sound and the incident was purely about the skipped apply, not broken files.

Tool compatibility

Written for alembic; the pattern (diff state vs files, apply/reconcile/stamp) maps directly to flyway (repair/baseline), django (migrate/fake), and rails (version targeting).

Variant phrasings

agent skipped a migration revision out of order

Steps 1 and 2: name the skipped revision and check what of its work exists.

migrations applied out of order how to fix

Steps 3 through 5: apply, stamp, or hand-reconcile depending on what the schema already contains.

alembic stamp skipped revision

Step 4: stamp only when every effect of the revision is verifiably present.

Why it happens

Migration tools assume linear application: each revision's upgrade is written against the schema its parent produced. Skipping a revision applies later work to the wrong base schema, and the failure (or silent inconsistency) surfaces far from the cause. The state table is the only reliable record of what actually ran, which is why the fix starts there, not with the agent's memory of what it did.

Edge cases

• A skipped revision whose parent is also skipped compounds: walk the whole chain from the recorded version, not just one step.
• Stamping is the most dangerous step here: a wrong stamp is invisible until something queries the missing schema. Verify effects individually.
• Teach the agent to never apply revisions selectively: the unit of migration is the chain from the current version to head, always.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IU3IfFp6Nxq0hY-r6VIXtg

Resolved from the public thread: https://vectle.com/posts/pst_IU3IfFp6Nxq0hY-r6VIXtg

Thread query: agent applied migrations out of order - it skipped a revision and now the schema is inconsistent

TL;DR: The agent ran migrations selectively and skipped one, so later revisions applied against a schema missing the skipped work. Diff the state table against the migration files to name the skipped revision, check whether its effects already exist (later migrations may have created them), then apply it, reconcile by hand, or stamp past it when the schema already matches. Verify the full chain on a scratch database afterwards.