Gate absence-based deletes with two-phase marking, grace windows and outage repair

When to use

Use this when a job compares a local set with a remote or upstream listing and takes an action because something was absent: deleting a local record, removing a cache entry, deprovisioning an account, marking an item as gone, or creating a replacement because no existing one was found. Typical cases are sync and reconciliation jobs, cache invalidation sweeps, orphan cleanup, membership pruning and "create if missing" logic.

This skill assumes the caller already tells "the source answered and was empty" apart from "the source was unavailable or incomplete". It covers the next question: even when an answer looks trustworthy, how do you act on absence so that a wrong or stale answer does no permanent damage, and how do you repair the damage when an earlier answer turns out to have been wrong?

The failure pattern

A single listing can be wrong in ways that no status flag catches at the time:

• The listing was served by a lagging replica or a partly rebuilt index, so it was complete for an old moment, not for now.
• The listing was complete, but the item was briefly absent because of an in-flight move, rename, re-import or ownership change.
• The job trusted an incomplete listing because a bug or misclassification reported it as complete. The bug is found days later.

If the job deletes on first sight of absence, every one of these produces an irreversible action from one bad observation. If the job has no record of which observation justified which action, there is no way to find out afterwards what was deleted for a bad reason.

Rule

Never take a destructive or conclusive action on the first observation of absence. Split the action into two phases:

1. Mark. Record that the item was absent, together with evidence about the observation that saw the absence.
2. Act. Perform the real action only when the marks meet an explicit threshold, and only after a grace window has passed.

Any observation that shows the item present clears the mark completely. An observation that is unavailable or incomplete neither advances nor clears the mark; it is simply not counted.

How to apply

1. Give every listing an observation record

For each run of the listing, store an observation record with at least: an observation id, the start and end time, the source name, the status (ok, empty, unavailable), whether it was confirmed complete, and the freshness evidence used to trust it (snapshot id, cursor watermark, replication position, or index build marker). This record is the thing that later actions point to. Without it, repair is guesswork.

2. Mark absence instead of acting

When an item is missing from a listing that was ok and complete, do not delete it. Write or update an absence mark on the item holding: the time of the first absent observation, the number of confirming complete observations since then, and the id of the most recent confirming observation.

When the same item appears in any later ok listing, delete the mark entirely. Do not decrement it. Presence is definitive; absence is only evidence.

When a listing is unavailable or incomplete, leave all marks unchanged. Do not count the run as a confirmation and do not reset marks. A long outage therefore delays cleanup rather than triggering it.

3. Require both a count and a time span

Act only when two conditions hold together:

• At least N separate complete observations have confirmed the absence, with N of at least two.
• The elapsed time since the first absent observation is at least the grace window.

The count protects against one wrong listing. The time span protects against several listings that were all served from the same stale state within a short period. Neither alone is enough: three listings in ten seconds from a lagging replica agree with each other and are all wrong.

Choose the grace window from known staleness bounds of the source: replication lag ceiling, index rebuild duration, the longest expected in-flight move or re-import, and the time it takes a human to notice and reverse a mistaken change. The window must exceed the largest of these. If none is known, choose a conservative window and say so in the configuration comment, rather than defaulting to zero.

4. Make the action itself reversible where possible

Prefer a soft delete or tombstone with a retention period over an immediate hard delete. Keep the payload or enough of a reference to restore it. Run the hard delete as a separate later sweep over tombstones older than the retention period. This gives a third safety layer after the mark and the grace window, and it makes repair a restore rather than a recreation.

Where the action cannot be reversed (an external deprovisioning call, a notification that is sent, a billing event), require a stronger threshold: a larger N, a longer window, or a second independent source confirming the absence.

5. Log every absence-based action with its justification

Each action must record the item, the action taken, the time, and the id of the observation that satisfied the threshold, plus the first-absent time. This is the index used for repair.

6. Repair after a suspect window

When you learn that listings between two points in time cannot be trusted (an outage that was misreported as healthy, a replica that was silently behind, an index that served partial data, a bug that marked incomplete listings as complete), run a repair:

1. Define the suspect window as the earliest possibly-affected observation through the last one before the fix, with a margin on each side.
2. Collect every absence-based action whose justifying observation, or whose first-absent time, falls inside the window. Actions whose entire confirmation history lies outside the window can be kept.
3. Also collect every current absence mark whose confirmations include a suspect observation, and reduce those marks by removing the suspect confirmations. If a mark loses all confirmations, delete it.
4. Obtain a fresh listing that is ok, confirmed complete, and fresh by evidence, not by assumption. If you cannot obtain one, stop the repair and escalate; do not reason from the suspect data.
5. For each collected action, check the item against the fresh listing. If the item is present, reverse the action: restore the tombstone, undo the deprovisioning, or remove the duplicate that was created. If the item is truly absent, keep the action but re-justify it by pointing its log entry at the fresh observation.
6. For actions that cannot be reversed, produce a list for manual handling and a compensating action where one exists.
7. Record the repair itself as an observation with a note, so a later audit can see that the log was rewritten and why.

7. Pause the acting phase during known trouble

When the source is in a known degraded state (maintenance, migration, replica rebuild, a recent incident), disable only the act phase. Marking can continue, because marks are harmless and are cleared by presence. This keeps the job running without letting it destroy anything until the source is trustworthy again.

Reasoned example

A directory sync removes local accounts that no longer appear in the upstream directory. Upstream is served from a replica that can lag by up to thirty minutes. Nightly at two in the morning the replica falls behind, and the listing omits forty accounts created that day.

• Without gating: forty accounts are removed on the spot. They reappear in the next listing and are recreated as new accounts, which may lose their settings and history.
• With gating and a grace window of one day and N of two: the forty accounts receive an absence mark. The next listing at three in the morning, now caught up, shows them present, and the marks are cleared. Nothing was removed.
• If instead the replica stayed behind for a week and a bug had marked its listings complete, the accounts would be removed after the window. Once the bug is found, the repair collects every removal whose justification points at observations during that week, checks each against a fresh confirmed listing, and restores the ones that are present.

This example is reasoning about the procedure. It is not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Make an item vanish for one listing and reappear on the next. The item should be marked and then cleared, with no action taken.
• Make an item vanish for N complete listings inside a period shorter than the grace window. No action should run until the window has also elapsed.
• Make listings unavailable during the grace window. Marks should stay exactly as they were; the window should not be treated as satisfied by outage runs.
• Run the repair with a suspect window that covers a known set of actions. Every action in the window should be re-checked, present items restored, and absent items re-justified against the fresh observation.
• Confirm that a soft-deleted item can be restored with its original data after the repair.

Pitfalls

• Counting observations without a time span, or a time span without a count. Each catches a case the other misses.
• Resetting a mark to zero on an unavailable listing. That silently extends cleanup delays after every blip and hides real absences.
• Decrementing a mark on presence instead of clearing it. Presence is proof; it should not be outvoted by earlier absences.
• Logging actions without the justifying observation id. Repair then has to guess from timestamps alone.
• Running the repair against the same suspect source state. The repair needs a fresh, confirmed-complete listing or it must stop.
• Hard-deleting inside the same run that reached the threshold. Keep the hard delete as a later sweep over tombstones so a repair can still restore.