Make absence-based deletions reversible: soft marking, grace windows and post-incident repair

When to use

Use this when a job takes a destructive or conclusive action **because a record was missing** from something it read: deleting local rows absent from a remote listing, archiving accounts absent from a directory export, revoking access absent from a permissions feed, or creating a "new" record because a lookup found no prior one.

This is the companion to the rule that you must not act on an absence unless the source answered *and* its answer is confirmed complete. That rule is about the decision. This guidance is about the two things it does not cover:

- The completeness signal itself can be wrong. A source can report an end-of-list marker and still have served a truncated listing; a snapshot can be taken from a replica that had not caught up; a filter change can make records legitimately drop out of scope and then come back.

- Once wrong deletions have run, someone has to find and undo them, and a naive rollback usually makes it worse.

Assume your completeness check will eventually be wrong, and design so that being wrong is survivable.

Rule

**An absence must be observed more than once, across a window longer than the source can plausibly take to recover, before anything irreversible happens — and every absence-based action must carry enough evidence to be undone.**

Procedure

1. Split the action into a mark phase and an execute phase

The mark phase is cheap and reversible. It records a *candidate* for deletion; it does not delete. It stores, alongside the record:

- the identity of the source listing that did not contain the record,

- the position or watermark that bounded that listing (final cursor, snapshot id, export sequence),

- the time the observation was made,

- the completeness evidence that was accepted (end-of-list marker, exact count, snapshot id).

The execute phase runs later and independently. It deletes only records whose mark is still standing and which satisfy the confirmation conditions below.

The value of the split is that a mark is auditable while it is still harmless. A wrong completeness check produces a visible spike in marks *before* it produces data loss, which is the only cheap moment to catch it.

2. Require repeated, independent confirmation

Do not execute on a single observation, however complete it looked. Require that the record was absent from **at least two listings that do not share a failure mode**. Two consecutive pages of the same truncated export are one observation, not two. Useful separations:

- different fetches separated in time, so a transient truncation is unlikely to repeat identically,

- a different access path where one exists — for example a direct lookup of that specific identifier, rather than another full listing.

A direct per-record existence check is the strongest confirmation available and is usually cheap, because the number of marked candidates is small even when the listing is large. Prefer it whenever the source offers one. A listing says "I did not mention it"; a direct lookup says "it is not there", and only the second statement is about the record.

3. Size the grace window to the source's recovery time

The window between mark and execute must exceed the longest plausible time for the source to return to a correct answer. Derive it, do not round it:

- replication or index rebuild lag at its observed worst, not its median,

- retention of the source's own backfill or catch-up process,

- the interval at which the source itself is refreshed — a feed regenerated nightly cannot confirm anything twice within one night, so its window is at least two cycles.

A window shorter than one refresh cycle of the source provides no confirmation at all: it just re-reads the same stale artifact. This is the most common way a grace window becomes decorative.

4. Clear marks on reappearance, and treat clearing as a signal

If a marked record appears in a later listing, clear the mark and record that it was cleared. The clear rate is a free, continuous test of your completeness logic. A steady trickle is expected; a burst means a listing you accepted as complete was not, and the execute phase for that source should be paused before it drains the backlog.

Alert on the clear rate, not only on listing errors. Listing errors are the case you already handle; wrongly-confident successes are the case that hurts, and the clear rate is the only place they surface early.

5. Make the execute phase recoverable rather than destructive

Prefer an action that can be reversed without reconstructing state: a soft-delete flag, a move to a quarantine area, a revocation that can be re-granted. Keep the reversible form for at least as long as it would take a human to notice the problem — which is typically far longer than the grace window, because nobody looks until a user complains.

Whatever form it takes, each executed action writes a repair record containing the record identity, the full prior state (or a pointer to a retained copy), both justifying observations, and the execution time. Without the justifying observations you cannot scope a repair later; you will only be able to ask "what did this job delete", not "what did it delete *because of the incident*", and the difference is between repairing a hundred records and reviewing a million.

6. Repair by re-verification, not by rollback

After an incident where absence-based actions ran on bad data, do not restore everything the job touched during the incident window. Some of those deletions were correct, and restoring them resurrects records that were genuinely removed at the source — which then get deleted again on the next run, producing a confusing flap, or worse, get treated as new.

Instead:

1. **Scope by evidence, not by time.** Select repair records whose justifying observations came from the affected source and overlap the incident, using the recorded listing identity and watermark. Time alone over-selects, because unrelated correct deletions ran in the same period.

2. **Wait for the source to be demonstrably healthy.** Repairing against a source that is still degraded reproduces the original bug. Require a fresh listing that passes the completeness check *and* whose size is consistent with the pre-incident baseline. A listing that is complete by its own marker but a third smaller than yesterday's is still broken.

3. **Re-verify each candidate individually.** Direct lookup per record beats another listing. Records that exist at the source were wrongly removed and should be restored; records genuinely absent should stay removed, with the repair record annotated so the same candidate is not re-examined on a later pass.

4. **Restore idempotently under stable identity.** Restore keyed on the source's own identifier, not on a locally generated one, so that a repair run interrupted halfway and restarted does not create second copies.

7. Repairing wrongful creation is a different job

The mirror-image failure — a lookup returned nothing because it was incomplete, so the job created a record that already existed — does not repair by deleting the new record. The new record may have accumulated its own references, activity or child rows since it was created.

Repairing that is a merge: pick a surviving identity, re-point references, then retire the other. It needs a stable natural key to match the pair, which is exactly what was missing when the duplicate was created. If your create path has no such key, the duplicate is not automatically repairable at all, and the mark-and-confirm discipline above matters more, because there is no cheap undo waiting at the end.

Boundary: what this does not make safe

Reversibility ends at your own storage. An action with an external effect — a notification sent, a payment issued, an external account deleted at a third party, a webhook delivered — cannot be repaired by this procedure, because the repair record can only restore your side. For those, confirmation must be complete *before* the action, and the grace window is not a safety net but the entire safety mechanism.

Similarly, the procedure assumes the absence decision is at worst *wrong*, not *adversarial*. A source that can be induced to omit records by whoever benefits from their deletion is an authorization problem, and no grace window addresses it.

Reasoned walkthroughs

These are worked examples, not executed tests. They are included because the failure modes only become visible when the timings are made concrete.

**A window shorter than the refresh cycle.** A directory export is regenerated once every twenty-four hours. A job marks absent accounts and executes after a six-hour grace window, re-reading the export twice more in that time. All three reads return the same file. One night the export is generated during a partial outage and omits a department. Six hours later, with "three independent confirmations", every account in that department is deleted. The window looked conservative and confirmed nothing, because all three observations came from one artifact. Two full cycles plus a direct lookup per candidate would have caught it at the first lookup.

**Scoping a repair by time instead of evidence.** A reconciliation job deletes roughly two hundred records a day in normal operation. During a four-hour incident it deletes nine thousand. Restoring everything deleted in a window that spans the incident restores the correct deletions from that period too. Those records are absent at the source, so the next run marks them again and deletes them again — and any downstream consumer that saw them reappear now has a spurious create-and-delete pair. Scoping by the recorded listing identity selects the wrongly deleted set and leaves the correct deletions alone.

Checks worth running before relying on this

- Make a listing return an end-of-list marker on a deliberately truncated result. Confirm that candidates are marked, that nothing is executed, and that the mark rate is visible somewhere a human would see it.

- Restore the truncated records before the window expires. Confirm every mark clears and that the clear burst raises an alert.

- Let the window expire with the truncation still in place, with per-record direct lookups enabled. Confirm the lookups override the listing and nothing is deleted.

- Run a repair pass twice against the same incident. The second pass should change nothing — no duplicate restores, no re-examination of candidates already confirmed genuinely absent.

- Interrupt a repair pass midway and restart it. Confirm restoration is keyed on source identity and produces no second copies.

Pitfalls

- A grace window measured in wall-clock time while the source refreshes on its own slower cycle, so repeated reads observe one artifact.

- Counting retries of the same failed fetch as independent confirmations.

- Hard-deleting immediately and relying on database backups as the repair path. Backups restore a whole point in time, not a selected set, and restoring one is usually more disruptive than the original loss.

- Repair records that store only the identifier and not the prior state, so the repair can identify the damage but not undo it.

- Treating a filter or scope change as a data outage. Records that legitimately left scope will mark and never clear; if you cannot tell the two apart, a configuration change silently drains the backlog. Version the filter definition in the mark, and invalidate outstanding marks when it changes.

- Running the execute phase from the same process and schedule as the listing fetch, so a bad fetch and its consequences land in the same run with nothing in between to inspect.

Evidence level

This guidance is reasoned design analysis. No tests, reproductions, incident data or benchmarks were executed for it, and it cites no external sources. The walkthroughs above are constructed examples used to expose the timing failures, not observed incidents. The suggested checks are proposed verification steps for adopters; they have not been run.