The Record Is Gone but the Job Is Still Running: A Tombstone Race Fixture
A staff member deletes a customer record. Moments earlier, an automation job was queued to update a related spreadsheet, send a message or sync another system. If the job runs after the deletion and still acts, it may resurrect data nobody wants. This article proposes a tombstone check and a small race fixture to test it. It is a conditional design recommendation, to be adapted to your own system.
The idea
Do not erase the record's identity at deletion. Keep a tombstone: record id, a generation number, a deleted-at time, and a version. Every job carries the id, the generation and the version it was created for. Immediately before any write or outward action, the job checks the current state. If the record is a tombstone, or the generation differs, the job stops and is marked rejected, with a reason.
This check narrows the race but does not remove it unless the check and the action are tied together. For internal writes, make the write conditional on the record still being live at that version. For external actions, treat the result as uncertain: a deletion could land between check and send. Log it, and do not repeat an uncertain external write automatically.
Primary artifact: the race fixture
A table of cases, each with an ordered event script and an expected outcome.
| Case | Order of events | Expected |
|---|---|---|
| 1 | Job queued, record deleted, job starts | Job rejected: tombstone found |
| 2 | Job starts, passes the check, record deleted, job writes | Conditional write fails, job rejected |
| 3 | Record deleted, then recreated with the same business reference, then old job starts | Rejected: generation differs |
| 4 | Job starts, record updated to the next version, job writes | Not a deletion. Stale version, re-read and decide |
Cases 1 to 3 are deletion branches. Case 4 is included so that the rule does not turn every conflict into a deletion rejection.
Worked hypothetical example
Record R is at generation 1, version 5. A job J1 is queued for generation 1, version 5.
Case 1: R is deleted, producing a tombstone at generation 1. J1 starts, finds the tombstone, and records "rejected: source deleted". Writes made: 0.
Case 3: R is recreated, now generation 2, version 1. A second job J2 created earlier carries generation 1. When J2 starts it compares 1 with 2 and is rejected. A new job J3 carries generation 2 and proceeds normally. Across the two jobs: 1 rejected, 1 accepted, 1 write.
The fixture result for the four cases is 3 rejected for source state and 1 re-read for a stale version. 4 cases in total, with no case unaccounted for.
Branch handling
- Rejected jobs are not retried. A retry would hit the same tombstone.
- Jobs failing for temporary reasons, such as a timeout before any write, may retry within a limit, with the check repeated each time.
- If the tombstone itself cannot be read, treat that as unknown and pause the job. Do not assume the record is live.
Stop conditions
Stop and escalate if:
- Jobs lack a generation or version, so the check is impossible.
- Tombstones are removed before queued jobs can expire.
- A rejected job appears in logs as a success.
Proposed acceptance tests (not yet performed)
- Run all four cases in a test environment. Each must match its expected outcome.
- Delete the record between the check and the write on purpose. The write must fail.
- Confirm a rejected job leaves a reason, an id and a timestamp, with no personal content copied into the log.
- Confirm a recreated record is never touched by a job from the earlier generation.
- Confirm that tombstone retention is longer than the longest time a job can sit in the queue.
The layers work together: a tombstone, a generation check, a conditional write and honest logging. Each one narrows the window, and none claims to close it alone.