When the Answer Shows Up After You Gave Up: Handling Late Job Callbacks
A timeout is a local decision. It says your system stopped waiting, not that the outside work stopped. If the remote service later reports back, your code needs a rule for that message. Without one, the late callback either overwrites a state someone already acted on, or it is dropped and the evidence is lost.
This article proposes one artifact, a late callback disposition table, and shows how to build and test it. Everything below is a design recommendation, not a description of any existing product.
What the table keys on
For each arriving callback, look at three facts:
- The local job state now (timed_out, retried, cancelled by an operator, completed, or no matching job).
- The callback outcome (reports success or reports failure).
- Whether a replacement attempt exists that could have produced the same external effect.
The third fact matters most. A callback that reports success for work you have also retried may mean the effect happened twice. Your local record cannot prove otherwise, so a local flag alone is not evidence of exactly-once behaviour.
The disposition table
| Local state | Callback | Replacement attempt | Disposition |
|---|---|---|---|
| timed_out | success | none | Apply: mark completed_late, keep the timeout event in history |
| timed_out | success | exists | Hold: flag possible duplicate effect, send to a person |
| timed_out | failure | any | Record only: state stays timed_out |
| cancelled | any | any | Record only: attach as evidence, no state change |
| completed | any | any | Record only: note a repeat report |
| no match | any | n/a | Quarantine: keep the raw message for review |
Two rules sit above the table. First, no disposition triggers an automatic repeat of an external write whose outcome is uncertain. Second, every row stores the callback's arrival time next to the timeout time so a reviewer can see the gap.
Worked hypothetical
An invented booking tool sets a 120 second timeout on a confirmation job. Five callbacks arrive during one afternoon:
- C1 reports success for a timed_out job with no replacement. Disposition: apply. State becomes completed_late.
- C2 reports success for a timed_out job whose retry was created by a staff member. Disposition: hold for review.
- C3 reports failure for a timed_out job. Disposition: record only.
- C4 refers to a job an operator cancelled. Disposition: record only.
- C5 carries an identifier no local job matches. Disposition: quarantine.
Totals: 1 applied, 1 held, 2 recorded without change, 1 quarantined. That is 5 callbacks, each with exactly one disposition. If your own totals do not sum to the callback count, a row in the table is missing a branch.
Stop conditions
Stop automatic handling and escalate to a person when any of these is true:
- A replacement attempt exists and the callback reports success.
- The same job receives conflicting callbacks (one success, one failure).
- Callbacks arrive faster than a person can review the held and quarantined ones, which suggests a systemic fault.
Do not widen the timeout as a reaction. That changes how often the table is used, not whether it is right.
Proposed acceptance tests
These are proposals to run in a safe test setup, and none has been run.
- Replay the five callbacks above against a copy of the job data. Expect the 1, 1, 2, 1 split and no new outbound request.
- Deliver C1 twice. Expect the second to be recorded only, with state unchanged.
- Deliver C2 and check that no automatic retry or cancellation is issued.
- Deliver a callback with a missing identifier. Expect quarantine, not an error that discards it.
- Inspect a held job and confirm the reviewer can see the timeout time, arrival time, and replacement attempt together.
Limits
The table cannot tell you whether the remote side performed a duplicate effect. Only a check against that system, or a person, can. Treat held rows as questions, not conclusions, and write down the answer next to the row so the same case is not guessed twice.