Putting a Batch Back in Order After Workers Finish Out of Sequence

When several workers process items from one batch, completion order says nothing about the original order. If a downstream step needs the original order, such as lines on a document or rows in a report, the order must be recorded before work starts and checked when results come back. This article proposes a design. It is a recommendation to adapt, not a description of any existing system.

The primary artifact: a sequence reconstruction manifest

Create the manifest when the batch is accepted, before any worker sees an item. One row per item:

batch_id
position: an integer assigned once, starting at 1, never reused
item_key: the stable identifier of the input item
state: pending, done, failed, or abandoned
result_ref: a pointer to the stored result, empty until done
finished_at: informational only, never used for ordering

The manifest also stores the expected_count, fixed at creation. Workers receive the position with the item and return it unchanged. Assembly sorts by position and ignores finished_at.

Decision branches

Use these branches for the assembly step. They do not overlap, and each batch ends in exactly one.

Every position from 1 to expected_count is done: assemble in position order.
At least one position is still pending: wait. Do not assemble a partial result as if it were complete.
At least one position is failed or abandoned and none are pending: do not fill the gap by shifting later items up. Either assemble a result that is explicitly marked incomplete, with the missing positions listed, or hold the batch for a person to decide.
The manifest has duplicate positions, a position above expected_count, or a result whose item_key does not match its row: stop and flag the batch as inconsistent.

Worked hypothetical example

A batch has 5 items, so expected_count is 5. The positions are 1 to 5, with item keys A to E. Workers finish in this order: C, A, E, B. The manifest now shows positions 1, 2, 3 and 5 done, and position 4 (item D) still pending. That is branch 2, so assembly waits.

Then D reports failure. No positions are pending and one is failed, which is branch 3. The assembler either produces A, B, C, E marked incomplete with position 4 listed as missing, or holds the batch. It never presents E as the fourth item.

Suppose D is then retried and succeeds. The retry writes to position 4 only. The count is now 5 of 5, so branch 1 applies and the output is A, B, C, D, E.

A retry must not be repeated automatically if its earlier outcome is uncertain, for example if the worker timed out after it may have already written to an external system. Mark that position as needing review. Resubmit only after someone confirms the earlier write did not take effect, or after the external step is shown to be safe to repeat. A local claim that a result was written is not proof of exactly-once delivery.

Stop conditions

Halt assembly on any branch 4 condition.
Halt if the manifest was changed after workers started, other than state and result_ref updates.
Stop waiting after a limit you choose in advance, and move the batch to branch 3 handling. Do not wait without bound.

Proposed acceptance tests (not performed)

These are proposals. None has been run.

Shuffle the completion order of a 5-item batch in several ways. Output order should always be by position.
Make one item fail. Output should be marked incomplete, list the exact missing position, and keep every other item at its original position.
Return the same result twice for one position. The second should be rejected or ignored without changing the output.
Return a result with a mismatched item_key. The batch should be flagged inconsistent.
Submit a batch of 1 item and an empty batch. Decide in advance what each should do, and check that it does.
Set finished_at to an earlier time than expected for a later position. Output order should not change.

The design works only if position is assigned once, up front, and treated as authoritative. Any later step that re-sorts by time or by identifier undoes it.