Letting Ready Jobs Pass a Job That Is Still Waiting for Its Attachment

A first-in, first-out queue is simple, but it has a weakness. If the oldest job is waiting for a large attachment to finish arriving, every job behind it can wait too, even though they could run. This article proposes a selection rule and an observation table to check it.

The rule

Give each queued job an explicit readiness state: ready, or blocked with a named reason (here, 'attachment pending'). The worker selects the oldest ready job. A blocked job keeps its original position, so when it becomes ready it is again the oldest candidate and is not pushed to the back.

The state must come from a checkable fact, such as 'the attachment record is marked complete', and not from a guess about time.

Decision branches

Branch A, independent jobs: nothing else depends on the blocked job. Later ready jobs may run past it.

Branch B, ordered by key: jobs for the same customer or record must run in sequence. A blocked job then holds back only later jobs with the same key, not the entire queue. Decide the key explicitly. If you cannot name one, treat everything as ordered and accept the blocking.

Branch C, wait too long: set a wait limit as a stated assumption. Past it, move the job to an 'needs attention' state with a visible reason rather than leaving it blocked indefinitely or failing it silently. Do not delete the partial attachment on a timer without an owner deciding.

Branch D, attachment arrives but fails a completeness check: the job is not ready. Move it to attention with the check result.

Worked hypothetical observation

Five jobs in arrival order. J2 awaits its attachment. J4 belongs to the same customer (C2) as J2 and must run after it. Others are independent.

Expected first pass: three jobs run (J1, J3, J5), two remain (J2 blocked, J4 held). Then the attachment completes. Now J2 is ready and is the oldest ready job, so it runs 4th, and J4 follows 5th. Total jobs run: 5, none run out of order within customer C2.

If the wait limit passes before completion, expected state: J2 in needs attention, and J4 still held, since its predecessor has not run. Someone must either resolve J2 or explicitly release J4 with a recorded reason. Silent release would break the ordering the key was meant to protect.

Starvation check

Because a blocked job keeps its position, it cannot be starved by later arrivals once ready. But a stream of ready jobs could keep a worker busy; if that matters, the observation below should show how long a newly ready job waits.

Stop conditions

• Stop skipping blocked jobs if the readiness fact cannot be read reliably; fall back to strict order and accept delay.
• Stop if the number of blocked jobs grows past a stated count, which may point to an upstream upload problem rather than normal waits.
• Stop any rule that lets a job run before its stated predecessor under Branch B.

Proposed acceptance tests (to be performed by you, none performed here)

1. Reproduce the five-job table in a test queue and record the actual order next to the expected one.
2. Complete J2's attachment and confirm J2 runs before J4.
3. Exceed the wait limit and confirm the attention state appears with a reason, and J4 stays held.
4. Submit an incomplete attachment and confirm J2 does not become ready.
5. Record how long J2 waited after becoming ready before running.

The states and limits here are design suggestions for your own system, not a description of any existing queue product.

Sources