Two Dials, Not One: Testing Concurrency and Rate Separately in a Background Worker

A background worker that calls an outside service has two separate dials. Concurrency is how many calls may be in flight at once. Rate is how many calls may start in a time window. Teams often raise one when the other is the real constraint, then wonder why throughput did not move. This article proposes a worksheet to separate them. Everything here is a conditional design suggestion to adapt, not a description of any existing system.

The relationship to test

If each call takes about L seconds and you allow c calls in flight, the most the worker can start per second is roughly c divided by L. If a rate cap R is also configured, the expected ceiling is the smaller of the two numbers. Treat this as a prediction to check against measurement, because real latency varies.

The two-axis worksheet

Make a grid. Rows are concurrency settings. Columns are rate cap settings, plus one column with no cap. Each cell has four entries: predicted starts per second, measured starts per second, measured median latency, and the count of rejections or slow-down responses from the remote side. Fill the predicted entry before running anything. A surprise is a gap between prediction and measurement, and the gap is the thing worth studying.

Worked hypothetical

Imagine a fictional remote service where a call takes about 0.8 seconds, and your trial rate cap is 3 starts per second. Predictions:

c = 1: 1 / 0.8 = 1.25 per second. Concurrency binds.
c = 2: 2.5 per second. Concurrency binds.
c = 4: 5 per second without a cap, so the cap of 3 binds.
c = 8: 10 per second without a cap, so the cap of 3 still binds.

So in this example, moving from c = 4 to c = 8 should change nothing except the queue of waiting callers and possibly latency. If the measured rows for c = 4 and c = 8 differ much from each other, the model is wrong somewhere, perhaps latency grows under load.

Decision branches

Branch A, concurrency binds: measured rate tracks c divided by L and stays under the cap. Raising c may help, within what the remote side tolerates.

Branch B, rate binds: measured rate sits near the cap regardless of c. Raising c only adds waiting. Look at the cap or at batching instead.

Branch C, neither binds: measured rate is below both predictions. The worker is probably starved by its own source, such as an empty queue or slow local work. Test the producer, not the dials.

Branch D, remote pushback: slow-down or rejection responses appear below your own cap. Treat the remote side's signal as authoritative and reduce pressure.

Stop conditions

Stop raising either dial when any of these appears: rejections rise from the previous row, median latency grows faster than throughput, or the local queue of waiting jobs grows without bound during the run. Also stop if a call type is a write whose outcome could be uncertain after a timeout. Do not let the trial retry such writes automatically; record them as unknown and review them by hand, because repeating an uncertain external write may duplicate its effect.

Proposed acceptance tests

These are proposals to run later, not results.

Against a stub with fixed 0.8 second latency, check that measured starts per second for c = 1 and c = 2 land near the predictions, and record the gap.
With the cap at 3, check that c = 4 and c = 8 give similar throughput.
Make the stub slow down and confirm throughput falls and does not climb.
Make the stub drop a write response and confirm the job is marked unknown, with no automatic second attempt.
Starve the producer and confirm the worksheet points to Branch C.

Keeping the worksheet honest

Write down the date, stub or target, and latency distribution used, so a later reader knows the numbers describe one setup only. Rerun the grid when the remote behaviour or your job mix changes, since a setting that fit one workload may not fit another.