One Busy Customer, Many Waiting Ones: Per-Tenant Limits and a Fairness Log
A shared worker pool processes jobs for several customers, called tenants here. A single first-in, first-out queue works until one tenant submits a huge batch. Then every other tenant's small job waits behind it. This article proposes per-tenant limits and a log that shows whether scheduling was actually fair. It is a conditional design proposal, and the numbers are illustrative.
Two controls, kept separate
- A per-tenant concurrency cap: how many of one tenant's jobs may run at once.
- A selection rule: when a worker is free, which tenant goes next. A simple proposal is round-robin across tenants that have waiting work.
The cap protects the pool from one tenant. The rotation protects each tenant from the others. Neither one replaces the other.
Decision branches
- Several tenants waiting: rotate between them, skipping any tenant already at its cap.
- Only one tenant waiting: you must choose between letting it use idle workers (work-conserving) or holding the cap strictly. Idle workers help throughput but could delay a newly arriving tenant, so decide in advance and write it down. A reasonable proposal is to allow spare use, since a new arrival gets the next free worker through the rotation.
- A tenant's jobs are failing repeatedly: that is a separate problem. Failures should not consume more than the tenant's own cap.
Primary artifact: fair scheduling observation log
Record one row per scheduling tick. Columns: tick, workers free, tenants waiting with queue lengths, tenants chosen, tenants skipped and why, cumulative completed per tenant.
Worked hypothetical example
Two workers. Every job takes exactly one tick. Per-tenant cap is 1 concurrent while others are waiting. Tenant A has 40 jobs, B has 3, C has 2. Rotation order is A, B, C.
| Tick | Chosen | A done | B done | C done |
|---|---|---|---|---|
| 1 | A, B | 1 | 1 | 0 |
| 2 | C, A | 2 | 1 | 1 |
| 3 | B, C | 2 | 2 | 2 |
| 4 | A, B | 3 | 3 | 2 |
After tick 4, B has completed 3 of 3 and C has completed 2 of 2. A has completed 3 of 40, with 37 remaining. In a strict first-in, first-out queue where A's 40 jobs were submitted first, B and C would have waited for roughly 20 ticks with two workers. Here they finish by tick 4. Treat that comparison as the output of this hypothetical, not as a measurement.
From tick 5, only A is waiting. Under the work-conserving choice, both workers take A jobs, so 37 jobs take 19 ticks, with 18 pairs and one single. If a new tenant D arrives at tick 8, D is added to the rotation and should get a worker at the next tick where a slot frees. The log should show D's wait in ticks.
Stop conditions
Pause and review the design if:
- Any tenant with waiting work and below its cap is skipped for more than one full rotation without a recorded reason.
- A single tenant occupies every worker while another tenant is waiting.
- Long-running jobs make the rotation meaningless. In that case consider time-bounded jobs or separate pools, which are different designs.
- Tenant identity on jobs is missing or untrusted. Fairness depends on it, so derive it from authenticated context, not from the job payload alone.
Proposed acceptance tests (not yet performed)
- Replay the A, B, C scenario in a test harness. B and C must complete in the four ticks shown.
- Add D mid-run. D must receive a worker within one rotation.
- Set the cap to 1 and submit 40 jobs from A with C waiting. A must never hold two workers while C is waiting.
- Remove all other tenants and confirm the chosen idle-capacity policy applies.
- Inspect the log. Every skipped tenant has a reason, and completed counts sum to the number of jobs finished.
Fairness here is a measured property of a log, not an assumption. If the log disagrees with the design, fix the design.