Lorixen.
EN

Background job queues: stop making customers wait

Short answer: If a customer clicks "Place order" and waits while your app sends emails, generates a PDF invoice, resizes photos and updates a stock system, they are waiting for work they do not need to watch. Move that work to a background job queue: the web request saves the order, adds a small job to the queue and responds straight away, while separate worker processes pick up jobs and run them. Make jobs small, safe to run twice and set to retry with backoff, and watch the queue so failures do not pile up unseen.

The symptoms of doing everything in the request

Most apps start by doing everything inside the request that triggered it. That is fine until the extra work grows:

  • Checkout or form submission takes several seconds, and some customers click twice.
  • A slow email provider or supplier API makes your own pages time out.
  • A burst of sign-ups or orders makes the whole site sluggish, because web servers are busy doing background chores.
  • One failing step, such as a PDF generator crashing, makes the customer see an error even though their order was saved.

All of these share a cause: work that could happen a few seconds later is happening while a person waits.

How a job queue works

A job queue has three parts:

  1. Producers, usually your web app, which add jobs such as "send order confirmation for order 4821".
  2. A queue store, often Redis or a database table, which holds jobs until they are processed.
  3. Workers, separate processes that take jobs off the queue and run them, one or several at a time.

Popular libraries include Sidekiq for Ruby, Celery for Python, BullMQ for Node.js and the queue systems built into frameworks such as Laravel. Cloud platforms also offer managed queues. The concepts are the same across all of them.

What belongs in the background

Good candidates are tasks the customer does not need to see finish before moving on:

  • Sending emails, text messages and push notifications.
  • Generating PDFs, exports and reports.
  • Resizing, compressing or scanning uploaded images and files.
  • Calling third-party APIs: accounting, shipping, CRM, marketing lists.
  • Search index updates and cache warming.

Keep in the request anything the customer must see the result of right now, such as a card payment being approved or declined, or a form validation error.

Rule one: pass IDs, not objects

Sidekiq's best-practice guide makes this its first rule: keep job parameters small and simple. Sidekiq stores the arguments as JSON in Redis, and the guide warns against passing a full record object; pass its ID and load fresh data when the job runs. That keeps the queue small and means the job uses current data, not a stale copy taken when it was queued.

Rule two: jobs must be safe to run more than once

The same guide's second rule is to make jobs idempotent and transactional. With automatic retries, a job might be half-processed, throw an error and run again. If your "send invoice" job emails the customer and then crashes while updating the database, a naive retry sends a second email.

Practical ways to make jobs idempotent:

  • Record that a step happened, such as "confirmation_sent_at", and check it before doing the step again.
  • Use idempotency keys when calling external APIs that support them.
  • Split large jobs into smaller ones, each doing one thing.

Rule three: retries with backoff and a limit

Temporary failures are normal. BullMQ's documentation shows the pattern: automatic retries are switched on with an attempts setting larger than 1, and failed jobs can be retried using backoff functions. Its built-in strategies are fixed, which waits the same delay each time, and exponential, where the delay grows with each attempt. Other libraries offer similar options.

Set a maximum number of attempts. When a job uses them all, it should land somewhere visible, such as a failed-jobs list, an alert or a dead-letter queue, rather than vanishing.

Rule four: watch the queue

A queue hides problems by design: the customer never sees a job fail. So someone on your side has to. Keep an eye on:

  • Queue length: a growing backlog means workers are too slow or stuck.
  • Job age: how long the oldest waiting job has been there.
  • Failure count: jobs that exhausted their retries.
  • Worker health: whether worker processes are actually running after a deploy or server restart.

Most queue libraries come with a dashboard. Add alerts on backlog and failures, and treat "workers not running" as an outage.

A worked example

Picture a hypothetical Christchurch outdoor clothing retailer whose checkout took up to eight seconds. After saving the order, the app sent a confirmation email, created a PDF invoice, posted the order to a warehouse system and updated a loyalty points service, all before showing the thank-you page. When the warehouse system was slow, checkouts timed out and some customers paid twice.

The developers kept payment and order creation in the request and moved the other four tasks into separate background jobs, each receiving only the order ID. Each job records when it completes, so a retry skips finished steps. Retries use exponential backoff with five attempts, and failed jobs alert the developer. The thank-you page now appears in about a second, and a slow warehouse API delays a warehouse update instead of a sale.

FAQ

Do I need Redis to run a job queue?

Not always. Several libraries can use your existing database, which is often enough for a small app. Redis-backed queues handle higher volumes.

How many workers should I run?

Start with one or two processes and increase based on queue length and job age during busy periods.

What happens to jobs during a deploy?

Workers should finish or release their current job before stopping. Check your library's guidance on shutting workers down gracefully.

Can I schedule jobs for later?

Yes. Most queues support delayed jobs, such as a reminder 24 hours after a quote is sent.

If your checkout or forms feel slow because of everything that happens after the click, describe what your app does after a customer submits.

Sources

WhatsAppBook a call