Lorixen.
EN

Graceful shutdown: deploy without dropping requests

Short answer: Every deploy, restart or scale-down stops some running copies of your app. If a process is killed mid-request, customers see errors, half-saved forms and failed payments. Graceful shutdown means your app listens for the stop signal (usually SIGTERM), stops accepting new work, finishes what it is already doing within a time limit, closes its database connections and then exits. Pair it with a load balancer that stops sending traffic first and a grace period long enough for your slowest normal request.

What actually happens when an app is stopped

Whether you run on a virtual server, Docker, Kubernetes or a platform that handles deploys for you, stopping a process usually works the same way: the system sends a polite signal, waits a while, then forces the issue.

Docker's documentation for docker stop puts it simply: the main process in the container receives SIGTERM and, after a grace period, SIGKILL. If no default is configured, the daemon uses 10 seconds for Linux containers. Kubernetes behaves similarly. Its pod lifecycle documentation explains that the container runtime sends a TERM signal to process 1 in each container, and that a pod is granted a term to terminate gracefully, which defaults to 30 seconds via terminationGracePeriodSeconds.

SIGKILL cannot be caught or delayed. Anything your app was doing at that moment simply stops. The window between SIGTERM and SIGKILL is your chance to tidy up.

The symptoms of no graceful shutdown

  • A handful of 502 or 503 errors on every deploy, often blamed on "the load balancer".
  • Orders saved but confirmation emails never sent, because the process died between steps.
  • Background jobs left half-finished and then retried, sometimes twice.
  • Database connection errors after a restart, because old connections were never closed cleanly.

If you only deploy at quiet times, these may be rare enough to ignore. Once you deploy several times a day, or autoscale up and down, they become a steady trickle of failed requests.

The shutdown sequence

A well-behaved web process does this when SIGTERM arrives:

  1. Stop taking new requests. Close the listening socket, or mark the instance as not ready so the load balancer stops routing to it.
  2. Finish requests in flight. Let current requests complete, up to a time limit shorter than the platform's grace period.
  3. Stop background workers cleanly. Let a worker finish its current job or return it to the queue rather than abandoning it mid-way.
  4. Close resources. Database pools, message broker connections and open files.
  5. Exit. With exit code zero if all went well.

In Node.js, the built-in HTTP server's close() method, as the Node documentation describes, stops the server accepting new connections and closes connections that are not sending a request or waiting for a response. You then wait for active requests to finish, close your database pool and call process.exit(). Most frameworks in other languages have equivalent hooks, and many web servers and process managers already handle part of this for you if configured.

Don't forget the load balancer

There is a timing gap that catches many teams. When an instance starts shutting down, the load balancer may not have noticed yet and can keep sending it new requests for a few seconds. If your app closes its socket immediately, those requests fail.

Two common fixes:

  • Fail the readiness check first. On SIGTERM, start returning "not ready" from your readiness endpoint, keep serving for a few seconds while the load balancer drains you, then begin closing.
  • Add a short pre-stop delay. Kubernetes supports a preStop hook that runs before the container is stopped; a few seconds' pause there gives routing time to update. The Kubernetes docs note that if the hook is still running when the grace period expires, the kubelet allows only a small one-off extension of 2 seconds, so budget the delay inside your grace period.

Choosing time limits

Your app's own shutdown timeout should be:

  • Longer than your slowest normal request, so ordinary traffic is never cut off.
  • Shorter than the platform's grace period, so your app exits on its own terms rather than being killed.

If an export or report endpoint routinely runs for a minute, either move it to a background job or raise the grace period. Do not set huge grace periods by default; they make deploys slow and can hide hung processes.

Common traps

  • The signal never reaches your app. If your container starts the app through a shell script, the shell may be process 1 and not forward SIGTERM. Use an exec form of the start command or a small init process.
  • Long-lived connections. WebSockets and streaming responses will not finish by themselves; tell clients to reconnect and close them deliberately.
  • Keep-alive connections. Idle keep-alive connections can hold a server open; close them explicitly once new requests have stopped.
  • Assuming the platform handles it. Managed platforms send the signal; whether your code reacts properly is still up to you.

A worked example

A hypothetical Bristol online ticketing start-up deployed several times a day and saw a burst of failed checkout requests each time. Logs showed the app was being killed while payments were still being confirmed, because the container started via a shell script that swallowed SIGTERM, so every shutdown ended in SIGKILL.

The team switched to an exec-form start command, added a SIGTERM handler that marks the instance not ready, waits five seconds, closes the server, lets in-flight requests finish within 20 seconds and closes the database pool. They kept the 30-second grace period. Deploy-time errors dropped to none in their monitoring, and they stopped scheduling releases for late evenings.

FAQ

Does serverless need graceful shutdown?

Less often, since the platform controls lifecycles, but long-running functions and connection pools still benefit from cleanup hooks where offered.

What about scheduled jobs that run for hours?

Make them resumable: save progress regularly so a restarted job continues instead of starting over.

How do I test it?

Send SIGTERM to a running instance under load and watch for failed requests in your logs or a load-test tool.

If every deploy leaves a trail of errors in your logs, tell us how your app is hosted and released.

Sources

WhatsAppBook a call