Streaming replies arrive all at once? Check proxy buffering
Short answer: If a response streams nicely on your laptop but arrives in one lump in production, something between your app and the browser is collecting the whole response before passing it on. The usual suspects are a reverse proxy such as nginx with buffering switched on (it is by default), compression middleware waiting for enough data, or a CDN or hosting platform that buffers by design. Turn off buffering for the streaming route only, for nginx with proxy_buffering off; or an X-Accel-Buffering: no response header, exclude the route from compression, and send small keep-alive messages so long pauses don't hit timeouts.
What it looks like
Streaming is everywhere now: AI chat answers appearing word by word, progress updates while a report is generated, live order status via server-sent events. Typical production symptoms:
- The AI answer appears all at once after 15 seconds of nothing.
- A progress bar jumps from 0% to 100%.
- Server-sent events arrive in bursts, or only when the connection closes.
- Long streams are cut off after about a minute with an error.
Locally, your app talks to the browser directly. In production, there's usually a proxy, a load balancer, a CDN or a platform layer in between, and each one can hold data back.
Cause 1: proxy buffering
nginx's documentation describes how buffering works for proxied responses. When buffering is enabled, nginx receives the response from your app as fast as it can, storing it in memory buffers or, if it doesn't fit, a temporary file on disk. When buffering is disabled, the response goes to the client synchronously, as it's received.
Buffering is a good default: it frees your app quickly for ordinary pages. For streams, it means the browser sees nothing until a buffer fills or the response ends.
Two ways to switch it off for streaming routes:
location /api/chat/stream {
proxy_pass http://app;
proxy_buffering off;
proxy_read_timeout 300s;
}
Or have your app send a response header. nginx's docs say buffering can be enabled or disabled by passing "yes" or "no" in the X-Accel-Buffering header. MDN's server-sent events example includes X-Accel-Buffering: no for exactly this reason. The header approach is handy when you can't easily edit the proxy configuration, but note that nginx can be configured to ignore it.
Cause 2: compression
Gzip or Brotli middleware compresses in blocks. To compress efficiently, it waits for a chunk of data before sending anything, so small streamed pieces sit in its buffer. Exclude streaming routes or content types such as text/event-stream from compression, or make sure your framework's compression flushes after each write.
Cause 3: CDNs, load balancers and platforms
Many CDNs and hosting platforms buffer responses by default, and some serverless platforms only support streaming in particular function types or modes. Check your provider's documentation for "streaming responses" or "server-sent events". Often there's a setting, a header or a specific runtime that enables it. If the CDN can't stream, route the streaming endpoint around it, for example through a separate subdomain that goes straight to your app.
Cause 4: your own code
Before blaming the infrastructure, confirm the app actually flushes:
- Some frameworks buffer output until the handler finishes unless you use their streaming response type.
- Writing to a response without flushing may hold data in a buffer.
- Logging or error-handling middleware that reads the full body to record it defeats streaming.
Test the app directly, bypassing the proxy, with curl -N (no buffering in curl) against the app's internal port. If it streams there but not through the public URL, the problem is in front of the app.
Don't forget timeouts
Streams can go quiet: an AI model may think for a while before producing text. nginx's proxy_read_timeout defaults to 60 seconds, and its documentation explains that this timeout applies between two successive reads, not to the whole response. If nothing arrives within that time, nginx closes the connection.
Two fixes, best used together:
- Raise the read timeout for the streaming route.
- Send a keep-alive every 15 to 30 seconds. With server-sent events, a comment line starting with a colon is ignored by the browser but keeps the connection active.
Load balancers and CDNs have their own idle timeouts, so check each layer.
Connection limits for server-sent events
MDN warns that server-sent events over HTTP/1.1 run into the browser's limit of six open connections per domain, shared across tabs. A customer with several tabs open can find new pages stuck loading. Serve the site over HTTP/2, which most hosts and CDNs support, and close streams when they're no longer needed.
A step-by-step check
curl -Nthe app directly: does it stream?curl -Nthrough the proxy on the server: does it stream?curl -Nthe public URL: does it stream?
The first step where data arrives in one block shows which layer is buffering.
A worked example
An illustrative Halifax tutoring app added an AI study helper that streamed answers. On a laptop the words flowed; in production, students waited with a spinner and then saw the whole answer at once, and long answers sometimes failed. Testing with curl -N showed the app streamed fine, but nginx buffered. The developer added X-Accel-Buffering: no to the streaming endpoint, excluded it from compression and sent a keep-alive comment every 20 seconds. Answers now appear as they're generated, and the timeouts stopped.
FAQ
Should I turn off buffering for the whole site?
No. Keep it for ordinary pages and turn it off only for streaming routes.
Do WebSockets have the same issue?
WebSockets need their own proxy settings to upgrade the connection, but once open they aren't buffered in the same way.
Why does it work in one browser and not another?
Usually it doesn't; check extensions, corporate proxies or antivirus software that may also buffer.
Will disabling buffering slow my server?
Slightly, for those routes, because the app stays busy while the client reads. That's the cost of streaming.
If your streaming features stall or arrive in one lump, describe your hosting setup and we'll help you find the layer that's holding them back.