Dark launches: test new back-end code on real traffic
Short answer: A dark launch runs new back-end code in production on real traffic without customers seeing its results, so you learn how it performs and whether it gives the right answers before anyone depends on it. Two common forms: calling a new feature behind the scenes and discarding its output, and mirroring a copy of live requests to a new service whose responses are ignored. Use it for risky back-end changes such as a new pricing engine, search service or database, compare outputs, watch load, and make sure the shadow copy can never send emails, charge cards or write to real data.
What a dark launch is
Martin Fowler defines dark launching as taking a new or changed back-end behaviour and calling it from existing users without the users being able to tell it is being called, in order to assess the additional load and performance impact before making a public announcement. His example is adding cross-selling recommendations to a retail checkout: the checkout calls the recommendation engine exactly as it would after launch, but does not show the results, so the engine does all its real work while nobody can see it. With a feature flag around the call, it can be switched off quickly if performance suffers.
Unlike a staged rollout, where some customers see the new behaviour, a dark launch shows the new behaviour to nobody. It answers "can this cope, and is it right?" before "do people like it?".
When it is worth the effort
Dark launches add complexity, so save them for changes where production behaviour is hard to predict:
- Replacing a core service, such as a new search engine, pricing calculator, shipping-rate calculator or recommendation engine.
- Moving to a new database or data store, where you want to compare reads before switching.
- A rewrite of a busy API, where test environments cannot reproduce real traffic patterns.
- New features with heavy computation, such as personalisation on every product page.
For a small UI change, a normal release with a feature flag is enough.
Pattern one: call and discard
In your existing code path, call the new component in addition to the old one, but use only the old result for the customer. Practical rules:
- Run the new call asynchronously or with a strict timeout, so it can never slow down the customer's request.
- Wrap it in error handling that logs failures but never surfaces them.
- Put it behind a switch so you can stop it instantly.
- Sample traffic, for example 5% of requests at first, then increase.
Pattern two: mirror traffic to a shadow service
Some web servers and proxies can copy incoming requests to a second destination. nginx's mirror module, for example, implements mirroring of an original request by creating background mirror subrequests, and its documentation states that responses to mirror subrequests are ignored. The customer gets the response from the existing back end, while the new service quietly handles the same request.
Service meshes and API gateways offer similar request mirroring. Mirroring is useful when the new service is a separate deployment and you want realistic load without changing application code.
Compare the answers, not just the speed
The most valuable output of a dark launch is often correctness. Log both results with a request ID and compare:
- Do the new search results include the same top products?
- Does the new pricing engine produce the same totals, to the cent?
- Where results differ, is the new one wrong, or was the old one wrong all along?
Build a small report of mismatches by type, fix the causes, and only go live when the differences are understood and acceptable.
Keep the shadow from touching the real world
This is the part that goes wrong. A shadow request that runs real code can also trigger real side effects:
- Emails and texts: the shadow must never send customer messages. Disable or stub the sender.
- Payments: never call a live payment provider from a shadow path.
- Writes: point the shadow at a separate database or make it read-only; a mirrored "place order" request must not create a second order.
- Third-party APIs: mirrored traffic can double your usage with partners, use up quotas or breach their terms. Stub them or sample lightly.
- Analytics: exclude shadow traffic so reports are not inflated.
Mirroring only read requests, such as searches and product pages, is the safest starting point.
Watch capacity on both sides
Dark launches deliberately add load. Watch database connections, CPU and response times for the existing system as well as the new one, and start with a small sample. If the existing system slows down, turn the shadow off first and investigate second.
Going live
When the new component has handled real load comfortably and its results match or beat the old one, switch customers over, often gradually, and keep the old path available for quick rollback for a while. Then remove the shadow code and switches, so they do not linger as hidden complexity.
A worked example
Picture a hypothetical Leeds outdoor-gear retailer replacing its slow product search with a new search service. Test environments could not reproduce the variety of real customer queries.
The team mirrored a sample of live search requests to the new service using their reverse proxy, with responses ignored, and logged both result lists against each request ID. Over two weeks they increased the sample, fixed mismatches caused by synonyms and out-of-stock filtering, and watched the new service's response times under peak load. Only read-only search traffic was mirrored, so nothing could create orders or send emails. When results matched and performance held, they switched customers to the new search with a quick rollback option ready.
FAQ
Is a dark launch the same as a beta?
No. A beta shows the feature to some users. A dark launch shows it to nobody.
Does it double hosting costs?
For the sampled share of traffic, roughly. Start small and only for the components you are testing.
Can I dark launch database changes?
Yes, by writing to or reading from the new store alongside the old one and comparing, with care around side effects.
If you're about to replace a core part of your site and want to see it under real traffic first, tell us what you're replacing.