The transactional outbox: never lose an order event again
Short answer: If your app saves something to its database and then separately sends a message about it, for example "order created" to a warehouse system, a webhook to a customer or an event to a search indexer, a crash between the two steps leaves them out of sync. The transactional outbox pattern fixes this: write the message into an "outbox" table in the same database transaction as the change, then have a separate process read the outbox and send the messages. The message goes out if and only if the change was saved. Consumers must handle the occasional duplicate.
The dual-write problem
Picture the code behind "Place order": insert the order row, commit, then publish an event to a message broker or call another service. Most of the time it works. But consider what can happen:
- The database commit succeeds, then the app crashes or the broker is unreachable. The order exists, but the warehouse never hears about it.
- The app publishes first, then the database commit fails. The warehouse starts packing an order that does not exist.
Chris Richardson's microservices.io catalogue describes this as the core problem: a service needs to update its database and send messages atomically to avoid inconsistencies, but a traditional distributed transaction spanning the database and the message broker (two-phase commit, 2PC) is often not viable. The database or broker might not support it, and even if they do, coupling the service to both is undesirable.
You do not need microservices to hit this. Any app that saves data and then calls a payment provider, sends a webhook, updates a CRM or indexes search content has the same gap.
How the outbox works
The pattern has two parts.
1. Write to the outbox in the same transaction. When you insert or update your business data, you also insert a row into an outbox table describing the message: an ID, the type of event (such as order_created), the record it concerns and a payload. Because it is the same database transaction, either both rows are saved or neither is.
2. Relay messages from the outbox. A separate process, the message relay, reads unsent rows and delivers them to the broker, webhook endpoint or other service, then marks them as sent or deletes them.
microservices.io lists two ways to build the relay. A polling publisher queries the outbox table on a short interval for unsent rows. Transaction log tailing reads the database's change log instead; tools such as Debezium do this, and its outbox event router documentation describes the outbox pattern as a way to safely and reliably exchange data between services while avoiding inconsistencies between a service's internal state and the events it publishes.
For a small app, polling every second or two is usually simple and sufficient.
What you gain
The microservices.io page lists the benefits plainly: 2PC is not used, messages are guaranteed to be sent if and only if the database transaction commits, and messages are sent in the order the application sent them. That last point helps when consumers care about sequence, such as "order created" before "order shipped".
The catch: duplicates
The same page is honest about the main drawback. The relay might publish a message more than once, for example by crashing after sending but before recording that it did. When it restarts, it sends again. So the consumer must be idempotent, perhaps by tracking the IDs of messages it has already processed. As the page notes, consumers usually need this anyway, because brokers can deliver messages more than once.
In practice: give every outbox message a unique ID, send that ID with the message, and have receivers ignore IDs they have already handled.
Building it sensibly
- Keep the outbox table simple: ID, event type, record reference, payload, created time, sent time and attempt count.
- Send in order where it matters, by processing rows in creation order per record.
- Retry failures with backoff and alert when a message has failed many times, rather than blocking all later messages forever.
- Clean up sent rows after a retention period, so the table does not grow without limit.
- Watch the backlog. The number of unsent rows and the age of the oldest one tell you if the relay is stuck.
- Avoid sending from inside the transaction. The whole point is that external calls happen only after commit, through the relay.
When you don't need it
If losing or duplicating a notification is harmless, such as an internal analytics ping, the extra moving parts may not be worth it. The outbox earns its keep where a missed or phantom message costs money or trust: orders, payments, stock, account changes and customer-facing webhooks. If you are unsure, list the messages your app sends and ask, for each one, what happens to a customer if it silently goes missing.
A worked example
Consider a hypothetical Vancouver furniture retailer whose app created orders and then immediately called a third-party delivery platform to book a slot. When the delivery platform timed out, the order still existed but had no delivery booked, and staff found out only when customers called.
The developers added an outbox table. The order and an "order_created" outbox row are now saved in one transaction. A relay process polls every two seconds, calls the delivery platform with the outbox message ID as an idempotency key, and marks the row sent. Failures retry with backoff and alert after an hour. Orders without bookings stopped appearing, and duplicate bookings were avoided because the delivery platform recognised repeated IDs.
FAQ
Is this the same as a job queue?
Related but different. A job queue stored in the same database can act as an outbox; the key is that the job is created in the same transaction as the data.
Does the relay add delay?
A little, usually seconds, depending on polling interval. For most business events that is acceptable.
Can I use this with a hosted database?
Yes. Polling needs only normal queries; log tailing needs access to the database's change stream.
If your app's orders, stock or webhooks sometimes fall out of sync with what other systems see, tell us how the pieces talk to each other.