The transactional outbox pattern is a way to save a change to your database and the message announcing that change in the same transaction, so the two can never disagree. A separate process then delivers those saved messages to a message broker, retrying until each one gets through.
This article explains why the pattern exists, how its four parts fit together (one transaction, an outbox table, a relay and a broker), and where teams usually go wrong. We'll follow a single order from checkout to shipment so you can see each step in practice.
- A database and a message broker can't commit together, so saving and sending can silently drift apart.
- The outbox pattern writes your data and its event as rows in one local transaction.
- A relay reads unsent rows by polling or change data capture and publishes them to the broker.
- The relay publishes first and marks rows sent afterwards, which means duplicates are possible.
- Consumers must dedupe by event ID, and old outbox rows need regular cleanup.
- 0:00Intro
- 0:25Saved order, missing event
- 1:13One transaction, two writes
- 2:02No shared commit button
- 2:55Write data and event together
- 3:45A relay finds unsent rows
- 4:36Publish, then mark sent
- 5:23Follow one order through
- 6:15Wherever services must stay in sync
- 7:06Where outbox setups go wrong
- 8:00The outbox in six points
- 8:35Save state and intent together
The problem: a saved order with a missing event
Most services do two things when something changes: they save it, then they announce it. An order service, for example, commits a new order to PostgreSQL and then sends an event to a message broker so the shipping service can act on it.
Now picture a crash between those two steps. The order is committed and durable. The customer has seen a confirmation. But the event never reached the broker, so the shipping service has no idea the order exists. The orders service says the order was placed, and shipping has no record of it.
What makes this dangerous is how quiet it is. No error reaches the user, and nobody gets paged. The data simply drifts out of sync between services, and you may only find out days later when a customer asks where their package is.
- Orders says yes: the order is committed and durable.
- Shipping never knows: no event arrived, so no package ships.
Why can't you just commit and then send?
The root cause is that a database and a message broker are separate systems. Each has its own commit, and there is no shared button that saves in both at once. Whichever you do first, a crash in between splits them.
Try both orders. If you commit and then crash, the event is lost. If you send and then crash before committing, other services react to a ghost event for an order that doesn't exist. Either the broker can be down while the database is fine, or your process can die between the two calls, and neither system knows about the other.
Retries alone don't solve this. Retry logic usually lives in memory, so if the process dies, the unsent message dies with it. A message can only survive a crash if it was written down somewhere durable first, and that is exactly the gap the outbox closes.
What is the transactional outbox pattern?
The fix fits in one sentence: write your business data and your outgoing messages in one database transaction. The message is stored as a row in an outbox table, right next to the data it describes.
A helpful comparison is a letter tray on your desk. Instead of running to the post office in the middle of a task, you drop the letter in your own outbox and carry on. Someone collects it later. The letter is safe the moment it's in the tray.
Because both rows live in the same database, that database's own guarantees cover both writes. There is no second system to coordinate at write time, and the database already knows how to commit atomically. Your intent to send becomes as safe as the data itself, so delivery can happen later and be retried as often as needed.
- One transaction covers the data and the event.
- An outbox table stores events waiting to be sent.
- A relay delivers them afterwards.
- A broker carries them to other services.
How do you write data and an event in one transaction?
Step one needs nothing exotic. Any relational database that supports transactions can do it with plain SQL: two inserts and one commit. BEGIN opens the transaction. The first insert adds order 42 to the orders table with a status of placed. The second insert adds an OrderPlaced event and its payload to the outbox table. COMMIT then makes both permanent at the same moment.
The key property is both or neither. If the second insert fails, the order rolls back too, so you can never end up with data that has no matching event. The outbox row itself is ordinary: usually an id, an event type, a JSON payload and a created timestamp.
Give every event a unique id when you create it. It looks like a small detail now, but consumers will rely on it later to recognise and skip duplicate deliveries.
BEGIN;
INSERT INTO orders (id, status)
VALUES (42, 'PLACED');
INSERT INTO outbox (type, payload)
VALUES ('OrderPlaced', '{...}');
COMMIT;How does the relay find unsent outbox rows?
With events stored safely, step two is getting them out. A separate process called the relay reads unsent rows from the outbox. There are two main ways it can notice new ones, and both find the same rows.
Polling runs a query on a timer that selects rows not yet sent. It's simple, just SQL and a loop, but it adds query load and a small delay. Change data capture (CDC) instead tails the database's change log, as tools like Debezium do. That gives near real-time delivery without polling queries, at the cost of more infrastructure to run.
Whichever you choose, read rows oldest first, by creation time or a sequence number, so events for the same order leave in the order they happened. The relay also runs outside the request path. The order service just commits and returns, so a slow or offline broker no longer slows down checkout.
- Polling: simple, but adds load and some delay.
- Change data capture: near real-time, but more to operate.
- Read in order: oldest first preserves event sequence.
Why does the relay publish first and mark sent later?
Step three closes the loop, and the order of actions is deliberate. The relay reads unsent rows, publishes each one to the broker, waits for the broker's acknowledgement that it's stored, and only then marks the row sent or deletes it. Then it starts again with the next batch.
Consider the reverse. If the relay marked a row sent and then crashed before publishing, that event would be gone for good, which is the very problem the pattern exists to prevent.
Publishing first has a trade-off: duplicates. If the relay crashes after publishing but before marking the row, it will send that event again when it restarts. That is acceptable, as long as the services receiving events are built to expect it.
- Read unsent rows
- Publish to the broker
- Wait for the broker ack
- Mark sent or delete the row
Following one order through the outbox
Here are all three steps in one concrete case. Checkout for order 42 starts it. The service commits one transaction containing the order row and the outbox row. The relay polls and finds one unsent row. It publishes OrderPlaced to the broker, then marks the row sent by setting sent_at to now(). Finally, the shipping service reacts and creates a shipment.
Each step happens in a different place: the service, the database, the relay and the broker. Now look at the two crash points. If the crash happens before commit, neither the order nor the event exists. The customer sees an error and tries again, and the services still agree.
If the crash happens after publishing but before marking, the relay publishes OrderPlaced again after restarting. Shipping receives it twice, so it must recognise the repeat and avoid creating a second shipment.
When should you use the outbox pattern?
The pattern isn't tied to a framework or a broker. It's tied to a need: one service's change must reliably reach another. In microservices, each service owns its database, so the outbox is how changes leave it safely. In sagas, each step's local transaction and its next message must never drift apart. Event-driven systems depend on every event actually being published.
It's a good fit whenever a database change must trigger a message and losing that message would leave another service wrong, such as an order, a payment or an account change. Skip it when nothing depends on the event, like a best-effort analytics ping. The pattern adds a table and a relay to run, so use it where reliability is worth that cost.
Common outbox pattern mistakes
The outbox guarantees delivery, not delivery exactly once. Most trouble comes from forgetting that, and it tends to appear in production rather than testing. A typical symptom is a customer receiving two confirmation emails.
Three mistakes come up again and again. Teams forget delivery is at-least-once, so duplicates surprise them. Consumers aren't idempotent, so each duplicate triggers the action again. And the outbox grows forever, slowing down every relay query.
The fixes are straightforward. Consumers should record the event IDs they've processed, ideally in the same transaction as their own changes, and skip any ID they've seen. Clean up sent rows by deleting them right after publishing or with a scheduled job, and add an index on unsent rows to keep the relay fast.
- Dedupe by event ID in every consumer.
- Delete or archive sent rows on a schedule.
- Index unsent rows so relay queries stay quick.
Key takeaways
- Dual writes to a database and a broker can silently disagree.
- Write data and event together in one local transaction.
- A relay picks up unsent rows by polling or change data capture.
- Publish, then mark sent, never the other way round.
- Expect duplicates and dedupe consumers by event ID.
- Clean up the outbox so it never grows without limit.
Frequently asked questions
Does the transactional outbox pattern guarantee exactly-once delivery?
No. It guarantees at-least-once delivery, because the relay may publish an event again if it crashes before marking the row sent. Consumers need to deduplicate using the event's unique ID.
Should I use polling or change data capture for the relay?
Polling is simpler, just SQL on a timer, but it adds query load and a small delay. Change data capture, using tools like Debezium, reads the database's change log for near real-time delivery, but it means more infrastructure to run.
What columns does an outbox table need?
Usually an id, an event type, a JSON payload and a created timestamp, plus a way to mark rows as sent, such as a sent_at column. Give each event a unique id so consumers can detect duplicates.
Why not just retry sending the message if the broker fails?
Retry logic typically lives in memory, so if the process crashes the unsent message is lost with it. Writing the event to the outbox first makes it durable, so it can be retried after any crash.
When is the outbox pattern not worth it?
When no other service depends on the event, such as a best-effort analytics ping. The pattern adds a table and a relay to operate, so reserve it for changes where a lost message would leave another service wrong.
