Change data capture (CDC) is a way of tracking every insert, update and delete in a database as it happens, so other systems can react to those changes. Instead of copying whole tables on a schedule, CDC records each change as a small event and sends it where it's needed.
This guide starts with the problem CDC solves, then covers how it works inside the database, where it's used, and the mistakes that most often break it. Along the way you'll meet the transaction log, change events, Apache Kafka, and tools like Debezium.
- CDC captures inserts, updates and deletes as they happen, instead of copying whole tables.
- It reads the database's transaction log, which already records every change in order.
- Each change becomes an event, often published to Kafka, that caches, search indexes and warehouses consume.
- The common pitfalls are schema changes, event ordering and forgetting the initial snapshot.
- 0:00Intro
- 0:27Your data is already out of date
- 1:33Capture every change as it happens
- 2:32Full copies vs change capture
- 3:29The database writes everything down
- 4:36A CDC tool reads the log
- 5:28Changes become events that flow
- 6:22One update, start to finish
- 7:06Keeping everything in sync
- 8:07A price change ripples out
- 8:56Where CDC goes wrong
- 9:57Order matters for every row
- 10:49Snapshot first, then stream
- 11:45Tools you'll actually meet
- 12:44CDC in five ideas
- 13:19Move only what changed
Why is copied data always out of date?
Picture a customer updating their address in your app. That afternoon, the team's report still shows the old address, and the search page shows something different again. The change was made hours ago, but the dashboard disagrees.
Behind this mismatch there's usually a simple habit. The team copies entire tables from one system to another every night or every hour and hopes that's fresh enough. Until the next copy job runs, every downstream system is showing yesterday.
This approach causes two kinds of pain, and both grow as your data grows. Between copies, people make decisions on numbers that are no longer true. During copies, re-reading millions of unchanged rows takes time, uses network bandwidth and slows down the main database.
- Stale data: reports and search lag hours behind the real database
- Heavy load: full copies hammer the database and the network
- Poor scaling: the bigger the data, the worse full copies get
What is change data capture?
Change data capture watches a database and records each change as a small event. A full copy asks what the whole table looks like right now. CDC asks a smaller question: what just changed? Then it writes that down and passes it on.
Think of reading a news feed instead of re-reading the whole newspaper every hour. Other systems receive a short stream of updates rather than a complete copy, so they only do work when something actually happens.
CDC tracks three kinds of change. Deletes deserve special attention because they're easy to forget. If a product is removed from the database but the search index never hears about it, customers can keep finding something that no longer exists.
- Inserts: a new row appears, such as a new order or a new user
- Updates: a row changes, and both before and after values can be captured
- Deletes: a row is removed, so downstream systems must remove it too
Full table copies vs CDC: what's the difference?
Both approaches have the same goal: getting data from your main database into the other systems that need it. The difference lies in how much work each one does to get there, and that difference is the reason CDC exists.
Imagine a huge customer table where only a handful of rows changed today. A full copy moves every single row anyway. CDC moves just that handful. Because CDC's cost tracks how much changes rather than how big the table is, it scales far better as data grows.
- Rows moved: full copy moves every row every time; CDC moves only changed rows
- Freshness: a copy is as old as its last run; CDC is seconds behind
- Database load: copies run heavy scans; CDC reads the log
- Cost: copies grow with table size; CDC grows with how much changes
How does CDC work? It starts with the transaction log
Step one happens inside the database itself. Before a change is applied to a table, the database records it in a special file called the transaction log. The app sends an update, the database appends it to the log, and only then applies it to the table.
This log exists for the database's own safety, not for CDC. If the server crashes mid-write, the database replays the log on restart so nothing is lost. The log is also an ordered history: each entry has a position, so you can tell exactly which change came first. Each database names it differently: MySQL has the binlog, PostgreSQL the WAL (write-ahead log), and SQL Server the transaction log.
Step two is where a CDC tool comes in. It connects to the database and reads the transaction log entry by entry, picking up each change moments after it's written. Because it reads a file that already exists rather than scanning tables, the extra load is very small, a bit like reading a shop's receipt roll instead of recounting the whole warehouse.
The tool also saves a bookmark: its current position in the log. If the tool restarts, it resumes exactly where it left off, so no changes are skipped, even after an outage.
How changes become events: one update, start to finish
In step three, the tool turns each change into an event, a small message describing what happened. A typical event names the table and the operation (insert, update or delete), carries the row's values before and after the change, and records its position in the log.
These events usually go into a streaming platform such as Apache Kafka, which acts as a central hub. From there, a cache refreshes its keys, a search index updates its listings, and a warehouse loads fresh rows, each reading at its own pace. Because consumers read the stream rather than the database, you can add a new one later without changing anything upstream.
Here's a single change followed all the way through, using an example order 1042 moving from pending to shipped. The whole trip takes moments. With a nightly copy job, the rest of the company would wait until tonight to see the shipped status.
- The app runs SQL: UPDATE orders, setting status to 'shipped'
- The database writes a log entry at position 881
- The CDC tool reads it and notes the before value: pending
- An event is built: operation u (update), new status shipped
- The event is delivered to a Kafka topic, ready for every consumer
Where is change data capture used?
CDC shows up anywhere one database's changes need to appear somewhere else, quickly and reliably. The common thread is that each system keeps its own copy of the data, and that copy has to stay in step with the original.
Take an online store where a manager lowers the price of a pair of headphones. The admin panel saves the new price to the products database. The CDC tool reads it from the log and sends an event to Kafka. The cache refreshes the price, the search index updates the listing, and the warehouse records it for sales reports.
Without CDC, a customer might see the old price in search and a different one at checkout, which means a support ticket or a lost sale. With CDC, every system agrees within seconds, and nobody had to write custom sync code for each one.
- Caches such as Redis: refresh values the moment they change
- Search indexes such as Elasticsearch: keep results matching the real database
- Analytics warehouses: fresh dashboards without nightly bulk loads
- Microservices: services share changes without touching each other's databases
Common CDC mistakes and why event order matters
CDC is powerful but easy to set up badly, and its problems rarely appear on day one. Picture a pipeline set up on Monday and broken by Friday: someone renamed a column, some events arrived out of order, and a new search index was missing every older product. Those are the three classic mistakes.
Schema changes alter the shape of events, so consumers expecting the old shape can fail or silently drop data. Ordering is the trickiest. Suppose an order is packed and then shipped. If a consumer receives those events swapped, it applies shipped first and packed second, and the order stays stuck on packed.
The fix is to key each event by its row's primary key. In Kafka, events with the same key go to the same partition, and a partition keeps its events in order. Consumers can also compare log positions and skip any event older than the last one they applied. Correct ordering comes from design, not luck.
- Ignoring schema changes: a new or renamed column breaks consumers
- Ignoring event ordering: out-of-order changes give wrong final values
- Forgetting the initial snapshot: the log holds recent changes, not existing rows
# Order 1042, as it happened:
1. status = 'packed'
2. status = 'shipped'
# Applied out of order:
2 then 1 → final = 'packed' ✗How to roll out CDC safely: snapshot first, then stream
The transaction log only holds recent changes, so a brand-new consumer can't rebuild your existing data from it alone. The safe pattern is like moving into a new house: bring all the furniture once, then just handle new deliveries. Most CDC tools support these steps directly, so you're usually switching on a feature rather than building it.
Marking the log position first is what makes this work. Any changes made while the snapshot runs are still in the log, so when streaming starts from the bookmark, nothing falls through the gap.
- Mark the current position in the transaction log
- Take a one-time snapshot of all existing rows
- Stream changes from the bookmark onward
- Manage schemas: add columns compatibly and version events with a schema registry
Which CDC tools will you actually meet?
You rarely need to read database logs yourself. The right tool usually depends on which database you use and where the data needs to go. Debezium is one of the best-known open-source options; it reads the logs of databases like MySQL, PostgreSQL and SQL Server and publishes changes as events, usually into Kafka.
Debezium often runs on Kafka Connect, a framework whose sink connectors deliver events onward into warehouses, search engines and caches. Cloud providers offer managed services such as AWS Database Migration Service, trading some control for not running the pipeline yourself. Some databases also have CDC built in, such as PostgreSQL logical replication and SQL Server's own CDC feature, so check what yours offers first.
Key takeaways
- CDC captures inserts, updates and deletes as they happen instead of copying whole tables.
- It exists because full copies are slow, costly and leave data out of date.
- The database already writes every change to its transaction log, and a CDC tool reads that log with little extra load.
- Each change becomes an event that Kafka, caches, search indexes and warehouses can consume.
- Plan ahead for schema changes, event ordering and the initial snapshot.
Frequently asked questions
Does change data capture slow down the source database?
Usually only slightly. Log-based CDC reads the transaction log, a file the database already writes for crash safety, instead of running heavy queries against your tables.
Do I need Kafka to use CDC?
Not strictly, but Kafka is a very common choice. It acts as a central hub where many systems can read the same stream of change events at their own pace, and you can add new consumers without touching the database.
What happens if the CDC tool crashes or restarts?
A CDC tool saves a bookmark of its position in the transaction log. When it comes back, it resumes from that position, so no changes are skipped.
Why does a new consumer need an initial snapshot?
The transaction log only holds recent changes, not your full history. A one-time snapshot of existing rows gives the new system a complete starting point, and streaming from a recorded log position covers everything after that.
How do I keep CDC events in the right order?
Key each event by the row's primary key so all events for one row land in the same Kafka partition, which preserves their order. Consumers can also compare log positions and ignore events older than the last one applied.
Is CDC better than a nightly batch copy?
For keeping systems in sync, it usually is: CDC moves only changed rows and is seconds behind, while a batch copy moves every row and is only as fresh as its last run. The cost of CDC grows with how much changes, not with table size.
