How Kafka Works: A Hands-On Guide to Messages, Partitions and Brokers

by Souy Soeng /October 14, 2026

Diagram of a rides topic split into three partitions on Kafka brokers, with a producer writing and consumers reading

How Kafka works comes down to one idea: Apache Kafka is an open-source platform that stores events in a shared, durable log, so many apps can write to it and read from it independently. Every click, payment or ride request is an event, and Kafka keeps them organized in topics, splits those topics into partitions, stores them on brokers, and lets readers track their place with offsets.

This guide follows the same path as the video. You will start a real Kafka broker on your laptop, create a topic called rides, send events into it, and read them back. Each step explains one moving part, so by the end, messages, partitions, brokers and offsets should feel concrete instead of fuzzy.

In short

Kafka is a shared, durable log: producers write events once, and any number of consumers read them.
Topics group events by meaning and are split into partitions, which are ordered, append-only logs stored on brokers.
Events with the same key always land in the same partition, so their order is kept.
Reading never deletes events; consumers track their position with offsets and resume after a crash.
Consumer lag, the gap between the newest event and a group's committed offset, is your main health check.

Gather what you need before you start

The goal is simple: one Kafka broker running locally, one topic named rides, and events flowing from a producer you type into, through to a consumer that prints them back. You do not need any prior Kafka experience. Everything is done from the command line, and each step builds on the one before it.

Docker is the shortcut that makes this easy. Kafka runs on Java and normally needs configuration files. The official container image bundles all of that, so a single command gives you a working broker, and removing it later is just as quick. You skip Java setup and config files entirely.

Docker Desktop or Docker Engine is installed and running
A terminal window is open
You are comfortable running a command and reading its output

See why direct app-to-app wiring breaks

Before any commands, picture the problem Kafka solves. Imagine Checkout, a Rides app, Billing, Analytics and Email, all wired straight to each other. Each pair needs its own connection, data format and retry logic. Every new service adds more wires, and every wire is something that can fail.

These links are fragile. If Billing goes down, Checkout's calls to it start failing too, because there is nowhere for events to wait. One slow or offline service drags down every app that depends on it, and data can be lost along the way.

Kafka replaces the mesh with one shared log. Producers write each event to Kafka once, and any number of apps read it whenever they are ready. Keep this picture in mind: the rest of Kafka exists to cut those wires.

Start a Kafka broker with Docker

A broker is simply a Kafka server: the program that receives events, stores them on disk and hands them out to readers. One docker run command downloads the official Apache Kafka image and starts a single broker in the background. The -d flag runs it detached, --name kafka gives the container a name, and -p 9092 maps Kafka's port to your laptop.

Port 9092 is Kafka's default listening port, and every producer and consumer in this guide connects to localhost:9092. If another program is already using that port, the container will not start. The broker also runs in KRaft mode, which is how modern Kafka manages itself. Older tutorials start ZooKeeper alongside Kafka, but recent versions run without it.

Run docker ps and look for a container named kafka
Its status should read Up
Nothing else on your machine is using port 9092

# start one Kafka broker in the background
docker run -d --name kafka \
  -p 9092:9092 apache/kafka:latest
# check it is running
docker ps

Create a topic and understand partitions

A topic is a named stream of events, much like a folder name for one kind of thing: rides, payments, clicks. Without a topic, producers have nowhere to write. Open a shell inside the container with docker exec, move to Kafka's scripts folder, then run kafka-topics.sh with --create, the topic name rides, three partitions, and the bootstrap server localhost:9092. You should see Created topic rides.

The bootstrap server is the first broker a tool contacts. The client asks that broker for a map of the whole cluster, then talks to the right brokers directly. The partition count matters more than it looks, because it sets how many readers in one group can work in parallel.

Each partition is an append-only log, like a notebook where you can only add pages at the end. In a real cluster, partition 0, 1 and 2 might sit on three different brokers; on your laptop they share one. Every new event gets the next number in its partition, and events stay on disk after being read. Order is guaranteed within a partition, but not across the whole topic.

The command prints Created topic rides
You chose a partition count with parallel readers in mind
You remember: order holds per partition, not per topic

docker exec -it kafka bash
cd /opt/kafka/bin
./kafka-topics.sh --create --topic rides \
  --partitions 3 \
  --bootstrap-server localhost:9092

Produce events and choose a partition key

A producer is the program that writes events. In a real app it is a few lines of code, but the console producer lets you type events by hand. Run kafka-console-producer.sh with the topic rides and the same bootstrap server, then type ride requested from Main St, ride accepted by driver Ana and ride completed, pressing Enter after each. Every line becomes one event appended to a partition.

Kafka uses event and message for the same thing. Each one carries a value (your data, often JSON in real apps), a timestamp and an optional key. The producer only waits for the broker to confirm the write; it never waits for a reader.

Those first events had no key, so Kafka spread them across partitions, and a reader could see ride completed before ride requested. A partition key fixes this. Kafka hashes the key and takes the result modulo the partition count. With rider-42 as the key, an example hash of 7021 mod 3 gives partition 1, and every future rider-42 event lands there too.

To try it, restart the producer with --property parse.key=true and key.separator=: then type rider-42:requested. The part before the colon becomes the key. Pick an ID whose events must stay in order, such as a rider ID or an order ID.

Each line you type in the producer becomes one event
Related events share the same key
The key is an ID whose events must stay in order

./kafka-console-producer.sh --topic rides \
  --bootstrap-server localhost:9092
> ride requested from Main St
> ride accepted by driver Ana
> ride completed

Consume events and track your place with offsets

A consumer reads events, and reading does not delete anything. Consumers pull events from the broker when they are ready, so one slow reader cannot hold up the rest. Open a second terminal, docker exec into the container again, and run kafka-console-consumer.sh with the topic rides, the group billing, --from-beginning and the bootstrap server. Your ride events should print within a second or two.

A consumer group is a named team of readers sharing the work. Kafka gives each partition to one member, so three partitions can be shared by up to three billing consumers. The --from-beginning flag tells a brand-new group to start at the oldest event still stored.

Consumers remember their place with offsets, the position number of each event in a partition. Offsets start at zero and count up separately in each partition. A consumer fetches events 0 and 1, commits offset 2, and if it crashes, it restarts by fetching from 2. Nothing is missed and nothing is repeated.

The safe habit is to commit only after an event is fully handled. Committing first and then crashing skips that event for good. Committing afterwards might repeat an event, so make your processing safe to run twice.

Your earlier ride events appear in the consumer terminal
The consumer belongs to a named group, here billing
Offsets are committed after processing, not before

# in a second terminal
docker exec -it kafka bash
cd /opt/kafka/bin
./kafka-console-consumer.sh --topic rides \
  --group billing --from-beginning \
  --bootstrap-server localhost:9092

Verify the setup and check consumer lag

Now prove everything works end to end. Keep the consumer running in one terminal and use a free shell in another. Run kafka-topics.sh with --describe, and you should see rides with three partitions, each with a leader broker. Then run kafka-consumer-groups.sh with --describe for the billing group. It lists each partition with its current offset, log end offset and lag.

For a live test, type a new event such as ride cancelled into the producer. It should appear in the consumer terminal almost straight away. Lag is the gap between the newest event and the group's committed offset. Near zero means your consumers are keeping up; a number that keeps growing means they are falling behind.

The topic shows three partitions, each with a leader
A new event appears in the consumer almost instantly
Lag for the billing group stays near zero

./kafka-topics.sh --describe --topic rides \
  --bootstrap-server localhost:9092
./kafka-consumer-groups.sh --describe \
  --group billing \
  --bootstrap-server localhost:9092

Fix common errors and avoid the queue mistake

Four errors cause most first-day trouble, and they usually come down to addresses, names or offsets. Connection refused on localhost:9092 usually means the container stopped. An UNKNOWN_TOPIC_OR_PARTITION warning points to a misspelled topic. An empty consumer is often an old group resuming from its saved offset. Events out of order mean related events were sent without a shared key.

The bigger mistake is treating Kafka like a queue. A queue is something you empty; a log is something you keep and can replay. Reading never deletes, retention decides cleanup based on time or size, and many groups can read the same events, each from its own offset. A new analytics service can read last week's rides without the producer changing a line.

Keys matter from day one. Changing a key or the partition count later sends events to different partitions, which can break ordering. Choose your key carefully before you go to production.

No connection: run docker ps and start the container
Unknown topic: check spelling and list topics with --list
Empty consumer: add --from-beginning or use a fresh group name
Wrong order: send related events with the same key
Anything else: run docker logs kafka and search the exact message

Recognize where Kafka fits best

The same three ideas, topics, partitions and offsets, power very different jobs. In activity tracking, every click, search and page view becomes an event; the website writes and moves on, while recommendation and reporting systems read at their own pace. In log collection, many servers send logs into Kafka instead of one fragile collector, and search and alerting tools read the same stream independently.

In real-time analytics, a stream processor reads events as they land, so a dashboard can show ride demand by area as it happens instead of waiting for a nightly report. In microservices, services share events instead of calling each other directly. The pattern is always the same: many writers, many readers, and events kept in order.

Key takeaways

Producers write events into named topics.
Topics split into ordered, append-only partitions.
Brokers store partitions as durable logs on disk.
Keys keep related events in order by sending them to one partition.
Consumers track progress with offsets, so each reader moves at its own pace.
Kafka is a log you replay, not a queue you empty: write once, read many times.

Frequently asked questions

What is the difference between a Kafka topic and a partition?

A topic is a named stream for one kind of event, such as rides or payments. Each topic is split into partitions, which are ordered, append-only logs stored on brokers. Partitions let several consumers in a group read in parallel.

Does Kafka delete a message after it is consumed?

No. Reading never deletes events in Kafka. Events stay on disk until retention removes them based on time or size, so many consumer groups can read the same events independently.

Does Kafka guarantee message order?

Kafka guarantees order within a single partition, not across a whole topic. To keep related events in order, give them the same key, such as a rider ID or order ID, so they always land in the same partition.

What is a Kafka offset?

An offset is the position number of an event inside a partition, starting at zero. Consumers commit the offset they have reached, so after a crash or restart they resume where they left off instead of starting over.

Do I still need ZooKeeper to run Kafka?

Modern Kafka runs in KRaft mode, which manages the cluster without ZooKeeper. Older tutorials start ZooKeeper alongside Kafka, but recent versions do not need it.

Why does my Kafka consumer show no messages?

A new consumer without --from-beginning, or an existing group, starts from its saved position rather than the oldest event. Add --from-beginning or use a fresh group name, and check the topic name is spelled correctly.

Watch the full video on YouTube →

Tag

Ad