How Docker, Kubernetes, Redis and Kafka Work Together in One App

by Souy Soeng /November 06, 2026

Docker, Kubernetes, Redis and Kafka shown as four connected stages of one small shop service

Docker, Kubernetes, Redis and Kafka work together by handling four separate jobs: Docker packages a service, Kubernetes keeps it running, Redis answers repeated reads from memory, and Kafka streams events to other services.

The easiest way to see how they fit is to build something with them. This guide follows one small shop service from a single image to a working system, adding one tool at a time and explaining the problem each one solves.

In short

Docker solves inconsistency by packaging a service into an image that runs the same way in different places.
Kubernetes reduces manual operations by continually working to match the number of replicas you asked for.
Redis reduces repeated database reads by serving frequently requested data from memory.
Kafka reduces tight coupling by letting services read the same events independently.
Add each tool only when you have the problem it solves.

Prepare your goal and local tools

This guide builds one small shop service in stages. By the end, it runs in containers, is managed by a local cluster, reads product data through a cache, and publishes order events to a stream. Each stage adds one tool and solves one problem, so you can check your work before moving on.

The order matters because each layer builds on the one before it. A Docker image packages the code, Kubernetes runs that image as pods, Redis caches product reads, and Kafka streams the order events. Having everything installed first means you can follow every step without stopping midway.

Docker Engine or Docker Desktop is installed.
kubectl and kind are available for a local cluster.
You have a web service with a Dockerfile.
Your service reads from a database.
You are comfortable with the terminal and HTTP.

Match each tool to the problem it solves

You rarely need these tools on day one. They become useful when growth starts to hurt in recognisable ways. Picture a shop on a quiet day and then during a big sale: the same code behaves very differently once the traffic, the number of services and the size of the team all grow.

Now follow one Place order click. It reaches Kubernetes, which routes it to a healthy shop-api container. The service checks Redis for product data and reads PostgreSQL on a miss. It then publishes to Kafka, and an email worker picks up the order later. Because the jobs are separate, you have a much better idea of which layer to check when something breaks.

Inconsistency: code works on one machine and fails on another. Use Docker.
Manual operations: people restart and add servers by hand. Use Kubernetes.
Slow database: repeated reads put it under strain. Use Redis.
Tight coupling: one slow service holds up the others. Use Kafka.

Build the shop service image with Docker

Docker turns your service and everything it needs into one image, so the same package runs on your laptop and later in the cluster. From the folder that holds your Dockerfile, build the image with a tag, start it in the background, then request the health route. Use your own service's port and route if they differ.

An image is a fixed package you can copy around, while a container is a running instance of it, isolated from other processes but sharing the host's kernel. Port mapping is the usual catch. In 8080:8080, the number after the colon must be the port your service listens on inside the container.

docker build finishes and names the image.
docker run prints a container ID.
curl on the health route returns status ok.

docker build -t shop-api:1.0 .
docker run -d -p 8080:8080 shop-api:1.0
curl localhost:8080/health

Deploy and scale the image on Kubernetes

A single container on one machine won't gain copies when traffic grows, and nothing replaces the machine if it fails. That is the job of Kubernetes. For local practice, run kind create cluster, then kind load docker-image shop-api:1.0. This copies your local image into the cluster, so Kubernetes can find it without a registry.

Kubernetes schedules pods rather than bare containers. A pod wraps one or more containers that share a network address, and the scheduler chooses a node with enough room to run it. The replica count is a statement of intent, not a one-time action: you say how many copies you want, and Kubernetes keeps working to match it.

That behaviour rests on a repeating loop. Kubernetes compares the desired state with what is actually running, and if it counts two pods instead of three, it starts another. A crashed container restarts inside its pod, and a missing pod gets replaced by the Deployment. Scaling uses the same loop, whether you change the number by hand or a Horizontal Pod Autoscaler changes it from metrics.

The deployment is reported as created.
The scale command reports the deployment as scaled.
kubectl get pods shows three pods, each Running.

kubectl create deploy shop --image=shop-api:1.0
kubectl scale deploy shop --replicas=3
kubectl get pods

Add Redis to reduce repeated database reads

Even with healthy pods, every product page still hits the database. Redis, an in-memory store, can answer the reads that keep repeating. A product detail is a good first candidate because many people read it and it rarely changes. Run Redis in the cluster, expose it as a Service, and confirm that PING returns PONG.

Inside the cluster, shop-api connects to the host redis on port 6379, the Redis default. Because the Service gives Redis a stable address, a restarted Redis pod doesn't change the address your code uses. Your connection setting can stay the same while the pods behind that name come and go.

Redis does nothing until your code uses it, so change the product lookup to follow the cache-aside pattern. For product 42, run GET product:42. A miss returns nil, so query the database, get Blue mug, and store it with SET product:42. The next read is a hit. Your application stays in charge, and Redis only holds a copy of data the database owns.

Set an expiry based on how stale the data may safely be, and delete the key when a price changes, so customers don't keep seeing the old value. The payoff is that popular products are answered from memory, leaving the database capacity for work only it can do, such as handling new orders.

PING returns PONG from redis-cli.
The first read of a product misses and queries the database.
The next read of that product is a cache hit.
Cached keys have an expiry.

kubectl create deploy redis --image=redis
kubectl expose deploy redis --port=6379
kubectl exec -it deploy/redis -- redis-cli

Publish order events to a Kafka topic

If shop-api calls email, billing and analytics directly, one slow service can hold up the rest. Instead, it records an event in Kafka. Start Kafka the same way as Redis, using kubectl create deploy kafka with the apache/kafka image. Give it a little time to start, then open a shell inside the pod.

Create a topic named orders with three partitions, then use the console producer to write one order. Each partition is an append-only sequence of events, and every event gets a position called an offset. Reading an event doesn't delete it. Events stay in the log until retention removes them.

Treat this single broker as a learning setup rather than a template for production. A production deployment needs configured listeners, replication and persistent storage. For now, one broker is enough to see how topics, partitions and offsets behave before you add the services that read from them.

The terminal confirms Created topic orders.
The topic has three partitions.
One test order has been sent through the console producer.

kubectl exec -it deploy/kafka -- sh
cd /opt/kafka/bin
./kafka-topics.sh --create --topic orders \
./kafka-console-producer.sh --topic orders \

Add consumers that read at their own pace

An event in the log does nothing until something reads it, and this is the stage where tight coupling goes away. The shop-api service publishes once to the orders topic. From there, an email worker sends a confirmation, analytics updates its reports, and a fraud check reviews the payment. None of them needs to know about the others.

For a real service, each consumer is usually its own deployment, using a Kafka client library and its own consumer group. That lets you scale or restart each one separately. Each group also tracks its own offset, so if analytics falls behind or goes offline, the email worker carries on and analytics catches up later.

Replay is what makes this more than a simple queue. If analytics ships a bug, you can fix it, reset its offset and reprocess the events Kafka still retains. Replay only reaches as far back as retention allows, so events that have already been removed can't be read again.

Each consumer has its own consumer group.
Each consumer can be restarted without stopping the others.
A consumer that falls behind can catch up from its offset.

Test recovery, cache hits and event replay

With all four tools in place, prove that each one does its job. First, run kubectl port-forward on the shop deployment and request product 42 twice through your app. That gives the cache something real to store. Then delete one pod and run kubectl get deploy. Within moments, you should see three of three ready again.

Next, open redis-cli and type GET product:42. A working cache returns Blue mug. Confirm it from the other side too: with database query logging on, the first request should appear and the second shouldn't. If both appear, your code isn't reading from Redis. Finally, run the console consumer on orders with --from-beginning, and your earlier order should appear.

The deployment returns to 3/3 after you delete a pod.
GET product:42 returns the cached product.
Database logs show only the first product request.
The console consumer shows your earlier order.

kubectl delete pod shop-6d4f7c9b8-x2kqp
kubectl get deploy shop
kubectl exec -it deploy/redis -- redis-cli

Fix common mistakes and avoid adding tools too early

Only the first of these common problems is a setup error. A pod stuck in ImagePullBackOff usually means the cluster can't find your image, so run kind load docker-image and check the tag. The other three come from wrong assumptions about the tools, so the fix is a change in how you use them.

The thought that you might need all of this someday lies behind many overbuilt systems. A better rule is to add a tool when its problem appears, which keeps your system simpler to run. These tools are powerful together, but none of them comes free, so start with the problem you have and choose accordingly.

Kubernetes: don't adopt it early for a simple service. Stay on Docker until operations actually hurt.
Redis: keep your source of truth in the database.
Kafka: ordering only holds within a partition, so give related events one key.

Key takeaways

Docker builds images and runs isolated containers.
Kubernetes schedules pods toward a desired state, restarting failures and scaling replicas.
Redis serves frequently read data from memory, which reduces load on the database.
Kafka stores events in partitioned, replayable logs, so consumers can work independently.
Add each tool only when its specific problem appears.

Frequently asked questions

Do I need Docker, Kubernetes, Redis and Kafka in every project?

No. Each tool answers a specific problem, and you rarely need them on day one. A simple service can stay on Docker until operations actually become painful, and the other tools can wait until their problems appear too.

What is the difference between Docker and Kubernetes?

Docker packages a service into an image and runs it as an isolated container. Kubernetes schedules pods that wrap those containers and keeps working to match the number of replicas you asked for.

Can Redis replace my database?

Not in the setup described here. Redis holds a copy of frequently read data in memory, while the database remains the source of truth. Setting an expiry and deleting keys when data changes helps limit how long stale values can survive.

Why publish events to Kafka instead of calling other services directly?

Direct calls let one slow service hold up the others. With Kafka, shop-api publishes an event once, and services such as email and analytics read it independently, each tracking its own progress through the log.

Does reading a Kafka event delete it?

No. Events stay in the partition until retention removes them, and each consumer group tracks its own offset. That is why a consumer can reset its offset and reprocess events Kafka still retains.

Why is my pod stuck in ImagePullBackOff when using kind?

This usually means the cluster can't find your image. Run kind load docker-image to copy the local image into the cluster, then check that the tag matches the one in your deployment.

Watch the full video on YouTube →

Tag

Ad