docker kafka cluster setup

What is Apache Kafka?

Apache Kafka as your might have known is a stream processing platform which means it processes data as it is generated in real-time. This helps in building real time data pipelines and streaming applications. I am not going to dive deep on explaining what is Apache Kafka as you can read the same from official documentation.


The only pre-requisite is to have docker installed on your local system. You can install the docker in your local system from here.

Docker Image selection

Let's go to docker hub explore section and search for the keyword Kafka. Don't forget to check all the trusted content filters. Let's go with the top selection. It is a Kafka docker image by bitnami.

docker-compose kafka

Local set up

Instead of directly pulling the image, let us go with docker-compose.yml file as a bit of configuration is needed like below.

Note that, if the container is cleaned up, the configurations are lost. If you don't want that, you can download use below docker-compose.yml file.

Create a directory and download one of the above docker-compose files. Then run below command
docker-compose up -d

To check the containers, run
docker ps

kafka docker compose yml

Kafka commands cheat sheet

Now the setup is complete and both Kafka and zookeeper are running. Let us create a topic and publish a message using console producer and consumer using console consumer.


I want to introduce a new tool that I generally use i.e kcat(Formally known as kafkacat). This is a more powerful tool than regular console consumer and producer and known as netcat for Kafka. 

kcat Kafka Installation

Install kcat from here. Detailed docs is found at Below are the commands to produce a record and consume with headers


Post a Comment