What is Apache Kafka?
Apache Kafka as your might have known is a stream processing platform which
means it processes data as it is generated in real-time. This helps in
building real time data pipelines and streaming applications. I am not going
to dive deep on explaining what is Apache Kafka as you can read the same from
official documentation.
Pre-requisite
The only pre-requisite is to have docker installed on your local system. You
can install the docker in your local system from
here.
Docker Image selection
Let's go to docker hub explore
section and search for the keyword Kafka. Don't forget to check all the
trusted content filters. Let's go with the top selection. It is a Kafka docker
image by bitnami.
Local set up
Instead of directly pulling the image, let us go with
docker-compose.yml file as a bit of configuration is needed like below.
Note that, if the container is cleaned up, the configurations are lost. If you
don't want that, you can download use below docker-compose.yml file.
Create a directory and download one of the above docker-compose files. Then
run below command
docker-compose up -dTo check the containers, run
docker ps
Kafka commands cheat sheet
Now the setup is complete and both Kafka and zookeeper are running. Let us
create a topic and publish a message using console producer and consumer using
console consumer.
Bonus
I want to introduce a new tool that I generally use i.e kcat(Formally
known as kafkacat). This is a more powerful tool than regular console consumer
and producer and known as netcat for Kafka.
kcat Kafka Installation
Install kcat from here.
Detailed docs is found at
docs.confluent.io. Below are the
commands to produce a record and consume with headers
Post a Comment