How to Install Apache Kafka on Ubuntu 18.04

In this blog, we will learn how to install and configure Apache Kafka, which is distribute publish subscribe messaging system and a robust queue which handle high volume of data and enables to pass message from one end point to another. It is use for both offline as well as online message consumption. Click here for more

Prerequisites :-
  • Ubuntu 18.04
  • Root privileges

What we will do?

  • Install Java OpenJDK 11
  • Install Apache Zookeeper
  • Download and Setup Apache Kafka
  • Add Apache Kafka and Zookeeper as s Service
  • Test complete setup

Install Java OpenJDK 11 :-

Apache Kafka is written in Java and Scala, so we need to install Java in our server. Update and upgrade the repository and packages before installing new any new packages.

Now install the OpenJDK 11 in ubuntu 18.04.

Install Apache Zookeeper :-

Apache Kafka uses zookeeper for electing leader in Kafka broker and use to store basic metadata like (information about topics, brokers, consumer offsets and so on).

Now we will install zookeeper in ubuntu 18.04 from ubuntu repository.

Download and setup Apache Kafka :-

In this part we will download Kafka from its official website, and then install and configure it as a non-root user.

Now add new user named ‘kafka’ and set password for the user.

Now go to /opt directory and download the Kafka.

Then create new Kafka directory.

Extract the kafka_*.tgz in Kafka directory and change the owner permissions for the directory.

Now login into Kafka user to configure some files.

Paste the following term to the end of file.

Then use “esc”, :wq! to save and exit from the file and then use exit to exit from kafka user.

The Kafka configuration has been completed.

Add Apache Kafka and Zookeeper as a Service :-

In this part of configuration we will add Kafka and Zookeeper as a service of our ubuntu machine.

For this go to /lib/systemd/system/ directory.

Paste this lines in the file.

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/kafka/bin/zookeeper-server-start.sh /kafka/config/zookeeper.properties
ExecStop=/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and exit the file with “esc”, :wq!.

Then create service file for Kafka.

Paste this lines in the file.

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c ‘/kafka/bin/kafka-server-start.sh /kafka/config/server.properties’
ExecStop=/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and exit the file with “esc”, :wq!.

Then after all this steps reload systemd configuration manager.

Now start Zookeeper and Kafka service.

Now the Kafka and Zookeeper both are in active state and zookeeper is running on port ‘2181’ and kafka is running on port ‘9092’.

Test Complete Setup :-

Login to the kafka user and go to bin directory.

Now create a topic in Kafka.

Then run ‘kafka-console-consumer.sh’ for the ‘CloudkulTesting’ topic.

Now open another terminal and follow the instructions.

Then run ‘kafka-console-producer.sh’ for the ‘CloudkulTesting’ topic.

In the right side of image consumer consumes the message and in the left side of image producer produce the messages.

The setup as well as configuration of Kafka and Zookeeper on ubuntu 18.04 is completed successfully.

Category(s) Uncategorized
. . .

Comment

Add Your Comment

Be the first to comment.

css.php