Quick start with Kafka
This article describes the first steps to get started with Kafka.
Install and run a Kafka environment via ADCM
-
To run a Kafka environment using ADCM in an ADS cluster, install the following services:
-
Kafka
-
ZooKeeper
-
Kafka-Manager (optional)
-
-
After installation in the ADCM interface for all installed services, apply the Start action by clicking the
icon in the Actions column one by one and wait for the launch to complete successfully.
Starting services in the ADCM interface
As a result, the Kafka environment is installed on the hosts, and the Kafka broker and ZooKeeper servers are running.
NOTE
|
Connect to Kafka
Command line
When installing Kafka on hosts, a new directory /usr/lib/kafka/bin is created. It contains .sh files, which are scripts for executing commands in the Kafka environment. To view the directory, run the following command:
$ ls -la /usr/lib/kafka/bin
Working with the Kafka service occurs by running the appropriate scripts on the command line. Typically, scripts are launched from the /usr/lib/kafka folder, specifying the bin/ part of the path in the command. To switch to the desired directory run the command:
$ cd /usr/lib/kafka
Kafka-Manager
-
In the address bar of the browser, enter the address of the host where the Kafka-Manager service is installed. This uses the port number specified when configuring the Kafka-Manager service (by default
9000
). The process of setting up services is given in the article Configure services. -
The Kafka-Manager UI then opens with the ADS cluster installed, indicating the version of the Kafka service.


Create a topic
Сommand line
In order to write messages to Kafka, it is necessary to create a topic. This is done using the kafka-topics.sh script and the --create
option from any cluster host that has a Kafka service installed.
To create a topic, run the following command:
$ bin/kafka-topics.sh --create --topic new-topic --bootstrap-server hostname:9092
In the command entry, options are applied to describe the topic:
-
new-topic
— topic name; -
hostname
— hostname where the topic is created; -
9092
— HTTP port for accessing the service Kafka.
As a result of the script execution, the following message appears:
Created topic new-topic.
NOTE
The topic name can include alphanumeric characters (including ASCII), as well as the characters |
In order to view data about an existing topic, the kafka-topics.sh script and the --describe
option are used.
To view data about topic, run the following command:
$ bin/kafka-topics.sh --describe --topic new-topic --bootstrap-server hostname:9092
As a result, the data about the requested topic appears in the form:
Topic: new-topic TopicId: STGaaB8QSF64s06MRtp-YA PartitionCount: 1 ReplicationFactor: 1 Configs: unclean.leader.election.enable=false Topic: new-topic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
It indicates that the created topic has the following parameters:
-
number of partitions —
1
; -
replication factor —
1
; -
broker leader designation for this partition —
1001
; -
unclean.leader.election.enable
parameter is set to false.
NOTE
Other actions that the kafka-topics.sh script performs can be seen in the list that appears after the command is executed:
|
Kafka-Manager
-
On the main page of the Kafka-Manager user interface, open the cluster settings by clicking on the cluster name.
Switching to cluster settingsSwitching to cluster settings -
In the window that opens with information about the cluster, select Topic → Create in the top menu.
Switching to topic creationSwitching to topic creation -
In the window that opens, enter the necessary parameters, for example:
-
topic name
new-topic-1
; -
number of partitions for topic
3
; -
replication factor
3
.Creating a topic with Kafka-ManagerCreating a topic with Kafka-Manager
-
-
Click Create.
-
Get the result
Done!
and follow the suggested link Go to topic view. -
As a result, a window appears with options and control buttons for the new topic.
Viewing topic parameters in Kafka-ManagerViewing topic parameters in Kafka-Manager -
To view all created topics, select Topic → List in the top menu.
Switching to the list of topicsSwitching to the list of topics -
As a result, a list of all topics created in this cluster appears.
Viewing a list of topics in a clusterViewing a list of topics in a cluster
All created topics are available for viewing in the list: the new-topic
, created using the command line, and the new-topic-1
, created using Kafka-Manager. The service topic __consumer_offsets
is also available for viewing.
Write a message to a topic
Writing a message to a topic is done using the kafka-console-producer.sh script from any cluster host on which the Kafka service is installed.
TIP
Topic creation occurs automatically when using the kafka-console-producer.sh script, if you enable the auto.create.topics.enable parameter in the server.properties group when configuring the Kafka service. |
To write a message to the new-topic
topic:
-
Run the following command:
$ bin/kafka-console-producer.sh --topic new-topic --bootstrap-server hostname:9092
This starts the message recording mode.
-
On the next line after entering the command enter the required messages, еach message on a new line. Writing a message and moving to a new line is done with
Enter
. For example, several messages are entered in this way, each on its own line:>Sunday >Monday >Tuesday >Wednesday >Thursday >Friday >Saturday
-
Exit the message recording mode, for this do the following:
-
after writing the last message go to the next line;
-
press
Ctrl+C
.
-
On the Kafka-Manager UI page, you can verify that the right number of partition offsets appeared in the topic; check the Sum of partition offsets item in the Topic Summary area.


NOTE
To view other options that can be applied when executing the kafka-console-producer.sh script, you must run the command:
|
Read messages from a topic
Reading messages from a topic is done using the kafka-console-consumer.sh script from any cluster host on which the Kafka service is installed.
Read from the beginning of the partition
To read messages from the new-topic
topic starting from the very first message written to the partition run the following command with the --from-beginning
option :
$ bin/kafka-console-consumer.sh --topic new-topic --from-beginning --bootstrap-server hostname:9092
This starts the message reading mode. Messages are read in the same order they are written:
Sunday Monday Tuesday Wednesday Thursday Friday Saturday
To exit from the message reading mode, press Ctrl+C
.
TIP
If you write messages to the same topic using kafka-console-producer.sh from another terminal session, you can immediately read them in read mode.
|
Read from the given offset
To read messages from the new-topic
topic starting at offset 4 (Thursday
), use the options to specify the partition --partition 0
and the offset --offset 4
:
$ bin/kafka-console-consumer.sh --topic new-topic --partition 0 --offset 4 --bootstrap-server hostname:9092
The result contains messages written starting from the offset 4:
Thursday Friday Saturday
NOTE
To view other options that can be applied when executing the kafka-console-consumer.sh script, you need to run the command:
|
Terminate the Kafka environment
To stop services via the ADCM interface, apply the Stop action by clicking the
icon in the Actions column one by one and wait for the action to complete successfully.

To delete local Kafka data including any generated messages, run the command:
$ rm -rf /tmp/kafka-logs /tmp/zookeeper