Quick start with Kafka

This article describes the first steps to get started with Kafka.

Install and run a Kafka environment via ADCM

  1. To run a Kafka environment using ADCM in an ADS cluster, install the following services:

    • Kafka

    • ZooKeeper

    • Kafka-Manager (optional)

  2. After installation in the ADCM interface for all installed services, apply the Start action by clicking the actions default dark actions default light icon in the Actions column one by one and wait for the launch to complete successfully.

    adcm services 01
    Starting services in the ADCM interface

As a result, the Kafka environment is installed on the hosts, and the Kafka broker and ZooKeeper servers are running.

NOTE
  • Installation of services is described in the articles Online installation, Offline installation.

  • The Kafka-Manager service is optional. It is based on the CMAK service that allows to perform some actions in Kafka without using the command line.

Connect to Kafka

Command line

When installing Kafka on hosts, a new directory /usr/lib/kafka/bin is created. It contains .sh files, which are scripts for executing commands in the Kafka environment. To view the directory, run the following command:

$ ls -la /usr/lib/kafka/bin

Working with the Kafka service occurs by running the appropriate scripts on the command line. Typically, scripts are launched from the /usr/lib/kafka folder, specifying the bin/ part of the path in the command. To switch to the desired directory run the command:

$ cd /usr/lib/kafka

Kafka-Manager

  1. In the address bar of the browser, enter the address of the host where the Kafka-Manager service is installed. This uses the port number specified when configuring the Kafka-Manager service (by default 9000). The process of setting up services is given in the article Configure services.

  2. The Kafka-Manager UI then opens with the ADS cluster installed, indicating the version of the Kafka service.

kafka manager ui 01 dark
Kafka-Manager user interface
kafka manager ui 01 light
Kafka-Manager user interface

Create a topic

Сommand line

In order to write messages to Kafka, it is necessary to create a topic. This is done using the kafka-topics.sh script and the --create option from any cluster host that has a Kafka service installed.

To create a topic, run the following command:

$ bin/kafka-topics.sh --create --topic new-topic --bootstrap-server hostname:9092

In the command entry, options are applied to describe the topic:

  • new-topic — topic name;

  • hostname — hostname where the topic is created;

  • 9092 — HTTP port for accessing the service Kafka.

As a result of the script execution, the following message appears:

Created topic new-topic.
NOTE

The topic name can include alphanumeric characters (including ASCII), as well as the characters ., _, and -.

In order to view data about an existing topic, the kafka-topics.sh script and the --describe option are used.

To view data about topic, run the following command:

$ bin/kafka-topics.sh --describe --topic new-topic --bootstrap-server hostname:9092

As a result, the data about the requested topic appears in the form:

Topic: new-topic	TopicId: STGaaB8QSF64s06MRtp-YA	PartitionCount: 1	ReplicationFactor: 1	Configs: unclean.leader.election.enable=false
Topic: new-topic	Partition: 0	Leader: 1001	Replicas: 1001	Isr: 1001

It indicates that the created topic has the following parameters:

  • number of partitions — 1;

  • replication factor — 1;

  • broker leader designation for this partition — 1001;

  • unclean.leader.election.enable parameter is set to false.

NOTE

Other actions that the kafka-topics.sh script performs can be seen in the list that appears after the command is executed:

$ bin/kafka-topics.sh

Kafka-Manager

  1. On the main page of the Kafka-Manager user interface, open the cluster settings by clicking on the cluster name.

    kafka manager ui 06 dark
    Switching to cluster settings
    kafka manager ui 06 light
    Switching to cluster settings
  2. In the window that opens with information about the cluster, select TopicCreate in the top menu.

    kafka manager ui 07 dark
    Switching to topic creation
    kafka manager ui 07 light
    Switching to topic creation
  3. In the window that opens, enter the necessary parameters, for example:

    • topic name new-topic-1;

    • number of partitions for topic 3;

    • replication factor 3.

      kafka manager ui 02 dark
      Creating a topic with Kafka-Manager
      kafka manager ui 02 light
      Creating a topic with Kafka-Manager
  4. Click Create.

  5. Get the result Done! and follow the suggested link Go to topic view.

  6. As a result, a window appears with options and control buttons for the new topic.

    kafka manager ui 03 dark
    Viewing topic parameters in Kafka-Manager
    kafka manager ui 03 light
    Viewing topic parameters in Kafka-Manager
  7. To view all created topics, select TopicList in the top menu.

    kafka manager ui 08 dark
    Switching to the list of topics
    kafka manager ui 08 light
    Switching to the list of topics
  8. As a result, a list of all topics created in this cluster appears.

    kafka manager ui 04 dark
    Viewing a list of topics in a cluster
    kafka manager ui 04 light
    Viewing a list of topics in a cluster

All created topics are available for viewing in the list: the new-topic, created using the command line, and the new-topic-1, created using Kafka-Manager. The service topic __consumer_offsets is also available for viewing.

Write a message to a topic

Writing a message to a topic is done using the kafka-console-producer.sh script from any cluster host on which the Kafka service is installed.

TIP

Topic creation occurs automatically when using the kafka-console-producer.sh script, if you enable the auto.create.topics.enable parameter in the server.properties group when configuring the Kafka service.

To write a message to the new-topic topic:

  1. Run the following command:

    $ bin/kafka-console-producer.sh --topic new-topic --bootstrap-server hostname:9092

    This starts the message recording mode.

  2. On the next line after entering the command enter the required messages, еach message on a new line. Writing a message and moving to a new line is done with Enter. For example, several messages are entered in this way, each on its own line:

    >Sunday
    >Monday
    >Tuesday
    >Wednesday
    >Thursday
    >Friday
    >Saturday
  3. Exit the message recording mode, for this do the following:

    • after writing the last message go to the next line;

    • press Ctrl+C.

On the Kafka-Manager UI page, you can verify that the right number of partition offsets appeared in the topic; check the Sum of partition offsets item in the Topic Summary area.

kafka manager ui 05 dark
View number of recorded offsets
kafka manager ui 05 light
View number of recorded offsets
NOTE

To view other options that can be applied when executing the kafka-console-producer.sh script, you must run the command:

$ bin/kafka-console-producer.sh

Read messages from a topic

Reading messages from a topic is done using the kafka-console-consumer.sh script from any cluster host on which the Kafka service is installed.

Read from the beginning of the partition

To read messages from the new-topic topic starting from the very first message written to the partition run the following command with the --from-beginning option :

$ bin/kafka-console-consumer.sh --topic new-topic --from-beginning --bootstrap-server hostname:9092

This starts the message reading mode. Messages are read in the same order they are written:

Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday

To exit from the message reading mode, press Ctrl+C.

TIP
If you write messages to the same topic using kafka-console-producer.sh from another terminal session, you can immediately read them in read mode.

Read from the given offset

To read messages from the new-topic topic starting at offset 4 (Thursday), use the options to specify the partition --partition 0 and the offset --offset 4:

$ bin/kafka-console-consumer.sh --topic new-topic --partition 0 --offset 4 --bootstrap-server hostname:9092

The result contains messages written starting from the offset 4:

Thursday
Friday
Saturday
NOTE

To view other options that can be applied when executing the kafka-console-consumer.sh script, you need to run the command:

$ bin/kafka-console-consumer.sh

Terminate the Kafka environment

To stop services via the ADCM interface, apply the Stop action by clicking the actions default dark actions default light icon in the Actions column one by one and wait for the action to complete successfully.

adcm services 02
Stopping services in the ADCM interface

To delete local Kafka data including any generated messages, run the command:

$ rm -rf /tmp/kafka-logs /tmp/zookeeper
Found a mistake? Seleсt text and press Ctrl+Enter to report it