Work with topics in ADS Control

Olga Semme

Topics page overview

The Clusters → <cluster name> → Topics page in the ADS Control web interface is designed to display information about topics and manage topics in the ADS cluster. The Topics page becomes available after selecting a cluster in the cluster management section and going to the desired tab on the General page.

Topics page
Topics page
Topics page
Topics page

The top of the page contains:

  • Section of general information about topics:

    • total topics field — the total number of topics in the cluster, including internal topics;

    • total partitions field — the total number of partitions, including partitions of internal topics, excluding replicas.

  • General topic management section:

    • field for searching a topic by name;

    • restart dark restart light button to reset the search results;

    • Show internal topics toggle off dark toggle off light switch, which enables display of system (internal) topics;

    • Create topic button for creating new topics.

The Topics page contains a table with information about each topic. The table columns are described below.

Field Description

Name

Topic name. By clicking the name, you can go to the topic overview page

Messages

Number of messages in the topic

Internal

Indicates whether this topic is internal

Size

Size of data contained in the topic

Cleanup policy

Log cleanup policy set for the topic (corresponds to the cleanup.policy topic parameter)

Retention,ms

Message storage time for a topic with a retention policy delete (corresponds to the retention.ms topic parameter)

Retention,bytes

The maximum partition size, upon reaching which old log segments are deleted for a topic with a storage policy delete (corresponds to the retention.bytes topic parameter)

Partitions

Number of partitions of the topic (corresponds to the num.partitions broker parameter or --partitions option when using the kafka-topics.sh script on the command line)

Replicas

Replication factor of the topic (corresponds to the default.replication.factor broker parameter or the --replication-factor option when using the kafka-topics.sh script on the command line)

Actions

The column contains icons for managing the topic:

  • clear dark clear light — when this icon is clicked, two actions are performed in the background — deleting and then creating a topic with the same parameters (cleaning the topic).

  • delete dark delete light — when this icon is clicked, topic removal starts. The delete operation is asynchronous. When a topic is deleted, it appears in the table until it is actually deleted from the ADS cluster.

Both actions require confirmation

Create a topic

After clicking Create topic, the topic creation window opens.

Topic creation window
Topic creation window
Topic creation window
Topic creation window

At the top of the window there are:

  • Expand content toggle off dark toggle off light switch, which reveals hidden content.

  • JSON view toggle off dark toggle off light switch, which opens the content in JSON format.

  • Save button to save the topic after setting the parameters and Cancel button to cancel the creation of the topic.

The parameters for the topic being created are divided into drop-down lists:

  • Common — basic parameters of the topic. Their description is located in the table information about topics (the description of the min.insync.replicas parameter is located in the description of the drop-down list Replication of the Configuration tab).

  • Advanced — field for specifying additional parameters that are not in the Common field. A description of possible additional parameters for a topic is located in the description of drop-down lists with parameters of the Configuration tab. If the Advanced field contains a parameter that already exists in the Common field, it will be overwritten.

The operation of creating a topic is asynchronous. The topic appears in the table after being physically created in the ADS cluster. The waiting time depends on the operation of the cluster.

When filling out topic parameters in JSON format, unspecified parameters will be assigned default values. After filling out the form, click Save to save the topic parameters or Cancel to cancel topic creation.

Topic creation window in JSON format
Topic creation window in JSON format
Topic creation window in JSON format
Topic creation window in JSON format

Topic overview

After clicking the topic name in the topic table, the topic overview page opens.

At the top of the page there is a section displaying the main parameters of the topic, as well as topic management icons in accordance with the table given above.

The Partitions and Configuration tabs are displayed at the bottom.

Partitions tab

The Clusters → <cluster name> → Topics → <topic name> → Partitions tab opens by default when you go to the topic overview page. The tab provides information about the topic partitions.

Partitions tab
Partitions tab
Partitions tab
Partitions tab

On the Partitions tab there are:

  • Field for searching a partition by number.

  • restart dark restart light button to reset search results.

  • Table with information about each partition. The table columns are described below.

Field Description

Partition ID

Partition number

Low

Minimum offset of messages in the topic partition

High

Maximum offset of messages in the topic partition

Messages

Number of messages in the partition

Brokers

Identifiers of the brokers on which the partition and its replicas are located. After clicking on the broker identifier, you are redirected to the broker overview page

Configuration tab

The Clusters → <cluster name> → Topics → <topic name> → Configuration tab opens from the topic overview page. The tab allows you to view and edit parameters that can be changed after creating a topic.

Configuration tab
Configuration tab
Configuration tab
Configuration tab

On the Configuration tab there are:

  • Field to search for a parameter by number.

  • Expand content toggle off dark toggle off light toggle, which reveals hidden content.

  • JSON view toggle off dark toggle off light toggle, which opens the content in JSON format.

  • Edit button to run the mode for editing topic parameters. After editing the parameters, click Save to save the changes or Cancel to cancel the changes.

  • Topic parameters divided into drop-down lists. Below is a brief description of the topic parameters that can be changed or specified on the Configuration tab.

Replication
Parameter Description

follower.replication.throttled.replicas

List of replicas for which log replication should be regulated on the subscriber side

leader.replication.throttled.replicas

List of replicas for which log replication should be regulated on the leader side

min.insync.replicas

The minimum number of replicas that must acknowledge a write for the write to be considered successful. You can read more about replica synchronization in the corresponding section of the Replication in Kafka article

unclean.leader.election.enable

Indicates whether to select non-ISR-replicas as a leader as a last resort, even though this may result in data loss
Storage
Parameter Description

cleanup.policy

Log cleanup policy (see Topics table description)

compression.type

Compression type for topic

delete.retention.ms

Storage time for deletion markers for compressed topics. You can read more about deletion markers in the corresponding section of the Log compaction in Kafka article

file.delete.delay.ms

Wait time before deleting a file from the file system

max.compaction.lag.ms

The maximum time during which a message will not be compressed in the log. You can read more about starting compression in the corresponding section of the Log compaction in Kafka article

max.message.bytes

Maximum record batch size allowed by Kafka (after compression, if compression is enabled)

message.format.version

The version of the message format that the broker will use to add messages to the logs

message.timestamp.difference.max.ms

The maximum allowed difference between the timestamp when the broker receives the message and the timestamp specified in the message

message.timestamp.type

Determines whether the timestamp in the message is the time the message was created or the time the log was added

min.compaction.lag.ms

The minimum amount of time a message will remain uncompressed in the log. You can read more about starting compression in the corresponding section of the Log compaction in Kafka article

retention.bytes

Maximum partition size before deleting old segments (see Topics table description)

retention.ms

Message storage time (see Topics table description)

segment.bytes

Log segment file size

segment.ms

The period of time after which Kafka forces it to scan the log for the possibility of deleting or compressing old data, even if the segment file is not full
Storage Performance
Parameter Description

flush.messages

Interval (number of messages) after which fsync (synchronization of application data with disk) will be forced

flush.ms

Time interval at which to force fsync of logged data

index.interval.bytes

Frequency of adding an index entry to the Kafka offset index

message.downconversion.enable

Enable downconversion of message formats to support consumption requests

min.cleanable.dirty.ratio

The ratio of the volume of dirty segments to the total volume of the log at which compression starts. You can read more about starting compression in the corresponding section of the Log compaction in Kafka article

segment.index.bytes

The size of the index that maps offsets to file positions

segment.jitter.ms

Maximum random jitter (phase jitter of a digital signal) of data
Others
Parameter Description

preallocate

Set to true if you need to pre-allocate the file on disk when creating a new log segment

If you change parameters in JSON format after filling out the form, click Save to save the changes or Cancel to cancel the changes.

Topic parameters in JSON format
Topic parameters in JSON format
Topic parameters in JSON format
Topic parameters in JSON format
Found a mistake? Seleсt text and press Ctrl+Enter to report it