Mirror Maker 2 in ADS Control

Olga Semme

Contents

Create connectors for Mirror Maker 2
Topic replication check

This article describes setting up topic replication to a remote cluster via the Mirror Maker 2 mechanism using the ADS Control.

The main concepts used in this article:

Source cluster — the cluster from which topics are replicated.
Target cluster — the cluster to which topics are replicated.
Source topic — a topic created on the source cluster and intended to be replicated to the target cluster.
Replica topic — a topic automatically created on the target cluster and having the same content and metadata as the source topic.

Note

Replication is shown based on the Active/Standby architecture when the target cluster is in passive mode (has no consumers and producers connected to it) and receives only replicated data.
Before configuring Mirror Maker 2, the Kafka Connect service must be installed and configured on the source and destination clusters. Kafka Connect is available for installation in ADS starting from version 1.7.1.

Create connectors for Mirror Maker 2

To perform a simple replication of a topic to a remote cluster, use the following connectors created on the basis of Kafka Connect:

MirrorSourceConnector
MirrorCheckpointConnector

To create connectors using the ADS Control, you need to:

In the ADS Control user interface, select the required ADS cluster.

Cluster selection in the ADS Control interface

Cluster selection in the ADS Control interface
On the page that opens, select a ADS cluster from the list of Kafka Connects.

Opening the page with Kafka Connect connectors

Opening the page with Kafka Connect connectors
On the page that opens, click Create Connector.

Creating a Kafka Connect Connector

Creating a Kafka Connect Connector
Select the desired connector to create.

Selecting the Kafka Connect connector to create

Selecting the Kafka Connect connector to create

Fill in the connector configuration parameters. If necessary, refer to the Kafka Connect service configuration information in ADS configuration parameters. You can use filling in the configuration in the form of a JSON file. To do this, enable the corresponding switch.

Connector configuration

Сonnector JSON configuration file

Example JSON file content for simple connector configuration for Active/Standby architecture

{
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "source.cluster.alias": "source",
    "target.cluster.alias": "dest",
    "name": "<connector_name>",
    "target.cluster.bootstrap.servers": "<dest_cluster_bootstrap>",
    "tasks.max": "4",
    "topics": "<test_topic_name>",
    "source.cluster.bootstrap.servers": "<source_cluster_bootstrap>"
}

Attribute Description

connector.class

The name of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector

source.cluster.alias

The name of the source cluster to be used in the replication

target.cluster.alias

The name of the target cluster to be used in the replication

name

The name of the connector to be used in the replication

target.cluster.bootstrap.servers

Bootstrap-servers of the target cluster, used with port 9092

tasks.max

The maximum number of tasks that can be used for a connector

topics

The comma-separated list of source topics to use

source.cluster.bootstrap.servers

Bootstrap-servers of the source cluster, used on port 9092

After filling, click Save and get a message about the successful creation of the connector.

Message about the successful creation of the connector

Message about the successful creation of the connector
Check that the Connectors for ads page displays the created connectors in the working status. The status is determined depending on the indicator in front of the connector name:
- — the connector/task is running;
- — the connector/task has been administratively paused;
- — the connector/task has failed (usually by raising an exception, which is reported in the status output);
- — the connector/task has not yet been assigned to a worker.
  
  Created connectors
  
  Created connectors

Topic replication check

After successful completion of the connector configuration, a service topic mm2-offset-syncs.<target_cluster_alias>.internal is created on the side of the source cluster, where <target_cluster_alias> corresponds to the target cluster designation entered during the connector configuration.

The mm2-offset-syncs topic is populated by the MirrorSourceConnector and then used by the MirrorCheckpointConnector to convert consumer group offsets.

In order to check for the presence of the service topic mm2-offset-syncs, enter the command on any broker of the source cluster to display the list of topics:

$ /usr/lib/kafka/bin/kafka-topics.sh --list --bootstrap-server hostname:9092

The list of topics that appears should contain the desired topic:

__consumer_offsets
mm-connect-configs
mm-connect-offsets
mm-connect-status
mm2-offset-syncs.dest.internal

The creation of a replicated source topic and the writing of messages to it occurs on the side of the source cluster.

To create an source topic, enter the command:

$ /usr/lib/kafka/bin/kafka-topics.sh --create --topic <test_topic_name> --bootstrap-server hostname:9092

where <test_topic_name> — the name of the source topic, as entered when configuring the connector.

To write messages to the original topic, enter the command:

$ /usr/lib/kafka/bin/kafka-console-producer.sh --topic <test_topic_name> --bootstrap-server hostname:9092

After writing messages to the source topic on the Connectors for ads page in the ADS Control user interface, the name of the replica topic appears in the line of the MirrorSourceConnector connector in the <source_cluster_alias>.<test_topic_name> format, where:

<source_cluster_alias> — the corresponding source cluster alias entered during connector configuration.
<test_topic_name> — the name of the source topic entered during connector configuration.

List of connectors indicating the replica topic

The replica topic also appears in the list of topics on the target cluster. In order to read messages from the replica topic on the target cluster, enter the command:

$ /usr/lib/kafka/bin/kafka-console-consumer.sh --topic <source_cluster_alias>.<test_topic_name> --from-beginning --bootstrap-server hostname:9092

Read messages in the target cluster completely repeat those written in the source cluster.

Found a mistake? Seleсt text and press Ctrl+Enter to report it