Kafka Connect configuration parameters

Olga Semme

To configure the service, use the following configuration parameters in ADCM.

NOTE

  • Some of the parameters become visible in the ADCM UI after the Advanced flag has been set.

  • The parameters that are set in the Custom group will overwrite the existing parameters even if they are read-only.
env_vars
Parameter Description Default value

LOG_DIR

Directory for logs

/var/log/kafka

KAFKA_OPTS

Kafka environment variables

 — 

KAFKA_HEAP_OPTS

Heap size allocated to the KAFKA server process

-Xms256M -Xmx2G

KAFKA_LOG4J_OPTS

Environment variable for LOG4J logging configuration

-Dlog4j.configuration=file:/etc/kafka/conf/connect-distributed-log4j.properties

Add property

Parameters and their values ​​entered in this field override the parameters specified in the ADCM user interface. This field also allows you to define the values of new environment variables that will be specified in the kafka-connect-java.yaml configuration file

 — 
connect-distributed.properties
Parameter Description Default value

group.id

Unique string that identifies a Kafka Connect group, to which this connector belongs

mm-connect

rest.port

Port for Kafka Connect REST API to work

8083

plugin.path

Path to Kafka Connect plugin

Preconfigured values:

  • /var/lib/kafka-connect/libs — general path to the plugin;

  • /usr/lib/kafka-connect/plugins — path to the Iceberg plugin.

config.storage.replication.factor

Replication factor for config.storage.topic

1

offset.storage.replication.factor

Replication factor for offset.storage.topic

1

status.storage.replication.factor

Replication factor for status.storage.topic

1

offset.flush.interval.ms

Interval at which to try committing offsets for tasks

1000

key.converter

Converter class for key Connect data

org.apache.kafka.connect.converters.ByteArrayConverter

value.converter

Converter class for value Connect data

org.apache.kafka.connect.converters.ByteArrayConverter

connector.client.config.override.policy

Class name or alias of implementation of ConnectorClientConfigOverridePolicy

None

Add key, value

Parameters and their values ​​entered in this field override the parameters specified in the ADCM user interface. This field also allows you to set values ​​for all user parameters that are not displayed in the interface, but are allowed in the kafka-rest.properties configuration file

Parameters and their values ​​entered in this field override the parameters specified in the ADCM user interface. This field also allows you to set values ​​for all user parameters that are not displayed in the interface, but are allowed in the configuration file connect-distributed.properties
connect-distributed-log4j.properties
Parameter Description Default value

rootLogger

Logging level

INFO

MaxBackupIndex

Maximum number of saved files

10

MaxFileSize

Maximum file size

100MB

Setting the structure of the logging configuration file for Kafka Connect.

Logger Default package names Default event level

loggers

org.reflections

ERROR

org.I0Itec.zkclient

ERROR

org.apache.zookeeper

ERROR

Add property

Parameters and their values ​​entered in this field override the parameters specified in the ADCM user interface. This field also allows you to set values ​​for all user parameters that are not displayed in the interface, but are allowed in the connect-distributed-log4j.properties configuration file
connect_distributed_log4j2_properties_template

The connect_distributed_log4j2_properties template is intended for specifying custom logging parameters.

Default value:

{% set p = kafka_connect_config.connect_distributed_log4j2_yaml_content %}

Configuration:
  status: warn

  Appenders:
    Console:
      name: stdout
      target: SYSTEM_OUT
      PatternLayout:
        pattern: "[%d] %p %m (%c:%L)%n"

    RollingFile:
      - name: connectDRFA
        fileName: "${sys:kafka.logs.dir}/connect-distributed.log"
        filePattern: "${sys:kafka.logs.dir}/connect-distributed.log.%d{yyyy-MM-dd-HH}"
        PatternLayout:
          pattern: "[%d] %p %m (%c:%L)%n"
        TimeBasedTriggeringPolicy:
          interval: 1
          modulate: true
        DefaultRolloverStrategy:
          max: {{ p['MaxBackupIndex'] }}

      - name: connectRFA
        fileName: "${sys:kafka.logs.dir}/connect-distributed.log"
        filePattern: "${sys:kafka.logs.dir}/connect-distributed.log.%i"
        append: true
        PatternLayout:
          pattern: "[%d] %p %m (%c)%n"
        SizeBasedTriggeringPolicy:
          size: "{{ p['MaxFileSize'] }}"
        DefaultRolloverStrategy:
          max: {{ p['MaxBackupIndex'] }}

  Loggers:
    Logger:
{% for key, value in p['loggers'] | dictsort %}
      - name: {{ key }}
        level: {{ value }}
        additivity: false
        AppenderRef:
          - ref: connectDRFA
          - ref: connectRFA
          - ref: stdout
{% endfor %}
    Root:
      level: {{ p['rootLogger'] }}
      AppenderRef:
        - ref: connectDRFA
        - ref: connectRFA
        - ref: stdout
SSL
Parameter Description Default value

ssl.keystore.location

Location of the keystore file. This is optional for client and can be used for two-way authentication for client

 — 

ssl.keystore.password

Store password for the keystore file. This is optional for client and only needed if ssl.keystore.location is configured

 — 

ssl.key.password

Password of the private key in the keystore file. This is optional for client

 — 

ssl.keystore.type

File format of the keystore file. This is optional for client

 — 

ssl.truststore.location

Location of the truststore file

 — 

ssl.truststore.password

Store password for the truststore file. This is optional for client and only needed if ssl.truststore.location is configured

 — 

ssl.truststore.type

File format of the truststore file

 — 
JAAS template

The user file template jaas.conf is intended for specifying user data for connecting clients of other services to the current service (paths to keytab files, the useTicketCache parameter, etc.). For more information, see Configure a custom jaas.conf.

Default value:

{% if cluster.config.kerberos_client and cluster.config.kerberos_client.enable_kerberos %}
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    serviceName="kafka"
    keyTab="{{ cluster.config.kerberos_client.keytab_dir }}/kafka-connect.service.keytab"
    principal="kafka-connect/{{ ansible_fqdn }}@{{ cluster.config.kerberos_client.realm }}";
};
{%- elif cluster.config.sasl_plain_auth_default_config is not none %}
    {%- set credential = cluster.config.sasl_plain_auth_default_config.sasl_plain_users_data %}
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="kafka-connect"
  password="{{ credential['kafka-connect'] }}";
};
{%- endif %}

Parameter

Description

Default value

JAAS config path

The path to the JAAS configuration file used by the service. By default, the package automatically creates its own file in the standard location. If you specify a different path, you’ll need to place the configuration file in the new location yourself

/etc/kafka-connect/jaas_config.conf

 

Kafka Connect Worker component configuration parameters:

JMX Monitoring
Parameter Description Default value

jmx_port

Port to which JMX metrics are sent

9996

JMX Exporter Port

Port for connecting to Prometheus JMX Exporter

11205

jmxremote_rmi_port

Remote JMX RMI port

10001
JMX Authentication

Enables authentication for JMX in the service (used when access to the JMX port needs to be protected).

Parameter Description Default value

Username

Username for authentication in JMX

monitoring

Password

User password for authentication in JMX

 — 
Found a mistake? Seleсt text and press Ctrl+Enter to report it