Configuration parameters

log.dirs

The directory to store the logs

/kafka-logs

listeners

Comma-separated list of URIs to listen on and the listener names.

If a non-PLAINTEXT protocol is used, then listener.security.protocol.map must also be specified.

To bind to all interfaces, specify the hostname as 0.0.0.0

PLAINTEXT://:9092

default.replication.factor

The default replication factors for automatically created topics

1

num.partitions

The default number of log partitions per topic

1

delete.topic.enable

Enables topics deletion. Topics deletion has no effect it this config is turned off

ON

log.retention.hours

The number of hours to keep a log file before deleting it

168

log.roll.hours

The maximum time before a new log segment is rolled out

168

log.cleanup.policy

Log cleanup policy

delete

log.cleanup.interval.mins

Log cleanup interval

10

log.cleaner.min.compaction.lag.ms

The minimum time a message remains uncompacted in the log. Only applicable for logs that are being compacted (in ms)

0

log.cleaner.delete.retention.ms

The amount of time to retain delete tombstone markers for log compacted topics (in ms)

86400000

xasecure.audit.destination.solr.batch.filespool.dir

The directory for Solr audit spool

/srv/ranger/kafka_plugin/audit_solr_spool

xasecure.audit.destination.solr.urls

Specifies Solr URL. Not setting when using ZooKeeper to connect to Solr

 — 

xasecure.audit.destination.solr.zookeepers

Enables Audit to Solr for the Ranger plugins

 — 

ranger.plugin.kafka.policy.rest.url

URL to Ranger Admin

 — 

ranger.plugin.kafka.service.name

Name of the Ranger Service containing policies for this Kafka instance

 — 

ranger.plugin.kafka.policy.cache.dir

The directory where Ranger policies are cached after successful retrieval from the source

/srv/ranger/kafka/policycache

ranger.plugin.kafka.policy.pollIntervalMs

How often to poll for changes in policies (in ms)

30000

ranger.plugin.kafka.policy.rest.client.connection.timeoutMs

Kafka plugin RangerRestClient connection timeout (in ms)

120000

ranger.plugin.kafka.policy.rest.client.read.timeoutMs

Kafka plugin RangerRestClient read timeout (in ms)

30000

PID_DIR

The directory to store the Kafka process ID

/var/run/kafka

LOG_DIR

The directory for logs

/var/log/kafka

JMX_PORT

Port on which Kafka sends JMX metrics

9999

auto.create.topics.enable

Enables automatic topic creation

OFF

auto.leader.rebalance.enable

Enables automatic leader balancing in the background at regular intervals

ON

queued.max.requests

Number of requests in the queue before blocking network flows

500

num.network.threads

The number of threads used by the server to receive requests from the network and send responses to the network

3

num.io.threads

Sets the number of threads spawned for IO operations

8

unclean.leader.election.enable

Specifies whether to include out-of-ISR replicas and set the last resort as the leader, even if doing so may result in data loss

OFF

offsets.topic.replication.factor

The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation does not occur until the cluster size meets this replication factor requirement

1

transaction.state.log.min.isr

Overrides the min.insync.replicas configuration for a transaction topic

1

transaction.state.log.replication.factor

The replication factor for the transaction topic (set higher to ensure availability). Internal topic creation fails until the cluster size meets this replication factor requirement

1

zookeeper.connection.timeout.ms

The max time that the client waits to establish a connection to ZooKeeper. If not set, the value in zookeeper.session.timeout.ms is used (in ms)

30000

zookeeper.session.timeout.ms

ZooKeeper session timeout (in ms)

30000

zookeeper.sync.time.ms

How far a ZooKeeper follower can be behind a ZooKeeper leader (in ms)

2000

security.inter.broker.protocol

Security protocol used to communicate between brokers

PLAINTEXT

ssl.keystore.location

The location of the keystore file. This is optional for client and can be used for two-way authentication for client

 — 

ssl.keystore.password

The store password for the keystore file. This is optional for client and only needed if ssl.keystore.location is configured

 — 

ssl.key.password

The password of the private key in the keystore file. This is optional for client

 — 

ssl.keystore.type

The file format of the keystore file. This is optional for client

 — 

ssl.truststore.location

The location of the trust store file

 — 

ssl.truststore.password

The store password for the truststore file. This is optional for client and only needed if ssl.truststore.location is configured

 — 

ssl.truststore.type

The file format of the trust store file

 — 

num.partitions

The default number of partitions for each topic. If the required setting is not in the list of the server.properties group, then to add it, use the Add key,value line, where you need to write the key and value in the appropriate fields

 — 

Add key,value

 — 

Kafka-Manager port

Kafka-Manager to listen port. Specified as JAVA_OPTS in kafka-manager-env file

9000

Enable JMX Polling

Enables or disables the polling thread for JMX

ON

JMX Auth Username

Adding a New Username for JMX Authorization

 — 

JMX Auth Password

Adding a New User Password for JMX Authorization

 — 

JMX with SSL

Enables or disables JMX activation by SSL authentication

OFF

Poll consumer information

Poll consumer information

ON

Filter out inactive consumers

Filter out inactive consumers

ON

Enable Logkafka

Enables or disables Logkafka

OFF

Enable Active OffsetCache

Enables Active OffsetCache

ON

Display Broker and Topic Size

Defines whether to display broker and topic size

OFF

brokerViewUpdatePeriodSeconds

Broker View Cycle Update Time/Cycle

30

clusterManagerThreadPoolSize

Cluster control thread pool size

10

clusterManagerThreadPoolQueueSize

Cluster control thread pool queue size

100

kafkaCommandThreadPoolSize

Kafka command thread pool size

10

kafkaCommandThreadPoolQueueSize

Kafka command thread pool queue size

100

logkafkaCommandThreadPoolSize

Logkafka command thread pool size

10

logkafkaCommandThreadPoolQueueSize

Logkafka command thread pool queue size

100

logkafkaUpdatePeriodSeconds

Logkafka update cycle time (in seconds)

30

partitionOffsetCacheTimeoutSecs

Logkafka update cycle time (in seconds)

5

brokerViewThreadPoolSize

Broker view thread pool size

10

brokerViewThreadPoolQueueSize

Broker view thread pool queue size

1000

offsetCacheThreadPoolSize

Cache offset thread pool size

10

offsetCacheThreadPoolQueueSize

Offset cache thread pool queue size

1000

kafkaAdminClientThreadPoolSize

Kafka control client thread pool size

10

kafkaAdminClientThreadPoolQueueSize

Kafka control client thread pool queue size

1000

kafkaManagedOffsetMetadataCheckMillis

Metadata offset check time

30000

kafkaManagedOffsetGroupCacheSize

Offset Group Cache Size

1000000

kafkaManagedOffsetGroupExpireDays

Offset Group Expire

7

Security Protocol

Security Protocol

PLAINTEXT

SASL Mechanism

Enables SASL authentication mechanism

DEFAULT

jaasConfig

Configurations for JAAS Authentication

 — 

rest_listener_port

REST Proxy listener port. Specified as listeners in kafka-rest.properties file

8082

LOG_DIR

The directory to store the logs

/var/log/kafka-rest

JMX_PORT

Port on which Kafka REST Proxy sends JMX metrics

9998

KAFKAREST_HEAP_OPTS

Heap size allocated to the Kafka REST Proxy process

-Xmx1024M

KAFKAREST_JMX_OPTS

JVM options in terms of JMX options (authorization, connection, ssl)

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.authenticate=false

-Dcom.sun.management.jmxremote.ssl=false

id

Unique ID for this REST server instance

kafka-rest-server

consumer.threads

The minimum number of threads to run consumer request on. You must set this value higher than the maximum number of consumers in a single consumer group

50

consumer.request.timeout.ms

The maximum total time to wait for messages for a request in the maximum request size has not yet been reached (in ms)

100

consumer.request.max.bytes

The maximum number of bytes in message keys and values returned by a single request

67108864

fetch.min.bytes

The minimum number of bytes in message keys and values returned by a single request

-1

client.ssl.endpoint.identification.algorithm

The endpoint identification algorithm to validate the server

 — 

Add key, value

 — 

rest_listener_port

KSQL server listener port. Specified as listeners in ksql-server.properties file

8088

LOG_DIR

The directory for storing logs

/var/log/ksql

JMX_PORT

Port on which KSQL-server sends JMX metrics

10099

KSQL_HEAP_OPTS

Heap size allocated to the KSQL-server process

-Xmx3g

KSQL_JVM_PERFORMANCE_OPTS

JVM options in terms of PERFORMANCE options

-server

-XX:+UseConcMarkSweepGC

-XX:+CMSClassUnloadingEnabled

-XX:+CMSScavengeBeforeRemark

-XX:+ExplicitGCInvokesConcurrent

-XX:NewRatio=1 -Djava.awt.headless=true

application.id

Application ID

ksql-server

ksql.internal.topic.replicas

The replication factor for the KSQL Servers internal topics

1

ksql.streams.state.dir

The storage directory for stateful operation

/usr/lib/ksql/state

ksql.streams.replication.factor

Underlying internal topics of Kafka Streams

1

ksql.streams.topic.min.insync.replicas

Minimum number of brokers that must have data written to synchronized replicas

2

ksql.streams.num.standby.replicas

Number of replicas for stateful operations

1

ksql.streams.producer.acks

Number of brokers that need to acknowledge receipt of a message before it is considered a successful write

all

ksql.streams.producer.delivery.timeout.ms

The batch expiry (in ms)

2147483647

ksql.streams.producer.max.block.ms

Maximum allowable time for the producer to block (in ms)

9223372036854775000

ssl.endpoint.identification.algorithm

Endpoint identification algorithm for server validation

 — 

Add key, value

 — 

rest_listener_port

Schema-Registry listener port. Specified as listeners in schema-registry.properties file

8088

LOG_DIR

The directory for storing logs

/var/log/schema-registry

JMX_PORT

Port on which Schema-Registry sends JMX metrics

9997

SCHEMA_REGISTRY_HEAP_OPTS

Heap size allocated to the Schema-Registry process

-Xmx1024M

SCHEMA_REGISTRY_JVM_PERFORMANCE_OPTS

JVM options in terms of PERFORMANCE options

-server

-XX:+UseG1G

-XX:MaxGCPauseMillis=20

-XX:InitiatingHeapOccupancyPercent=35

-XX:+ExplicitGCInvokesConcurrent

-Djava.awt.headless=true

kafkastore.topic

The durable single partition topic that acts as the durable log for the data. This topic must be compacted to avoid losing data due to retention policy

_schemas

debug

Boolean indicating whether extra debugging information is generated in some error response entities

OFF

kafkastore.ssl.endpoint.identification.algorithm

The endpoint identification algorithm to validate the server hostname using the server certificate

 — 

Add key,value

 — 

Nifi UI port

NiFi Server HTTP port. Specified as property nifi.web.http.port in the nifi.properties configuration file

9090

Nifi server Heap size

Heap size for Nifi server. Specified in bootstrap.conf configuration file

1024m

Nifi Registry UI

Nifi Registry HTTP port. Specified as property nifi.registry.web.http.port in nifi.properties configuration file

18080

Nifi Registry Heap size

Heap size for Nifi Registry. Specified in bootstrap.conf configuration file

512m

nifi.queue.backpressure.count

The default value for the number of FlowFile files (underlying NiFi processing object) that can be queued before backpressure is applied, i.e. the source stops sending data. The value must be an integer

10000

nifi.queue.backpressure.size

The default value for the maximum amount of data that must be queued before applying before backpressure is applied. The value must be the size of the data, including the unit of measure

1 GB

nifi.flowfile.repository.directory

FlowFile repository location

/usr/lib/nifi-server/flowfile_repository

nifi.content.repository.directory

Content repository location

/usr/lib/nifi-server/content_repository

nifi.provenance.repository.directory

Provenance repository location

/usr/lib/nifi-server/provenance_repository

nifi.database.directory

H2 database directory location

/usr/lib/nifi-server/database_repository

nifi.registry.db.directory

Location of the Registry database directory

/usr/lib/nifi-registry/database

nifi.nar.library.directory.lib

The parameter should be used in case of adding custom nars

 — 

nifi.analytics.predict.interval

Time interval in which analytic predictions should be made (e.g. queue saturation)

3 мин

nifi.analytics.query.interval

The time interval to query for past observations (for example, the last 3 minutes of snapshots). The value must be at least 3 times greater than the specified value nifi.components.status.snapshot.frequency

5 мин

nifi.analytics.connection.model.implementation

Implementation class for the state analysis model used for connection predictions

Ordinary Least Squares

nifi.analytics.connection.model.score.name

Name of the scoring type to use to score the model

rSquared

nifi.analytics.connection.model.score.threshold

Threshold for the scoring value (the score model must be above the specified threshold)

.90

Flow Persistence Provider Type

Flow provider type

File System

Flow Storage Directory

Flow storage directory

/usr/lib/nifi-registry/flow_storage

Bundle Persistence Provider Type

Bundle provider type

File System

Extension Bundle Storage Directory

Bundle storage directory

/usr/lib/nifi-registry/extension_bundles

Nifi Login Identity Provider Type

Pluggable mechanism for user authentication by username/password

None

Default Realm

The default realm provided when the user enters an incomplete user subject

 — 

Authentication Expiration

Validity period for successful user authentication

12 hours

nifi.kerberos.spnego.principal

The name of the NiFi Kerberos service principal

 — 

nifi.kerberos.spnego.keytab.location

The file path of the NiFi Kerberos keytab

 — 

nifi.kerberos.spnego.authentication.expiration

The expiration duration of a successful Kerberos user authentication, if used

12 hours

Nifi-Registry Identity Provider Type

Pluggable mechanism to authenticate users by their username/password

None

Default Realm

The default realm provided when the user enters an incomplete user subject

 — 

Authentication Expiration

Validity period for successful user authentication

12 hours

Enable Debug

Enables debugging

OFF

NIFI_HOME

The directory for NiFi installation

/usr/lib/nifi-server

NIFI_PID_DIR

The directory to store the NiFi process ID

/var/run/nifi

NIFI_LOG_DIR

The directory to store the logs

/var/log/nifi

NIFI_ALLOW_EXPLICIT_KEYTAB

Defines whether to prevent of the old free-form keytab properties that were left around for backwards compatibility

true

nifi.flow.configuration.file

The location of the XML-based flow configuration file

/etc/nifi/conf/flow.xml.gz

nifi.flow.configuration.archive.enabled

Enables NiFi to create a fallback schema for automatic stream updates

ON

nifi.cluster.node.connection.timeout

When connecting to another node in the cluster, specifies how long this node should wait before considering the connection a failure

5 sec

nifi.cluster.node.read.timeout

When communicating with another node in the cluster, specifies how long this node should wait to receive information from the remote node before considering the communication with the node a failure

5 sec

nifi.zookeeper.connect.timeout

How long to wait when connecting to ZooKeeper before considering the connection a failure

3 sec

nifi.zookeeper.session.timeout

How long to wait after losing a connection to ZooKeeper before the session is expired

3 sec

nifi.variable.registry.properties

Comma-separated list of file location paths for one or more custom property files

/etc/nifi/conf/extra-args.properties

app_file_max_history

Maximum number of files for applications

10

user_file_max_history

Maximum user files

10

boot_file_max_history

Maximum number of files for Boot

5

root_level

Event level

INFO

app_loggers

org.apache.nifi

INFO

org.apache.nifi.processors

WARN

org.apache.nifi.processors.standard.LogAttribute

INFO

org.apache.nifi.processors.standard.LogMessage

INFO

org.apache.nifi.controller.repository.StandardProcessSession

WARN

org.wali

WARN

org.apache.nifi.cluster

INFO

org.apache.nifi.server.JettyServer

INFO

org.eclipse.jetty

INFO

user_events_loggers

org.apache.nifi.web.security

INFO

org.apache.nifi.web.api.config

INFO

org.apache.nifi.authorization

INFO

org.apache.nifi.cluster.authorization

INFO

org.apache.nifi.web.filter.RequestLogger

INFO

bootstrap_loggers

org.apache.nifi.bootstrap

INFO

org.apache.nifi.bootstrap.Command

INFO

org.apache.nifi.StdOut

INFO

org.apache.nifi.StdErr

INFO

custom_logger

 — 

 — 

NIFI_REGISTRY_HOME

The directory for installing

/usr/lib/nifi-registry

NIFI_REGISTRY_PID_DIR

The directory to store the NiFi-Registry

/var/run/nifi-registry

NIFI_REGISTRY_LOG_DIR

The directory to store the logs

/var/log/nifi-registry

app_file_max_history

Maximum number of files for applications

10

events_file_max_history

Maximum number of files for events

5

boot_file_max_history

Maximum number of files for Boot

5

root_level

Event level

INFO

app_loggers

org.apache.nifi.registry

INFO

org.hibernate.SQL

WARN

org.hibernate.type

INFO

events_loggers

org.apache.nifi.registry.provider.hook.LoggingEventHookProvider

INFO

bootstrap_loggers

org.apache.nifi.registry.bootstrap

INFO

org.apache.nifi.registry.bootstrap.Command

INFO

org.apache.nifi.registry.StdOut

INFO

org.apache.nifi.registry.StdErr

ERROR

MiNiFi C2 Server port

MiNiFi Server HTTP port

10080

nifi.minifi.notifier.ingestors.pull.http.query

Query string to pull configuration

minifi

MINIFI_HOME

The directory for installing MiNiFi

/usr/lib/minifi

MINIFI_PID_DIR

The directory to store the MiNiFi process ID

/var/run/minifi

MINIFI_LOG_DIR

The directory to store the logs

/var/log/minifi

MiNiFi Agent Heap size

Agent heap size

256m

nifi.minifi.notifier.ingestors.pull.http.period.ms

Update check period (in ms)

300000

nifi.minifi.status.reporter.log.query

nifi.minifi.status.reporter.log.level

Log level at which the status is logged

INFO

nifi.minifi.status.reporter.log.period

Delay between each request (in ms)

60000

app_file_max_history

Maximum number of files for applications

10

boot_file_max_history

Maximum number of files for Boot

5

status_file_max_history

Maximum number of files for statuses

5

root_level

Event Level

INFO

app_loggers

org.apache.nifi

INFO

org.apache.nifi.processors

WARN

org.apache.nifi.processors.standard.LogAttribute

INFO

org.apache.nifi.controller.repository.StandardProcessSession

WARN

bootstrap_loggers

org.apache.nifi.bootstrap

INFO

org.apache.nifi.bootstrap.Command

INFO

org.apache.nifi.StdOut

INFO

org.apache.nifi.StdErr

ERROR

status_loggers

org.apache.nifi.minifi.bootstrap.status.reporters.StatusLogger

INFO

log_file_max_history

Maximum number of files for applications

10

root_level

Event Level

INFO

log_file_loggers

org.apache.nifi.minifi.c2

DEBUG

The ZooKeeper connection string that is used by other services or clusters. It is generated automatically

 — 

The location where ZooKeeper stores the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database

/var/lib/zookeeper

The port that Client connections, i.e. the port that Clients attempt to connect to

2181

Base unit of time used by ZooKeeper. It is used for heart contractions. The minimum session timeout becomes twice the tickTime (in ms)

2000

The timeouts, ZooKeeper uses to limit the length of the time that the ZooKeeper Servers in Quorum have to connect to the Leader

5

How far out of date each Server can be from the Leader

2

This property limits the number of active connections from the host, specified by IP address, to a single ZooKeeper Server

0

Enables storing the most recent snapshots and related transaction logs in dataDir and dataLogDir respectively, and deleting the rest. The minimum value is 3

3

The time interval in hours between runs of the purge task. Set to a positive integer (1 and above) to enable the auto purging

24

Add key,value

 — 

ZOO_LOG_DIR

The directory to store the logs

/var/log/zookeeper

ZOOPIDFILE

The directory to store the ZooKeeper process ID

/var/run/zookeeper/zookeeper_server.pid

SERVER_JVMFLAGS

It is used for setting different JVM parameters connected, for example, with garbage collecting

-Xmx1024m

JAVA

A path to Java

$JAVA_HOME/bin/java

ZOO_LOG4J_PROP

It is used for setting the LOG4J logging level and telling what log appenders to turn on. The effect of turning on the log appender CONSOLE is that logs now go to stdout. The effect of turning on ROLLINGFILE is that zookeeper.log file is created, rotated, and expired

INFO, CONSOLE, ROLLINGFILE

tasks.max

The maximum number of tasks that should be created for this connector

10

offset-syncs.topic.replication.factor

Replication factor for internal offset-syncs.topic

1

checkpoint.topic.replication.factor

Replication factor used for internal checkpoint.topic

1

sync.topic.acls.enabled

Enable monitoring of the source cluster for ACL changes

OFF

KAFKA_HEAP_OPTS

Heap size allocated to the KAFKA server process

-Xms256M -Xmx2G

LOG_DIR

The directory for logs

/var/log/kafka

KAFKA_LOG4J_OPTS

Environment variable for LOG4J logging configuration

-Dlog4j.configuration=file:/etc/kafka/conf/connect-distributed-log4j.properties

group.id

A unique string that identifies a Kafka Connect group, to which this connector belongs

mm-connect

rest.port

Port for Kafka Connect REST API to work

8083

plugin.path

Path to Kafka Connect plugin

 — 

config.storage.replication.factor

Replication factor for config.storage.topic

1

offset.storage.replication.factor

Replication factor for offset.storage.topic

1

status.storage.replication.factor

Replication factor for status.storage.topic

1

offset.flush.interval.ms

Interval at which to try committing offsets for tasks

1000

key.converter

Converter class for key Connect data

org.apache.kafka.connect.converters.ByteArrayConverter

value.converter

Converter class for value Connect data

org.apache.kafka.connect.converters.ByteArrayConverter

connector.client.config.override.policy

Class name or alias of implementation of ConnectorClientConfigOverridePolicy

None

Add key,value

 — 

rootLogger

logging level

INFO

MaxBackupIndex

Maximum number of saved files

10

MaxFileSize

Maximum file size

100MB

loggers

org.apache.zookeeper

ERROR

org.I0Itec.zkclient

ERROR

org.reflections

ERROR