Flink configuration parameters

Sources

A list of sources that will be written into flink-env.sh

/usr/lib/bigtop-utils/bigtop-detect-javahome

HADOOP_CLASSPATH

A list of files/directories to be added to the classpath

$(hadoop classpath)

HADOOP_HOME

Hadoop home directory

/usr/lib/hadoop

HADOOP_CONF_DIR

Directory that stores Hadoop configurations

/etc/hadoop/conf

FLINK_LOG_DIR

Flink server log directory

/var/log/flink

FLINK_CONF_DIR

Directory that stores Flink configurations

/etc/flink/conf

FLINK_HOME

Flink home directory

/usr/lib/flink

$HADOOP_OPTS

Additional Flink startup parameters

$HADOOP_OPTS -Dsecurity.kerberos.token.provider.hadoopfs.renewer=yarn

historyserver.archive.fs.dir

A comma-separated list of directories with job archives scanned by Flink History Server

hdfs:///apps/flink/completed-jobs

jobmanager.archive.fs.dir

Directory where JobManager stores completed job archives

hdfs:///apps/flink/completed-jobs

archive.fs.refresh-interval

Polling interval in milliseconds for scanning archive directories

10000

historyserver.web.port

Port number of the Flink History Server web UI

8082

historyserver.web.ssl.enabled

Enables/disables SSL for the History Server web UI

false

rest.port

Port that the client connects to. This option is respected only if the high-availability configuration is NONE

8081

jobmanager.rpc.port

RPC port through which the JobManager is reachable. In the high availability mode, this value is ignored and the port number to connect to JobManager is generated by ZooKeeper

6123

sql-gateway.endpoint.rest.port

A port to connect to the SQL Gateway service

8083

taskmanager.network.bind-policy

Automatic address binding policy used by the TaskManager

name

parallelism.default

System-wide default parallelism level for all execution environments

1

taskmanager.numberOfTaskSlots

Number of task slots that each TaskManager offers. Each slot runs one parallel pipeline

1

taskmanager.cpu.cores

Number of CPU cores used by TaskManager. By default, the value is set to the number of slots per TaskManager

1

taskmanager.memory.flink.size

The total memory size for the TaskExecutors

 — 

taskmanager.memory.process.size

The total process memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, including the total Flink memory, JVM Metaspace, and JVM Overhead. In containerized setups, this parameter should be equal to the container memory

2048m

jobmanager.memory.flink.size

The total memory size for the JobManager

 — 

jobmanager.memory.process.size

The total process memory size for the JobManager. This includes all the memory that a JobManager JVM consumes, including the total Flink memory, JVM Metaspace, and JVM Overhead. In containerized setups, this parameter should be equal to the container memory

2048m

taskmanager.heap.size

The Java heap size for the TaskManager JVM

1024m

jobmanager.memory.heap.size

The heap size for the JobManager JVM

 — 

flink.yarn.appmaster.vcores

Number of virtual cores (vcores) used by YARN application master

1

taskmanager.host

The external address of the network interface where TaskManager runs

 — 

taskmanager.memory.task.heap.size

The size of the JVM heap memory reserved for tasks

256m

taskmanager.memory.task.off-heap.size

The size of the off-heap memory reserved for tasks

256m

taskmanager.memory.managed.size

The size of the managed memory for TaskExecutors. This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results, and the RocksDB state backend

256m

taskmanager.memory.framework.heap.size

The size of the JVM heap memory reserved for TaskExecutor framework that will not be allocated to task slots

256m

taskmanager.memory.framework.off-heap.size

The size of the off-heap memory reserved for TaskExecutor framework that will not be allocated to task slots

256m

taskmanager.memory.network.min

Minimum network memory size for TaskExecutors. The network memory is the off-heap memory reserved for ShuffleEnvironment (e.g. network buffers)

256m

taskmanager.memory.network.max

Maximum network memory size for TaskExecutors. The network memory is the off-heap memory reserved for ShuffleEnvironment (e.g. network buffers)

256m

taskmanager.memory.jvm-overhead.max

Maximum JVM overhead size for the TaskExecutors. This is the off-heap memory reserved for JVM overhead, such as thread stack space, compile cache, etc.

256m

taskmanager.memory.jvm-metaspace.size

The JVM metaspace size for TaskExecutors

256m

yarn.provided.lib.dirs

A semicolon-separated list of directories with provided libraries. Flink uses these libraries to exclude the local Flink JARs uploading to accelerate the job submission process

hdfs:///apps/flink/

flink.yarn.resourcemanager.scheduler.address

The address of the scheduler interface

 — 

flink.yarn.containers.vcores

Sets the number of vcores for Flink YARN containers

1

flink.yarn.application.classpath

high-availability.cluster-id

The ID of the Flink cluster used to separate multiple Flink clusters from each other

default

high-availability.storageDir

A file system path (URI) where Flink persists metadata in the HA mode

 — 

high-availability

Defines the High Availability (HA) mode used for cluster execution

ZOOKEEPER

high-availability.zookeeper.quorum

The ZooKeeper quorum to use when running Flink in the HA mode with ZooKeeper

 — 

high-availability.zookeeper.path.root

The root path for Flink ZNode in Zookeeper

/flink

sql-gateway.session.check-interval

The check interval to detect idle sessions. A value <= 0 disables the checks

1 min

sql-gateway.session.idle-timeout

Timeout to close a session if no successful connection was made during this interval. A value <= 0 never closes the sessions

10 min

sql-gateway.session.max-num

Maximum number of sessions to run simultaneously

1000000

sql-gateway.worker.keepalive-time

Time to keep an idle worker thread alive. When the worker thread count exceeds sql-gateway.worker.threads.min, excessive threads are killed after this time interval

5 min

sql-gateway.worker.threads.max

Maximum number of worker threads on the SQL Gateway server

500

sql-gateway.worker.threads.min

Minimum number of worker threads. If the current number of worker threads is less than this value, the worker threads are not deleted automatically

5

env.java.opts.all

Java options used to start all Flink JVMs

-Djava.io.tmpdir={{ cluster.config.java_tmpdir | d('/tmp') }}

security.kerberos.login.use-ticket-cache

Indicates whether to read from the Kerberos ticket cache

false

security.delegation.token.provider.HiveServer2.enabled

Defines whether to obtain credentials from HiveServer2 when security is enabled

false

security.kerberos.login.keytab

The absolute path to the Kerberos keytab file that stores user credentials

 — 

security.kerberos.login.principal

Flink Kerberos principal

 — 

security.delegation.tokens.hive.renewer

Flink Kerberos principal for Hive

 — 

security.kerberos.login.contexts

A comma-separated list of login contexts to provide the Kerberos credentials to

 — 

security.ssl.internal.enabled

Enables SSL for internal communication channels between Flink components. This includes the communication between TaskManagers, transporting of blobs from JobManager to TaskManager, RPC-connections, etc.

false

security.ssl.internal.keystore

Path to the keystore file to be used by Flink’s internal endpoints

 — 

security.ssl.internal.truststore

Path to the truststore file used by internal Flink’s endpoints

 — 

security.ssl.internal.keystore-password

The password to the keystore file used by internal Flink’s endpoints

 — 

security.ssl.internal.truststore-password

The password to the truststore file used by internal Flink’s endpoints

 — 

security.ssl.internal.key-password

The password to decrypt the key in the keystore

 — 

security.ssl.rest.enabled

Turns on SSL for external communication via REST endpoints

false

security.ssl.rest.keystore

The Java keystore file with SSL keys and certificates to be used by Flink’s external REST endpoints

 — 

security.ssl.rest.truststore

The truststore file containing public CA certificates to verify the peer for Flink’s external REST endpoints

 — 

security.ssl.rest.keystore-password

The secret to decrypt the keystore file for Flink external REST endpoints

 — 

security.ssl.rest.truststore-password

The password to decrypt the truststore for Flink’s external REST endpoints

 — 

security.ssl.rest.key-password

The secret to decrypt the key in the keystore for Flink’s external REST endpoints

 — 

security.ssl.protocol

The TLS protocol version to be used for SSL. Accepts a single value, not a list

TLSv1.2

zookeeper.sasl.disable

Defines the SASL authentication in Zookeeper

false

Logging level

Defines the logging level for Flink activity

INFO

yarn.classpath.include-user-jar

ORDER

Custom flink_env.sh

 — 

Custom flink-conf.yaml

 — 

log4j.properties

The contents of the log4j.properties configuration file

log4j-cli.properties

The contents of the log4j-cli.properties configuration file