Airflow configuration parameters

Konstantin Alpashkin

To configure the service, use the following configuration parameters in ADCM.

NOTE

  • Some of the parameters become visible in the ADCM UI after the Advanced flag has been set.

  • The parameters that are set in the Custom group will overwrite the existing parameters even if they are read-only.
Redis configuration
Parameter Description Default value

redis.conf

Redis configuration file

 — 

sentinel.conf

Sentinel configuration file

 — 

redis_port

Redis broker listen port

6379

sentinel_port

Sentinel port

26379
airflow.cfg
Parameter Description Default value

db_user

The user to connect to Metadata DB

airflow

db_password

The password to connect to Metadata DB

 — 

db_port

The port to connect to Metadata DB

3307

admin_password

The password for the web server’s admin user

 — 

server_port

The port to run the web server

8080

flower_port

The port that Celery Flower runs on

5555

worker_port

When you start an Airflow Worker, Airflow starts a tiny web server subprocess to serve the Workers local log files to the Airflow main web server, which then builds pages and sends them to users. This defines the port, on which the logs are served. The port must be free and accessible from the main web server to connect to the Workers

8793

fernet_key

The secret key to save connection passwords in the database

 — 

security

Defines which security module to use. For example, kerberos

 — 

keytab

Path to the keytab file

 — 

reinit_frequency

Sets the ticket renewal frequency

3600

principal

The Kerberos principal

 — 

ssl_active

Defines if SSL is active for Airflow

false

web_server_ssl_cert

Path to SSL certificate

/etc/ssl/certs/host_cert.cert

web_server_ssl_key

Path to SSL certificate key

/etc/ssl/host_cert.key

Logging level

Specifies the logging level for Airflow activity

INFO

Logging level for Flask-appbuilder UI

Specifies the logging level for Flask-appbuilder UI

WARNING

cfg_properties_template

The Jinja template to initialize environment variables for Airflow

cfg_properties_template
External database
Parameter Description Default value

Database type

The external database type. Possible values: PostgreSQL, MySQL/MariaDB

MySQL/MariaDB

Hostname

The external database host

 — 

Custom port

The external database port

 — 

Airflow database name

The external database name

airflow
External Broker
Parameter Description Default value

Broker URL

The URL of an external broker

 — 
LDAP Security manager
Parameter Description Default value

AUTH_LDAP_SERVER

The LDAP server URI

 — 

AUTH_LDAP_BIND_USER

Path of the LDAP proxy user to bind on to the top level. Example: cn=airflow,ou=users,dc=example,dc=com

 — 

AUTH_LDAP_BIND_PASSWORD

The password of the bind user

 — 

AUTH_LDAP_SEARCH

Update with the LDAP path under which you’d like the users to have access to Airflow. Example: dc=example, dc=com

 — 

AUTH_LDAP_UID_FIELD

The UID (unique identifier) field in LDAP

 — 

AUTH_ROLES_MAPPING

The parameter for mapping the internal roles to the LDAP Active Directory groups

 — 

AUTH_LDAP_GROUP_FIELD

The LDAP user attribute which has their role DNs

 — 

AUTH_ROLES_SYNC_AT_LOGIN

A flag that indicates if all the user’s roles should be replaced on each login, or only on registration

true

PERMANENT_SESSION_LIFETIME

Sets an inactivity timeout after which users have to re-authenticate (to keep roles in sync)

1800

AUTH_LDAP_USE_TLS

Boolean whether TLS is being used

false

AUTH_LDAP_ALLOW_SELF_SIGNED

Boolean to allow self-signed certificates

true

AUTH_LDAP_TLS_CACERTFILE

Location of the certificate

 — 
Airflow2 Broker component
Parameter Description Default value

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Airflow2 Flower component
Parameter Description Default value

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Airflow2 Scheduler component
Parameter Description Default value

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Airflow2 Server component
Parameter Description Default value

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Airflow2 Worker component
Parameter Description Default value

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Found a mistake? Seleсt text and press Ctrl+Enter to report it