Airflow configuration parameters

Manage sensitive configuration data

When enabled, ADO takes over the creation of secrets (transferring them from configurations to Vault) as well as updating them. Requires the right to create secrets. Affects the Rotate fernet key action (see fernet key rotation)

true

Secrets backend

A secret backend to use

airflow.providers.hashicorp.secrets.vault.VaultBackend

url

Base URL for a Vault instance being addressed. Has to include protocol and port (e.g. http://127.0.0.1:8200)

 — 

auth_type

Authentication type for Vault. Possible values: approle, github, kubernetes, ldap, token, userpass

token

mount_point

The path the secret engine was mounted on. Note that this mount_point is not used for authentication if authentication is done via a different engine. For authentication mount points, see auth_mount_point

secret

config_path

Specifies the path of the Airflow configuration secret to read. If set to None (null), requests for configurations will not be sent to Vault

config

connections_path

Specifies the path of the secret to read to get connections. If set to None (null), requests for connections will not be sent to Vault

connections

variables_path

Specifies the path of the secret to read to get variables. If set to None (null), requests for variables will not be sent to Vault

variables

auth_mount_point

Defines a mount point for a chosen authentication type. The default value depends on the authentication method used

 — 

kv_engine_version

The engine version to run

2

token

Authentication token to include in requests sent to Vault (for the token and github authentication methods)

 — 

token_path

Path to the file containing authentication token to include in requests sent to Vault (for the token and github authentication methods)

 — 

username

Username for the ldap and userpass authentication methods

 — 

password

Password for the ldap and userpass authentication methods

 — 

secret_id

Secret ID for the approle authentication method

 — 

role_id

Role ID for the approle authentication method

 — 

admin_password

The password of the webserver’s admin user

 — 

db_user

The name of the metadata DB user

airflow

db_password

The password of the metadata DB user

 — 

Database type

The external database type. Possible values: PostgreSQL, MySQL/MariaDB

PostgreSQL

Database connection string

Parameters for connecting to the database

Airflow database name

The external database name

airflow

dags_folder

The absolute path to the Airflow pipelines directory

/opt/airflow/dags

hostname_callable

A path to a callable, which will resolve the hostname. The format is package.function. The default value (airflow.utils.net.getfqdn) means that result from patched version of socket.getfqdn(). No argument should be required in the function specified. If using IP address as hostname is preferred, use value airflow.utils.net.get_host_ip_address

airflow.utils.net.getfqdn

might_contain_dag_callable

A callable to check if a Python file has Airflow DAGs defined or not with argument as: (file_path: str, zip_file: zipfile.ZipFile | None = None). Returns True if it has DAGs, otherwise False. If this is not provided, Airflow uses its own heuristic rules

airflow.utils.file.might_contain_dag_via_default_heuristic

default_timezone

Default timezone. Can be UTC (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)

utc

executor

The executor class that Airflow should use. Choices include SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor, KubernetesExecutor, CeleryKubernetesExecutor, or a full import path to the class if a custom executor is used

CeleryExecutor

auth_manager

The auth manager class that Airflow should use. Full import path to the auth manager class

airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager

execution_api_server_url

Task Execution API URL. If empty, the bundle sets it from the Airflow API base URL. If HAProxy is used, the URL should be the HAProxy URL

http://<FQDN>:8080/execution/

parallelism

This defines the maximum number of task instances that can run concurrently per scheduler in Airflow, regardless of the worker count. Generally this value, multiplied by the number of schedulers in your cluster, is the maximum number of task instances with the running state in the metadata database

32

max_active_tasks_per_dag

The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as max_active_tasks_per_dag. An example scenario when this would be useful is when you want to stop a new dag with an early start date from stealing all the executor slots in a cluster

16

dags_are_paused_at_creation

The flag that indicates if DAGs are paused by default at creation

true

max_active_runs_per_dag

The maximum number of active DAG runs per DAG. The scheduler will not create more DAG runs if it reaches the limit. This is configurable at the DAG level with max_active_runs, which is defaulted as max_active_runs_per_dag

16

mp_start_method

The name of the method used in order to start Python processes via the multiprocessing module. This corresponds directly with the options available in the Python docs. Must be one of the values returned by multiprocessing

 — 

load_examples

Whether to load the DAG examples that ship with Airflow

true

plugins_folder

Path to the folder containing Airflow plugins

/usr/lib/airflow/plugins

execute_tasks_new_python_interpreter

Should tasks be executed via forking of the parent process (False, the speedier option) or by spawning a new python process (True slow, but means plugin changes picked up by tasks straight away)

false

fernet_key

The secret key to save connection passwords in the database

 — 

donot_pickle

Whether to disable pickling DAGs

true

dagbag_import_timeout

How long before timing out a Python file import

30

dagbag_import_error_tracebacks

Should a traceback be shown in the UI for dagbag import errors instead of just the exception message

true

dagbag_import_error_traceback_depth

If tracebacks are shown, how many entries from the traceback should be shown

2

task_runner

The class to use for running task instances in a subprocess. Choices include StandardTaskRunner, CgroupTaskRunner or the full import path to the class when using a custom task runner

StandardTaskRunner

default_impersonation

If set, tasks without a run_as_user argument will be run with this user. Can be used to de-elevate a sudo user running Airflow when executing tasks

 — 

security

Defines which security module to use. For example, kerberos

 — 

unit_test_mode

Turn unit test mode on (overwrites many configuration options with test values at runtime)

false

enable_xcom_pickling

Whether to enable pickling for xcom (note that this is insecure and allows for RCE exploits)

false

allowed_deserialization_classes

What classes can be imported during deserialization. This is a multi line value. The individual items will be parsed as regexp. Python built-in classes (like dict) are always allowed. Bare . will be replaced so you can set airflow.*

airflow.*

killed_task_cleanup_time

When a task is killed forcefully, this is the amount of time in seconds that it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED

60

dag_run_conf_overrides_params

Whether to override params with dag_run.conf. If you pass some key-value pairs through airflow dags backfill -c or airflow dags trigger -c, the key/value pairs will override the existing ones in params

true

dag_discovery_safe_mode

If enabled, Airflow will only scan files containing both DAG and airflow (case-insensitive)

true

dag_ignore_file_syntax

The pattern syntax used in the .airflowignore files in the DAG directories. Valid values are regexp or glob

regexp

default_task_retries

The number of retries each task is going to have by default. Can be overridden at DAG or task level

0

default_task_retry_delay

The number of seconds each task is going to wait by default between retries. Can be overridden at dag or task level

300

max_task_retry_delay

The maximum delay (in seconds) each task is going to wait by default between retries. This is a global setting and cannot be overridden at task or DAG level

86400

default_task_weight_rule

The weighting method used for the effective total priority weight of the task

downstream

default_task_execution_timeout

The default task execution_timeout value for the operators. Expected an integer value to be passed into timedelta as seconds. If not specified, then the value is considered as None, meaning that the operators are never timed out by default

 — 

min_serialized_dag_update_interval

Updating serialized DAG cannot be faster than a minimum interval to reduce database write rate

30

compress_serialized_dags

If True, serialized DAGs are compressed before writing to DB. This will disable the DAG dependencies view

false

min_serialized_dag_fetch_interval

Fetching serialized DAG cannot be faster than a minimum interval to reduce database read rate. This config controls when your DAGs are updated in the Webserver

10

max_num_rendered_ti_fields_per_task

Maximum number of rendered task instance fields (template fields) per task to store in the database. All the template_fields for each of task instance are stored in the database. Keeping this number small may cause an error when you try to view Rendered tab in TaskInstance view for older tasks

30

check_slas

On each dagrun check against defined SLAs

true

xcom_backend

Path to custom XCom class that will be used to store and resolve operators results

airflow.models.xcom.BaseXCom

lazy_load_plugins

By default, Airflow plugins are lazily-loaded (only loaded when required). Set it to False if you want to load plugins whenever airflow is invoked via CLI or loaded from module

true

lazy_discover_providers

By default, Airflow providers are lazily-discovered (discovery and imports happen only when required). Set it to False if you want to discover providers whenever airflow is invoked via CLI or loaded from a module

true

hide_sensitive_var_conn_fields

Hide sensitive variables or extra JSON connection keys from UI and task logs when set to True (connection passwords are always hidden in logs)

true

sensitive_var_conn_names

A comma-separated list of extra sensitive keywords to look for in variables names or connection’s extra JSON

 — 

default_pool_task_slot_count

Task slot counts for default_pool. This setting would not have any effect in an existing deployment where the default_pool is already created. For existing deployments, users can change the number of slots using webserver, API, or the CLI

128

max_map_length

The maximum list/dict length an XCom can push to trigger task mapping. If the pushed list/dict has a length exceeding this value, the task pushing the XCom will be failed automatically to prevent the mapped tasks from clogging the scheduler

1024

daemon_umask

The default umask to use for process when run in daemon mode (scheduler, worker, etc.) This controls the file-creation mode mask which determines the initial value of file permission bits for newly created files. This value is treated as an octal-integer

0o077

dataset_manager_class

Class to use as dataset manager

 — 

dataset_manager_kwargs

Kwargs to supply to dataset manager

 — 

database_access_isolation

Experimental feature. The flag that indicates whether components should use Airflow Internal API for DB connectivity

false

internal_api_url

Experimental feature. Airflow Internal API URL. Only used if the database_access_isolation core setting is True

 — 

max_consecutive_failed_dag_runs_per_dag

Maximum number of consecutive failed DAG runs allowed for a DAG before it is considered failed

0

fernet_key_secret

ADO secret reference used as the source for the Fernet key value

fernet_key_value

allowed_deserialization_classes_regexp

Regular expression defining additional classes allowed during deserialization

 — 

task_success_overtime

Maximum additional time, in seconds, available for auxiliary processes after a task is marked successful

20

strict_dataset_uri_validation

Enables strict validation of dataset URIs

false

internal_api_secret_key

Secret key used to authenticate requests to the Airflow internal API

 — 

test_connection

Controls whether connection testing is available through the Airflow UI, API, and CLI

Disabled

max_templated_field_length

Maximum length of a templated field that Airflow will render/store

4096

sql_alchemy_conn

The SQLAlchemy connection string to the metadata database. The value of the parameter is automatically populated based on the input values in the Database settings section. It is not displayed in the UI for security reasons. SQLAlchemy supports many different database engines

 — 

sql_alchemy_engine_args

Extra engine specific keyword args passed to SQLAlchemy’s create_engine, as a JSON-encoded value

 — 

sql_engine_encoding

The encoding for the databases

utf-8

sql_engine_collation_for_ids

Collation for dag_id, task_id, key, external_executor_id columns in case they have different encoding. By default, this collation is the same as the database collation, however for mysql and mariadb the default is utf8mb3_bin so that the index sizes of index keys will not exceed the maximum size of allowed index when collation is set to utf8mb4 variant

 — 

sql_alchemy_pool_enabled

If SQLAlchemy should pool database connections

true

sql_alchemy_pool_size

The SQLAlchemy pool size is the maximum number of database connections in the pool. 0 indicates no limit

5

sql_alchemy_max_overflow

The maximum overflow size of the pool. When the number of checked-out connections reaches the size set in pool_size, additional connections will be returned up to this limit. When those additional connections are returned to the pool, they are disconnected and discarded. The total number of simultaneous connections the pool will allow is pool_size + max_overflow, and the total number of sleeping connections the pool will allow is pool_size. max_overflow can be set to -1 to indicate no overflow limit; no limit will be placed on the total number of concurrent connections. Defaults to 10

10

sql_alchemy_pool_recycle

The SQLAlchemy pool recycle is the number of seconds a connection can be idle in the pool before it is invalidated. This config does not apply to Sqlite. If the number of DB connections is ever exceeded, a lower config value will allow the system to recover faster

1800

sql_alchemy_pool_pre_ping

Check connection at the start of each connection pool checkout

true

sql_alchemy_schema

The schema to use for the metadata database. SQLAlchemy supports databases with the concept of multiple schemas

 — 

sql_alchemy_connect_args

Import path for connection arguments in SQLAlchemy. Defaults to an empty dictionary. This is useful when you want to configure DB engine arguments that SQLAlchemy won’t parse in connection string

 — 

load_default_connections

Whether to load the default connections that ship with Airflow

true

max_db_retries

Number of times the code should be retried in case of DB operational errors. Not all transactions will be retried as it can cause undesired state. Currently, it is only used in DagFileProcessor.process_file to retry dagbag.sync_to_db

3

check_migrations

Whether to run alembic migrations during Airflow start up. Sometimes this operation can be expensive, and the users can assert the correct version through other means (e.g. through a Helm chart). Accepts True or False

true

alembic_ini_file_path

Path to the alembic.ini file

alembic.ini

sql_alchemy_conn_secret

ADO secret reference used as the source for the SQLAlchemy connection value

sql_alchemy_conn_value

sql_alchemy_session_maker

Import path for a custom SQLAlchemy session maker

 — 

base_log_folder

The absolute path to the Airflow log files directory. There are a few existing configurations that assume this is set to the default. If you choose to override this, you may need to update the dag_processor_manager_log_location and dag_processor_manager_log_location settings as well

/var/log/airflow

remote_logging

Airflow can store logs remotely in AWS S3, Google Cloud Storage, or Elastic Search. Set this to True if you want to enable remote logging

false

remote_log_conn_id

Users must supply an Airflow connection ID that provides access to the storage location. Depending on your remote logging service, this may only be used for reading logs, not writing them

 — 

delete_local_logs

Whether the local log files for GCS, S3, WASB, and OSS remote logging should be deleted after they are uploaded to the remote location

false

google_key_path

Path to Google Credential JSON file. If omitted, authorization based on the Application Default Credentials will be used

 — 

remote_base_log_folder

Storage bucket URL for remote logging. S3 buckets should start with s3://, Cloudwatch log groups should start with cloudwatch://, GCS buckets should start with gs://, WASB buckets should start with wasb just to help Airflow select correct handler, Stackdriver logs should start with stackdriver://

 — 

remote_task_handler_kwargs

The remote_task_handler_kwargs param is loaded into a dictionary and passed to __init__ of remote task handler and it overrides the values provided by Airflow config. For example, if you set delete_local_logs=False and you provide {{"delete_local_copy": true}}, then the local log files will be deleted after they are uploaded to remote location

 — 

encrypt_s3_logs

Use server-side encryption for logs stored in S3

false

logging_level

Logging level. Supported values: CRITICAL, ERROR, WARNING, INFO, DEBUG

INFO

celery_logging_level

Logging level for celery

WARNING

fab_logging_level

Logging level for Flask-appbuilder UI. Supported values: CRITICAL, ERROR, WARNING, INFO, DEBUG

WARNING

logging_config_class

The name of the class that specifies the logging configuration. This class has to be on the Python classpath

 — 

colored_console_log

Flag to enable/disable colored logs

true

colored_log_format

The log format for colored logs if they are enabled. The value must be taken in a tag raw/endraw

{% raw %}[%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s{% endraw %}

colored_formatter_class

Specifies the class utilized by Airflow to implement colored logging

airflow.utils.log.colored_log.CustomTTYColoredFormatter

log_format

Format of log line. The value must be taken in a tag raw/endraw

{% raw %}[%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s{% endraw %}

simple_log_format

Defines the format of log messages for simple logging configuration

%%(asctime)s %%(levelname)s - %%(message)s

dag_processor_log_target

Where to store DAG parser logs. If set to file, logs are sent to log files defined in child_process_log_directory

file

dag_processor_log_format

DAG processor log line format. The value must be taken in a tag raw/endraw

{% raw %}[%%(asctime)s] [SOURCE:DAG_PROCESSOR]{{%%(filename)s:%%(lineno)d}} %%(levelname)s - %%(message)s{% endraw %}

log_formatter_class

Determines the formatter class used by Airflow for structuring its log messages. The default formatter class is timezone-aware, which means that timestamps attached to log entries will be adjusted to reflect the local timezone of the Airflow instance

airflow.utils.log.timezone_aware.TimezoneAware

secret_mask_adapter

An import path to a function to add adaptations of each secret added with airflow.utils.log.secrets_masker.mask_secret to be masked in log messages. The given function is expected to require a single parameter: the secret to be adapted. It may return a single adaptation of the secret or an iterable of adaptations to each be masked as secrets. The original secret will be masked as well as any adaptations returned

 — 

task_log_prefix_template

Prefix pattern specified with stream handler TaskHandlerWithCustomFormatter

 — 

log_filename_template

The format of generated Airflow file and path names for each task run. The value must be taken in a tag raw/endraw

{% raw %}dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number }}.log{% endraw %}

log_processor_filename_template

The format of generated Airflow file and path names for logs. The value must be taken in a tag raw/endraw

{% raw %}{{ filename }}.log{% endraw %}

dag_processor_manager_log_location

Full path of dag_processor_manager logfile

/var/log/airflow/dag_processor_manager/dag_processor_manager.log

task_log_reader

Name of handler to read task instance logs. Defaults to use task handler

task

extra_logger_names

A comma-separated list of third-party logger names that will be configured to print messages to consoles

 — 

worker_log_server_port

When you start an Airflow worker, the service starts a tiny web server subprocess to serve the workers local log files to the Airflow main web server, who then builds pages and sends them to users. This defines the port on which the logs are served. It must be unused, open, and visible from the main web server to connect into the workers

8793

trigger_log_server_port

Port to serve logs from for triggerer. See worker_log_server_port description for more information

8794

interleave_timestamp_parser

Import path to callable, which takes a string log line and returns the timestamp (datetime.datetime compatible)

 — 

file_task_handler_new_folder_permissions

Permissions in the form of octal string as understood by chmod. The permissions are important when you use impersonation, when logs are written by a different user than airflow. The most secure way of configuring it is to add both users to the same group and make it the default group of both users. Group-writeable logs are default in Airflow. For cases when the logs other-writeable, set the value to 0o777. You might decide to add more security if you do not use impersonation and change it to 0o755 to make it only owner-writeable. You can also make it just readable only for owner by changing it to 0o700, if all the access (read/write) for your logs happens from the same user

0o775

file_task_handler_new_file_permissions

Permissions in the form of octal string as understood by chmod. The permissions are important when you use impersonation, when logs are written by a different user than airflow. The most secure way of configuring it is to add both users to the same group and make it the default group of both users. For cases when the logs other-writeable, set the value to 0o666. You might decide to add more security if you do not use impersonation and change it to 0o644 to make it only owner-writeable. You can also make it just readable only for owner by changing it to 0o600, if all the access (read/write) for your logs happens from the same user

0o664

dag_processor_manager_log_stdout

Controls whether the DAG processor manager writes its logs to stdout

false

celery_stdout_stderr_separation

Separates lower-severity Celery logs to stdout and higher-severity logs to stderr

false

enable_task_context_logger

Enables the task context logger

true

color_log_error_keywords

Comma-separated keywords whose presence causes a log line to be displayed as an error color

error,exception

color_log_warning_keywords

Keywords whose presence causes a log line to be displayed as a warning color

warn

metrics_allow_list

If you want to avoid emitting all the available metrics, you can configure a list of prefixes (comma-separated) to send only the metrics that start with the elements of the list (e.g. scheduler,executor,dagrun)

 — 

metrics_block_list

If you want to avoid emitting all the available metrics, you can configure a list of prefixes (comma-separated) to filter out metrics that start with the elements of the list (e.g. scheduler,executor,dagrun). If metrics_allow_list and metrics_block_list are both configured, metrics_block_list is ignored

 — 

statsd_on

Enables sending metrics to StatsD

true

statsd_host

Specifies the host address where the StatsD daemon (or server) is running

localhost

statsd_port

Specifies the port on which the StatsD daemon (or server) is listening to

8125

statsd_prefix

Defines the namespace for all metrics sent from Airflow to StatsD

airflow

stat_name_handler

A function that validates the StatsD stat name, applies changes to the stat name if necessary, and returns the transformed stat name. The function should have the following signature: def func_name(stat_name: str) → str

 — 

statsd_datadog_enabled

Enables datadog integration to send Airflow metrics

false

statsd_datadog_tags

List of datadog tags attached to all metrics(e.g. key1:value1,key2:value2)

 — 

statsd_datadog_metrics_tags

Set to False to disable metadata tags for some of the emitted metrics

true

statsd_custom_client_path

If you want to use your own custom StatsD client, set the relevant module path in this value. The module path must exist on your PYTHONPATH

 — 

statsd_disabled_tags

If you want to avoid sending all the available metrics tags to StatsD, you can configure a list of prefixes (comma-separated) to filter out metric tags that start with the elements of the list (e.g. job_id,run_id)

job_id,run_id

statsd_influxdb_enabled

Enables sending Airflow metrics with StatsD-Influxdb tagging convention

false

otel_on

Enables sending metrics to OpenTelemetry

false

otel_host

Specifies the hostname or IP address of the OpenTelemetry Collector to which Airflow sends traces

localhost

otel_port

Specifies the port of the OpenTelemetry Collector that is listening to

8889

otel_prefix

The prefix for the Airflow metrics

airflow

otel_interval_milliseconds

Defines the interval, in milliseconds, at which Airflow sends batches of metrics and traces to the configured OpenTelemetry Collector

60000

metrics_use_pattern_match

Controls whether metric allow/block lists are interpreted as patterns

false

otel_debugging_on

Enables additional OpenTelemetry debugging output

false

otel_service

Service name reported to OpenTelemetry

Airflow

otel_ssl_active

Enables SSL for OpenTelemetry

false

otel_on

Enables sending traces to OpenTelemetry

false

otel_host

Host name or IP address of the OpenTelemetry Collector to which Airflow sends traces

localhost

otel_port

Port of the OpenTelemetry endpoint

8889

otel_debugging_on

Enables additional OpenTelemetry debugging output

false

otel_service

Service name reported to OpenTelemetry

Airflow

otel_ssl_active

Enables SSL for OpenTelemetry

false

otel_task_log_event

JSON configuration passed to the secrets backend when it is initialized

 — 

backend

Full class name of secrets backend to enable

 — 

backend_kwargs

JSON configuration passed to the secrets backend when it is initialized

 — 

use_cache

Enables local caching of variables during DAG parsing

false

cache_ttl_seconds

Lifetime of cached secret/variable values, in seconds

900

api_client

Defines the format of access to the API. The LocalClient will use the database directly, while the json_client will use the API running on the webserver

airflow.api.client.local_client

endpoint_url

If you set web_server_url_prefix, append it here as follows: endpoint_url = http://localhost:8080/myroot. So that the API URI looks like this: http://localhost:8080/myroot/api/experimental/...

http://localhost:8080

fail_fast

Used only with DebugExecutor. If set to True, DAG will fail with the first failed task

false

base_url

Base URL of the Airflow API server. If not specified, ADCM sets it based on the first server host

 — 

host

IP address on which the API server listens

0.0.0.0

port

Port on which the API server listens

8080

workers

Number of workers to run on the API server

1

worker_timeout

Timeout in seconds that the API server waits for a worker process to respond

120

server_type

Server implementation used by API Server: uvicorn or gunicorn

uvicorn

worker_refresh_interval

Interval in seconds between rolling API server worker refreshes. This parameter is used only with server_type=gunicorn

0

worker_refresh_batch_size

The number of API server workers to refresh at a time. This parameter is used only with server_type=gunicorn

1

secret_key

Shared API server secret key. It must be identical across all API server instances

 — 

ssl_cert

Path to the SSL certificate for the API server

 — 

ssl_key

Path to the SSL key for the API server certificate

 — 

maximum_page_limit

Maximum page size allowed for API requests

100

fallback_page_limit

Default page size used when an API request does not specify a positive limit

100

access_control_allow_headers

CORS headers allowed by the API server

 — 

access_control_allow_methods

CORS methods allowed by the API server

 — 

access_control_allow_origins

CORS origins allowed by the API server. Separate multiple URLs with spaces

 — 

enable_swagger_ui

Enables Swagger UI on the API server

true

expose_config

Exposes Airflow configuration through the API UI, with sensitive values being masked

false

expose_stacktrace

Exposes stack traces through the API server

false

log_config

Path to the Uvicorn logging configuration file

 — 

log_stream_buffer_size

Number of log lines to buffer before flushing streaming task logs to the client

500

config_file

Path to the FAB webserver configuration file

/usr/lib/airflow/webserver_config.py

auth_backends

Comma separated list of FAB auth backends for Airflow API authentication

airflow.providers.fab.auth_manager.api.auth.backend.basic_auth

update_fab_perms

Specifies whether to update FAB permissions on API Server startup

true

jwt_secret

Secret used for signing and validating JWTs

<secret>

jwt_algorithm

Algorithm used when generating and validating JWT Task Identities

HS512

jwt_issuer

Issuer claim used when generating and validating JWTs

 — 

jwt_audience

Audience claim used when generating and validating API JWTs

 — 

jwt_expiration_time

Lifetime of API authentication JWTs, in seconds

86400

jwt_cli_expiration_time

Lifetime of JWTs used for CLI authentication, in seconds

3600

jwt_leeway

Allowed clock skew when validating JWT timestamps, in seconds

10

jwt_expiration_time

Lifetime of Execution API JWTs, in seconds

600

jwt_audience

Audience claim used for Execution API task JWTs

urn:airflow.apache.org:task

min_file_process_interval

Interval in seconds between DAG file parse runs

30

refresh_interval

Number of seconds between DAG bundle refreshes/checks for new DAG files

300

dag_file_processor_timeout

Timeout for parsing a single DAG file in seconds

50

parsing_processes

Number of parallel DAG parsing processes

2

file_parsing_sort_mode

modified_time

max_callbacks_per_loop

Maximum callbacks fetched per DAG Processor loop

20

stale_dag_threshold

Seconds to wait before deactivating stale DAGs

50

email_backend

Email backend to use

airflow.utils.email.send_email_smtp

email_conn_id

An Airflow connection that contains SMTP credentials

smtp_default

default_email_on_retry

Whether email alerts should be sent when a task is retried

true

default_email_on_failure

Whether email alerts should be sent when a task failed

true

subject_template

File that will be used as the template for email subject (which will be rendered using Jinja2). If not set, Airflow uses a base template

 — 

html_content_template

File that will be used as the template for email content (which will be rendered using Jinja2). If not set, Airflow uses a base template

 — 

from_email

Email address that will be used as sender address. It can either be raw email or the complete address in a format Sender Name <sender@email.com>

 — 

ssl_context

SSL context used for SMTP and IMAP SSL connections. Set to none to disable certificate verification (not recommended)

default

smtp_host

Specifies the host server address used by Airflow when sending out email notifications via SMTP

localhost

smtp_starttls

Determines whether to use the STARTTLS command when connecting to the SMTP server

true

smtp_ssl

Determines whether to use an SSL connection when talking to the SMTP server

false

smtp_user

Username to authenticate when connecting to SMTP server

 — 

smtp_password

Password to authenticate when connecting to SMTP server

 — 

smtp_port

Defines the port number on which Airflow connects to the SMTP server to send email notifications

25

smtp_mail_from

Specifies the default from email address used when Airflow sends email notifications

airflow@example.com

smtp_timeout

Determines the maximum time (in seconds) the Apache Airflow system will wait for a connection to the SMTP server to be established

30

smtp_retry_limit

Defines the maximum number of times Airflow will attempt to connect to the SMTP server

5

sentry_on

Enables error reporting to Sentry

false

sentry_dsn

A Sentry DSN URL

 — 

before_send

Dotted path to a before_send function that the sentry SDK should be configured to use

 — 

celery_app_name

The app name that will be used by Celery

airflow.executors.celery_executor

worker_concurrency

The concurrency that will be used when starting workers with the airflow celery worker command. This defines the number of task instances that a worker will take, so size up your workers based on the resources on your worker box and the nature of your tasks

16

worker_autoscale

The maximum and minimum concurrency that will be used when starting workers with the airflow celery worker command (always keep minimum processes, but grow to maximum if necessary). The value should be in format max_concurrency,min_concurrency. If autoscale option is available, worker_concurrency will be ignored

 — 

worker_prefetch_multiplier

Used to increase the number of tasks that a worker prefetches, which can improve performance. The number of processes multiplied by worker_prefetch_multiplier is the number of tasks that are prefetched by a worker. A value greater than 1 can result in tasks being unnecessarily blocked if there are multiple workers and one worker prefetches tasks that sit behind long running tasks while another worker has unutilized processes that are unable to process the already claimed blocked tasks. For more information, see Celery documentation

1

worker_enable_remote_control

Specify if remote control of workers is enabled. In some cases, when the broker does not support remote control, Celery creates lots of .*reply-celery-pidbox queues. You can prevent this by setting this parameter to false. However, with this option is disabled, Flower won’t work. For more information, see Celery documentation

true

broker_url

The Celery broker URL. Celery supports RabbitMQ, Redis, and experimentally a SQLAlchemy database. Refer to the Celery documentation for more information

redis://{{groups['redis.server'][0]|d(omit)}}:6379/0

result_backend

The Celery backend for storing job metadata. When a job finishes, it needs to update the metadata of the job. Therefore it will post a message on a message bus or insert it into a database (depending of the backend). This status is used by the scheduler to update the state of the task. The use of a database is highly recommended. When not specified, sql_alchemy_conn with a db+ scheme prefix will be used. For more information, see Celery documentation

 — 

result_backend_sqlalchemy_engine_options

Optional configuration dictionary to pass to the Celery result backend SQLAlchemy engine

 — 

flower_host

Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start it airflow celery flower. This defines the IP that Celery Flower runs on

0.0.0.0

flower_url_prefix

The root URL for Flower

 — 

flower_port

The port that Celery Flower runs on

5555

flower_basic_auth

Enable basic authentication for Flower. This parameter takes a string in the format user:password, which will be required when accessing the Flower UI

 — 

sync_parallelism

How many processes CeleryExecutor uses to sync task state. 0 means to use max

0

celery_config_options

Import path for Celery configuration options

airflow.providers.celery.executors.default_celery.DEFAULT_CELERY_CONFIG

ssl_active

Defines if SSL is active for Airflow

false

ssl_key

Path to the client key

 — 

ssl_cert

Path to the client certificate

 — 

ssl_cacert

Path to the CA certificate

 — 

pool

Celery pool implementation. Possible choices are: prefork (default), eventlet, gevent, or solo. For more information, see Celery documentation

prefork

operation_timeout

The number of seconds to wait before timing out send_task_to_executor or fetch_celery_task_state operations

1

task_track_started

Celery task will report its status as started when the task is executed by a worker. This is used in Airflow to keep track of the running tasks and if a Scheduler is restarted or run in HA mode, it can adopt the orphan tasks launched by previous SchedulerJob

true

task_publish_max_retries

The maximum number of retries for publishing task messages to the broker when failing due to AirflowTaskTimeout error before giving up and marking a task as failed

3

worker_precheck

Worker initialisation check to validate metadata database connection

false

broker_url_secret

ADO secret reference used as the source for the Celery broker URL

broker_url_value

result_backend_secret

ADO secret reference used as the source for the Celery result backend

result_backend_value

job_heartbeat_sec

Defines the frequency (in seconds) at which task instances should listen for external kill signal (when you clear tasks from the CLI or the UI)

5

scheduler_heartbeat_sec

The scheduler constantly tries to trigger new tasks. This defines how often the scheduler should run (in seconds)

5

num_runs

The number of times to try to schedule each DAG file. -1 indicates unlimited number

-1

scheduler_idle_sleep_time

Controls how long the scheduler will sleep between loops. If there was nothing to schedule, the next loop starts straight away

1

parsing_cleanup_interval

How often (in seconds) to check for stale DAGs (DAGs which are no longer present in the expected files) which should be deactivated, as well as datasets that are no longer referenced and should be marked as orphaned

60

print_stats_interval

How often should stats be printed to the logs. Setting to 0 will disable printing stats

30

pool_metrics_interval

How often (in seconds) should pool usage stats be sent to StatsD (if statsd_on is enabled)

5

scheduler_health_check_threshold

If the last scheduler heartbeat happened more than scheduler_health_check_threshold ago (in seconds), scheduler is considered unhealthy. This is used by the health check in the /health endpoint and in airflow jobs check CLI for SchedulerJob

30

enable_health_check

When you start a scheduler, Airflow starts a tiny webserver subprocess to serve a health check if this is set to True

false

scheduler_health_check_server_port

When you start a scheduler, Airflow starts a tiny webserver subprocess to serve a health check on this port

8974

orphaned_tasks_check_interval

How often (in seconds) should the scheduler check for orphaned tasks and SchedulerJobs

300

child_process_log_directory

Determines the directory where logs for the child processes of the scheduler will be stored

/var/log/airflow/scheduler

scheduler_zombie_task_threshold

Local task jobs periodically heartbeat to the DB. If the job has not heartbeat in this many seconds, the scheduler will mark the associated task instance as failed and will re-schedule the task

300

zombie_detection_interval

How often (in seconds) should the scheduler check for zombie tasks

10

catchup_by_default

Turn off scheduler catchup by setting this to False. Default behavior is unchanged and command line backfills still work, but the scheduler will not do scheduler catchup if this is False, however it can be set on a per DAG basis in the DAG definition (catchup)

true

ignore_first_depends_on_past_by_default

Setting this to True will make first task instance of a task ignore depends_on_past setting. A task instance will be considered as the first task instance of a task when there is no task instance in the DB with an execution_date earlier than it, i.e. no manual marking success will be needed for a newly added task to be scheduled

true

max_tis_per_query

This changes the batch size of queries in the scheduling main loop. If this is too high, SQL query performance may be impacted by complexity of query predicate, and/or excessive locking. Additionally, you may hit the maximum allowable query length for your db. Set this to 0 for no limit (not advised)

16

use_row_level_locking

Should the scheduler issue SELECT …​ FOR UPDATE in relevant queries. If this is set to False then you should not run more than a single scheduler at once

true

max_dagruns_to_create_per_loop

Max number of DAGs to create DagRuns for per scheduler loop

10

max_dagruns_per_loop_to_schedule

How many DagRuns should a scheduler examine (and lock) when scheduling and queuing tasks

20

schedule_after_task_execution

Should the Task supervisor process perform a mini scheduler to attempt to schedule more tasks of the same DAG. Leaving this on will mean tasks in the same DAG execute quicker, but might starve out other DAGs in some circumstances

true

parsing_pre_import_modules

The scheduler reads DAG files to extract the Airflow modules that are going to be used, and imports them ahead of time to avoid having to re-do it for each parsing process. This flag can be set to False to disable this behavior in case an Airflow module needs to be freshly imported each time (at the cost of increased DAG parsing time)

true

use_job_schedule

Turn off scheduler use of cron intervals by setting this to False. DAGs submitted manually in the web UI or with trigger_dag will still run

true

allow_trigger_in_future

Allows externally triggered DagRuns for Execution Dates in the future. Only has effect if schedule_interval is set to None in DAG

false

trigger_timeout_check_interval

How often to check for expired trigger requests that have not run yet

15

task_queued_timeout

Amount of time a task can be in the queued state before being retried or set to failed

600

task_queued_timeout_check_interval

How often to check for tasks that have been in the queued state for longer than [scheduler] task_queued_timeout

120

allowed_run_id_pattern

The run_id pattern used to verify the validity of user input to the run_id parameter when triggering a DAG. This pattern cannot change the pattern used by scheduler to generate run_id for scheduled DAG runs or DAG runs triggered without changing the run_id parameter

^[A-Za-z0-9_.~:+-]+$

local_task_job_heartbeat_sec

Frequency, in seconds, at which a local task job sends heartbeat signals

0

scheduler_health_check_server_host

Host on which the scheduler health-check server listens

0.0.0.0

create_cron_data_intervals

Controls creation of cron-based data intervals

true

default_capacity

How many triggers a single Triggerer will run at once, by default

1000

job_heartbeat_sec

How often to heartbeat the Triggerer job to ensure it hasn’t been killed

5

triggerer_health_check_threshold

Number of seconds after the last Triggerer heartbeat before it is considered unhealthy

30

ccache

Location of your ccache file once kinit has been performed

/usr/lib/airflow/krb5_ccache

principal

Kerberos principal

 — 

reinit_frequency

Kerberos reinit frequency

3600

kinit_path

Path to the kinit executable

kinit

keytab

Designates the path to the Kerberos keytab file for the Airflow user

 — 

forwardable

Allows you to disable ticket forwardability

true

include_ip

Allows you to remove source IP from token, useful when using token behind NATted Docker host

true

default_timeout

Sensor default timeout, 7 days by default (7 * 24 * 60 * 60)

604800

AIRFLOW_HOME

The home directory for Airflow service

/usr/lib/airflow

AIRFLOW_CONFIG

The location of Airflow configuration file

/etc/airflow/conf/airflow.cfg

AIRFLOW_PYTHON_PATH

The location of Python used by Airflow

/usr/lib/airflow/bin/python3.10

DAG_PROCESSOR_SUBDIR

The location of Airflow stored DAGs

/opt/airflow/dags

VIRTUAL_ENV

Path to the virtual environment for Airflow

/usr/lib/airflow

PATH

Airflow PATH directory

/usr/lib/airflow/bin:/sbin:/bin:/usr/sbin:/usr/bin

AUTH_LDAP_SERVER

The LDAP server URI

 — 

AUTH_LDAP_BIND_USER

The path of the LDAP proxy user to bind on to the top level. Example: cn=airflow,ou=users,dc=example,dc=com

 — 

AUTH_LDAP_BIND_PASSWORD

The password of the bind user

 — 

AUTH_LDAP_SEARCH

Update with the LDAP path under which you’d like the users to have access to Airflow. Example: dc=example, dc=com

 — 

AUTH_LDAP_UID_FIELD

The UID (unique identifier) field in LDAP

 — 

AUTH_ROLES_MAPPING

The parameter for mapping the internal roles to the LDAP Active Directory groups

 — 

AUTH_LDAP_GROUP_FIELD

The LDAP user attribute which has their role DNs

 — 

AUTH_ROLES_SYNC_AT_LOGIN

A flag that indicates if all the user’s roles should be replaced on each login, or only on registration

true

PERMANENT_SESSION_LIFETIME

Sets an inactivity timeout after which users have to re-authenticate (to keep roles in sync)

1800

AUTH_LDAP_USE_TLS

Boolean whether TLS is being used

false

AUTH_LDAP_ALLOW_SELF_SIGNED

Boolean to allow self-signed certificates

true

AUTH_LDAP_TLS_CACERTFILE

Location of the certificate

 — 

Constraints file

Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not

 — 

Base constraints file

Includes all the Python dependencies required for Airflow installation

Extra requirements

List of Python packages to be installed on Airflow hosts. Use the standard requirements.txt format: <package_name>==<version>

 — 

index-url

Base URL of the Python Package Index (default: https://pypi.org/simple). The URL must point to a repository that complies with PEP 503 (the simple API) or to a local directory with the same structure

 — 

index-url-user

Username used for authenticating with the repository specified in index-url

 — 

index-url-password

Password used for authenticating with the repository specified in index-url

 — 

proxy

Address of the proxy server through which package installation requests will be routed

 — 

proxy-user

Username for authenticating with the proxy server

 — 

proxy-password

Password used for authenticating with the proxy server

 — 

trusted-host

IP address of the host or the <host IP>:<port> pair to be treated as trusted, even if it lacks valid HTTPS. Useful for internal or self-hosted repositories

 — 

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the Ulimit settings table

auto_refresh

Enables automatic refresh for the Workers view. By default, the Workers view automatically refreshes at regular intervals to provide up-to-date information about the workers. Set this option to False to disable automatic refreshing

true

ca_cert

Sets the path to the ca_certs file containing a set of concatenated certification authority certificates

 — 

cert_file

Sets the path to the SSL certificate file

 — 

keyfile

Sets the path to the SSL key file

 — 

db

Sets the database file to use if persistent mode is enabled

flower

tasks_columns

Specifies the list of comma-separated columns to display on the Tasks page

name,uuid,state,args,kwargs,result,received,started,runtime,worker

persistent

When persistent mode is enabled, Flower saves its current state and reloads it upon restart. This ensures that Flower retains its state and configuration across restarts. Flower stores its state in a database file specified by the db option

false

debug

Enables the debug mode

false

enable_events

When enabled, Flower periodically sends Celery enable_events commands to all workers. Enabling Celery events allows Flower to receive real-time updates about task events from the Celery workers

false

inspect_timeout

Sets the timeout for the worker inspect commands in milliseconds

1000

max_workers

Sets the maximum number of workers to keep in memory

5000

max_tasks

Sets the maximum number of tasks to keep in memory

100000

natural_time

Enables showing time relative to the page refresh time in a human-readable format

false

state_save_interval

Sets the interval for saving the Flower state. Flower state includes information about workers, tasks. The state is saved periodically to ensure data persistence and recovery upon restart

100000

xheaders

Enables support for X-Real-Ip and X-Scheme headers

false

purge_offline_workers

Time (in seconds) after which offline workers are automatically removed from the Workers view. By default, offline workers will remain on the dashboard indefinitely

 — 

task_runtime_metric_buckets

Sets the task runtime latency buckets. You can provide the buckets value as a comma-separated list of values

 — 

auth_provider

Sets the authentication provider for Flower. By default, the auth_provider option is set to None, indicating that no authentication provider is configured

 — 

auth

Enables authentication. auth is a regular expression of emails to grant access. The auth option allows you to enable authentication in Flower. By default, the auth option is set to an empty string, indicating that authentication is disabled

 — 

oauth2_key

Sets the OAuth 2.0 key (client ID) issued by the OAuth 2.0 provider

 — 

oauth2_secret

Sets the OAuth 2.0 secret issued by the OAuth 2.0 provider

 — 

oauth2_redirect_uri

Sets the URI to which an OAuth 2.0 server redirects the user after successful authentication and authorization

 — 

cookie_secret

Sets a secret key for signing cookies

 — 

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the Ulimit settings table

haproxy-airflowcfg

Jinja template of HAProxy configuration file

Specifies the location of the certificate in standard .PEM format

Path to the PEM certificate for HTTPS

 — 

/etc/syslog-ng/conf.d/haproxy-airflow.conf

Syslog-ng configuration for Haproxy. It will be used if syslog-ng is installed in your operating system

/etc/rsyslog.d/haproxy-airflow.conf

Rsyslog configuration for Haproxy. It will be used if rsyslog is installed in your operating system

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

DefaultLimitFSIZE

The maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

DefaultLimitDATA

The maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

DefaultLimitSTACK

The maximum stack size allocated to a process, in kilobytes

stack size ( -s)

DefaultLimitCORE

The maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

DefaultLimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

DefaultLimitNOFILE

The maximum number of open file descriptors allowed for the process

open files ( -n)

DefaultLimitAS

The maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

DefaultLimitNPROC

The maximum number of processes

max user processes ( -u)

DefaultLimitMEMLOCK

The maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

DefaultLimitLOCKS

The maximum number of files locked by a process

file locks ( -x)

DefaultLimitSIGPENDING

The maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

DefaultLimitMSGQUEUE

The maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

DefaultLimitNICE

The maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

DefaultLimitRTPRIO

The maximum real-time scheduling priority level

real-time priority ( -r)

DefaultLimitRTTIME

The maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)