Airflow Server high availability with HAProxy

Elena Kostyuchenko

Starting with ADO 3.0, the Airflow Server component uses Airflow API Server as the HTTP layer. API Server provides Airflow Web UI, REST API, and Task Execution API used by Airflow workers.

Multiple Airflow Server instances can be deployed to improve service availability (High Availability, HA). In this configuration, Airflow application (workers, UI, REST API requests) must use a single endpoint instead of connecting to individual Airflow Server instances. The HAProxy component provides this endpoint and routes requests to available Airflow Server instances.

The Airflow service includes the Airflow Haproxy component for this purpose. It automatically generates the HAProxy configuration based on the cluster topology and the selected traffic routing mode.

Airflow Haproxy component

The Airflow Haproxy component acts as a reverse proxy between Airflow clients and server instances. It provides a single HTTP/HTTPS endpoint for the Airflow Web UI, REST API, and requests from Airflow workers.

The component performs health checks and excludes unavailable servers from request routing. When the current server becomes unavailable, HAProxy can switch requests to another available instance.

Two traffic routing modes are supported:

  • Failover — requests are sent to one server instance. If this instance becomes unavailable, HAProxy switches requests to another available instance. This mode is enabled by default.

  • Load balancing — requests are distributed between available Airflow Server instances.

To use Airflow in the HA mode, clients must connect to the Airflow Haproxy endpoint rather than directly to an Airflow Server.

The Airflow Haproxy component can be installed on one or more hosts. When several Airflow Haproxy instances are deployed, they can be used together with external network-level HA mechanisms to eliminate the HAProxy host as a single point of failure.

NOTE

The Airflow Haproxy and Airflow Server components must be installed on different hosts.

Airflow Haproxy configuration

When multiple Airflow Server instances are deployed, ADCM automatically generates the HAProxy configuration. The generated configuration contains the addresses of the available Airflow Server instances and the selected traffic routing algorithm.

The Airflow Haproxy configuration parameters are available in the Airflow service configuration in ADCM.

To modify the Airflow Haproxy configuration via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Airflow.

  3. Navigate to the Components tab and select Airflow Haproxy.

  4. Select the required parameter and make the necessary changes.

  5. Confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

The main HAProxy configuration is generated from the haproxy-airflowcfg Jinja template:

{% set tls_haproxy_pem = '' %}
{%- if airflow_ssl_enable -%}
    {% set service_ssl = services.airflow.haproxy.config.config_properties_ssl | default('', true) | trim %}
    {% set cluster_ssl = cluster.config.ssl_default_config.tls_haproxy_pem | default('', true) | trim %}
    {% if service_ssl %}
        {% set tls_haproxy_pem = 'ssl crt ' ~ service_ssl %}
    {% elif cluster_ssl %}
        {% set tls_haproxy_pem = 'ssl crt ' ~ cluster_ssl %}
    {% endif %}
{% endif %}
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global (1)
    log /dev/log sample 1:10  local0
    log 127.0.0.1:514  local0
    log-tag airflow-haproxy

    chroot      /var/lib/ado-haproxy/airflow
    maxconn     1024
    user        haproxy
    group       haproxy
    daemon

    ssl-server-verify none

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults (2)
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

listen stats (3)
    bind *:7001 {{ tls_haproxy_pem }}
    stats enable
    stats uri /

frontend airflow_http_in (4)
    bind *:{{ roles_airflow_vars_component_ports['server']['web'] }} {{ tls_haproxy_pem }}
    default_backend airflow_http_servers

backend airflow_http_servers (5)
    balance first

{% for host in haproxy_conf_hosts %}
    {{ host }}
{% endfor %}
1 Global HAProxy parameters, including logging, the chroot directory, connection limits, and SSL verification.
2 Default HTTP mode parameters, logging options, retries, and timeout settings.
3 Enables the HAProxy statistics endpoint. The endpoint is available at http://<haproxy_host>:7001/ or, when SSL is enabled, at https://<haproxy_host>:7001/.
4 Defines the listening socket for Airflow HTTP/HTTPS traffic.
5 Defines the backend containing the available Airflow Server instances and the traffic routing mode.
NOTE

The haproxy-airflowcfg parameter is a Jinja template. It is generated and applied by ADO. Do not edit the generated configuration file directly on Airflow hosts because changes made on the host can be overwritten when the configuration is regenerated.

For more details on HAProxy configuration directives, see the HAProxy reference.

Traffic routing

The traffic routing mode is configured in the Airflow service settings in ADCM. The default mode is failover.

In failover mode (balance first), HAProxy uses one Airflow Server as the active server. Other available server instances remain available as failover targets. When HAProxy detects that the active server is unavailable, it stops sending new requests to that server and selects another available instance.

This mode is suitable when the primary Airflow Server should handle requests whenever it is available and other Airflow Server instances should primarily serve as failover targets.

In load balancing mode (roundrobin), HAProxy distributes requests between available server instances.

The HAProxy backend is regenerated when the Airflow Server topology or traffic routing configuration changes. This ensures that newly added instances are included in routing and removed or unavailable instances are excluded.

To change the traffic routing method to load balancing via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Airflow.

  3. Navigate to the Components tab and select Airflow Haproxy.

  4. Select the haproxy-airflowcfg parameter and make changes to the configuration as follows:

    backend airflow_http_servers
    balance roundrobin

  5. Confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Access to the Airflow Web UI and REST API

The Airflow Haproxy endpoint is the common entry point for Airflow HTTP traffic.

When HAProxy is enabled, users should open the Airflow Web UI through the HAProxy endpoint. REST API users should also use this endpoint instead of an individual API Server address.

For example, a REST API request through HAProxy has the following format:

$ curl --insecure https://<haproxy_host>:<port>/api/v2/<endpoint>

where:

  • <haproxy_host> — host where the Airflow Haproxy component is installed;

  • <port> — HTTP or HTTPS port configured for the Airflow Haproxy endpoint;

  • <endpoint> — REST API endpoint.

For more information about the Airflow REST API, see Connect to Airflow via REST API.

The same endpoint is used for the Airflow Web UI. The current Airflow Web UI link is available on the Info tab of the Airflow service in ADCM.

Airflow Worker configuration

Airflow workers use the Task Execution API provided by the Airflow API Server. When the Airflow Haproxy component is used, workers are configured to address the HAProxy endpoint through the execution_api_server_url setting.

This allows workers to continue communicating with the Airflow HTTP layer when an individual API Server instance becomes unavailable.

The configuration is generated by ADO according to the deployed Airflow Haproxy and Airflow Server components. No manual configuration of individual worker hosts is required.

Cluster topology changes

ADO automatically updates the HAProxy configuration when the cluster topology changes.

When a server is added, its address is included in the generated HAProxy backend configuration. When a server is removed, its address is removed from the backend configuration.

HAProxy health checks also prevent requests from being sent to a server that is temporarily unavailable.

If the Airflow Haproxy component is moved from one host or cluster to another, the Airflow Server components affected by the topology change are automatically restarted by ADO. This ensures that the generated Airflow configuration, including the HAProxy endpoint used by workers, is applied consistently.

SSL configuration

The Airflow Haproxy component supports HTTP and HTTPS endpoints.

When SSL is enabled for the Airflow service, the Jinja template determines the certificate location from the service-level SSL configuration or, if it is not specified there, from the cluster-level default SSL configuration.

The following template fragment selects the certificate:

{% set tls_haproxy_pem = '' %}
{%- if airflow_ssl_enable -%}
{% set service_ssl = services.airflow.haproxy.config.config_properties_ssl | default('', true) | trim %}
{% set cluster_ssl = cluster.config.ssl_default_config.tls_haproxy_pem | default('', true) | trim %}
{% if service_ssl %}
{% set tls_haproxy_pem = 'ssl crt ' ~ service_ssl %}
{% elif cluster_ssl %}
{% set tls_haproxy_pem = 'ssl crt ' ~ cluster_ssl %}
{% endif %}
{% endif %}

The certificate must be provided in standard PEM format.

The path to the certificate must be added via ADCM parameter Specifies the location of the certificate in standard .PEM format.

The service-level certificate configuration takes precedence over the cluster-level default certificate when both are specified.

Logging

The Airflow Haproxy component sends its logs to a dedicated local syslog socket. By default, logs are stored in /var/log/ado-haproxy/haproxy-airflow.log when rsyslog is used.

The logging behavior is defined by the following syslog-ng configuration:

/etc/syslog-ng/conf.d/haproxy-airflow.conf
source s_haproxy_airflow {
    unix-dgram("/var/lib/ado-haproxy/airflow/dev/log");
};

destination d_haproxy_airflow {
    file("/var/log/ado-haproxy/haproxy-airflow.$YEAR-$MONTH-$DAY.log"
    template("$DATE $FULLHOST $PROGRAM: ${HAPROXY.CLIENT_IPPORT} \\[${HAPROXY.DATE}\\] ${HAPROXY.FRONTEND} ${HAPROXY.BACKEND} ${HAPROXY.TIME} ${HAPROXY.STATUS_CODE} ${HAPROXY.BYTES_READ} ${HAPROXY.COOKIE_REQ} ${HAPROXY.COOKIE_RESP} ${HAPROXY.TERM_STATE} ${HAPROXY.RUN_STATE} ${HAPROXY.QUEUE_STATE} \\{${HAPROXY.HOST}|${HAPROXY.USER_AGENT}\\} \\"${HAPROXY.METHOD} ${HAPROXY.FRONTEND_PROTOCOL}://${HAPROXY.HOST}${HAPROXY.URL} ${HAPROXY.HTTP_VERSION}\\"\n")
    group(adm)
    perm(0640)
    dir_perm(0750)
    template_escape(no)
    );
};

filter f_haproxy_airflow {
    program("haproxy_airflow") or program("haproxy");
};

filter f_haproxy_airflow_ssl {
    message("~ ");
};

rewrite r_haproxy_airflow_set_frontend_protocol {
    set("http", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_haproxy_airflow)));
    set("https", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_haproxy_airflow_ssl)));
};

parser p_haproxy_airflow {
    csv-parser(
        columns(
            "HAPROXY.CLIENT_IPPORT",
            "HAPROXY.DATE",
            "HAPROXY.FRONTEND",
            "HAPROXY.BACKEND",
            "HAPROXY.TIME",
            "HAPROXY.STATUS_CODE",
            "HAPROXY.BYTES_READ",
            "HAPROXY.COOKIE_REQ",
            "HAPROXY.COOKIE_RESP",
            "HAPROXY.TERM_STATE",
            "HAPROXY.RUN_STATE",
            "HAPROXY.QUEUE_STATE",
            "HAPROXY.HEADERS_REQ",
            "HAPROXY.REQUEST"
        )
        flags(escape-double-char, strip-whitespace, greedy)
        delimiters(" ")
        quote-pairs('""[]{}')
        template("${MESSAGE}")
    );
};

parser p_haproxy_airflow_request {
    csv-parser(
        columns(
            "HAPROXY.METHOD",
            "HAPROXY.URL",
            "HAPROXY.HTTP_VERSION"
        )
        delimiters(" ")
        flags(escape-none, greedy)
        template("${HAPROXY.REQUEST}")
    );
};

parser p_haproxy_airflow_headers_req {
    csv-parser(
        columns(
            "HAPROXY.HOST",
            "HAPROXY.USER_AGENT"
        )
        delimiters("|")
        flags(escape-none, greedy)
        template("${HAPROXY.HEADERS_REQ}")
    );
};

log {
    source(s_haproxy_airflow);
    filter(f_haproxy_airflow);
    parser(p_haproxy_airflow);
    parser(p_haproxy_airflow_request);
    parser(p_haproxy_airflow_headers_req);
    rewrite(r_haproxy_airflow_set_frontend_protocol);
    destination(d_haproxy_airflow);
};

To edit the syslog-ng configuration via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Airflow.

  3. Navigate to the Components tab and select Airflow Haproxy.

  4. Select the /etc/syslog-ng/conf.d/haproxy-airflow.conf parameter.

  5. Make the necessary changes and click Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

If rsyslog is used instead of syslog-ng, the corresponding configuration is stored in /etc/rsyslog.d/haproxy-airflow.conf.

/etc/rsyslog.d/haproxy-airflow.conf
module(load="imuxsock")
input(type="imuxsock" Socket="/var/lib/ado-haproxy/airflow/dev/log")

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "airflow-haproxy" {
    /var/log/ado-haproxy/haproxy-airflow.log
    stop
}

To edit the rsyslog configuration via ADCM, select the /etc/rsyslog.d/haproxy-airflow.conf parameter in the Airflow Haproxy component configuration.

Found a mistake? Seleсt text and press Ctrl+Enter to report it