Use HAProxy to enable HA for ADS services

Olga Semme

Overview

HAProxy is an ADS service used to ensure high availability (HA), load balancing, and routing of HTTP/TCP traffic between servers.

This service is based on the HAProxy project and allows to efficiently use multiple server instances in a cluster (for example, several Kafka REST Proxy components) by distributing incoming HTTP requests.

High availability in ADS is supported for services:

IMPORTANT

To implement HA at the network level, you can deploy multiple instances of HAProxy Server.

HAProxy configuration in ADCM

After installing the HAProxy service in an ADS cluster, ADCM automatically generates a main configuration.

The main configuration file is /etc/ads-haproxy/haproxy.cfg. This file is generated from a Jinja template in ADCM and it is not recommended to edit it manually on the hosts.

To change the HAProxy configuration, do the following:

  1. Open the haproxy_cfg_template (available after activating the Advanced switch) on the HAProxy Server component configuration page (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration).

  2. Make the necessary changes in the template, apply them using the Apply button, and confirm the configuration changes by clicking Save.

  3. After making the changes, update the service configuration using the Reload action.

    NOTE

    Some HAProxy configuration parameters can be changed in the ADCM interface without using a template on the HAProxy Server component configuration page of the HAProxy service (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration). After changing the parameter values, you also need to update the service configuration.

When you start the service or after updating the service configuration, the resulting /etc/ads-haproxy/haproxy.cfg file will be generated from the following template:

#jinja2: lstrip_blocks:"True", trim_blocks:"True", keep_trailing_newline: True
# {{ ansible_managed }}
{% set stats_username = haproxy_server_config.haproxy_cfg.listener_stats.haproxy_server_listener_user %}
{% set stats_password = haproxy_server_config.haproxy_cfg.listener_stats.haproxy_server_listener_password %}
{% if ads_ssl_enabled %}
    {% set merged_pem_certificate = haproxy_server_config.haproxy_cfg.haproxy_ssl_frontend %}
    {% set haproxy_ssl_frontend = "ssl crt " ~ merged_pem_certificate | default('/etc/ssl/merged.pem', true) %}
{% endif %}


#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global (1)
    log /dev/log    local0
    chroot      /var/lib/haproxy/
    maxconn     1024
    user        haproxy
    group       haproxy
    daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults (2)
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
#---------------------------------------------------------------------
# HAProxy statistics
#---------------------------------------------------------------------
listen stats (3)
    bind *:{{ haproxy_server_config.haproxy_cfg.listener_stats.haproxy_server_listener_port }} {% if ads_ssl_enabled %} {{ haproxy_ssl_frontend }}{% endif %}

    stats enable
    stats uri /stats
    stats hide-version
    stats auth {{ stats_username }}:{{ stats_password }}
1 Global HAProxy parameters (logging, chroot directory, SSL, etc.).
2 Default HTTP mode parameters and timeout settings.
3 Enables the HAProxy statistics and monitoring endpoint (default: http://<haproxy_host>:7001/).
Example of the resulting configuration file
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log    local0
    chroot      /var/lib/haproxy/
    maxconn     1024
    user        haproxy
    group       haproxy
    daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
#---------------------------------------------------------------------
# HAProxy statistics
#---------------------------------------------------------------------
listen stats
    bind *:7001
    stats enable
    stats uri /stats
    stats hide-version
    stats auth haproxy:haproxy

For more details on HAProxy configuration, see HAProxy reference.

Schema-Registry high availability

After installing the HAProxy service in an ADS cluster, ADCM automatically generates a configuration for routing incoming requests to all available instances of the Schema-Registry service in addition to the general HAProxy configuration.

IMPORTANT

To use the service in HA mode, the Schema-Registry client should connect to the HAProxy Server component, not directly to the server. The current connection link (Load Balancer (HAProxy) links) is available in ADCM (Clusters → <ads_cluster_name> → Services → Schema-Registry → Info).

The configuration file for setting up a connection to Kafka REST Proxy is /etc/ads-haproxy/conf.d/schema-registry.cfg. This file is generated from a Jinja template in ADCM, and it is not recommended to edit it manually on the hosts.

To change the HAProxy configuration, do the following:

  1. Open the Schema-Registry HAProxy config template (available after enabling the Advanced switch) in the Listener Schema Registry group on the HAProxy Server component configuration page (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration).

  2. Make the necessary changes in the template, apply them using the Apply button, and confirm the configuration changes by clicking Save.

  3. After making the changes, update the service configuration by using the Reload action.

    NOTE

    Some HAProxy configuration settings for connecting to Schema-Registry can be changed in the ADCM interface without using a template on the HAProxy Server component configuration page (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration) in the Listener Schema Registry group. After changing the parameter values, the service configuration also needs to be updated.

When starting the service or after updating the service configuration, the resulting /etc/ads-haproxy/conf.d/schema-registry.cfg file will be generated from the following template:

#jinja2: lstrip_blocks:"True", trim_blocks:"True", keep_trailing_newline: True
# {{ ansible_managed }}
{% if ads_ssl_enabled %}
    {% set merged_pem_certificate = haproxy_server_config.haproxy_cfg.haproxy_ssl_frontend %}
    {% set haproxy_ssl_frontend = "ssl crt " ~ merged_pem_certificate | default('/etc/ssl/merged.pem', true) %}
    {% set haproxy_ssl_backend = "ssl verify required ca-file @system-ca" %}
{% endif %}
{% set sr_port = services.schema_registry.config.Main.schema_registry_listener_port%}
{% set sr_group = groups['schema_registry'] %}

# Schema Registry frontend (1)
frontend schema_registry_frontend
    default_backend schema_registry_backend
    http-response add-header X-Backend %s
    bind *:{{ haproxy_server_config.haproxy_cfg.listener_schema_registry.schema_registry_front_port }} {% if ads_ssl_enabled %} {{ haproxy_ssl_frontend }}{% endif %}

# Schema Registry backend (2)
backend schema_registry_backend
    balance {{ haproxy_server_config.haproxy_cfg.listener_schema_registry.haproxy_server_backend_balance }}
    option {{ haproxy_server_config.haproxy_cfg.listener_schema_registry.haproxy_server_backend_option }}
{% if ads_basic_auth_enabled %}
    http-check send hdr Authorization "Basic {{ haproxy_check_auth_b64 }}"
{% endif %}
{% if ads_ssl_enabled %}
    http-send-name-header Host
{% endif %}
    {# # `ipv4@{{ host }}` - Force IPv4 address resolution. Without an explicit IP protocol prefix,
    ALT Linux may resolve FQDN via IPv6 by default. #}
{% for host in sr_group %}
    {% set __host = ("ipv4@" if haproxy_server_config.haproxy_cfg.backend_network.force_ipv4 else "") ~ host %}
    {% set _ssl_settings = '' %}
    {% if ads_ssl_enabled %}
        {% set _ssl_settings = haproxy_ssl_backend ~ ' sni str(' ~ host ~ ')' %}
    {% endif %}
    {#  Server name must stay the host FQDN: http-send-name-header sends it as the Host header,
        and the backend rejects a Host absent from its certificate (400 Invalid SNI). #}
    server {{ host }} {{ __host }}:{{ sr_port }} check inter 10s rise 2 fall 3 {{ _ssl_settings }}
{% endfor %}
1 IP addresses and ports that Schema-Registry clients can connect to.
2 The list of Schema-Registry servers to which the load balancer will route requests. Inside the loop {% for host in …​ %}, it generates a list of Schema-Registry servers available in the ADS cluster. balance roundrobin evenly distributes requests across all available hosts.
Example of the resulting configuration file
# Schema Registry frontend
frontend schema_registry_frontend
    default_backend schema_registry_backend
    http-response add-header X-Backend %s
    bind *:8085
# Schema Registry backend
backend schema_registry_backend
    balance roundrobin
    option httpchk GET /subjects
    server sov-ads-3.ru-central1.internal ipv4@sov-ads-3.ru-central1.internal:8081 check inter 10s rise 2 fall 3
    server sov-ads-2.ru-central1.internal ipv4@sov-ads-2.ru-central1.internal:8081 check inter 10s rise 2 fall 3
    server sov-ads-1.ru-central1.internal ipv4@sov-ads-1.ru-central1.internal:8081 check inter 10s rise 2 fall 3

After installing the HAProxy service in an ADS cluster, the configuration file parameters for the Schema-Registry service in /etc/schema-registry/schema-registry.properties are automatically determined, as described below.

Parameter Description Default value

leader.eligibility

Determines whether a specific node (server) can be elected as the leader (primary) in the cluster. This should be set to true for all nodes (Schema-Registry components)

true

schema.registry.group.id

Consumer group ID that the Schema-Registry uses to read from the _schemas topic. It should be the same for all nodes (Schema-Registry components) in the cluster

schema-registry

Kafka REST Proxy high availability

After installing the HAProxy service in an ADS cluster, ADCM automatically generates a configuration for routing incoming requests to all available instances of the Kafka REST Proxy service in addition to the general HAProxy configuration.

IMPORTANT

To use the service in HA mode, the Kafka REST Proxy client should connect to the HAProxy Server component, not directly to the server. The current connection link (Load Balancer (HAProxy) links) is available in ADCM (Clusters → <ads_cluster_name> → Services → Kafka REST Proxy → Info).

The configuration file for setting up a connection to Kafka REST Proxy is /etc/ads-haproxy/conf.d/kafka-rest.cfg. This file is generated from a Jinja template in ADCM, and it’s not recommended to edit it manually on the hosts.

To change the HAProxy configuration, do the following:

  1. Open the Kafka REST HAProxy config template (available after enabling the Advanced switch) in the Kafka REST Listener group on the configuration page of the HAProxy Server component of the HAProxy service (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration).

  2. Make the necessary changes to the template, apply them using the Apply button, and confirm the changes in the configuration by clicking Save.

  3. After making the changes, update the service configuration using the Reload action.

    NOTE

    Some configuration parameters of HAProxy for setting up connections to Kafka REST Proxy can be changed in the ADCM interface without using a template on the HAProxy Server component configuration page (Clusters → <ads_cluster_name> → Services → HAProxy → Components → HAProxy Server → Primary configuration) in the Listener Kafka REST group. After changing the parameter values, you also need to update the service configuration.

When starting the service or after updating the service configuration, the resulting /etc/ads-haproxy/conf.d/kafka-rest.cfg file will be generated from the following template:

#jinja2: lstrip_blocks:"True", trim_blocks:"True", keep_trailing_newline: True
# {{ ansible_managed }}
{% if ads_ssl_enabled %}
    {% set merged_pem_certificate = haproxy_server_config.haproxy_cfg.haproxy_ssl_frontend %}
    {% set haproxy_ssl_frontend = "ssl crt " ~ merged_pem_certificate | default('/etc/ssl/merged.pem', true) %}
    {% set haproxy_ssl_backend = "ssl verify required ca-file @system-ca" %}
{% endif %}
{% set kafka_rest_port = services.kafka_rest.config.Main.rest_listener_port%}
{% set kafka_rest_group = groups['kafka_rest'] %}

# Kafka REST frontend (1)
frontend kafka_rest_frontend
    default_backend kafka_rest_backend
    http-response add-header X-Backend %s
    bind *:{{ haproxy_server_config.haproxy_cfg.listener_kafka_rest.kafka_rest_front_port }} {% if ads_ssl_enabled %} {{ haproxy_ssl_frontend }}{% endif %}

# Kafka REST backend (2)
backend kafka_rest_backend
    balance {{ haproxy_server_config.haproxy_cfg.listener_kafka_rest.haproxy_server_backend_balance }}
    option {{ haproxy_server_config.haproxy_cfg.listener_kafka_rest.haproxy_server_backend_option }}
{% if ads_basic_auth_enabled %}
    http-check send hdr Authorization "Basic {{ haproxy_check_auth_b64 }}"
{% endif %}
{% if ads_ssl_enabled %}
    http-send-name-header Host
{% endif %}
    {# # `ipv4@{{ host }}` - Force IPv4 address resolution. Without an explicit IP protocol prefix,
    ALT Linux may resolve FQDN via IPv6 by default. #}
{% for host in kafka_rest_group %}
    {% set __host = ("ipv4@" if haproxy_server_config.haproxy_cfg.backend_network.force_ipv4 else "") ~ host %}
    {% set _ssl_settings = '' %}
    {% if ads_ssl_enabled %}
        {% set _ssl_settings = haproxy_ssl_backend ~ ' sni str(' ~ host ~ ')' %}
    {% endif %}
    {# # Server name must stay the host FQDN: http-send-name-header sends it as the Host header,
        and the backend rejects a Host absent from its certificate (400 Invalid SNI). #}
    server {{ host }} {{ __host }}:{{ kafka_rest_port }} check inter 10s rise 2 fall 3 {{ _ssl_settings }}
{% endfor %}
1 IP addresses and ports that Kafka REST Proxy clients can connect to.
2 The list of Kafka REST Proxy servers to which the load balancer will route requests. In the loop {% for host in …​ %}, it generates a list of Kafka REST Proxy servers available in the ADS cluster. balance roundrobin evenly distributes requests across all available hosts.
Example of the resulting configuration file
# Kafka REST frontend
frontend kafka_rest_frontend
    default_backend kafka_rest_backend
    http-response add-header X-Backend %s
    bind *:8086
# Kafka REST backend
backend kafka_rest_backend
    balance roundrobin
    option httpchk GET /topics
    server sov-ads-3.ru-central1.internal ipv4@sov-ads-3.ru-central1.internal:8082 check inter 10s rise 2 fall 3
    server sov-ads-2.ru-central1.internal ipv4@sov-ads-2.ru-central1.internal:8082 check inter 10s rise 2 fall 3
    server sov-ads-1.ru-central1.internal ipv4@sov-ads-1.ru-central1.internal:8082 check inter 10s rise 2 fall 3

After installing the HAProxy service in an ADS cluster for the Kafka REST Proxy service, the configuration file /etc/kafka-rest/kafka-rest.properties is automatically set up with the parameters described below.

Parameter Description Default value

id

Unique ID for this Kafka REST Proxy server instance in the cluster; when HA mode is enabled, it is set in the kafka-rest-server-<n> format

 — 

advertised.listeners

Names of HAProxy Server listeners in the <haproxy_fqdn>:<port> format for connecting to the Kafka REST Proxy service in HA mode and for generating correct URLs in API responses

 — 
Found a mistake? Seleсt text and press Ctrl+Enter to report it