Ozone S3 Gateway high availability

Elena Kostyuchenko

Ozone S3 Gateway (S3G) provides an S3-compatible interface to the Ozone storage. It’s designed to be stateless, meaning it does not store any data locally. In Apache Ozone, data is stored on local Ozone Datanode disks, and metadata is managed separately by Ozone Manager and Ozone Storage Container Manager.

This architectural choice allows you to run multiple identical S3 Gateway instances simultaneously without complex synchronization concerns, which enables the high availability (HA) mode. If one gateway instance becomes unavailable, other instances continue serving requests without data loss or inconsistency.

HAProxy Ozone S3G component

Switching between multiple Ozone S3 Gateway instances is performed by the HAProxy Ozone S3G component. It acts as a reverse proxy between S3 clients and Ozone S3 Gateway servers in ADH.

This component is based on the HAProxy project and uses its configuration model described in the HAProxy documentation.

IMPORTANT

  • To use Ozone S3 Gateway in the HA mode, S3 clients (AWS CLI, SDK, or other S3-compatible tools) must connect to the HAProxy Ozone S3G endpoint, not directly to individual S3G instances.

  • The HAProxy Ozone S3G and Ozone S3G components must be installed on different hosts to avoid port conflicts.

  • You can deploy several HAProxy Ozone S3G instances to implement high availability at the network level.

HAProxy configuration in ADCM

When the Ozone S3G service runs in HA mode, ADCM automatically generates the HAProxy configuration to route incoming requests to all available S3 Gateway instances.

The main configuration file is /etc/adh-haproxy/conf/haproxy-ozone-s3g.cfg. This file is generated from a Jinja template in ADCM and should not be edited manually on the hosts.

To modify the configuration via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. Navigate to the Components tab and select HAProxy Ozone S3G.

  4. Select the haproxy-ozone-s3g.cfg parameter and make the necessary changes.

  5. Confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

During the service startup, the final haproxy-ozone-s3g.cfg file will be generated from the following template:

{% set ssl_cert = '' %}
{%- if ozone_ssl_enable -%}
    {% set ssl_cert = ozone_haproxy_conf_ssl_vars %}
{% endif %}

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global (1)
    log /dev/log sample 1:10  local0
    log 127.0.0.1:514  local0
    log-tag hp-ozone-s3g

    chroot      /var/lib/adh-haproxy/ozone-s3g
    maxconn     1024
    user        haproxy
    group       haproxy
    daemon

    ssl-server-verify none

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults (2)
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

listen stats (3)
    bind *:7002 {{ ssl_cert }}
    stats enable
    stats uri /

frontend ozone_s3g_http_in (4)
    bind *:{{ roles_ozone_vars_component_ports['haproxy_ozone_s3g']['s3'] }} {{ ssl_cert }}
    default_backend ozone_s3g_http_servers

backend ozone_s3g_http_servers (5)
    balance roundrobin
    {% for host in haproxy_ozone_s3g_conf_hosts %}
        {{ host }}
    {% endfor %}

frontend ozone_s3g_webui_http_in (6)
    bind *:{{ roles_ozone_vars_component_ports['haproxy_ozone_s3g']['webadmin'] }} {{ ssl_cert }}
    default_backend ozone_s3g_webui_http_servers

backend ozone_s3g_webui_http_servers (7)
    balance roundrobin
    {% for host in haproxy_ozone_s3g_webadmin_conf_hosts %}
        {{ host }}
    {% endfor %}
1 Global HAProxy parameters (logging, chroot directory, SSL verification, etc.).
2 Default HTTP mode parameters and timeout settings.
3 Enables HAProxy statistics and monitoring endpoint (default: http://<haproxy_host>:7002/).
4 Defines a set of listening sockets that accept incoming S3 protocol requests.
5 Lists Ozone S3 Gateway servers to which HAProxy forwards S3 traffic. In this template, the {% for host in …​ %} loop produces a list of Ozone S3G servers available in the ADH cluster. balance roundrobin distributes requests evenly across all available instances.
6 Defines a set of listening sockets for the Ozone S3G web UI.
7 Lists S3 gateway servers to which HAProxy will forward incoming requests for Ozone S3G web UI.
Example of the resulting configuration file

 

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log sample 1:10  local0
    log 127.0.0.1:514  local0
    log-tag haproxy-ozone-s3g

    chroot      /var/lib/adh-haproxy/ozone-s3g
    maxconn     1024
    user        haproxy
    group       haproxy
    daemon

    ssl-server-verify none

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

listen stats
    bind *:7002 ssl crt /etc/ssl/serverkey.pem
    stats enable
    stats uri /

frontend ozone_s3g_http_in
    bind *:9879 ssl crt /etc/ssl/serverkey.pem
    default_backend ozone_s3g_http_servers

backend ozone_s3g_http_servers
    balance roundrobin
    server server1 test-adh-ozone1.ru-central1.internal:9879 ssl check inter 1s fall 2 rise 5
    server server2 test-adh-ozone2.ru-central1.internal:9879 ssl check inter 1s fall 2 rise 5
    server server3 test-adh-ozone3.ru-central1.internal:9879 ssl check inter 1s fall 2 rise 5

frontend ozone_s3g_webui_http_in
    bind *:19879 ssl crt /etc/ssl/serverkey.pem
    default_backend ozone_s3g_webui_http_servers

backend ozone_s3g_webui_http_servers
    balance roundrobin
    server server1 test-adh-ozone1.ru-central1.internal:19879 ssl check inter 1s fall 2 rise 5
    server server2 test-adh-ozone2.ru-central1.internal:19879 ssl check inter 1s fall 2 rise 5
    server server3 test-adh-ozone3.ru-central1.internal:19879 ssl check inter 1s fall 2 rise 5

For more details on HAProxy configuration, see the HAProxy reference.

Ozone S3G configuration

To enable the Web UI and the REST API endpoint for retrieving secrets, configure one of the following parameters in the /etc/ozone/conf/ozone-site.xml configuration file for Ozone S3G.

Property Description Default value

ozone.s3g.webadmin.http-address

HTTP address to use for the S3 Gateway Web UI and to get secrets via REST API

0.0.0.0:19878

ozone.s3g.webadmin.https-address

HTTPS address to use for the S3 Gateway Web UI and to get secrets via REST API

0.0.0.0:19879

To modify the configuration via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. Navigate to the Components tab and select Ozone S3G.

  4. Select the required parameter and make the necessary changes.

  5. Confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Specifics for Kerberos

The Ozone S3 Gateway web UI provides a REST API endpoint for obtaining S3 secrets (if Kerberos/SPNEGO is enabled).

To enable Kerberos authentication:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. Navigate to the Components tab and select Ozone S3G.

  4. Check the Advanced flag and click Add property in the Custom ozone-site.xml section.

  5. Add the ozone.s3g.secret.http.enabled with the value True and confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Only administrators can get secrets via REST API. You can add an Ozone administrator in the ozone.administrators parameter in the ozone-site.xml section in ADCM. A user who has the administrator role will automatically have S3 administrative privileges.

If you want to add an administrator role specifically for Ozone S3G, define the ozone.s3.administrators parameter using the Custom ozone-site.xml section in the Ozone S3G component configuration.

To give administrative access to a group of users, fill in the ozone.administrators.groups or ozone.s3.administrators.groups parameter respectively.

As a value for all parameters, specify a comma-separated list of user names or group names who have administrative access.

To add an administrator or an administrative group via ADCM:

  • For Ozone

  • For Ozone S3G

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. In the ozone-site.xml section, fill in the values in the ozone.administrators or ozone.administrators.groups parameter.

  4. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. Navigate to the Components tab and select Ozone S3G.

  4. Check the Advanced flag and click Add property in the Custom ozone-site.xml section.

  5. Add an ozone.s3.administrators or ozone.s3.administrators.groups parameter, fill in the values and confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Example authentication request:

$ curl --insecure -X PUT --negotiate -u : https://<haproxy_host>:19879/secret/<username>

where:

  • <haproxy_host> — HAProxy gateway host;

  • <username> — the name of the user for authentication.

Example response:

<S3Secret>
  <awsAccessKey>user</awsAccessKey>
  <awsSecret>***</awsSecret>
</S3Secret>

Logging

By default, logs are stored in /var/log/adh-haproxy/ozone-s3g.log. The logging behavior is defined in the global section of haproxy-ozone-s3g.cfg.

Rsyslog configuration example:

$AddUnixListenSocket /var/lib/adh-haproxy/ozone-s3g/dev/log

:programname, startswith, "haproxy-ozone-s3g" {
/var/log/adh-haproxy/ozone-s3g.log
stop
}

To edit the Rsyslog configuration via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click on Ozone.

  3. Navigate to the Components tab and select HAProxy Ozone S3G.

  4. Select the /etc/rsyslog.d/haproxy-ozone-s3g.conf parameter and make the necessary changes.

  5. Confirm changes by clicking Save.

  6. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Found a mistake? Seleсt text and press Ctrl+Enter to report it