Enable SSL encryption

Elena Dvoryadkina

You can use the SSL encryption protocol to secure communication between an ADQM server and client, as well as between servers in a cluster. This article provides instructions on how to configure ADQM to enable SSL certificate validation for incoming connections and use secure protocols for interserver communication. The example uses an ADQM cluster that includes the following hosts:

  • 10.92.17.146, adqm-host-1.arenadata.local;

  • 10.92.17.106, adqm-host-2.arenadata.local;

  • 10.92.17.140, adqm-host-3.arenadata.local;

  • 10.92.16.245, adqm-host-4.arenadata.local.

NOTE

In this article, a self-signed CA certificate is used to create server SSL certificates for demonstration and testing purposes. For production systems, use a trusted certificate authority to sign certificates.

Create SSL certificates

To run commands below, you need openssl installed (minimum version 1.1.1) — it is a command-line tool for using functions of the OpenSSL cryptographic library.

  1. Generate a private key:

    $ openssl genrsa -out arenadata_ca.key 2048

    This key will be used to create a CA certificate and to sign other certificates (server certificates and certificates for user authentication), so keep it in a secure place.

  2. Create a self-signed CA certificate:

    $ openssl req \
  -x509 \
  -subj "/CN=arenadata.local CA" \
  -nodes \
  -key arenadata_ca.key \
  -days 1095 \
  -out arenadata_ca.crt

  3. For each ADQM server, generate a key and create a certificate signing request (CSR):

    $ openssl req -newkey rsa:2048 -nodes -subj "/CN=adqm-host-1.arenadata.local" -addext "subjectAltName = DNS:adqm-host-1.arenadata.local,IP:10.92.17.146" -keyout adqm-host-1.key -out adqm-host-1.csr
$ openssl req -newkey rsa:2048 -nodes -subj "/CN=adqm-host-2.arenadata.local" -addext "subjectAltName = DNS:adqm-host-2.arenadata.local,IP:10.92.17.106" -keyout adqm-host-2.key -out adqm-host-2.csr
$ openssl req -newkey rsa:2048 -nodes -subj "/CN=adqm-host-3.arenadata.local" -addext "subjectAltName = DNS:adqm-host-3.arenadata.local,IP:10.92.17.140" -keyout adqm-host-3.key -out adqm-host-3.csr
$ openssl req -newkey rsa:2048 -nodes -subj "/CN=adqm-host-4.arenadata.local" -addext "subjectAltName = DNS:adqm-host-4.arenadata.local,IP:10.92.16.245" -keyout adqm-host-4.key -out adqm-host-4.csr

    The CN (Common Name) certificate identifier should be the fully qualified domain name (FQDN) of a server for which the certificate is being created. Otherwise, the certificate cannot be verified.

  4. For each ADQM server, create and sign a certificate using the private key and the CA certificate:

    $ openssl x509 -req -in adqm-host-1.csr -out adqm-host-1.crt -CAcreateserial -CA arenadata_ca.crt -CAkey arenadata_ca.key -days 365
$ openssl x509 -req -in adqm-host-2.csr -out adqm-host-2.crt -CAcreateserial -CA arenadata_ca.crt -CAkey arenadata_ca.key -days 365
$ openssl x509 -req -in adqm-host-3.csr -out adqm-host-3.crt -CAcreateserial -CA arenadata_ca.crt -CAkey arenadata_ca.key -days 365
$ openssl x509 -req -in adqm-host-4.csr -out adqm-host-4.crt -CAcreateserial -CA arenadata_ca.crt -CAkey arenadata_ca.key -days 365

  5. For each ADQM host, copy the CA certificate, as well as the certificate and key of the corresponding server. These files should be stored in the same path and named the same on each host of the cluster — paths and file names of the CA certificate, server certificate and server key are specified in cluster settings (see the SSL parameters in service configurations section below).

     
    For example, copy files to the adqm-host-1.arenadata.local host as follows:

    • arenadata_ca.crt to /etc/clickhouse-server/ca.crt;

    • adqm-host-1.crt to /etc/clickhouse-server/server.crt;

    • adqm-host-1.key to /etc/clickhouse-server/server.key.

    The /etc/clickhouse-server/server.crt and /etc/clickhouse-server/server.key files are set by default in the Server certificate file and Server private key file parameters of the ADQMDB service, respectively. If the certificate and key files on your ADQM hosts have different names or are located in a different directory, specify the corresponding paths in these parameters.

    The self-signed CA certificate should be specified in one of the following ways:

    • Pass the path and file name of the CA certificate to the Certificate authority file parameter of the ADQMDB service configuration (see the example below), which has an empty value by default.

    • Add the CA certificate to trusted certificates. For example, to do this on CentOS, perform the following commands:

      $ sudo cp ./arenadata_ca.crt /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

Configure ADQM to enable SSL encryption

You can enable SSL encryption of connections in ADQM using the ADCM interface, which provides the following functionality for this:

  • Manage SSL cluster action — allows you to enable SSL for multiple ADQM services at once, as well as disable ports for unsecured connections.

  • SSL settings in configurations of ADQM services — allow you to configure SSL for each service separately (for example, you can change the default paths to the certificate and key files, enable client certificate verification, specify the CA certificate, etc.).

Manage SSL cluster action

  1. On the Clusters page, click the icon actions default dark actions default light in the ADQM cluster row to open the list of cluster actions and select Manage SSL.

    Launch the Manage SSL action
    Launch the "Manage SSL" action

    ADCM will display a dialog window. When you turn on the Enable SSL switch, it provides options allowing you to specify which ADQM services should use secure connections.

    Options for enabling SSL in the Manage SSL action configuration window
    Options for enabling SSL in the "Manage SSL" action configuration window

  2. Fill in the required parameters in the Manage SSL action configuration window:

    • Parameters in the Enable SSL section — to enable SSL encryption of incoming and internal connections for various ADQM services.

      Parameters to enable SSL for services

       

      Parameter values ​​specified in the Manage SSL action window overwrite the values ​​of the corresponding parameters in service configurations after the action execution — see the ADQM service parameter column in the table below.

      Manage SSL parameter Parameter description ADQM service parameter

      [ADQMDB] → Enable HTTP

      Allows you to disable the port for connecting to ADQM over HTTP (to use HTTPS only)

      ADQMDB service: Network → Enable http parameter

      [ADQMDB] → Enable TCP

      Allows you to disable the non-secure TCP port so that only the secure TCP port (SSL Configuration → TCP secure port) is used

      ADQMDB service: Network → Enable tcp parameter

      [ADQMDB] → Secure gRPC

      Enables SSL for secure communication with clients via gRPC

      ADQMDB service: Enable gRPC → Enable SSL parameter

      [ADQMDB] → Secure intra-cluster communication

      If this parameter is enabled, only secure protocols (HTTPS, secure TCP) will be used to exchange data between ClickHouse servers

      ADQMDB service: Intra-cluster communication security → Enable secure intra-cluster communication parameter

      [Zookeeper] → Secure Incoming Connections

      Enables secure incoming connections in ZooKeeper

      Zookeeper service: SSL configuration → Enable SSL parameter

      [Zookeeper] → Secure SSL Quorum

      Enables encrypted quorum communication in ZooKeeper

      Zookeeper service: SSL configuration → sslQuorum parameter

      [Chproxy] → Secure Incoming Connections

      Enables secure incoming connections in Chproxy

      Chproxy service: Network configuration → Enable HTTPS parameter

      [Chproxy] → Use HTTP to connect to Clickhouse if possible

      Specifies which protocol (HTTP or HTTPS) Chproxy will use to connect to ClickHouse

      Chproxy service: Network configuration → Use HTTP to connect to ADQMDB parameter

    • The Run service checks parameter — to specify whether to check the cluster health and availability after applying the settings.

  3. To launch the action, click Next and then Run in the confirmation window.

During the Manage SSL action execution, ADCM stops all services one by one, reconfigures them to enable or disable SSL encryption, then starts the services again and checks them.

You can view the result of the action and its execution process on the Jobs page, which displays the execution details for all internal steps of actions, including verbose Ansible stdout/stderr outputs.

SSL parameters in service configurations

An alternative way to set up secure connections in ADQM is to configure SSL settings for each service separately.

  1. Open the Services tab of your ADQM cluster and select the service for which you want to enable SSL encryption — ADQMDB, Zookeeper, or Chproxy.

  2. On the Primary configuration tab, configure the SSL parameters — the name of the section where these parameters are located depends on the service.

    ADQMDB: SSL Configuration

     
    In the SSL Configuration section of the ADQMDB service configuration, the following parameters are available:

    • Enable SSL — enables HTTPS and TCP secure ports for SSL-protected connections.

    • HTTPS port — port for secure communication via HTTPS (default is 8443).

    • TCP secure port — TCP port for secure communication with clients and inter-server communication through the binary protocol (default is 9440).

    • Disable protocols — protocols that are not allowed to be used (for example, sslv2, sslv3).

    • Server certificate file — path to the server SSL certificate file in the PEM format.

    • Server private key file — path to the file with the private key of the server SSL certificate.

    • Certificate authority file — path to the file with the trusted CA certificate in the PEM format used to verify client certificates. It is required if you need to enable TLS authentication.

    • Client certificate verification mode — method of checking user SSL certificates signed by the CA certificate (the CA certificate file should be specified in the Certificate authority file field).

    • Client certificate file, Client private key file, Client certificate authority file — parameters to configure a secure connection between ClickHouse and ZooKeeper.

    ADQMDB service configuration parameters to check SSL certificates of connections
    ADQMDB service configuration parameters to check SSL certificates of connections
    ADQMDB: Enable gRPC

     

    If the Enable SSL (in the SSL Configuration section) and Enable gRPC options are enabled for the ADQMDB service, you can also configure the following SSL settings for gRPC connections in the Enable gRPC section:

    • Enable SSL — enables SSL for secure communication with clients via gRPC.

    • SSL certificate file — path to the file with the SSL certificate in the PEM format for connections via gRPC.

    • SSL private key file — path to the file with the secret key of the SSL certificate for connections via gRPC.

    • Client auth required — indicates whether the server should request a client for a certificate.

    • Certificate authority file — path to the file with the trusted CA certificate in the PEM format that will be used to verify client certificates. It should be specified if Client auth required is enabled.

    ADQMDB service configuration parameters for SSL connections via gRPC
    ADQMDB service configuration parameters for SSL connections via gRPC
    ADQMDB: Network

     
    In the Network section of the ADQMDB service configuration, you can find options to disable non-secure ports:

    • Enable http — allows you to disable the port for connections via the HTTP protocol (in favor of connections via the HTTPS port configured by the SSL Configuration → HTTPS port parameter).

    • Enable tcp — allows you to disable the port for non-secure connections via the ClickHouse TCP protocol (in favor of the SSL Configuration → TCP secure port port).

    ADQMDB: Secure intra-cluster communication

     

    In the Secure intra-cluster communication section of the ADQMDB service configuration, there is the Enable secure intra-cluster communication parameter that enables the use of only secure protocols (HTTPS, secure TCP) for data exchange between ClickHouse servers. To activate the parameter, turn on also SSL Configuration → Enable SSL.

    Zookeeper: SSL configuration

     

    The SSL configuration section of the Zookeeper service configuration (available if the Show advanced option is enabled) contains parameters to set up SSL connections between ClickHouse and ZooKeeper:

    • Enable SSL — enables secure incoming connections in ZooKeeper;

    • sslQuorum — enables encrypted quorum communication;

    • serverCnxnFactory — specifies ServerCnxnFactory implementation (to use TLS-based server communication, set it to NettyServerCnxnFactory);

    • ssl.quorum.keyStore.location — path to the server keystore file;

    • ssl.quorum.keyStore.password — password for the keystore specified in ssl.quorum.keyStore.location;

    • ssl.quorum.trustStore.location — path to the server truststore file;

    • ssl.quorum.trustStore.password — password for the truststore specified in ssl.quorum.trustStore.location;

    • ssl.keyStore.location — path to the server keystore file to be used for client TLS connections;

    • ssl.keyStore.password — password for the keystore specified in ssl.keyStore.location;

    • ssl.trustStore.location — path to the server truststore file to be used for client TLS connections;

    • ssl.trustStore.password — password for the truststore specified in ssl.trustStore.location;

    • ssl.protocol — protocol to be used in client TLS negotiation;

    • ssl.quorum.protocol — protocol to be used in quorum TLS negotiation;

    • secureClientPort — port to be used for secure client connections.

    Zookeeper service configuration parameters for SSL connections
    Zookeeper service configuration parameters for SSL connections
    Chproxy: Network configuration

     

    The Network configuration section of the Chproxy service configuration includes the following parameters to set up secure connections:

    • Enable HTTPS — enables listening for HTTPS requests;

    • HTTPS listen port — port to listen for HTTPS requests;

    • HTTPS allowed networks — list of networks from which HTTPS access is allowed;

    • Certificate file — path to the server SSL certificate file in the PEM format;

    • Private key file — path to the file with the private key of the server SSL certificate;

    • Use HTTP to connect to ADQMDB — specifies which protocol (HTTP or HTTPS) Chproxy should use to connect to ClickHouse.

    Chproxy service configuration parameters for SSL connections
    Chproxy service configuration parameters for SSL connections

  3. After specifying all the necessary parameters for configuring SSL connections, click Save and run Reconfig and restart for a service to apply the changes.

Example

  1. In the Certificate authority file parameter of the ADQMDB service configuration, specify the path to the self-signed CA certificate file — /etc/clickhouse-server/ca.crt.

    SSL settings for the ADQMDB service
    SSL settings for the ADQMDB service

    Click Save.

  2. Run the Manage SSL action for the cluster. In the configuration window that appears, activate Enable SSL and also set Secure intra-cluster communication to true to encrypt internal traffic between ClickHouse servers.

    Enable SSL in ADQM via the Manage SSL cluster action
    Enable SSL in ADQM via the "Manage SSL" cluster action

    Click Next and then Run to start the action execution.

Test connection

Connect via clickhouse-client

To ensure that the connection to ADQM is secured with the SSL encryption, you can connect to any server of the ADQM cluster via clickhouse-client with the --secure flag, specifying the TCP port for secure connection over SSL (9440). For example:

$ clickhouse-client --host adqm-host-1.arenadata.local --secure --port 9440 --user default

The output confirms the connection on the 9440 secure port:

ClickHouse client version 24.8.8.17.
Connecting to adqm-host-1.arenadata.local:9440 as user default.
Connected to ClickHouse server version 24.8.8.

Note that the Client certificate verification mode option in the SSL Configuration section of the ADQMDB service configuration should be disabled (set to none) as this test connection does not use the SSL certificate user authentication.

Check interserver connections

Run a distributed query, for example, that creates a new ADQM user on each host:

CREATE USER test_user ON CLUSTER default_cluster IDENTIFIED WITH sha256_password BY 'qwerty';

The query output shows that the communication between ADQM hosts also occurs via the 9440 port:

   ┌─host────────────────────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
1. │ adqm-host-4.arenadata.local │ 9440 │      0 │       │                   3 │                0 │
2. │ adqm-host-1.arenadata.local │ 9440 │      0 │       │                   2 │                0 │
3. │ adqm-host-2.arenadata.local │ 9440 │      0 │       │                   1 │                0 │
4. │ adqm-host-3.arenadata.local │ 9440 │      0 │       │                   0 │                0 │
   └─────────────────────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

Connect via HTTPS

To test the connection to ADQM over HTTPS, you can run the following command on the cluster host:

$ echo 'SHOW TABLES' | curl 'https://adqm-host-1.arenadata.local:8443/?query=' --data-binary @- --cacert arenadata_ca.crt

If the connection is successful, the query will return a list of tables in the default database on the adqm-host-1.arenadata.local host.

Disable SSL

To disable SSL, run the Manage SSL action with the Enable SSL switch inactive or turn off the Enable SSL option in the configuration of a service.

Disable SSL with the Manage SSL action
Disable SSL with the "Manage SSL" action
Found a mistake? Seleсt text and press Ctrl+Enter to report it