Requirements for SSL certificates

To implement encryption between the nodes the following is required:

  1. Create a certificate for each node using the RSA algorithm. The key length should be 2048 bit, CN should be the FQDN of the corresponding node. It is not recommended to use the DSA algorithm. It is also not recommended to use the ECDSA algorithm due to this algorithm not being widely supported by certification centers. The key length that equals 2048 bits is necessary and sufficient to implement secure encryption.

  2. Create the trustore.jks keystore file that will store the certificates. The path to this file should be specified for SSL to be started.

    NOTE
    • Certificates should be signed by the key that belongs to one of the trusted root certificates included into the Java certificate storage.

    • Self-signed certificates are allowed. In this case, the root certificate should be added to the trustore.jks storage.

    • Nginx requires OpenSSL certificates. For a cluster that uses Nginx (with Hive Tez UI or Airflow), generate an OpenSSL key and a certificate.

    • The OpenSSL certificate for your host name should be added to the ca-bundle.crt storage.

    • The account used for the installation of certificates should have the rights to write to the following paths:

      • /etc/pki/tls/certs/

      • /etc/pki/java/

    • Make sure to use an FQDN as a key alias in keystore.jks, not a short name.

  3. Enable the traffic between the nodes according to the table below.

    The script below creates all the necessary entities and can be used as a reference.

    SSL preparation script
    #!/bin/bash
    set -euo pipefail
    
    echo "# SSL preparation script started"
    
    if [ -f ./crtdistr-config.cfg ]; then
      echo "## Config found"
    else
      echo "## Error: configuration file ./crtdistr-config.cfg not found"
      exit 1
    fi
    
    
    HOSTS_MAKE="${HOSTS-}"
    
    source ./crtdistr-config.cfg
    
    if [ -n "$HOSTS_MAKE" ]; then
      echo "## HOSTS provided from MakeFile"
      read -r -a HOSTS <<< "$HOSTS_MAKE"
    fi
    
    SSH_HOSTS=()
    SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=error'
    TMP_DIR="/tmp/crtdistr"
    CA_KEY="$TMP_DIR/ca_admprom.key"
    CA_CERT="$TMP_DIR/ca_certs.crt"
    CA_ALIAS="root-ca"
    
    get_cert_paths() {
      local host="$1"
    
      if ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'astra'; then
        CA_CERTS_PATH="/usr/local/share/ca-certificates/"
        CA_UPDATE_CMD="update-ca-certificates"
        HOST_CERT_NAME="host_cert.crt"
        JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
        JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
      elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'ubuntu'; then
        CA_CERTS_PATH="/usr/local/share/ca-certificates"
        CA_UPDATE_CMD="update-ca-certificates"
        HOST_CERT_NAME="host_cert.crt"
        JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
        JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
      elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'redos'; then
        CA_CERTS_PATH="/etc/pki/ca-trust/source/anchors"
        CA_UPDATE_CMD="update-ca-trust"
        HOST_CERT_NAME="host_cert.cert"
        JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
        JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
      else
        CA_CERTS_PATH=""
        CA_UPDATE_CMD=""
        HOST_CERT_NAME=""
        JAVA_CA_PATH=""
        JAVA_CA_PATH_17=""
      fi
    }
    
    
    echo "# Pre-check"
    for HOST in "${HOSTS[@]}"; do
      echo "## Checking ssh connection to host $HOST"
      if ssh $SSH_OPTS -q "$HOST" "exit"; then
        get_cert_paths "$HOST"
        if [[ -z "$CA_CERTS_PATH" ]]; then
          echo "### Unknown OS type on $HOST, skipped"
        elif [[ -z "$JAVA_CA_PATH" ]]; then
          echo "### Java cacerts path not detected on $HOST, skipped"
        else
          echo "### $HOST is OK"
          SSH_HOSTS+=("$HOST")
        fi
      else
        echo "### SSH connection failed for $HOST"
      fi
    done
    
    if [[ "${#SSH_HOSTS[@]}" -eq 0 ]]; then
      echo "## No reachable hosts found"
      exit 1
    fi
    
    rm -rf "$TMP_DIR"
    mkdir -p "$TMP_DIR"
    
    echo "# Creating local CA"
    openssl genrsa -out "$CA_KEY" 4096
    openssl req -x509 \
      -nodes \
      -key "$CA_KEY" \
      -days 1095 \
      -out "$CA_CERT" \
      -subj "/CN=marsnet.local CA"
    
    echo "# Creating server certificates"
    for SSH_HOST in "${SSH_HOSTS[@]}"; do
      echo "## Generate key and CSR for $SSH_HOST"
      openssl req \
        -newkey rsa:4096 \
        -nodes \
        -keyout "$TMP_DIR/$SSH_HOST.key" \
        -out "$TMP_DIR/$SSH_HOST.host.csr" \
        -subj "/C=RU/ST=Denial/L=MSK/O=AD/CN=$SSH_HOST"
    
      echo "## Sign certificate for $SSH_HOST"
      openssl x509 \
        -req \
        -in "$TMP_DIR/$SSH_HOST.host.csr" \
        -CA "$CA_CERT" \
        -CAkey "$CA_KEY" \
        -CAcreateserial \
        -out "$TMP_DIR/$SSH_HOST.crt" \
        -days 365 \
        -extfile <(printf "basicConstraints=critical,CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=DNS:$SSH_HOST\n")
    done
    
    echo "# Distributing certificates"
    for SSH_HOST in "${SSH_HOSTS[@]}"; do
      echo "## Copy files to $SSH_HOST"
      scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.crt" "$SSH_HOST:/tmp/"
      scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.key" "$SSH_HOST:/tmp/"
      scp $SSH_OPTS "$CA_CERT" "$SSH_HOST:/tmp/"
    
      for node in "${SSH_HOSTS[@]}"; do
        if [[ "$SSH_HOST" != "$node" ]]; then
          scp $SSH_OPTS "$TMP_DIR/$node.crt" "$SSH_HOST:/tmp/$node.crt"
        fi
      done
    done
    
    echo "# Configuring remote hosts"
    for SSH_HOST in "${SSH_HOSTS[@]}"; do
      echo "## Install certificate and build keystore/truststore on $SSH_HOST"
      get_cert_paths "$SSH_HOST"
    
      ssh $SSH_OPTS "$SSH_HOST" "
        set -eo pipefail
        export KEYSTOREPASS='$KEYSTOREPASS'
        export TRUSTSTOREPASS='$TRUSTSTOREPASS'
    
        sudo rm -f /etc/ssl/host.pem /etc/ssl/host.key /etc/ssl/ca_certs.crt
    
        sudo mkdir -p /etc/ssl
    
        sudo cp /tmp/$SSH_HOST.crt /etc/ssl/host.pem
        sudo cp /tmp/$SSH_HOST.key /etc/ssl/host.key
        sudo cp /tmp/ca_certs.crt /etc/ssl/ca_certs.crt
    
        set +u
        source /usr/lib/bigtop-utils/bigtop-detect-javahome
        set -u
    
        sudo \$JAVA_HOME/bin/keytool -delete \
          -alias $SSH_HOST \
          -keystore $KEYSTOREPATH \
          -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true
    
        sudo \$JAVA_HOME/bin/keytool -delete \
          -alias $CA_ALIAS \
          -keystore $KEYSTOREPATH \
          -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true
    
        sudo \$JAVA_HOME/bin/keytool -delete \
          -alias $CA_ALIAS \
          -keystore $TRUSTSTOREPATH \
          -storepass \$TRUSTSTOREPASS >/dev/null 2>&1 || true
    
        sudo cp /etc/ssl/host.pem $CA_CERTS_PATH/$HOST_CERT_NAME
        sudo cp /tmp/ca_certs.crt $CA_CERTS_PATH/ca_certs.crt
        sudo $CA_UPDATE_CMD
        sudo chmod 644 /etc/ssl/host.key
    
        openssl pkcs12 -export \
          -in /tmp/$SSH_HOST.crt \
          -inkey /tmp/$SSH_HOST.key \
          -certfile /tmp/ca_certs.crt \
          -name $SSH_HOST \
          -out /tmp/$SSH_HOST.p12 \
          -password pass:\$KEYSTOREPASS
    
        sudo \$JAVA_HOME/bin/keytool -importkeystore \
          -srckeystore /tmp/$SSH_HOST.p12 \
          -srcstoretype PKCS12 \
          -srcstorepass \$KEYSTOREPASS \
          -destkeystore $KEYSTOREPATH \
          -deststoretype JKS \
          -deststorepass \$KEYSTOREPASS \
          -alias $SSH_HOST \
          -noprompt
    
        sudo \$JAVA_HOME/bin/keytool -import \
          -noprompt \
          -alias $CA_ALIAS \
          -file /tmp/ca_certs.crt \
          -keystore $KEYSTOREPATH \
          -storepass \$KEYSTOREPASS \
          -trustcacerts
    
        sudo \$JAVA_HOME/bin/keytool -import \
          -noprompt \
          -alias $CA_ALIAS \
          -file /tmp/ca_certs.crt \
          -keystore $TRUSTSTOREPATH \
          -storepass \$TRUSTSTOREPASS \
          -trustcacerts
    
        #For Haproxy
        sudo bash -c 'cat /etc/ssl/host.key /etc/ssl/host.pem /etc/ssl/ca_certs.crt > /etc/ssl/serverkey.pem'
        sudo chmod 600 /etc/ssl/serverkey.pem
      "
    
      for node in "${SSH_HOSTS[@]}"; do
        ssh $SSH_OPTS "$SSH_HOST" "
          set +u
          source /usr/lib/bigtop-utils/bigtop-detect-javahome
          set -u
    
          sudo \$JAVA_HOME/bin/keytool -delete \
            -alias $node \
            -keystore $TRUSTSTOREPATH \
            -storepass $TRUSTSTOREPASS >/dev/null 2>&1 || true
    
          sudo \$JAVA_HOME/bin/keytool -import \
            -noprompt \
            -alias $node \
            -file /tmp/$node.crt \
            -keystore $TRUSTSTOREPATH \
            -storepass $TRUSTSTOREPASS \
            -trustcacerts
        "
      done
    
      #Import trustore to JKS
      ssh $SSH_OPTS "$SSH_HOST" "
        set -eo pipefail
    
        set +u
        source /usr/lib/bigtop-utils/bigtop-detect-javahome
        set -u
    
        sudo \$JAVA_HOME/bin/keytool -importkeystore \
          -noprompt \
          -srckeystore $TRUSTSTOREPATH \
          -srcstorepass $TRUSTSTOREPASS \
          -destkeystore $JAVA_CA_PATH_17 \
          -deststorepass changeit || true
    
        sudo \$JAVA_HOME/bin/keytool -importkeystore \
          -noprompt \
          -srckeystore $TRUSTSTOREPATH \
          -srcstorepass $TRUSTSTOREPASS \
          -destkeystore $JAVA_CA_PATH \
          -deststorepass changeit || true
    
        sudo rm -f /tmp/$SSH_HOST.p12 /tmp/*.crt /tmp/*.key
      " || {
        echo "Failed to configure certs on $SSH_HOST"
        exit 1
      }
    done
    
    echo "# Cleanup local temp files"
    rm -rf "$TMP_DIR"
    
    echo "# All done successfully!"
Matching between ports and HTTP/HTTPS services
Service Component/Server Port Protocol Description

Airflow

Server

8080/8080

HTTP/HTTPS

Airflow Server Web UI port

Server/Worker

8793/8793

HTTP/HTTPS

Celery Worker API port

Server/Flower

5555/5555

HTTP/HTTPS

Flower Web UI port

Flink

Job Manager

8081/8081

HTTP/HTTPS

Flink Job Manager Web UI port

HBase

HBase Region

16030/16030

HTTP/HTTPS

HBase Region Server Web UI port

Phoenix Query Server

8765/8765

HTTP/HTTPS

API port

HBase REST Server

60080/60080

HTTP/HTTPS

API port

HBase REST Server

8085/8085

HTTP/HTTPS

Web UI Port

HBase Master

16010/16010

HTTP/HTTPS

HBase Master Web UI port

HBaseThrift2Server

Trift2 Server

9095/9095

HTTP/HTTPS

Thrift2 Server Web UI port

HDFS

NameNode

9870/9871

HTTP/HTTPS

NameNode Web UI port

DataNode

9864/9865

HTTP/HTTPS

DataNode Web UI port

JournalNode

8480/8481

HTTP/HTTPS

JournalNode Web UI port

HTTPfs server

14000/14000

HTTP/HTTPS

HTTPfs API port

Monitoring

Grafana

11210

HTTP/HTTPS

Grafana Web UI port

Node Exporter

11203

HTTP/HTTPS

Node Exporter listen port

Prometheus Server

11200

HTTP/HTTPS

Prometheus Web UI port

Hive

Hive Server

10002/10002

HTTP/HTTPS

Hive Server Web UI port

Tez

9999/9999

HTTP/HTTPS

Tez Web UI port

Knox

Knox Gateway

8443

HTTPS

Gateway port

Ranger

Ranger Admin

6080/6182

HTTP/HTTPS

Ranger Admin web UI and API port

Ranger KMS

9292/9393

HTTP/HTTPS

Port for Ranger KMS

Solr

Solr Server

8983/8985

HTTP/HTTPS

Solr Server Web UI and API port

Spark

History Server

18082/18082

HTTP/HTTPS

HS Web UO port

Thrift Server

4040/4040

HTTP/HTTPS

Thrift Server Web UI

Livy Server

8998/8998

HTTP/HTTPS

Livy Server Web UI port

YARN

Resource Manager

8088/8090

HTTP/HTTPS

RM Web UI port

Node Manager

8042/8044

HTTP/HTTPS

NM Web UI port

MapReduce History Server

19888/19890

HTTP/HTTPS

HS Web UI port

Timeline Server

8188/8190

HTTP/HTTPS

TS Web UI port

Zeppelin

Server

8180/8180

HTTP/HTTPS

Zeppelin Web UI port

Found a mistake? Seleсt text and press Ctrl+Enter to report it