Home
Arenadata Hyperwave
Cluster-level operations
Security
SSL
Requirements for SSL certificates

Requirements for SSL certificates

Mikhail Serov

Collapse content Expand content

Contents

Permissions for keystore files
HTTPS ports usage

To implement encryption between ADH nodes the following is required:

Create a certificate for each node using the RSA algorithm. The key length should be 2048 bit, CN should be the FQDN of the corresponding node. It is not recommended to use the DSA algorithm. It is also not recommended to use the ECDSA algorithm due to this algorithm not being widely supported by certification centers. The key length that equals 2048 bits is necessary and sufficient to implement secure encryption.

Create the trustore.jks keystore file that will store the certificates. The path to this file should be specified for SSL to be started.

NOTE

Certificates should be signed by the key that belongs to one of the trusted root certificates included into the Java certificate storage.
Self-signed certificates are allowed. In this case, the root certificate should be added to the trustore.jks storage.
Nginx requires SSL certificates different from Java-based. For a cluster that uses Nginx (with Hive Tez UI or Airflow), generate a non-Java SSL key and a certificate. The SSL certificate for your host name should be added to the trusted system store.
The account used for the installation of certificates should have the rights to write to the following paths:
- /etc/pki/tls/certs/
- /etc/pki/java/
Make sure to use an FQDN as a key alias in keystore.jks, not a short name.

Enable the traffic between the nodes according to the table below.

The script below creates all the necessary entities and can be used as a reference.

SSL preparation script

#!/bin/bash
set -euo pipefail

echo "# SSL preparation script started"

if [ -f ./crtdistr-config.cfg ]; then
  echo "## Config found"
else
  echo "## Error: configuration file ./crtdistr-config.cfg not found"
  exit 1
fi

HOSTS_MAKE="${HOSTS-}"

source ./crtdistr-config.cfg

if [ -n "$HOSTS_MAKE" ]; then
  echo "## HOSTS provided from MakeFile"
  read -r -a HOSTS <<< "$HOSTS_MAKE"
fi

SSH_HOSTS=()
SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=error'
TMP_DIR="/tmp/crtdistr"
CA_KEY="$TMP_DIR/ca_admprom.key"
CA_CERT="$TMP_DIR/ca_certs.crt"
CA_ALIAS="root-ca"

get_cert_paths() {
  local host="$1"

  if ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'astra'; then
    CA_CERTS_PATH="/usr/local/share/ca-certificates/"
    CA_UPDATE_CMD="update-ca-certificates"
    HOST_CERT_NAME="host_cert.crt"
    JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
    JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
  elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'ubuntu'; then
    CA_CERTS_PATH="/usr/local/share/ca-certificates"
    CA_UPDATE_CMD="update-ca-certificates"
    HOST_CERT_NAME="host_cert.crt"
    JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
    JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
  elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'rhel|centos|altlinux|redos'; then
    CA_CERTS_PATH="/etc/pki/ca-trust/source/anchors"
    CA_UPDATE_CMD="update-ca-trust"
    HOST_CERT_NAME="host_cert.cert"
    JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts"
    JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts"
  else
    CA_CERTS_PATH=""
    CA_UPDATE_CMD=""
    HOST_CERT_NAME=""
    JAVA_CA_PATH=""
    JAVA_CA_PATH_17=""
  fi
}

echo "# Pre-check"
for HOST in "${HOSTS[@]}"; do
  echo "## Checking ssh connection to host $HOST"
  if ssh $SSH_OPTS -q "$HOST" "exit"; then
    get_cert_paths "$HOST"
    if [[ -z "$CA_CERTS_PATH" ]]; then
      echo "### Unknown OS type on $HOST, skipped"
    elif [[ -z "$JAVA_CA_PATH" ]]; then
      echo "### Java cacerts path not detected on $HOST, skipped"
    else
      echo "### $HOST is OK"
      SSH_HOSTS+=("$HOST")
    fi
  else
    echo "### SSH connection failed for $HOST"
  fi
done

if [[ "${#SSH_HOSTS[@]}" -eq 0 ]]; then
  echo "## No reachable hosts found"
  exit 1
fi

rm -rf "$TMP_DIR"
mkdir -p "$TMP_DIR"

echo "# Creating local CA"
openssl genrsa -out "$CA_KEY" 4096
openssl req -x509 \
  -nodes \
  -key "$CA_KEY" \
  -days 1095 \
  -out "$CA_CERT" \
  -subj "/CN=marsnet.local CA"

echo "# Creating server certificates"
for SSH_HOST in "${SSH_HOSTS[@]}"; do
  echo "## Generate key and CSR for $SSH_HOST"
  openssl req \
    -newkey rsa:4096 \
    -nodes \
    -keyout "$TMP_DIR/$SSH_HOST.key" \
    -out "$TMP_DIR/$SSH_HOST.host.csr" \
    -subj "/C=RU/ST=Denial/L=MSK/O=AD/CN=$SSH_HOST"

  echo "## Sign certificate for $SSH_HOST"
  openssl x509 \
    -req \
    -in "$TMP_DIR/$SSH_HOST.host.csr" \
    -CA "$CA_CERT" \
    -CAkey "$CA_KEY" \
    -CAcreateserial \
    -out "$TMP_DIR/$SSH_HOST.crt" \
    -days 365 \
    -extfile <(printf "basicConstraints=critical,CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=DNS:$SSH_HOST\n")
done

echo "# Distributing certificates"
for SSH_HOST in "${SSH_HOSTS[@]}"; do
  echo "## Copy files to $SSH_HOST"
  scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.crt" "$SSH_HOST:/tmp/"
  scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.key" "$SSH_HOST:/tmp/"
  scp $SSH_OPTS "$CA_CERT" "$SSH_HOST:/tmp/"

  for node in "${SSH_HOSTS[@]}"; do
    if [[ "$SSH_HOST" != "$node" ]]; then
      scp $SSH_OPTS "$TMP_DIR/$node.crt" "$SSH_HOST:/tmp/$node.crt"
    fi
  done
done

echo "# Configuring remote hosts"
for SSH_HOST in "${SSH_HOSTS[@]}"; do
  echo "## Install certificate and build keystore/truststore on $SSH_HOST"
  get_cert_paths "$SSH_HOST"

  ssh $SSH_OPTS "$SSH_HOST" "
    set -eo pipefail
    export KEYSTOREPASS='$KEYSTOREPASS'
    export TRUSTSTOREPASS='$TRUSTSTOREPASS'

    sudo rm -f /etc/ssl/host.pem /etc/ssl/host.key /etc/ssl/ca_certs.crt

    sudo mkdir -p /etc/ssl

    sudo cp /tmp/$SSH_HOST.crt /etc/ssl/host.pem
    sudo cp /tmp/$SSH_HOST.key /etc/ssl/host.key
    sudo cp /tmp/ca_certs.crt /etc/ssl/ca_certs.crt

    set +u
    source /usr/lib/bigtop-utils/bigtop-detect-javahome
    set -u

    # Removing old aliases from the keystore
    sudo \$JAVA_HOME/bin/keytool -delete \
      -alias $SSH_HOST \
      -keystore $KEYSTOREPATH \
      -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true

    sudo \$JAVA_HOME/bin/keytool -delete \
      -alias $CA_ALIAS \
      -keystore $KEYSTOREPATH \
      -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true

    # Removing the old CA alias from the truststore
    sudo \$JAVA_HOME/bin/keytool -delete \
      -alias $CA_ALIAS \
      -keystore $TRUSTSTOREPATH \
      -storepass \$TRUSTSTOREPASS >/dev/null 2>&1 || true

    sudo cp /etc/ssl/host.pem $CA_CERTS_PATH/$HOST_CERT_NAME
    sudo cp /tmp/ca_certs.crt $CA_CERTS_PATH/ca_certs.crt
    sudo $CA_UPDATE_CMD
    sudo chown root:hadoop /etc/ssl/host.key
    sudo chmod 640 /etc/ssl/host.key
    sudo setfacl -m u:airflow:r /etc/ssl/host.key

    openssl pkcs12 -export \
      -in /tmp/$SSH_HOST.crt \
      -inkey /tmp/$SSH_HOST.key \
      -certfile /tmp/ca_certs.crt \
      -name $SSH_HOST \
      -out /tmp/$SSH_HOST.p12 \
      -password pass:\$KEYSTOREPASS

    sudo \$JAVA_HOME/bin/keytool -importkeystore \
      -srckeystore /tmp/$SSH_HOST.p12 \
      -srcstoretype PKCS12 \
      -srcstorepass \$KEYSTOREPASS \
      -destkeystore $KEYSTOREPATH \
      -deststoretype JKS \
      -deststorepass \$KEYSTOREPASS \
      -alias $SSH_HOST \
      -noprompt

    sudo \$JAVA_HOME/bin/keytool -import \
      -noprompt \
      -alias $CA_ALIAS \
      -file /tmp/ca_certs.crt \
      -keystore $KEYSTOREPATH \
      -storepass \$KEYSTOREPASS \
      -trustcacerts

    sudo \$JAVA_HOME/bin/keytool -import \
      -noprompt \
      -alias $CA_ALIAS \
      -file /tmp/ca_certs.crt \
      -keystore $TRUSTSTOREPATH \
      -storepass \$TRUSTSTOREPASS \
      -trustcacerts

    # For Haproxy
    sudo bash -c 'cat /etc/ssl/host.key /etc/ssl/host.pem /etc/ssl/ca_certs.crt > /etc/ssl/serverkey.pem'
    sudo chmod 600 /etc/ssl/serverkey.pem
  "

  # Importing certificates of all nodes into the truststore, removing old aliases
  for node in "${SSH_HOSTS[@]}"; do
    ssh $SSH_OPTS "$SSH_HOST" "
      set +u
      source /usr/lib/bigtop-utils/bigtop-detect-javahome
      set -u

      sudo \$JAVA_HOME/bin/keytool -delete \
        -alias $node \
        -keystore $TRUSTSTOREPATH \
        -storepass $TRUSTSTOREPASS >/dev/null 2>&1 || true

      sudo \$JAVA_HOME/bin/keytool -import \
        -noprompt \
        -alias $node \
        -file /tmp/$node.crt \
        -keystore $TRUSTSTOREPATH \
        -storepass $TRUSTSTOREPASS \
        -trustcacerts
    "
  done

  # Importing truststore into system Java cacerts with preliminary removal of aliases
  ssh $SSH_OPTS "$SSH_HOST" "
    set -eo pipefail

    set +u
    source /usr/lib/bigtop-utils/bigtop-detect-javahome
    set -u

    # Remove all known aliases from target cacerts before importing
    for ALIAS in $CA_ALIAS ${SSH_HOSTS[@]}; do
      sudo \$JAVA_HOME/bin/keytool -delete \
        -alias \$ALIAS \
        -keystore $JAVA_CA_PATH_17 \
        -storepass changeit >/dev/null 2>&1 || true
      sudo \$JAVA_HOME/bin/keytool -delete \
        -alias \$ALIAS \
        -keystore $JAVA_CA_PATH \
        -storepass changeit >/dev/null 2>&1 || true
    done

    # Importing cacerts into Java 17
    sudo \$JAVA_HOME/bin/keytool -importkeystore \
      -noprompt \
      -srckeystore $TRUSTSTOREPATH \
      -srcstorepass $TRUSTSTOREPASS \
      -destkeystore $JAVA_CA_PATH_17 \
      -deststorepass changeit || true

    # Importing cacerts into Java 8
    sudo \$JAVA_HOME/bin/keytool -importkeystore \
      -noprompt \
      -srckeystore $TRUSTSTOREPATH \
      -srcstorepass $TRUSTSTOREPASS \
      -destkeystore $JAVA_CA_PATH \
      -deststorepass changeit || true

    sudo rm -f /tmp/$SSH_HOST.p12 /tmp/*.crt /tmp/*.key
  " || {
    echo "Failed to configure certs on $SSH_HOST"
    exit 1
  }
done

echo "# Cleanup local temp files"
rm -rf "$TMP_DIR"

echo "# All done successfully!"

crtdistr-config.cfg config file for the script

#!/usr/bin/env sh
# KEYSTORE password
KEYSTOREPASS='bigdata'
# Path to create a KEYSTORE
KEYSTOREPATH='/etc/ssl/keystore.jks'
# Truststore password
TRUSTSTOREPASS='bigdata'
# Path to create a TRUSTORE
TRUSTSTOREPATH='/etc/ssl/truststore.jks'
# DNS hostnames to use for certificate mapping
HOSTS=(
    "prefix-adh-1.ru-central1.internal"
    "prefix-adh-2.ru-central1.internal"
    "prefix-adh-3.ru-central1.internal"
    "prefix-adps-1.ru-central1.internal"
    )

Permissions for keystore files

After generating keystore/truststore files, sufficient permissions must be granted to these files in order to allow ADH services to access them.

An example of granting permissions (safe from the security point of view) is below:

$ sudo chown root:hadoop "$KEYSTOREPATH"
$ sudo chmod 640 "$KEYSTOREPATH"

$ sudo chown root:hadoop "$TRUSTSTOREPATH"
$ sudo chmod 640 "$TRUSTSTOREPATH"

However, 640 permissions may be insufficient for some ADH services, such as Zookeeper and Airflow2. Particularly, you may observe an error like the following:

java.io.FileNotFoundException: /etc/ssl/keystore.jks (Permission denied)

There are two ways to address this issue, described below.

Grant read access via ACL (recommended)

You can allow the zookeeper and airflow users to read the keystore file by adding the ACL entries. For example:

$ sudo setfacl -m u:zookeeper:r "$KEYSTOREPATH"
$ sudo setfacl -m u:airflow:r "$KEYSTOREPATH"

Grant read permissions to all users (not recommended)

You can change the keystore file permission bits to allow all users to read the keystore. This approach is less favourable from the security perspective. For example:

$ sudo chmod 644 "$KEYSTOREPATH"

HTTPS ports usage

For information on ports used by ADH services for HTTPS communication, see Network requirements.

Found a mistake? Seleсt text and press Ctrl+Enter to report it