Requirements for SSL certificates
To implement encryption between the nodes the following is required:
-
Create a certificate for each node using the RSA algorithm. The key length should be 2048 bit, CN should be the FQDN of the corresponding node. It is not recommended to use the DSA algorithm. It is also not recommended to use the ECDSA algorithm due to this algorithm not being widely supported by certification centers. The key length that equals 2048 bits is necessary and sufficient to implement secure encryption.
-
Create the trustore.jks keystore file that will store the certificates. The path to this file should be specified for SSL to be started.
NOTE-
Certificates should be signed by the key that belongs to one of the trusted root certificates included into the Java certificate storage.
-
Self-signed certificates are allowed. In this case, the root certificate should be added to the trustore.jks storage.
-
Nginx requires OpenSSL certificates. For a cluster that uses Nginx (with Hive Tez UI or Airflow), generate an OpenSSL key and a certificate.
-
The OpenSSL certificate for your host name should be added to the ca-bundle.crt storage.
-
The account used for the installation of certificates should have the rights to write to the following paths:
-
/etc/pki/tls/certs/
-
/etc/pki/java/
-
-
Make sure to use an FQDN as a key alias in keystore.jks, not a short name.
-
-
Enable the traffic between the nodes according to the table below.
The script below creates all the necessary entities and can be used as a reference.
SSL preparation script#!/bin/bash set -euo pipefail echo "# SSL preparation script started" if [ -f ./crtdistr-config.cfg ]; then echo "## Config found" else echo "## Error: configuration file ./crtdistr-config.cfg not found" exit 1 fi HOSTS_MAKE="${HOSTS-}" source ./crtdistr-config.cfg if [ -n "$HOSTS_MAKE" ]; then echo "## HOSTS provided from MakeFile" read -r -a HOSTS <<< "$HOSTS_MAKE" fi SSH_HOSTS=() SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=error' TMP_DIR="/tmp/crtdistr" CA_KEY="$TMP_DIR/ca_admprom.key" CA_CERT="$TMP_DIR/ca_certs.crt" CA_ALIAS="root-ca" get_cert_paths() { local host="$1" if ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'astra'; then CA_CERTS_PATH="/usr/local/share/ca-certificates/" CA_UPDATE_CMD="update-ca-certificates" HOST_CERT_NAME="host_cert.crt" JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts" JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts" elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'ubuntu'; then CA_CERTS_PATH="/usr/local/share/ca-certificates" CA_UPDATE_CMD="update-ca-certificates" HOST_CERT_NAME="host_cert.crt" JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts" JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts" elif ssh $SSH_OPTS "$host" 'grep "^ID=" /etc/os-release' 2>/dev/null | grep -qE 'redos'; then CA_CERTS_PATH="/etc/pki/ca-trust/source/anchors" CA_UPDATE_CMD="update-ca-trust" HOST_CERT_NAME="host_cert.cert" JAVA_CA_PATH="/usr/lib/jvm/java-arenadata-openjdk-8/jre/lib/security/cacerts" JAVA_CA_PATH_17="/usr/lib/jvm/java-arenadata-openjdk-17/lib/security/cacerts" else CA_CERTS_PATH="" CA_UPDATE_CMD="" HOST_CERT_NAME="" JAVA_CA_PATH="" JAVA_CA_PATH_17="" fi } echo "# Pre-check" for HOST in "${HOSTS[@]}"; do echo "## Checking ssh connection to host $HOST" if ssh $SSH_OPTS -q "$HOST" "exit"; then get_cert_paths "$HOST" if [[ -z "$CA_CERTS_PATH" ]]; then echo "### Unknown OS type on $HOST, skipped" elif [[ -z "$JAVA_CA_PATH" ]]; then echo "### Java cacerts path not detected on $HOST, skipped" else echo "### $HOST is OK" SSH_HOSTS+=("$HOST") fi else echo "### SSH connection failed for $HOST" fi done if [[ "${#SSH_HOSTS[@]}" -eq 0 ]]; then echo "## No reachable hosts found" exit 1 fi rm -rf "$TMP_DIR" mkdir -p "$TMP_DIR" echo "# Creating local CA" openssl genrsa -out "$CA_KEY" 4096 openssl req -x509 \ -nodes \ -key "$CA_KEY" \ -days 1095 \ -out "$CA_CERT" \ -subj "/CN=marsnet.local CA" echo "# Creating server certificates" for SSH_HOST in "${SSH_HOSTS[@]}"; do echo "## Generate key and CSR for $SSH_HOST" openssl req \ -newkey rsa:4096 \ -nodes \ -keyout "$TMP_DIR/$SSH_HOST.key" \ -out "$TMP_DIR/$SSH_HOST.host.csr" \ -subj "/C=RU/ST=Denial/L=MSK/O=AD/CN=$SSH_HOST" echo "## Sign certificate for $SSH_HOST" openssl x509 \ -req \ -in "$TMP_DIR/$SSH_HOST.host.csr" \ -CA "$CA_CERT" \ -CAkey "$CA_KEY" \ -CAcreateserial \ -out "$TMP_DIR/$SSH_HOST.crt" \ -days 365 \ -extfile <(printf "basicConstraints=critical,CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=DNS:$SSH_HOST\n") done echo "# Distributing certificates" for SSH_HOST in "${SSH_HOSTS[@]}"; do echo "## Copy files to $SSH_HOST" scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.crt" "$SSH_HOST:/tmp/" scp $SSH_OPTS "$TMP_DIR/$SSH_HOST.key" "$SSH_HOST:/tmp/" scp $SSH_OPTS "$CA_CERT" "$SSH_HOST:/tmp/" for node in "${SSH_HOSTS[@]}"; do if [[ "$SSH_HOST" != "$node" ]]; then scp $SSH_OPTS "$TMP_DIR/$node.crt" "$SSH_HOST:/tmp/$node.crt" fi done done echo "# Configuring remote hosts" for SSH_HOST in "${SSH_HOSTS[@]}"; do echo "## Install certificate and build keystore/truststore on $SSH_HOST" get_cert_paths "$SSH_HOST" ssh $SSH_OPTS "$SSH_HOST" " set -eo pipefail export KEYSTOREPASS='$KEYSTOREPASS' export TRUSTSTOREPASS='$TRUSTSTOREPASS' sudo rm -f /etc/ssl/host.pem /etc/ssl/host.key /etc/ssl/ca_certs.crt sudo mkdir -p /etc/ssl sudo cp /tmp/$SSH_HOST.crt /etc/ssl/host.pem sudo cp /tmp/$SSH_HOST.key /etc/ssl/host.key sudo cp /tmp/ca_certs.crt /etc/ssl/ca_certs.crt set +u source /usr/lib/bigtop-utils/bigtop-detect-javahome set -u sudo \$JAVA_HOME/bin/keytool -delete \ -alias $SSH_HOST \ -keystore $KEYSTOREPATH \ -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true sudo \$JAVA_HOME/bin/keytool -delete \ -alias $CA_ALIAS \ -keystore $KEYSTOREPATH \ -storepass \$KEYSTOREPASS >/dev/null 2>&1 || true sudo \$JAVA_HOME/bin/keytool -delete \ -alias $CA_ALIAS \ -keystore $TRUSTSTOREPATH \ -storepass \$TRUSTSTOREPASS >/dev/null 2>&1 || true sudo cp /etc/ssl/host.pem $CA_CERTS_PATH/$HOST_CERT_NAME sudo cp /tmp/ca_certs.crt $CA_CERTS_PATH/ca_certs.crt sudo $CA_UPDATE_CMD sudo chmod 644 /etc/ssl/host.key openssl pkcs12 -export \ -in /tmp/$SSH_HOST.crt \ -inkey /tmp/$SSH_HOST.key \ -certfile /tmp/ca_certs.crt \ -name $SSH_HOST \ -out /tmp/$SSH_HOST.p12 \ -password pass:\$KEYSTOREPASS sudo \$JAVA_HOME/bin/keytool -importkeystore \ -srckeystore /tmp/$SSH_HOST.p12 \ -srcstoretype PKCS12 \ -srcstorepass \$KEYSTOREPASS \ -destkeystore $KEYSTOREPATH \ -deststoretype JKS \ -deststorepass \$KEYSTOREPASS \ -alias $SSH_HOST \ -noprompt sudo \$JAVA_HOME/bin/keytool -import \ -noprompt \ -alias $CA_ALIAS \ -file /tmp/ca_certs.crt \ -keystore $KEYSTOREPATH \ -storepass \$KEYSTOREPASS \ -trustcacerts sudo \$JAVA_HOME/bin/keytool -import \ -noprompt \ -alias $CA_ALIAS \ -file /tmp/ca_certs.crt \ -keystore $TRUSTSTOREPATH \ -storepass \$TRUSTSTOREPASS \ -trustcacerts #For Haproxy sudo bash -c 'cat /etc/ssl/host.key /etc/ssl/host.pem /etc/ssl/ca_certs.crt > /etc/ssl/serverkey.pem' sudo chmod 600 /etc/ssl/serverkey.pem " for node in "${SSH_HOSTS[@]}"; do ssh $SSH_OPTS "$SSH_HOST" " set +u source /usr/lib/bigtop-utils/bigtop-detect-javahome set -u sudo \$JAVA_HOME/bin/keytool -delete \ -alias $node \ -keystore $TRUSTSTOREPATH \ -storepass $TRUSTSTOREPASS >/dev/null 2>&1 || true sudo \$JAVA_HOME/bin/keytool -import \ -noprompt \ -alias $node \ -file /tmp/$node.crt \ -keystore $TRUSTSTOREPATH \ -storepass $TRUSTSTOREPASS \ -trustcacerts " done #Import trustore to JKS ssh $SSH_OPTS "$SSH_HOST" " set -eo pipefail set +u source /usr/lib/bigtop-utils/bigtop-detect-javahome set -u sudo \$JAVA_HOME/bin/keytool -importkeystore \ -noprompt \ -srckeystore $TRUSTSTOREPATH \ -srcstorepass $TRUSTSTOREPASS \ -destkeystore $JAVA_CA_PATH_17 \ -deststorepass changeit || true sudo \$JAVA_HOME/bin/keytool -importkeystore \ -noprompt \ -srckeystore $TRUSTSTOREPATH \ -srcstorepass $TRUSTSTOREPASS \ -destkeystore $JAVA_CA_PATH \ -deststorepass changeit || true sudo rm -f /tmp/$SSH_HOST.p12 /tmp/*.crt /tmp/*.key " || { echo "Failed to configure certs on $SSH_HOST" exit 1 } done echo "# Cleanup local temp files" rm -rf "$TMP_DIR" echo "# All done successfully!"
| Service | Component/Server | Port | Protocol | Description |
|---|---|---|---|---|
Airflow |
Server |
8080/8080 |
HTTP/HTTPS |
Airflow Server Web UI port |
Server/Worker |
8793/8793 |
HTTP/HTTPS |
Celery Worker API port |
|
Server/Flower |
5555/5555 |
HTTP/HTTPS |
Flower Web UI port |
|
Flink |
Job Manager |
8081/8081 |
HTTP/HTTPS |
Flink Job Manager Web UI port |
HBase |
HBase Region |
16030/16030 |
HTTP/HTTPS |
HBase Region Server Web UI port |
Phoenix Query Server |
8765/8765 |
HTTP/HTTPS |
API port |
|
HBase REST Server |
60080/60080 |
HTTP/HTTPS |
API port |
|
HBase REST Server |
8085/8085 |
HTTP/HTTPS |
Web UI Port |
|
HBase Master |
16010/16010 |
HTTP/HTTPS |
HBase Master Web UI port |
|
HBaseThrift2Server |
Trift2 Server |
9095/9095 |
HTTP/HTTPS |
Thrift2 Server Web UI port |
HDFS |
NameNode |
9870/9871 |
HTTP/HTTPS |
NameNode Web UI port |
DataNode |
9864/9865 |
HTTP/HTTPS |
DataNode Web UI port |
|
JournalNode |
8480/8481 |
HTTP/HTTPS |
JournalNode Web UI port |
|
HTTPfs server |
14000/14000 |
HTTP/HTTPS |
HTTPfs API port |
|
Monitoring |
Grafana |
11210 |
HTTP/HTTPS |
Grafana Web UI port |
Node Exporter |
11203 |
HTTP/HTTPS |
Node Exporter listen port |
|
Prometheus Server |
11200 |
HTTP/HTTPS |
Prometheus Web UI port |
|
Hive |
Hive Server |
10002/10002 |
HTTP/HTTPS |
Hive Server Web UI port |
Tez |
9999/9999 |
HTTP/HTTPS |
Tez Web UI port |
|
Knox |
Knox Gateway |
8443 |
HTTPS |
Gateway port |
Ranger |
Ranger Admin |
6080/6182 |
HTTP/HTTPS |
Ranger Admin web UI and API port |
Ranger KMS |
9292/9393 |
HTTP/HTTPS |
Port for Ranger KMS |
|
Solr |
Solr Server |
8983/8985 |
HTTP/HTTPS |
Solr Server Web UI and API port |
Spark |
History Server |
18082/18082 |
HTTP/HTTPS |
HS Web UO port |
Thrift Server |
4040/4040 |
HTTP/HTTPS |
Thrift Server Web UI |
|
Livy Server |
8998/8998 |
HTTP/HTTPS |
Livy Server Web UI port |
|
YARN |
Resource Manager |
8088/8090 |
HTTP/HTTPS |
RM Web UI port |
Node Manager |
8042/8044 |
HTTP/HTTPS |
NM Web UI port |
|
MapReduce History Server |
19888/19890 |
HTTP/HTTPS |
HS Web UI port |
|
Timeline Server |
8188/8190 |
HTTP/HTTPS |
TS Web UI port |
|
Zeppelin |
Server |
8180/8180 |
HTTP/HTTPS |
Zeppelin Web UI port |