Requirements for SSL certificates

To implement encryption between the nodes the following is required:

  1. Create a certificate for each node using the RSA algorithm. The key length should be 2048 bit, CN should be the FQDN of the corresponding node. It is not recommended to use the DAS algorithm due to the maximum key length being 1024. It is also not recommended to use the ECDSA algorithm due to this algorithm not being widely supported by certification centers. The key length that equals 2048 bits is necessary and sufficient to implement secure encryption.

  2. Create the trustore.jks keystore file that will store the certificates. The path to this file should be specified for SSL to be started.

    NOTE
    • Certificates should be signed by the key that belongs to one of the trusted root certificates included into the Java certificate storage.

    • Self-signed certificates are allowed. In this case, the root certificate should be added to the trustore.jks storage.

    • Nginx requires OpenSSL certificates. For a cluster that uses Nginx (with Hive TezUI or Airflow), generate an OpenSSL key and a certificate.

    • The OpenSSL certificate for your host name should be added to the ca-bundle.crt storage.

    • The account used for the installation of certificates should have the rights to write to the following paths:

      • /etc/pki/tls/certs/

      • /etc/pki/java/

  3. Enable the traffic between the nodes according to the table below.

    The script below creates all the necessary entities and can be used as a reference.

    #!/bin/bash
    
    # Script for generating and import self-signed certificates to java keystore and openssl ca-bundle
    # Edit NUMHOSTS and HOSTS as it will be suitable for your case.
    
    
    declare -a NUMHOSTS
    declare -a HOSTS
    
    SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
    NUMHOSTS=(1 2 3)
    HOSTS=$(for i in "${NUMHOSTS[@]}"; do echo "adh-$i.ru-central1.internal"; done)
    
    echo Generate keystore.jks on each host
    for HOST in $HOSTS; do
      echo "Generating keypair"
    	ssh $SSH_OPTS $HOST "keytool -genkeypair -noprompt -keyalg RSA -alias $HOST -dname \"CN=$HOST, OU=AD, O=AD, L=MSK, S=MO, C=RU\" -keystore /tmp/keystore.jks -storepass bigdata -keypass bigdata -validity 360 -keysize 2048";
    done
    
    echo
    echo Export certificates
    for HOST in $HOSTS;do
    	ssh $SSH_OPTS $HOST "keytool -exportcert -file /tmp/$HOST.crt -keystore /tmp/keystore.jks -storepass bigdata -alias $HOST -rfc";
    done
    
    echo
    echo Collect all certificates
    for HOST in $HOSTS; do
    	scp $SSH_OPTS $HOST:/tmp/$HOST.crt /tmp/
    done
    
    echo
    echo Transfer certificates on hosts
    for HOST in $HOSTS; do
    	scp $SSH_OPTS /tmp/*.crt $HOST:/tmp/
    done
    
    echo
    echo Import certificates on each host
    for HOST in $HOSTS; do
    	ssh $SSH_OPTS $HOST "for CERT in $(echo ${HOSTS[*]}); do
    	    keytool -importcert -noprompt -alias \$CERT -file /tmp/\$CERT.crt -keystore /tmp/truststore.jks -storepass bigdata;
    	    sudo bash -c \"cat /tmp/\$CERT.crt >> /etc/pki/tls/certs/ca-bundle.crt\";
    	done";
    done
    
    echo
    echo Import truststore to Java CA store
    for HOST in $HOSTS; do
    	ssh $SSH_OPTS $HOST "sudo keytool -importkeystore -noprompt -srckeystore /tmp/truststore.jks -destkeystore /etc/pki/java/cacerts -deststorepass changeit -srcstorepass bigdata"
    done
    
    echo
    echo Create and import OpenSSL cert for Nginx
    for HOST in $HOSTS; do
            ssh $SSH_OPTS $HOST "sudo openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -sub
    j \"/C=RU/ST=Denial/L=MSK/O=AD/CN=$HOST\" -keyout /etc/ssl/host_cert.key  -out /etc/ssl/certs/h
    ost_cert.cert"
            ssh $SSH_OPTS $HOST "sudo bash -c \"cat /etc/ssl/certs/host_cert.cert >> /etc/pki/tls/c
    erts/ca-bundle.crt\""
    done
Matching between ports and HTTP/HTTPS services
Service Component/Server Port Protocol Description

Airflow

Server

8080/8080

HTTP/HTTPS

Airflow Server Web UI port

Server/Worker

8793/8793

HTTP/HTTPS

Celery Worker API port

Server/Flower

5555/5555

HTTP/HTTPS

Flower Web UI port

Flink

Job Manager

8081/8081

HTTP/HTTPS

Flink Job Manager Web UI port

HBase

HBase Region

16030/16030

HTTP/HTTPS

HBase Region Server Web UI port

Phoenix Query Server

8765/8765

HTTP/HTTPS

API port

HBase REST Server

60080/60080

HTTP/HTTPS

API port

HBase Master

16010/16010

HTTP/HTTPS

HBase Master Web UI port

HBaseThrift2Server

Trift2 Server

9095/9095

HTTP/HTTPS

Thrift2 Server Web UI port

HDFS

NameNode

9870/9871

HTTP/HTTPS

NameNode Web UI port

DataNode

9864/9865

HTTP/HTTPS

DataNode Web UI port

JournalNode

8480/8481

HTTP/HTTPS

JournalNode Web UI port

HTTPfs server

14000/14000

HTTP/HTTPS

HTTPfs API port

Hive

Hive Server

10002/10002

HTTP/HTTPS

Hive Server Web UI port

Tez

9999/9999

HTTP/HTTPS

Tez Web UI port

Solr

Solr Server

8983/8985

HTTP/HTTPS

Solr Server Web UI and API port

Spark

History Server

18082/18082

HTTP/HTTPS

HS Web UO port

Thrift Server

4040/4040

HTTP/HTTPS

Thrift Server Web UI

Livy Server

8998/8998

HTTP/HTTPS

Livy Server Web UI port

YARN

Resource Manager

8088/8090

HTTP/HTTPS

RM Web UI port

Node Manager

8042/8044

HTTP/HTTPS

NM Web UI port

MapReduce History Server

19888/19890

HTTP/HTTPS

HS Web UI port

Timeline Server

8188/8188

HTTP/HTTPS

TS Web UI port

Zeppelin

Server

8180/8180

HTTP/HTTPS

Zeppelin Web UI port

Found a mistake? Seleсt text and press Ctrl+Enter to report it