Requirements for SSL certificates
To implement encryption between the nodes the following is required:
-
Create a certificate for each node using the RSA algorithm. The key length should be 2048 bit, CN should be the FQDN of the corresponding node. It is not recommended to use the DSA algorithm. It is also not recommended to use the ECDSA algorithm due to this algorithm not being widely supported by certification centers. The key length that equals 2048 bits is necessary and sufficient to implement secure encryption.
-
Create the trustore.jks keystore file that will store the certificates. The path to this file should be specified for SSL to be started.
NOTE-
Certificates should be signed by the key that belongs to one of the trusted root certificates included into the Java certificate storage.
-
Self-signed certificates are allowed. In this case, the root certificate should be added to the trustore.jks storage.
-
Nginx requires OpenSSL certificates. For a cluster that uses Nginx (with Hive TezUI or Airflow), generate an OpenSSL key and a certificate.
-
The OpenSSL certificate for your host name should be added to the ca-bundle.crt storage.
-
The account used for the installation of certificates should have the rights to write to the following paths:
-
/etc/pki/tls/certs/
-
/etc/pki/java/
-
-
Make sure to use an FQDN as a key alias in keystore.jks, not a short name.
-
-
Enable the traffic between the nodes according to the table below.
The script below creates all the necessary entities and can be used as a reference.
#!/bin/bash # Script for generating and import self-signed certificates to java keystore and openssl ca-bundle # Edit NUMHOSTS and HOSTS as it will be suitable for your case. declare -a NUMHOSTS declare -a HOSTS SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' NUMHOSTS=(1 2 3) HOSTS=$(for i in "${NUMHOSTS[@]}"; do echo "adh-$i.ru-central1.internal"; done) echo Generate keystore.jks on each host for HOST in $HOSTS; do echo "Generating keypair" ssh $SSH_OPTS $HOST "keytool -genkeypair -noprompt -keyalg RSA -alias $HOST -dname \"CN=$HOST, OU=AD, O=AD, L=MSK, S=MO, C=RU\" -keystore /tmp/keystore.jks -storepass bigdata -keypass bigdata -validity 360 -keysize 2048"; done echo echo Export certificates for HOST in $HOSTS;do ssh $SSH_OPTS $HOST "keytool -exportcert -file /tmp/$HOST.crt -keystore /tmp/keystore.jks -storepass bigdata -alias $HOST -rfc"; done echo echo Collect all certificates for HOST in $HOSTS; do scp $SSH_OPTS $HOST:/tmp/$HOST.crt /tmp/ done echo echo Transfer certificates on hosts for HOST in $HOSTS; do scp $SSH_OPTS /tmp/*.crt $HOST:/tmp/ done echo echo Import certificates on each host for HOST in $HOSTS; do ssh $SSH_OPTS $HOST "for CERT in $(echo ${HOSTS[*]}); do keytool -importcert -noprompt -alias \$CERT -file /tmp/\$CERT.crt -keystore /tmp/truststore.jks -storepass bigdata; sudo bash -c \"cat /tmp/\$CERT.crt >> /etc/pki/tls/certs/ca-bundle.crt\"; done"; done echo echo Import truststore to Java CA store for HOST in $HOSTS; do ssh $SSH_OPTS $HOST "sudo keytool -importkeystore -noprompt -srckeystore /tmp/truststore.jks -destkeystore /etc/pki/java/cacerts -deststorepass changeit -srcstorepass bigdata" done echo echo Create and import OpenSSL cert for Nginx for HOST in $HOSTS; do ssh $SSH_OPTS $HOST "sudo openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -sub j \"/C=RU/ST=Denial/L=MSK/O=AD/CN=$HOST\" -keyout /etc/ssl/host_cert.key -out /etc/ssl/certs/h ost_cert.cert" ssh $SSH_OPTS $HOST "sudo bash -c \"cat /etc/ssl/certs/host_cert.cert >> /etc/pki/tls/c erts/ca-bundle.crt\"" done
Service | Component/Server | Port | Protocol | Description |
---|---|---|---|---|
Airflow |
Server |
8080/8080 |
HTTP/HTTPS |
Airflow Server Web UI port |
Server/Worker |
8793/8793 |
HTTP/HTTPS |
Celery Worker API port |
|
Server/Flower |
5555/5555 |
HTTP/HTTPS |
Flower Web UI port |
|
Flink |
Job Manager |
8081/8081 |
HTTP/HTTPS |
Flink Job Manager Web UI port |
HBase |
HBase Region |
16030/16030 |
HTTP/HTTPS |
HBase Region Server Web UI port |
Phoenix Query Server |
8765/8765 |
HTTP/HTTPS |
API port |
|
HBase REST Server |
60080/60080 |
HTTP/HTTPS |
API port |
|
HBase REST Server |
8085/8085 |
HTTP/HTTPS |
Web UI Port |
|
HBase Master |
16010/16010 |
HTTP/HTTPS |
HBase Master Web UI port |
|
HBaseThrift2Server |
Trift2 Server |
9095/9095 |
HTTP/HTTPS |
Thrift2 Server Web UI port |
HDFS |
NameNode |
9870/9871 |
HTTP/HTTPS |
NameNode Web UI port |
DataNode |
9864/9865 |
HTTP/HTTPS |
DataNode Web UI port |
|
JournalNode |
8480/8481 |
HTTP/HTTPS |
JournalNode Web UI port |
|
HTTPfs server |
14000/14000 |
HTTP/HTTPS |
HTTPfs API port |
|
Hive |
Hive Server |
10002/10002 |
HTTP/HTTPS |
Hive Server Web UI port |
Tez |
9999/9999 |
HTTP/HTTPS |
Tez Web UI port |
|
Knox |
Knox Gateway |
8443 |
HTTPS |
Gateway port |
Ranger |
Ranger Admin |
6080/6182 |
HTTP/HTTPS |
Ranger Admin web UI and API port |
Ranger KMS |
9292/9393 |
HTTP/HTTPS |
Port for Ranger KMS |
|
Solr |
Solr Server |
8983/8985 |
HTTP/HTTPS |
Solr Server Web UI and API port |
Spark |
History Server |
18082/18082 |
HTTP/HTTPS |
HS Web UO port |
Thrift Server |
4040/4040 |
HTTP/HTTPS |
Thrift Server Web UI |
|
Livy Server |
8998/8998 |
HTTP/HTTPS |
Livy Server Web UI port |
|
YARN |
Resource Manager |
8088/8090 |
HTTP/HTTPS |
RM Web UI port |
Node Manager |
8042/8044 |
HTTP/HTTPS |
NM Web UI port |
|
MapReduce History Server |
19888/19890 |
HTTP/HTTPS |
HS Web UI port |
|
Timeline Server |
8188/8190 |
HTTP/HTTPS |
TS Web UI port |
|
Zeppelin |
Server |
8180/8180 |
HTTP/HTTPS |
Zeppelin Web UI port |