SSL channel security

This article describes the steps to enable SSL channel security for Kafka authentication.

Create SSL certificate for Certification Authority

Certificate generation sequence

  1. Generate an RSA key pair for each host.

  2. Create a keystore.jks keystore on each host.

  3. Create an SSL certificate for each host.

  4. Collect all certificates in one place.

  5. Copy all certificates for each host to the keystore.jks store.

  6. Sign the certificates with a key that belongs to one of the trusted root certificates included in the Java certificate store.

  7. Create a trustore.jks store containing the required certificates.

  8. Import trustore.jks into Java CA store.

  9. Create and import an OpenSSL certificate for Nginx.

Creating an SSL certificate for Certification Authority using a script

For test and reference purposes, when generating certificates for each host and importing certificates into the Java keystore, you can use the special script generate.sh.

It is launched on a host that has access rights to cluster hosts via FQDN and internal IP.

Below there is the sequence of creating the generate.sh script.

  1. Enter the command to create the script file:

    $ sudo vim /tmp/ssl/generate.sh
  2. Replace for the following script lines (only for them!) the following values ​​with values ​​that apply to your case:

    • NUMHOSTS= — in this line specify the values ​​(letters or numbers), individual for each host. The values ​​(1 2 3) are given as an example for a list of hosts:

      • sov-ads-test-1.ru-central1.internal;

      • sov-ads-test-2.ru-central1.internal;

      • sov-ads-test-3.ru-central1.internal.

    • HOSTS= — specify the common part of the FQDN of the host in this line, replacing $i with the value (letters or numbers) specified in the NUMHOSTS parameter. The value sov-ads-test-$i.ru-central1.internal is given as an example.

    • bigdata — password for stores, recommended to change.

  3. Fill in generate.sh with the script text given below, taking into account individual parameters.

    #!/bin/bash
    
    # Script for generating and import self-signed certificates to java keystore and openssl ca-bundle
    # Edit NUMHOSTS and HOSTS as it will be suitable for your case.
    
    
    declare -a NUMHOSTS
    declare -a HOSTS
    
    SSH_OPTS='-o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
    NUMHOSTS=(1 2 3)
    HOSTS=$(for i in "${NUMHOSTS[@]}"; do echo "sov-ads-test-$i.ru-central1.internal"; done)
    
    echo Generate keystore.jks on each host
    for HOST in $HOSTS; do
      echo "Generating keypair"
        ssh $SSH_OPTS $HOST "keytool -genkeypair -noprompt -keyalg RSA -alias $HOST -dname \"CN=$HOST, OU=AD, O=AD, L=MSK, S=MO, C=RU\" -keystore /tmp/keystore.jks -storepass bigdata -keypass bigdata -validity 360 -keysize 2048";
    done
    
    echo
    echo Export certificates
    for HOST in $HOSTS;do
        ssh $SSH_OPTS $HOST "keytool -exportcert -file /tmp/$HOST.crt -keystore /tmp/keystore.jks -storepass bigdata -alias $HOST -rfc";
    done
    
    echo
    echo Collect all certificates
    for HOST in $HOSTS; do
        scp $SSH_OPTS $HOST:/tmp/$HOST.crt /tmp/
    done
    
    echo
    echo Transfer certificates on hosts
    for HOST in $HOSTS; do
        scp $SSH_OPTS /tmp/*.crt $HOST:/tmp/
    done
    
    echo
    echo Import certificates on each host
    for HOST in $HOSTS; do
        ssh $SSH_OPTS $HOST "for CERT in $(echo ${HOSTS[*]}); do
            keytool -importcert -noprompt -alias \$CERT -file /tmp/\$CERT.crt -keystore /tmp/truststore.jks -storepass bigdata;
            sudo bash -c \"cat /tmp/\$CERT.crt >> /etc/pki/tls/certs/ca-bundle.crt\";
        done";
    done
    
    echo
    echo Import truststore to Java CA store
    for HOST in $HOSTS; do
        ssh $SSH_OPTS $HOST "sudo keytool -importkeystore -noprompt -srckeystore /tmp/truststore.jks -destkeystore /etc/pki/java/cacerts -deststorepass changeit -srcstorepass bigdata"
    done
    
    echo
    echo Create and import OpenSSL cert for Nginx
    for HOST in $HOSTS; do
            ssh $SSH_OPTS $HOST "sudo openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj \"/C=RU/ST=Denial/L=MSK/O=AD/CN=$HOST\" -keyout /etc/ssl/host_cert.key  -out /etc/ssl/certs/host_cert.cert"
            ssh $SSH_OPTS $HOST "sudo bash -c \"cat /etc/ssl/certs/host_cert.cert >> /etc/pki/tls/certs/ca-bundle.crt\""
    done
  4. Allow script execution:

    $ sudo chmod +x /tmp/ssl/generate.sh
  5. Run the script:

    $ /tmp/ssl/generate.sh
  6. Monitor script execution.

  7. Check each host for storages and files:

    • /etc/ssl/host_cert.key

    • /tmp/truststore.jks

    • /tmp/keystore.jks

Start SSL on an ADS Cluster using ADCM

  1. Initiate enabling SSL from the ADCM panel in the Actions cluster.

    ads ssl
    Enabling SSL on the ADS cluster
  2. In the window that opens, enter the appropriate configurations:

    • Keystore path — path to keystore.jks storage on hosts with Kafka service.

    • Keystore password — password for keystore.jks storage, script uses bigdata.

    • Truststore path — path to truststore.jks storage on hosts with Kafka service.

    • Truststore password — password for truststore.jks storage, script uses bigdata.

      ads ssl 01
      Setting configurations to run SSL
  3. Click Run.

  4. Wait until SSL is enabled. Analyze and correct errors if they occur.

    ads ssl 03
    SSl startup process

Checking the installed SSL

  1. Verify that the listeners parameter in the Kafka service settings changed from SASL_PLAINTEXT://:9092 to SASL_SSL://:9092.

    ads ssl 04
    Kafka service settings
  2. Check for a change in the server.properties file on each Kafka server.

    Open file server.properties:

    $ sudo vim /usr/lib/kafka/config/server.properties
  3. Make sure that for each Kafka broker the line defining the security protocol is changed to SASL_SSL and there are lines defining the server RSA keystores and passwords for them:

security.inter.broker.protocol=SASL_SSL
...
...
...
ssl.keystore.location=/tmp/keystore.jks
ssl.keystore.password=bigdata
ssl.key.password=bigdata
ssl.keystore.type=JKS
ssl.truststore.location=/tmp/truststore.jks
ssl.truststore.password=bigdata
ssl.truststore.type=JKS

Connecting to Kafka (creating tickets) and working with .sh files (scripts) with participation of different users after installing SSL

  1. To create a configuration file .properties for a user with SSL in mind, run the command:

    $ sudo vim /tmp/client_ssl.properties

    Fill the file with data:

    security.protocol=SASL_SSL
    sasl.mechanism=GSSAPI
    sasl.kerberos.service.name=kafka
  2. Follow the steps to connect to Kafka as user writer from the article Kerberos SASL client side in Kafka:

    • Open terminal session 1 and connect to one of the Kafka brokers.

    • Create a ticket for user writer.

    • Check ticket.

    • Export the client.jaas file as a JVM option for the given user using the KAFKA_OPTS environment variable.

  3. Create a topic by specifying the path to the created client_ssl.properties file:

    $ /usr/lib/kafka/bin/kafka-topics.sh --create --topic test-topic1 --bootstrap-server sov-ads-test-1.ru-central1.internal:9092,sov-ads-test-2.ru-central1.internal:9092,sov-ads-test-3.ru-central1.internal:9092 --command-config /tmp/client_ssl.properties
    NOTE
    The client.jaas file does not need to be changed after installing SSL.
  4. Write a message to a topic:

    $ /usr/lib/kafka/bin/kafka-console-producer.sh --topic test-topic1 --bootstrap-server sov-ads-test-1.ru-central1.internal:9092,sov-ads-test-2.ru-central1.internal:9092,sov-ads-test-3.ru-central1.internal:9092 --producer.config /tmp/client_ssl.properties
  5. Follow the steps to connect to Kafka as the user reader from the article Kerberos SASL client side in Kafka:

    • Open terminal session 2 and connect to one of the Kafka brokers.

    • Create a ticket for user reader.

    • Check ticket.

    • Export the generated client.jaas file as a JVM option for the given user using the KAFKA_OPTS environment variable.

  6. Read messages from a topic by specifying the path to the created client_ssl.properties file:

    $ /usr/lib/kafka/bin/kafka-console-consumer.sh --topic test-topic1 --from-beginning --bootstrap-server sov-ads-test-1.ru-central1.internal:9092,sov -ads-test-2.ru-central1.internal:9092,sov-ads-test-3.ru-central1.internal:9092 --consumer.config /tmp/client_ssl.properties

    Make sure messages are read correctly.

Found a mistake? Seleсt text and press Ctrl+Enter to report it