Kyuubi integration with Spark Connect

Sergei Tikhomirov

Overview

Spark Connect is a protocol that allows client applications to communicate with a remote Spark server via gRPC without the need to co-locate the client with the Spark driver.

Kyuubi is a multi-tenant gateway that enables submitting Spark workloads and SQL queries on top of various compute engines such as Spark, Hive, and Flink.

A standard Spark Connect server does not provide user isolation and authentication. The gRPC channel between a Spark client app and the Spark Connect server remains insecure. But Kyuubi provides the user isolation and authentication features. Therefore, Spark Connect support was introduced to Kyuubi.

Spark Connect supports the following authentication types: KERBEROS, LDAP, and NONE. A client sends an authentication header: Authorization: Negotiate …​ (for Kerberos, SPNEGO) or Authorization: Basic …​ (for LDAP). If authentication is correct, the server issues a token that lives for the time specified in the kyuubi.frontend.spark.connect.token.ttl Kyuubi Server configuration parameter (default is 24 hours).

After that, a client can make any Spark request using the token (the client sends this token in the Bearer authentication header on every request). TTL of this token is renewed on every Spark request. When a user closes the Spark session (by calling spark.stop()), the Spark client calls the ReleaseSession RPC method, and the server revokes the token automatically.

Limitations

  • Spark Connect doesn’t support high availability. There is only one Kyuubi server specified in a connection string.

  • There is no authentication and encryption between Kyuubi and the Spark engine.

  • Kyuubi supports only Spark3 applications.

These restrictions will be lifted in future releases.

Client libraries

There are client libraries available for Python and Java/Scala.

The Python package contains the module which provides the Spark Connect authentication. The corresponding PySpark package is also provided. These libraries are already present on all ADH hosts (in the /usr/lib/spark3/python and /opt/pyspark3-python directories), but you need to activate the environment before running your Python programs:

$ source /opt/pyspark3-python/bin/activate

If you use a standard PySpark package, the ReleaseSession RPC method will not be called in spark.stop(). Therefore, after closing the Spark session, the authentication token will not be revoked. And if Kyuubi share level is CONNECTION, the YARN session will still be open after closing it with spark.stop().

The Java/Scala package is also available.

Setup

In ADCM, add the SPARK_CONNECT value to the kyuubi.frontend.protocols configuration parameter of the Kyuubi Server component and restart the Kyuubi service. That is, the final parameter value may look like THRIFT_BINARY,SPARK_CONNECT.

Examples

Python

This example demonstrates how to connect to Kyuubi via Spark Connect from Python with SSL and Kerberos enabled in the ADH cluster.

  1. Prepare a shell script that sets the correct environment:

    prepare_env.sh
    source /opt/pyspark3-python/bin/activate
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt

  2. Prepare a Python program:

    spark_connect_client_kerberos.py
    from kyuubi.spark_connect import KyuubiSessionBuilder

HOST = "tsn-adh3-3.ru-central1.internal"
PORT = 10199

spark = KyuubiSessionBuilder(f"sc://{HOST}:{PORT}/;use_ssl=true",
    auth="kerberos").getOrCreate()

spark.sql("SELECT current_user()").show()

spark.stop()

    The code for LDAP is similar, but the session parameters should be adjusted as follows:

    spark = KyuubiSessionBuilder(f"sc://{HOST}:{PORT}/;use_ssl=true",
    auth="ldap",
    username="<ldap_username>",
    password="<ldap_password>").getOrCreate()

    where <ldap_username> and <ldap_password> are the username and password of a user in LDAP.

    If you don’t use LDAP or Kerberos, omit the auth parameter.

  3. Run the Python program:

    $ python3 spark_connect_client_kerberos.py

    The expected output should be similar to:

    +--------------+
|current_user()|
+--------------+
|   stikhomirov|
+--------------+

Java/Scala

  1. Prepare a Java program:

    SparkConnectClientKerberos.java
    import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.kyuubi.KyuubiAuthType;
import org.apache.spark.sql.kyuubi.KyuubiSessionBuilder;

public class SparkConnectClientKerberos {

  public static void main(String[] args) {
    String url = System.getenv("KYUUBI_URL");
    if (url == null || url.isEmpty()) {
      throw new IllegalArgumentException("KYUUBI_URL environment variable is required");
    }

    String authStr = System.getenv().getOrDefault("KYUUBI_AUTH", "NONE");
    KyuubiAuthType authType = KyuubiAuthType.valueOf(authStr.toUpperCase());

    System.out.println("connection string: " + url);
    System.out.println("auth type: " + authType);

    SparkSession spark = new KyuubiSessionBuilder(url, authType).getOrCreate();
    try {
      spark.sql("SELECT current_user()").show();
    } finally {
      spark.stop();
    }
  }
}

  2. In your pom.xml file, add the following dependency:

    <dependency>
  <groupId>org.apache.kyuubi</groupId>
  <artifactId>kyuubi-spark-connect-client-standalone_2.13</artifactId>
  <version>1.11.1.1-4.3.0-4</version>
</dependency>
    pom.xml
    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.apache.kyuubi.examples</groupId>
  <artifactId>kyuubi-spark-connect-client-jvm-examples</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <kyuubi.version>1.11.1.1-4.3.0-0</kyuubi.version>
    <spark.version>3.5.4</spark.version>
  </properties>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.11.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.5.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals><goal>shade</goal></goals>
            <configuration>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/versions/*/module-info.class</exclude>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>SparkConnectClientKerberos</mainClass>
                  <manifestEntries>
                    <Multi-Release>true</Multi-Release>
                  </manifestEntries>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <dependencies>
    <dependency>
      <groupId>org.apache.kyuubi</groupId>
      <artifactId>kyuubi-spark-connect-client-standalone_2.13</artifactId>
      <version>${kyuubi.version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>1.7.36</version>
    </dependency>
  </dependencies>

</project>

    Build a JAR file with the created POM.

  3. Prepare a shell script that sets the correct environment and runs the application:

    run.sh
    export KYUUBI_AUTH=kerberos
export KYUUBI_URL = "sc://tsn-adh3-3.ru-central1.internal:10199/;use_ssl=true"
java --add-opens=java.base/java.nio=ALL-UNNAMED -jar target/kyuubi-spark-connect-client-jvm-examples-1.0-SNAPSHOT.jar

  4. Run the prepared shell script:

    $ source run.sh
Found a mistake? Seleсt text and press Ctrl+Enter to report it