Kyuubi configuration parameters

Konstantin Alpashkin

To configure the service, use the following configuration parameters in ADCM.

NOTE

  • Some of the parameters become visible in the ADCM UI after the Advanced flag has been set.

  • The parameters that are set in the Custom group will overwrite the existing parameters even if they are read-only.
The Kyuubi Server component
kyuubi-defaults.conf
Parameter Description Default value

kyuubi.frontend.rest.bind.port

Port on which the REST frontend service runs

10099

kyuubi.frontend.thrift.binary.bind.port

Port on which the Thrift frontend service runs via a binary protocol

10099

kyuubi.frontend.thrift.http.bind.port

Port on which the Thrift frontend service runs via HTTP

10010

kyuubi.frontend.thrift.http.path

The path component of the URL endpoint for the HTTP version of Thrift

cliservice

kyuubi.engine.share.level

An engine share level. Possible values: CONNECTION (one engine per connection), USER (one engine per user), GROUP (one engine per group), SERVER (one engine per server)

USER

kyuubi.engine.type

An engine type supported by Kyuubi. Possible values: SPARK_SQL, FLINK_SQL, TRINO, HIVE_SQL, JDBC

SPARK_SQL

kyuubi.operation.language

Programming language used to interpret inputs. Possible values: SQL, SCALA, PYTHON

SQL

kyuubi.frontend.protocols

A comma-separated list for supported frontend protocols. Possible values: THRIFT_BINARY, THRIFT_HTTP, REST

THRIFT_BINARY

kyuubi.frontend.thrift.binary.ssl.disallowed.protocols

Forbidden SSL versions for Thrift binary frontend

SSLv2,SSLv3,TLSv1.1

kyuubi.frontend.thrift.http.ssl.protocol.blacklist

Forbidden SSL versions for Thrift HTTP frontend

SSLv2,SSLv3,TLSv1.1

kyuubi.ha.addresses

External Kyuubi instance addresses

<hostname_1>:2181, …​, <hostname_N>:2181

kyuubi.ha.namespace

The root directory for the service to deploy its instance URI

kyuubi

kyuubi.metadata.store.jdbc.database.type

A database type for the server metadata store. Possible values: SQLITE, MYSQL, POSTGRESQL

POSTGRESQL

kyuubi.metadata.store.jdbc.url

A JDBC URL for the server metadata store

jdbc:postgresql://{{ groups['adpg.adpg'][0] | d(omit) }}:5432/kyuubi

kyuubi.metadata.store.jdbc.driver

A JDBC driver classname for the server metadata store

org.postgresql.Driver

kyuubi.metadata.store.jdbc.user

A username for the server metadata store

kyuubi

kyuubi.metadata.store.jdbc.password

A password for the server metadata store

 — 

kyuubi.metrics.enabled

Enables metrics collection from Kyuubi Server

false

kyuubi.metrics.reporters

Monitoring service for collecting metrics

PROMETHEUS

kyuubi.metrics.prometheus.port

Prometheus service port

10019

kyuubi.metrics.prometheus.path

Prometheus service metrics endpoint

/metrics

kyuubi.frontend.thrift.binary.ssl.enabled

Indicates whether to use the SSL encryption in the Thrift binary mode

false

kyuubi.frontend.thrift.http.use.SSL

Indicates whether to use the SSL encryption in the Thrift HTTP mode

false

kyuubi.frontend.ssl.keystore.type

Type of the SSL certificate keystore

 — 

kyuubi.frontend.ssl.keystore.path

Path to the SSL certificate keystore

 — 

kyuubi.frontend.ssl.keystore.password

Password for the SSL certificate keystore

 — 

kyuubi.frontend.thrift.http.ssl.keystore.path

Path to the SSL certificate keystore

 — 

kyuubi.frontend.thrift.http.ssl.keystore.password

Password for the SSL certificate keystore

 — 

kyuubi.authentication

Authentication type. Possible values: NONE, LDAP, or LDAP,KERBEROS

NONE

kyuubi.ha.zookeeper.acl.enabled

Indicates whether the ZooKeeper ensemble is kerberized

false

kyuubi.ha.zookeeper.auth.type

ZooKeeper authentication type. Possible values: NONE, KERBEROS

NONE

kyuubi.ha.zookeeper.auth.principal

Kerberos principal name used for ZooKeeper authentication

 — 

kyuubi.ha.zookeeper.auth.keytab

Path to Kyuubi Server’s keytab used for ZooKeeper authentication

 — 

kyuubi.kinit.principal

Name of the Kerberos principal

 — 

kyuubi.kinit.keytab

Path to Kyuubi Server’s keytab

 — 

kyuubi.spnego.principal

Name of the SPNego service principal. Set only if using SPNego in authentication

 — 

kyuubi.spnego.keytab

Path to the SPNego service keytab. Set only if using SPNego in authentication

 — 

kyuubi.engine.hive.java.options

Extra Java options for the Hive query engine

 — 
LDAP Security
Parameter Description Default value

kyuubi.authentication.ldap.url

A whitespace-separated list of LDAP connection URLs

 — 

kyuubi.authentication.ldap.domain

An LDAP domain

 — 

kyuubi.authentication.ldap.binddn

The distinguished name of the user to bind to for user/group searches. If not set, the name of the user trying to authenticate will be used. For example, CN=bindUser,CN=Users,DC=subdomain,DC=domain,DC=com

 — 

kyuubi.authentication.ldap.bindpw

The password for the bind user. Required only if kyuubi.authentication.ldap.binddn was specified

 — 

kyuubi.authentication.ldap.baseDN

An LDAP base DN

 — 

kyuubi.authentication.ldap.groupClassKey

An LDAP attribute name on the group entry to be used in the LDAP group searches. For example: group, groupOfNames, or groupOfUniqueNames

 — 

kyuubi.authentication.ldap.groupDNPattern

A colon-separated list of the patterns to use to find DNs for group entities in this directory. Use %s where the actual group name needs to be put. For example: CN=%s,CN=Groups,DC=subdomain,DC=domain,DC=com

 — 

kyuubi.authentication.ldap.groupFilter

A comma-separated list of the LDAP group names (short names, not full DNs). For example: HiveAdmins,HadoopAdmins,Administrators

 — 

kyuubi.authentication.ldap.groupMembershipKey

An LDAP attribute name on the group object that contains the list of distinguished names for the user, group, and contact objects that are members of the group. For example: member, uniqueMember, or memberUid

 — 

kyuubi.authentication.ldap.guidKey

An LDAP attribute name whose values are unique in this LDAP server. For example: uid or CN

 — 

kyuubi.authentication.ldap.userDNPattern

A colon-separated list of patterns to use to find DNs for users in this directory. Use %s where the actual user name needs to be put. For example: CN=%s,CN=Users,DC=subdomain,DC=domain,DC=com

 — 

kyuubi.authentication.ldap.userFilter

A comma-separated list of the LDAP usernames (short names, not full DNs). For example: hiveuser,impalauser,hiveadmin,hadoopadmin

 — 

kyuubi.authentication.ldap.userMembershipKey

An LDAP attribute name on the user object that contains groups of which the user is a direct member, except for the primary group, which is represented by the primaryGroupId. For example: memberOf

 — 
kyuubi-env.sh
Parameter Description Default value

KYUUBI_HOME

Kyuubi home directory

/usr/lib/kyuubi

KYUUBI_CONF_DIR

Directory that stores Kyuubi configurations

/etc/kyuubi/conf

KYUUBI_LOG_DIR

Kyuubi server log directory

/var/log/kyuubi

KYUUBI_PID_DIR

Directory that stores the Kyuubi instance .pid-file

/var/run/kyuubi

KYUUBI_ADDITIONAL_CLASSPATH

Additional classpath entries to be added to Kyuubi’s classpath. For example, SSM libraries for statistics collection. A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • /usr/lib/ssm/lib/smart*

HADOOP_HOME

Hadoop home directory

/usr/lib/hadoop

HADOOP_LIB_DIR

Directory that stores Hadoop libraries

${HADOOP_HOME}/lib

KYUUBI_JAVA_OPTS

Java parameters for Kyuubi

-Djava.library.path=${HADOOP_LIB_DIR}/native/ -Djava.io.tmpdir={{ cluster.config.java_tmpdir | d('/tmp') }}

HADOOP_CLASSPATH

A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • $HADOOP_CLASSPATH

  • /usr/lib/ssm/lib/smart*

HADOOP_CONF_DIR

Directory that stores Hadoop configurations

/etc/hadoop/conf

SPARK_HOME

Spark home directory

/usr/lib/spark3

SPARK_CONF_DIR

Directory that stores Spark configurations

/etc/spark3/conf

FLINK_HOME

Flink home directory

/usr/lib/flink

FLINK_CONF_DIR

Directory that stores Flink configurations

/etc/flink/conf

FLINK_HADOOP_CLASSPATH

Additional Hadoop JAR files required to use the Kyuubi Flink engine. A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • $(hadoop classpath)

  • /usr/lib/ssm/lib/smart*

HIVE_HOME

Hive home directory

/usr/lib/hive

HIVE_CONF_DIR

Directory that stores Hive configurations

/etc/hive/conf

HIVE_HADOOP_CLASSPATH

Additional Hadoop JAR files required to use the Kyuubi Hive engine A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • $(hadoop classpath)

  • /etc/tez/conf/

  • /usr/lib/tez/*

  • /usr/lib/tez/lib/*

  • /usr/lib/ssm/lib/smart*
Others
Parameter Description Default value

Custom properties

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed for the Kyuubi service

 — 

Custom kyuubi-env.sh

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file kyuubi-env.sh

 — 

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

 
[Service]
LimitCPU=
LimitFSIZE=
LimitDATA=
LimitSTACK=
LimitCORE=
LimitRSS=
LimitNOFILE=
LimitAS=
LimitNPROC=
LimitMEMLOCK=
LimitLOCKS=
LimitSIGPENDING=
LimitMSGQUEUE=
LimitNICE=
LimitRTPRIO=
LimitRTTIME=

Custom log4j2-repl.xml

The contents of the log4j2-repl.xml configuration file

log4j2-repl.xml

log4j2.xml

The contents of the log4j2.xml configuration file

log4j2.xml
Ulimit settings
Parameter Description Corresponding option of the ulimit command in CentOS

LimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

LimitFSIZE

Maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

LimitDATA

Maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

LimitSTACK

Maximum stack size allocated to a process, in kilobytes

stack size ( -s)

LimitCORE

Maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

LimitRSS

The maximum amount of RAM memory (resident set size) that can be allocated to a process, in kilobytes

max memory size ( -m)

LimitNOFILE

Maximum number of open file descriptors allowed for the process

open files ( -n)

LimitAS

Maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

LimitNPROC

Maximum number of processes

max user processes ( -u)

LimitMEMLOCK

Maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

LimitLOCKS

Maximum number of files locked by a process

file locks ( -x)

LimitSIGPENDING

Maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

LimitMSGQUEUE

Maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

LimitNICE

Maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

LimitRTPRIO

Maximum real-time scheduling priority level

real-time priority ( -r)

LimitRTTIME

Maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)
Found a mistake? Seleсt text and press Ctrl+Enter to report it