Share levels of Kyuubi engines
Overview
Kyuubi share levels feature enables configuring conditions for sharing engines between users. This feature allows for more flexibility in cluster workload management: more isolated engines provide better engine stability and higher shareability of the engines means more frugal cluster resources management.
The current supported share levels are:
-
CONNECTION
-
USER
-
GROUP
-
SERVER
The default share level is USER. Beyond the main four share levels, you can configure the subdomain setting to create several engines.
All supported share levels can be used together in a single Kyuubi server or cluster. The share levels work the same way for both single-node configurations and in clusters with high availability (HA).
| Share level | Number of engines | Scenario | Isolation degree | Shareability |
|---|---|---|---|---|
CONNECTION |
One engine per session |
Large-scale ad hoc ETL |
High |
Low |
USER |
One engine per user |
Small-scale ad hoc ETL |
Medium |
Medium |
GROUP |
One engine per primary group |
Small-scale ad hoc ETL |
Low |
High |
SERVER |
One engine per cluster |
Admin |
Highest if secured, lowest if unsecured |
Admin only if secured |
Share levels
CONNECTION
The CONNECTION share level creates an isolated engine, available only to one session. Within that session, a user can send multiple operation requests, including metadata calls or queries.
Once the session that created the engine is closed, the engine shuts down as well.
USER
The USER share level isolates the engine by user. Each engine has a specific user as the owner of that engine. Any requests sent from one or multiple clients are redirected to that user’s engine.
If the user has several sessions open, they will belong to the one same SparkContext, but each session will have its own SparkSession instance, which contains separate session state, including temporary views, SQL config, UDFs, etc.
To make a singleton SparkSession instance, which would be shared across sessions, set the value of the kyuubi.engine.single.spark.session parameter to true manually or using ADCM.
The engine with the USER share level shuts down after all sessions connected to it are closed and its time-to-live (TTL) has expired.
GROUP
In the GROUP share level, an engine is shared between users from the same group. All sessions of users in the same group will belong to the group engine.
A group engine has the group name as the name of its owner. The grouping of users follows the Hadoop GroupsMapping. If the primary group is not found, it falls back to the USER level.
The engine with the GROUP share level shuts down after all sessions of all users in the group are closed and its time-to-live (TTL) has expired.
SERVER
The SERVER share level works similar to Spark Thrift Server with high availability. A single engine is created for each single Kyuubi instance.
Subdomain
Using the kyuubi.engine.share.level.subdomain property, you can configure additional isolation policies for USER, GROUP, or SERVER share levels. The subdomain configuration enables creating multiple engines for a single user, group, or server.
If several engines are available, you can choose which engine to use in the JDBC connection URL.
Configuration
| Property | Value | Default | Description |
|---|---|---|---|
kyuubi.engine.share.level (kyuubi.session.engine.share.level) |
USER, CONNECTION, GROUP, SERVER |
USER |
The share level type. It can be set both in the server configuration file or in a connection URL. The latter has a higher priority |
kyuubi.session.engine.idle.timeout |
A number with a time units indicator |
PT30 (30 min) |
Time to live since an engine becomes idle. It can be set both in the server configuration file or in a connection URL. The latter has a higher priority |
kyuubi.engine.share.level.subdomain (kyuubi.engine.share.level.sub.domain) |
A valid ZooKeeper node name |
— |
Additionall engine isolation by znode. It can be set both in the server configuration file or in a connection URL. The latter has a higher priority |