Zeppelin performance tuning

Konstantin Alpashkin

This article describes optimization techniques for the Zeppelin service and highlights key factors that affect the Zeppelin performance.

Memory

The major factor that affects the Zeppelin service is the amount of RAM available for the Zeppelin Server component. You can speed up the Zeppelin service performance by allocating more RAM for the Zeppelin JVM process. To increase -Xms/-Xmx parameters, use Zeppelin Server Heap Memory settings in ADCM (Clusters → <clusterName> → Services → Zeppelin → Primary configuration) and restart the service.

When estimating the amount of memory that will satisfy your setup, take into account the following major characteristics:

  • The number of users that will connect to the Zeppelin server in parallel.

  • The binding mode used by interpreters.

Interpreter binding mode

Zeppelin interpreters run as separate JVM processes that execute notes code and communicate with the Zeppelin daemon. Zeppelin provides several interpreter binding modes that define ways for launching new interpreter processes. For a multi-user environment, this mode is the most RAM-expensive. Be sure the ADH host with the Zeppelin Server component can handle the number of interpreter JVMs for each user note. These modes are shared, scoped, and isolated.

  • Shared.

    In this mode, a single JVM process with one interpreter session serves all notebooks. This approach ensures less memory consumption for a multi-user environment and also provides cross-note collaboration capabilities, such as accessing variables created in other notes. The latter can be especially beneficial for reusing "heavy" objects like SparkContext, SparkSession, etc.

    To increase the amount of RAM allocated by the Zeppelin Server to each interpreter process, specify the ZEPPELIN_INTP_MEM parameter in the Custom zeppelin-env.sh field in ADCM (Clusters → <clusterName> → Services → Zeppelin → Primary configuration and enable Show advanced). For example:

    ZEPPELIN_INTP_MEM="-Xms2048m -Xmx4096m -XX:MaxMetaspaceSize=1024m"

  • Scoped.

    In this mode, Zeppelin runs a single JVM per interpreter and executes each note in its own dedicated session, depending on whether Per Note or Per User scope is selected.

  • Isolated.

    In this mode, each note spawns a new interpreter JVM process (in the Per note scope). The overall memory consumption increases, however, Zeppelin users can run notes in a fully isolated environment, avoiding possible conflicts. Additionally, the restart of an interpreter by one user does not affect others.

Tweaking with binding modes can help optimize resources consumed by Zeppelin and select an approach that best fits your setup.

Spark optimizations

When working with Spark interpreters, to speed up Spark jobs execution, you can use the optimization techniques described in the Spark performance tuning article. To specify Spark configuration properties, use the Custom zeppelin-env.sh field in ADCM (Clusters → <clusterName> → Services → Zeppelin → Primary configuration and enable Show advanced). For example:

export SPARK_SUBMIT_OPTIONS="--executor-memory 4G"
NOTE
The first run of a Spark note is always slower since the Spark interpreter has to instantiate heavy context objects and create Application Master.

Logging

Setting the Zeppelin log level to DEBUG may slow down the UI response, especially when saving an interpreter configuration during the creation of a new notebook. Enable the DEBUG level only for troubleshooting.

To change the logging level in Zeppelin, use the Custom log4j.properties settings in ADCM (Clusters → <clusterName> → Services → Zeppelin → Primary configuration and enable Show advanced).

Browser

Use a modern web browser capable of loading large notebooks (10+ MB), visualizing tens and hundreds of megabytes of output data, supporting third-party visualization libraries, etc. For example, older versions of IE have been reported to have issues with login/authentication actions.

Clearing massive paragraphs' output periodically will reduce the load on the browser/JVM since Zeppelin stores the last execution output internally.

Other

  • Restart interpreters periodically. Long-running interpreters may slow down over time.

  • Limit the number of output rows returned by SQL queries. For example, for Spark interpreters, this can be achieved using the zeppelin.spark.maxResult interpreter configuration property.

  • Use %table for more efficient visualization of large result sets.

Found a mistake? Seleсt text and press Ctrl+Enter to report it