Logging in Zeppelin

Konstantin Alpashkin

Zeppelin server generates text logs that can be useful for analyzing the cause of different errors that may occur while working with interpreters, notes, etc. Under the hood, Zeppelin uses the Log4j framework for logging. To configure Zeppelin logging processes, use the Custom log4j.properties settings in ADCM. More details on using these settings are available in the Configure logging with ADCM section below.

Logs location and format

By default, Zeppelin logs are located in the /var/log/zeppelin directory on the hosts where Zeppelin Server is deployed. To view these logs, do the following:

  1. Connect to the Zeppelin Server host (via SSH, etc.) and run the command:

    $ ls -la /var/log/zeppelin

    The result can look like this:

    total 1768
drwxr-xr-x.  2 zeppelin zeppelin   4096 Dec  8 10:22 .
drwxr-xr-x. 19 root     root       4096 Dec  8 06:10 ..
-rw-r--r--.  1 root     root        555 Dec  7 06:54 install-interpreter.log
-rw-r--r--.  1 root     root       1665 Dec  6 16:13 install-interpreter.log.2021-12-06
-rw-r--r--.  1 zeppelin zeppelin   2656 Dec  8 10:30 zeppelin-interpreter-jdbc-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin   5918 Dec  6 12:24 zeppelin-interpreter-jdbc-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-06
-rw-r--r--.  1 zeppelin zeppelin  82200 Dec  7 18:15 zeppelin-interpreter-jdbc-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-07
-rw-r--r--.  1 zeppelin zeppelin   2376 Dec  7 17:27 zeppelin-interpreter-phoenix2-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin   4638 Dec  8 10:31 zeppelin-interpreter-phoenix-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin 112258 Dec  6 16:11 zeppelin-interpreter-phoenix-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-06
-rw-r--r--.  1 zeppelin zeppelin  17404 Dec  7 17:27 zeppelin-interpreter-phoenix-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-07
-rw-r--r--.  1 zeppelin zeppelin   3951 Dec  7 18:15 zeppelin-interpreter-python-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin   4134 Dec  6 16:01 zeppelin-interpreter-sh-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin  61835 Dec  8 12:09 zeppelin-zeppelin-bds-adh-2.ru-central1.internal.log
-rw-r--r--.  1 zeppelin zeppelin 722037 Dec  6 16:30 zeppelin-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-06
-rw-r--r--.  1 zeppelin zeppelin 662981 Dec  7 18:15 zeppelin-zeppelin-bds-adh-2.ru-central1.internal.log.2021-12-07
-rw-r--r--.  1 zeppelin zeppelin  78400 Dec  8 06:11 zeppelin-zeppelin-bds-adh-2.ru-central1.internal.out
    NOTE
    Logs are stored on the local file system of the host, not in HDFS.

  2. To view the content of a specific log file, run the following:

    $ tail /var/log/zeppelin/zeppelin-zeppelin-bds-adh-2.ru-central1.internal.log

    The output can be similar to:

     INFO [2021-12-08 10:31:02,382] ({pool-2-thread-4} RemoteInterpreterManagedProcess.java[start]:115) - Thrift server for callback will start. Port: 38183
 INFO [2021-12-08 10:31:02,882] ({pool-2-thread-4} RemoteInterpreterManagedProcess.java[start]:190) - Run interpreter process [/usr/lib/zeppelin/bin/interpreter.sh, -d, /usr/lib/zeppelin/interpreter/jdbc, -c, 10.92.6.9, -p, 38183, -r, :, -l, /usr/lib/zeppelin/local-repo/phoenix, -g, phoenix]
 INFO [2021-12-08 10:31:04,138] ({pool-11-thread-1} RemoteInterpreterManagedProcess.java[callback]:123) - RemoteInterpreterServer Registered: CallbackInfo(host:10.92.6.9, port:34481)
 INFO [2021-12-08 10:31:04,140] ({pool-2-thread-4} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zeppelin.jdbc.JDBCInterpreter
 INFO [2021-12-08 10:31:04,239] ({pool-2-thread-4} RemoteInterpreter.java[call]:142) - Open RemoteInterpreter org.apache.zeppelin.jdbc.JDBCInterpreter
 INFO [2021-12-08 10:31:04,240] ({pool-2-thread-4} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:436) - Push local angular object registry from ZeppelinServer to remote interpreter group phoenix:shared_process
 INFO [2021-12-08 10:31:04,830] ({pool-2-thread-4} NotebookServer.java[afterStatusChange]:2314) - Job 20211207-082338_940138511 is finished successfully, status: FINISHED
 INFO [2021-12-08 10:31:04,836] ({pool-2-thread-4} VFSNotebookRepo.java[save]:196) - Saving note:2GS5T8MS7
 INFO [2021-12-08 10:31:04,838] ({pool-2-thread-4} SchedulerFactory.java[jobFinished]:120) - Job 20211207-082338_940138511 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-phoenix:shared_process-shared_session
 INFO [2021-12-08 12:09:00,355] ({qtp837764579-66} NotebookServer.java[onClose]:372) - Closed connection to 10.92.2.6 : 54643. (1001) null

Log types

There are several types of logs generated by Zeppelin. Their description is listed below.

Log name Attributes Description

zeppelin-zeppelin-<host_name>.log

<host_name> — Zeppelin host name

Contains information about basic operations with connections, jobs, notes: opening, closing, saving, etc.

zeppelin-zeppelin-<host_name>.out

<host_name> — Zeppelin host name

Stores Zeppelin console output for performed operations, i.e. errors, returned in the web UI

zeppelin-interpreter-<interpreter_name>-zeppelin-<host_name>.log

  • <host_name> — Zeppelin host name.

  • <interpreter_name> — interpreter name.

Contains information reflecting the activity of a specific interpreter

install-interpreter.log

 — 

Contains information about installing new interpreters

Configure logging with ADCM

With the use of Log4j, configuring Zeppelin logging assumes editing the log4j.properties configuration file. This file is located in the /etc/zeppelin/conf directory on the host where the Zeppelin Server component is installed, and Zeppelin reads this file during the startup. However, instead of editing the file manually on each host, ADCM provides a way to update the Log4j configuration on all ADH hosts at once. For this:

  1. In ADCM, go to Clusters → <your_cluster_name> → Services → Zeppelin → Primary Configuration and enable the Show advanced option.

  2. Click Custom log4j.properties and edit the configuration. The Log4j configuration specified via ADCM UI will overwrite the contents of the /etc/zeppelin/conf/log4j.properties file during the service restart. For detailed reference on supported Log4j properties, see Log4j documentation.

  3. Save the Zeppelin service configuration.

  4. Restart the Zeppelin service.

Default log4j.properties

Below is the default log4j.properties file used by Zeppelin. The major properties are highlighted to help you get started with Log4j configuration.

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
log4j.rootLogger = INFO, dailyfile (1)

log4j.appender.stdout = org.apache.log4j.ConsoleAppender (2)
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n (5)

log4j.appender.dailyfile.DatePattern=.yyyy-MM-dd
log4j.appender.dailyfile.Threshold = INFO
log4j.appender.dailyfile = org.apache.log4j.DailyRollingFileAppender (3)
log4j.appender.dailyfile.File = ${zeppelin.log.file} (4)
log4j.appender.dailyfile.layout = org.apache.log4j.PatternLayout
log4j.appender.dailyfile.layout.ConversionPattern=%5p [%d] ({%t} %F[%M]:%L) - %m%n (5)
1 Sets the root logging level (INFO) for the default logger. Events with lower severity are not included in logs. For example, using DEBUG generates more verbose logs, whereas using ERROR will force Zeppelin to log only errors and more critical events.
2 Defines a ConsoleAppender that outputs log events to STDOUT.
3 Defines a DailyRollingFileAppender that writes log events to a file, which is rotated every day.
4 Specifies a path to the log file. The default ${zeppelin.log.file} property points to /var/log/zeppelin.
5 Specifies a pattern for converting event objects to strings. The default pattern has the following structure: 
[Severity] [Date] ([Thread] [File name][Method][Line number]) [Log message]

Configure logging via REST API

The Zeppelin service exposes a REST API through which you can interact with Zeppelin, including logs configuration commands. The REST endpoint to change the logging level is as follows:

PUT http://<zeppelin-host>:<zeppelin-port>/api/log/level/<LOG_LEVEL>

Where:

  • <zeppelin-host> is the address of the host with the running Zeppelin Server.

  • <zeppelin-port> is the port number used by the Zeppelin REST API. The port number can be changed with the zeppelin.server.port property in ADCM. Defaults to 8180.

  • <LOG_LEVEL> is the logging level to be set. The possible severity levels are: ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.

Example:

$ curl -X PUT http://localhost:8180/api/log/level/WARN

The response:

{"status":"OK"}

Common tasks

Below are some common tasks that may arise while configuring Zeppelin logging with Log4j.

Change the default log level

 
To change the root logging level (defaults to INFO), use the log4j.rootLogger=<LEVEL>,<APPENDER> property in the Custom log4j.properties section in ADCM. You can set a root log level, which will be inherited by non-root Log4j loggers, or you can set log levels individually for a specific logger, for example log4j.logger.org.apache.zeppelin.interpreter=<OTHER_LEVEL>.

Change default logs location

 
By default, Zeppelin logs are stored under the /var/log/zeppelin directory. To change this directory, modify the log4j.appender.<appender-name>.File property where <appender-name> is the name of the Log4j appender bound to the root logger.

Add a custom logger

 
The following example shows how to add a custom logger that writes logs to a separate directory. Also, the example shows how to send log events from a Spark interpreter code to the log file.

  1. Add the following properties to your Custom log4j.properties section in ADCM. The major properties are highlighted with comments.

    # ...
# New logger definition:
log4j.logger.MyCustomLogger=DEBUG, mycustomappender (1)
log4j.appender.mycustomappender=org.apache.log4j.RollingFileAppender (2)
log4j.appender.mycustomappender.File = /var/log/zeppelin/customlogger/test.log (3)
log4j.appender.mycustomappender.rollingPolicy.FileNamePattern=/var/log/zeppelin/customlogger/test.%i.log (4)
log4j.appender.mycustomappender.layout=org.apache.log4j.PatternLayout
log4j.appender.mycustomappender.layout.conversionPattern=[%p] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{10} - %m %n
log4j.appender.mycustomappender.MaxFileSize=10MB (5)
log4j.appender.mycustomappender.MaxBackupIndex=10 (6)
{% endraw %}
    1 Defines a new logger and binds an appender.
    2 Defines a RollingFileAppender to back up the log files when they reach a certain size.
    3 Specifies the location for log files generated by the logger.
    4 Specifies the file name pattern when a log file gets rolled over.
    5 Sets the maximum log file size for making a rollover.
    6 Sets the maximum number of backup log files to keep.

  2. Save the configuration and restart the Zeppelin service.

  3. Run a sample Spark notebook to generate some log entries. For this, create a new Spark note using Zeppelin UI and run the following code in the notebook:

    import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

val logger: Logger = LoggerFactory.getLogger("MyCustomLogger");
logger.info("Test info message");

  4. Check the logs generated by the custom logger. For this, connect to the Zeppelin Server host and run the command:

    $ ls -l /var/log/zeppelin/customlogger

    The output:

    -rw-r--r-- 1 zeppelin zeppelin 114 Nov 25 13:14 test.log

    The test.log file contains the only log record generated from the Spark notebook:

    [INFO] 2024-11-22 13:14:28.569 [FIFOScheduler-interpreter_453440904-Worker-1] MyCustomLogger - Test info message
Found a mistake? Seleсt text and press Ctrl+Enter to report it