Logging in YARN

Elena Kostyuchenko

Contents

Overview
Grep logs
Logging levels
Container logs
Log aggregation

Overview

YARN records command output and processes events in text logs. This information can be useful when diagnosing technical issues.

Each YARN component stores logs locally, on the host it occupies. The logs have the .log extension and are located in the var/log/hadoop-yarn/ directory. In the same directory, you can find .out files that contain component restart information.

All logs of all YARN components are also available in YARN UIs.

The logs naming convention is hadoop-yarn-<component>-<host>.log.

Where:

<component> — a component name, for example, ResourceManager;
<host> — the FQDN of the component host.

The YARN logs configuration shares the same log4j.properties file with HDFS.

Grep logs

You can search through the logs for a specific information, like error messages. To do this, connect to the host with the component whose logs you want to inspect, and use a grep command.

For example:

$ cat /var/log/hadoop-yarn/hadoop-yarn-nodemanager-elenas-adh3.ru-central1.internal.log | grep -i -A3 -B1 error

This command searches for messages containing the word error in the CLI log, located on the elenas-adh3.ru-central1.internal host. The -i option allows you to ignore case distinctions. The -A3 -B1 options expand the output to one line before and three lines after the line containing the error.

The example output:

2024-04-08 13:26:24,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager bound to 0.0.0.0/0.0.0.0:8041
2024-04-08 13:26:24,912 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler: Unknown event arrived at ContainerScheduler: EventType: RECOVERY_COMPLETED
2024-04-08 13:26:24,915 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:8042
2024-04-08 13:26:24,935 INFO org.eclipse.jetty.util.log: Logging initialized @56117ms
2024-04-08 13:26:25,004 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.

Logging levels

YARN uses the Log4j2 library for logging which supports the following log levels (from least to most informative):

FATAL — indicates that an operation can’t continue execution and will terminate.
ERROR — notifies that a program is not working correctly or has stopped.
WARN — warns about potential problems. This doesn’t mean that a program is not working, but raises a concern.
INFO — informs regarding the program lifecycle or state.
DEBUG — prints debugging information about internal states of the program.
TRACE — prints messages tracing the execution flow of a program.

The Log4j4 loggers also accept logging level values: OFF — for switching off the logging, and ALL — for allowing all types of messages.

Enabling one level of logging will enable this level and all levels above it. For example, if you set the logging level to WARN, then only warnings, errors, and fatal messages would get into the log files, but not INFO, DEBUG, and TRACE.

Container logs

Applications submitted to YARN generate their own logs. To read application logs, use the logs command.

Log aggregation

Log aggregation is a YARN feature that, when enabled, copies application log files into a single file in HDFS after an application has finished.

The default directory for aggregated application logs is hdfs://<NameNode address>:8020/logs/yarn/logs/, where <NameNode address> is the IP address or FQDN of the active NameNode.

You can configure log aggregation by changing its properties in the yarn-site.xml configuration file.

YARN log aggregation parameters
Parameter	Description	Default value
yarn.log.server.url	A URL for displaying the application logs. If set to the JobHistory server, the logs will be available in the Resource Manager UI	—
yarn.log-aggregation-enable	Enables log aggregation	true
yarn.log-aggregation.retain-seconds	The amount of time (in seconds) before the logs will be deleted. Setting this to `-1` or lower will disable the deletion of logs. Entering a value too small will overload the NameNode	172800
yarn.nodemanager.remote-app-log-dir	The application logs directory in HDFS	/logs

To change logging properties via ADCM:

On the Clusters page, select the desired cluster.
Go to the Services tab and click at YARN.
Find the yarn-site.xml field.
Select the required parameter and make the necessary changes.
Confirm changes by clicking Save.
In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true, and click Run.

Found a mistake? Seleсt text and press Ctrl+Enter to report it