Logging in YARN

YARN records command output and processes events in text logs. This information can be useful when diagnosing technical issues.

Each YARN component stores logs locally, on the host it occupies. The logs have the .log extension and are located in the var/log/hadoop-yarn/ directory. In the same directory, you can find .out files that contain component restart information.

All logs of all YARN components are also available in YARN UIs.

The logs naming convention is hadoop-yarn-<component>-<host>.log.

Where:

The YARN logs configuration shares the same log4j.properties file with HDFS.

You can search through the logs for a specific information, like error messages. To do this, connect to the host with the component whose logs you want to inspect, and use a grep command.

For example:

This command searches for messages containing the word error in the CLI log, located on the elenas-adh3.ru-central1.internal host. The -i option allows you to ignore case distinctions. The -A3 -B1 options expand the output to one line before and three lines after the line containing the error.

The example output:

YARN uses the Log4j2 library for logging which supports the following log levels (from least to most informative):

The Log4j4 loggers also accept logging level values: OFF — for switching off the logging, and ALL — for allowing all types of messages.

Enabling one level of logging will enable this level and all levels above it. For example, if you set the logging level to WARN, then only warnings, errors, and fatal messages would get into the log files, but not INFO, DEBUG, and TRACE.

Applications submitted to YARN generate their own logs. To read application logs, use the logs command.

Log aggregation is a YARN feature that, when enabled, copies application log files into a single file in HDFS after an application has finished.

The default directory for aggregated application logs is hdfs://<NameNode address>:8020/logs/yarn/logs/, where <NameNode address> is the IP address or FQDN of the active NameNode.

You can configure log aggregation by changing its properties in the yarn-site.xml configuration file.

To change logging properties via ADCM: