Logging in Sqoop
Overview
Sqoop records information related to its components in text logs. This information can be useful when diagnosing technical issues.
Sqoop Metastore keeps logs locally, on the host it occupies. The logs have the .log extension and are located in the /var/log/sqoop/ directory.
The logs naming convention is sqoop-metastore.log.<date>, where <date> is the day, month, and a year the log has been created.
Since some of the Sqoop commands, like import or export, use YARN, the information related to those commands will be available in YARN logs and YARN UIs.
By default, the Sqoop client doesn’t create text logs, it outputs all information about command execution to the console.
Logging configuration
You can configure logging settings for Sqoop via the log4j.properties configuration file. Sqoop has separate logging configuration files for the metastore and for clients.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
org.apache.sqoop=INFO, console
org.apache.hadoop=INFO, console
org.apache.hive=INFO, console
sqoop.root.logger=INFO,console,DRFA
hive.root.logger=INFO,console,DRFA
sqoop.log.dir=/var/log/sqoop/
sqoop.log.file=sqoop-metastore.log
# Define the root logger to the system property "sqoop.root.logger".
log4j.rootLogger=${sqoop.root.logger}
#
# DRFA
# Daily Rolling File Appender
#
log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA.File=${sqoop.log.dir}/${sqoop.log.file}
log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFA.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
#
# console
# Add "console" to rootlogger above if you want to use this
#
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
org.apache.sqoop=INFO, console
org.apache.hadoop=INFO, console
org.apache.hive=INFO, console
sqoop.root.logger=INFO,console
hive.root.logger=INFO,console
# Define the root logger to the system property "sqoop.root.logger".
log4j.rootLogger=${sqoop.root.logger}
#
# console
# Add "console" to rootlogger above if you want to use this
#
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
The log4j.properties file contains logger settings, appender settings, and other logging properties. The parameters you set for a logger define what Sqoop reports to the system. The parameters you set for the appenders influence the information from loggers that ends up in the log files.
The default logging configuration of the Sqoop Client doesn’t have a FileAppender, so it outputs everything to the console.
For more detailed information on Log4j architecture, see the Apache Logging Services documentation.
To edit log4j.properties on all hosts at once, use ADCM:
-
On the Clusters page, select the desired cluster.
-
Go to the Services tab and click at Sqoop.
-
Toggle the Show advanced option and select either log4j_properties template or log4j-cli.properties template.
-
Make the necessary changes to the default file.
-
Confirm changes by clicking Save.
-
In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to
true, and click Run.
Logging levels
Sqoop uses the Log4j library for logging which supports the following log levels (from least to most informative):
-
FATAL— indicates that an operation can’t continue execution and will terminate. -
ERROR— notifies that a program is not working correctly or has stopped. -
WARN— warns about potential problems. This doesn’t mean that a program is not working, but raises a concern. -
INFO— informs regarding the program lifecycle or state. -
DEBUG— prints debugging information about internal states of the program. -
TRACE— prints messages tracing the execution flow of a program.
The Log4j loggers also accept logging level values: OFF — for switching off the logging, and ALL — for allowing all types of messages.
Enabling one level of logging will enable this level and all levels above it. For example, if you set the logging level to WARN, then only warnings, errors, and fatal messages would get into the log files, but not INFO, DEBUG, and TRACE.
Console output
Sqoop outputs all command and procces information to the console, which can generate lengthy output.
You can generate logs with command output by adding an option like 1><file-name>.out and 2><file-name>.err.
For example:
$ sqoop eval --connect jdbc:postgresql://127.0.0.1:5432/books --username <user> --password <password> --query "SELECT * FROM book LIMIT 3" 1>select.out 2>select.errThis command saves the SELECT query process information to the select.err file and the result of the query to the select.out file.
The select.out example:
------------------------------------------------------------- | id | title | author_id | public_year | ------------------------------------------------------------- | 1 | Mrs. Dalloway | 1 | 1925 | | 2 | To the Lighthouse | 1 | 1927 | | 3 | To Kill a Mockingbird | 2 | 1960 | -------------------------------------------------------------
The select.err example:
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/sqoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] 2024-06-18 13:52:09,940 (main) [INFO - org.apache.sqoop.Sqoop.<init>(Sqoop.java:94)] Running Sqoop version: 1.4.7_arenadata2 2024-06-18 13:52:09,997 (main) [WARN - org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1153)] Setting your password on the command-line is insecure. Consider using -P instead. 2024-06-18 13:52:10,066 (main) [INFO - org.apache.sqoop.manager.SqlManager.initOptionDefaults(SqlManager.java:98)] Using default fetchSize of 1000
Grep logs
You can search through the logs for a specific information, like error messages. To do this, connect to the host with the component whose logs you want to inspect, and use a grep command.
For example:
$ cat /var/log/sqoop/sqoop-metastore.log.2024-06-14 | grep -i -A3 -B1 errorThis command searches for messages containing the word error in the metastore log. The -i option allows you to ignore case distinctions. The -A3 -B1 options expand the output to one line before and three lines after the line containing the error.