Logging in Solr
Under the hood, Solr uses the Log4j 2 framework for logging. The service generates several types of logs and also provides a Logging UI to view and analyze Solr logs. All the logging settings are defined through Log4j2 configuration files that can be edited via ADCM.
Logs location and format
Solr generates several types of logs to store all the system events. On the file system, these logs are stored as plain text .log files under the /var/log/solr/ directory (default location). Most Solr log events have the following structure:
2024-10-21 15:04:36.923 INFO (coreLoadExecutor-27-thread-1-processing-n:ka-adh-3.ru-central1.internal:8983_solr) [c:test_collection s:shard1 r:core_node2 x:test_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_collection_shard1_replica_n1' using configuration from configset test_collection, trusted=true
The path to the log files is customizable and can be changed by editing the log4j2.xml configuration via ADCM. The sample /var/log/solr/ directory contents is as follows:
-rw-r--r-- 1 solr solr 3141 Nov 6 12:50 solr-8983-console.log (1)
-rw-r--r-- 1 solr solr 703758 Nov 10 20:11 solr_gc.log.0.current (2)
-rw-r--r-- 1 solr solr 4686227 Nov 9 13:00 solr.log (3)
-rw-r--r-- 1 solr solr 113261 Oct 21 15:23 solr.log.1
-rw-r--r-- 1 solr solr 9054280 Oct 28 23:32 solr.log.10
-rw-r--r-- 1 solr solr 178374 Oct 21 15:52 solr.log.2
-rw-r--r-- 1 solr solr 1829020 Oct 22 19:02 solr.log.3
-rw-r--r-- 1 solr solr 40809 Oct 22 19:10 solr.log.4
-rw-r--r-- 1 solr solr 53809 Oct 22 19:14 solr.log.5
-rw-r--r-- 1 solr solr 463368 Oct 22 19:43 solr.log.6
-rw-r--r-- 1 solr solr 68754 Oct 22 19:50 solr.log.7
-rw-r--r-- 1 solr solr 57230 Oct 22 19:54 solr.log.8
-rw-r--r-- 1 solr solr 234734 Oct 22 20:31 solr.log.9
-rw-r--r-- 1 solr solr 0 Oct 21 14:21 solr_slow_requests.log (4)| 1 | Stores Solr’s console output redirected to STDOUT. |
| 2 | Stores details on GC invocations by Solr. |
| 3 | The main log files reflecting Solr Server’s activity. By default, the solr.log.{n} files get rolled over upon reaching certain size. |
| 4 | Stores details about slow queries in your system. |
Configure logging
With the use of Log4j2, configuring logging in Solr assumes editing the corresponding Log4j2 files:
-
log4j2.xml — the main configuration file responsible for logging all Solr Server’s events.
-
log4j2-console.xml — the configuration file responsible for logging the console client’s activity.
Both files are available at /usr/lib/solr/server/resources, and Solr reads these files during the startup. ADCM provides an editor where you can modify the contents of these files. To edit the Log4j2 configuration using ADCM:
-
In ADCM, go to Clusters → <clusterName> → Services → Solr → Components → Solr Server → Primary configuration.
-
Click the Log4j2 configuration file you want to edit (for example, log4j2.xml).
-
Update the configuration. Below you can find the default log4j2.xml configuration with major configuration parameters highlighted. The detailed information about supported Log4j2 properties can be found in Log4j 2 documentation.
-
Save the Solr service configuration.
-
Restart the Solr service.
Default log4j2.xml
Below you can find the default log4j2.xml configuration used by Solr. The major configuration parameters are highlighted with comments.
<?xml version="1.0" encoding="UTF-8"?>
<!-- Configuration for asynchronous logging -->
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT"> (1)
<PatternLayout>
<Pattern> (2)
%maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n
</Pattern>
</PatternLayout>
</Console>
<RollingRandomAccessFile (3)
name="MainLogFile"
fileName="/var/log/solr/solr.log" (4)
filePattern="/var/log/solr/solr.log.%i" > (5)
<PatternLayout>
<Pattern>
%maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy /> (6)
<SizeBasedTriggeringPolicy size="32 MB"/> (7)
</Policies>
<DefaultRolloverStrategy max="10"/> (8)
</RollingRandomAccessFile>
<RollingRandomAccessFile (9)
name="SlowLogFile"
fileName="/var/log/solr/solr_slow_requests.log"
filePattern="/var/log/solr/solr_slow_requests.log.%i" >
<PatternLayout>
<Pattern>
%maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingRandomAccessFile>
</Appenders>
<Loggers> (10)
<AsyncLogger name="org.apache.hadoop" level="warn"/>
<AsyncLogger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
<AsyncLogger name="org.apache.zookeeper" level="warn"/>
<AsyncLogger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">
<AppenderRef ref="SlowLogFile"/>
</AsyncLogger>
<AsyncRoot level="info">
<AppenderRef ref="MainLogFile"/>
<AppenderRef ref="STDOUT"/>
</AsyncRoot>
</Loggers>
</Configuration>
<!-- Configuration for synchronous logging
there _may_ be a very small window where log messages will not be flushed
to the log file on abnormal shutdown. If even this risk is unacceptable, use
the configuration below
-->
<!--Configuration>
...
</Configuration-->| 1 | Defines the console appender named STDOUT that writes logs to STDOUT. |
| 2 | Specifies a pattern for each log record written to the log file by the appender.
For details on PatternLayout syntax, see Log4j2 layouts. |
| 3 | Defines the RollingRandomAccessFile appender that writes main Solr logs to files, which are rotated based on size. |
| 4 | Specifies a log file name. |
| 5 | Specifies a log file name pattern for the rotated log files when a rollover occurs. |
| 6 | The policy triggers a rollover on every Solr startup. |
| 7 | The policy triggers a rollover when the file reaches 32 MB. |
| 8 | Defines the maximum number of archived log files to keep. When this limit is reached, the oldest files are deleted. |
| 9 | Defines the RollingRandomAccessFile appender that writes information about slow requests to files, rotated based on size. |
| 10 | Defines a list of all loggers that log Solr events. |
|
NOTE
By default, the log4j2.xml file has two configurations — for synchronous and asynchronous logging.
The synchronous configuration is disabled by default, as the Log4j’s asynchronous logging provides better performance by executing all I/O operations in a separate thread.
However, in case of an abnormal Solr shutdown, there is a chance that some log messages cannot be flushed to the disk, and thus, may get lost.
If this is unacceptable, use the synchronous Logj2 configuration.
For this, in log4j2.xml, comment the asynchronous configuration part, then uncomment the synchronous configuration block, and restart Solr.
|
Loglevel API
Solr exposes a REST endpoint to update the logging configuration for Solr components.
For example, to set a new logging level for a single Solr component (org.apache.solr.security), submit the following request:
$ curl http://ka-adh-3.ru-central1.internal:8983/solr/admin/info/logging --data-binary "set=org.apache.solr.security:WARN"A request to set a logging level for several Solr components at once (for example, all org.apache.zookeeper children) looks as follows:
$ curl http://ka-adh-3.ru-central1.internal:8983/solr/admin/info/logging --data-binary "set=org.apache.zookeeper:WARN"As a response, Solr returns a JSON document, reflecting the current Log4j2 levels for each Solr component. A sample response is below.
{
"responseHeader":{
"status":0,
"QTime":1},
"watcher":"Log4j2",
"levels":["ALL",
"TRACE",
"DEBUG",
"INFO",
"WARN",
"ERROR",
"FATAL",
"OFF"],
"loggers":[{
"name":"root",
"level":"INFO",
"set":true},
{
"name":"com",
"level":null,
"set":false},
{
"name":"com.codahale",
"level":null,
"set":false},
{
"name":"com.codahale.metrics",
"level":null,
"set":false},
{
"name":"com.codahale.metrics.jmx",
"level":null,
"set":false},
{
"name":"com.codahale.metrics.jmx.JmxReporter",
"level":"INFO",
"set":true},
{
"name":"org",
"level":"INFO",
"set":true},
{
"name":"org.apache",
"level":null,
"set":false},
{
"name":"org.apache.calcite",
"level":null,
"set":false},
{
"name":"org.apache.calcite.avatica",
"level":null,
"set":false},
{
"name":"org.apache.calcite.avatica.remote",
"level":null,
"set":false},
{
"name":"org.apache.calcite.avatica.remote.Driver",
"level":"INFO",
"set":true},
{
"name":"org.apache.hadoop",
"level":"WARN",
"set":true},
....Log slow queries
For search-intensive applications, logging every single query can generate a large amount of excessive logs, and depending on the volume, may potentially impact the performance.
If you need the detailed logs (for example, for automatic log analysis or early detection of possible issues), then logging every query may be useful.
On the other hand, if you are only concerned about warnings and errors related to requests, then you can set the log verbosity to WARN.
However, this brings a potential problem — Solr will not raise alerts for queries that take long, since such queries are still logged at the INFO level.
If you want to use the WARN verbosity level, but still be able to detect slow queries, Solr provides a way to set a latency threshold above which a request is considered slow.
Such long-running requests will be logged at the WARN level to help you identify slow queries in your application.
To enable this feature, add the slowQueryThresholdMillis parameter in the query section of solrconfig.xml.
For example:
<requestHandler name="/select" class="solr.SearchHandler">
<int name="slowQueryThresholdMillis">3000</int>
</requestHandler>Any queries that take longer than the specified threshold will be logged as slow at the WARN level.
These logs are written to a separate log file, which defaults to solr_slow_requests.log.
Logging UI
You can view Solr logs on the Logging page of Solr Admin UI as shown below.
Also, using the Level tab, you can control the amount of logging information generated by individual Solr components and classes. For this, set the desired logging level for the Solr components available on the page. Note that the component nodes are hierarchical, i.e. you can define a logging level for a node, and it will be inherited by all of its children components. This allows changing many categories at once by adjusting the logging level of one parent.
|
NOTE
Using this page, you can only change logging settings for a running Solr instance.
These setting are not persisted for the next Solr run.
|