Logging
ADQM writes general and error text logs that can be useful for analyzing the causes of different errors. Additionally, you can enable logs of other types (for example, collect information about query execution, dependent views, threads, stack traces) that will be saved to special system tables in ADQM.
Log files
By default, ADQM saves the following logs to the /var/log/clickhouse-server directory:
-
clickhouse-server.log — main log of an ADQM server (the log verbosity level is configured by a special parameter described below);
-
clickhouse-server.err.log — error log (contains records of the
warning,error,critical, andfatallevels).
You can change the storage directory for log files using the Log path parameter in the Directories section on the ADQMDB service’s configuration page of the ADCM interface.
On the same page, you can configure logging parameters in the Log settings section that is visible when the Show advanced option is activated:
-
Logging level — verbosity level of messages written to the clickhouse-server.log file. Possible values:
trace,debug,information,notice,warning,error,critical,fatal,none.The logging level for clickhouse-server.err.log cannot be separately configured.
-
Log file size, MB — log file size (in MB). Once a file reaches the specified size, ADQM archives and renames it, and creates a new log file in its place.
-
Log files count — number of archived log files that ADQM stores.
|
IMPORTANT
After you have set necessary parameters, do not forget to click Save and execute the Reconfig and restart action for the ADQMDB service to save configuration changes and restart the service.
|
To view the content of a log file, use the following command:
$ sudo less /var/log/clickhouse-server/clickhouse-server.logThe output can be similar to:
2023.09.08 10:59:14.549245 [ 31027 ] {} <Information> Application: Will watch for the process with pid 31036
2023.09.08 10:59:14.549357 [ 31036 ] {} <Information> Application: Forked a child process to watch
2023.09.08 10:59:14.549768 [ 31036 ] {} <Information> SentryWriter: Sending crash reports is disabled
2023.09.08 10:59:14.549851 [ 31036 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2023.09.08 10:59:14.674600 [ 31036 ] {} <Information> : Starting ClickHouse 22.3.7.28 with revision 54460, build id: 6F5FF3E355A49AA523CB708AFAC0EB6C5EE73278, PID 31036
2023.09.08 10:59:14.674685 [ 31036 ] {} <Information> Application: starting up
2023.09.08 10:59:14.674703 [ 31036 ] {} <Information> Application: OS name: Linux, version: 3.10.0-1160.66.1.el7.x86_64, architecture: x86_64
2023.09.08 10:59:14.678336 [ 31036 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2023.09.08 10:59:14.678369 [ 31036 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2023.09.08 10:59:14.678516 [ 31036 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2023.09.08 10:59:14.678536 [ 31036 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2023.09.08 10:59:14.981938 [ 31036 ] {} <Warning> Application: Calculated checksum of the binary: 1841F46AA9C864447AE24AC0D983E86C. There is no information about the reference checksum.
2023.09.08 10:59:14.991365 [ 31036 ] {} <Debug> Application: rlimit on number of file descriptors is 500000
2023.09.08 10:59:14.991394 [ 31036 ] {} <Debug> Application: Initializing DateLUT.
2023.09.08 10:59:14.991397 [ 31036 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2023.09.08 10:59:14.991426 [ 31036 ] {} <Debug> Application: Setting up /var/lib/clickhouse/tmp/ to store temporary data in it
2023.09.08 10:59:14.993872 [ 31036 ] {} <Debug> Application: Initiailizing interserver credentials.
2023.09.08 10:59:14.994558 [ 31036 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/config.xml'
2023.09.08 10:59:14.995658 [ 31036 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-server/config.xml', performing update on configuration
...
If an ADQM server has started successfully, the log contains the following messages:
-
<Information> Application: starting up— server starts. -
<Information> Application: Ready for connections.— server is running and ready to handle client connections.
If an ADQM server start has failed with a configuration error, the log provides an <Error> string with an error description. For example:
...
2023.09.08 10:59:14.991426 [ 31036 ] {} <Error> Application: DB::Exception: A setting 'query_profiler_cpu_time_period_ns' appeared at top level in config /etc/clickhouse-server/config.xml. But it is user-level setting that should be located in users.xml inside <profiles> section for specific profile. You can add it to <profiles><default> if you want to change default value of this setting. You can also disable the check - specify <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> in the main configuration file.
...
If there is no an error message at the end of the log file, look through the entire file starting with the <Information> Application: starting up string.
Log tables
ADQM supports a set of system tables to store logs of different types. To start logging, activate the corresponding parameter in the Log settings section on the configuration page of the ADQMDB service — ADQM will create a system table with the same name as the parameter and will begin writing the log to it. Parameters responsible for enabling different logging types are listed in the table below. This table also describes which kind of information each log contains.
| Parameter to start logging | Log table | Log table data |
|---|---|---|
query_log |
Information about executed queries (for example, query start time, duration of query processing, error messages) |
|
trace_log |
Stack traces collected by the sampling query profiler (useful for deep performance analysis) |
|
query_views_log |
Information about dependent views executed when processing queries received with the |
|
query_thread_log |
Information about threads that execute queries received with the |
|
part_log |
Information about events related to data parts of MergeTree tables (for example, inserting/merging/deleting data parts, downloading data from replicas, moving data between disks) |
|
crash_log |
Information about stack traces for fatal errors |
|
metric_log |
Historical values of metrics from the |
|
asynchronous_metric_log |
Historical values of metrics from the |
|
asynchronous_insert_log |
Information about asynchronous inserts |
|
NOTE
Data of tables with system logs is stored in a file system and is not deleted automatically. If you remove a table from a file system, ADQM will create an empty table again the next time when data is written to the log. If a system table schema changed when the ADQM server was updated, the current table is renamed and a new one is created. |
In the Log settings section, you can set the following settings that are common to all log tables:
-
Database — database to which log tables belong;
-
Flush interval, milliseconds — interval for flushing data from the memory buffer to tables (in milliseconds);
-
TTL, days — lifetime of logs (in days).
