Monitoring service

Elena Dvoryadkina

The Monitoring service deploys the Prometheus server inside ADQM to collect and store ADQM cluster monitoring metrics, and also supports the ability to use the Grafana web application for visualization and analysis of information. This article describes steps required to add this service to an ADQM cluster.

Overview

When you add the Monitoring service to an ADQM cluster, node exporter is installed on all hosts. It is a monitoring agent that reads host system metrics that Prometheus will collect. Prometheus will also collect metrics from ADQM services (ClickHouse, ZooKeeper, ClickHouse Keeper, Chproxy). These metrics will be available in the Prometheus format on ports and endpoints specified in service configuration settings. You can also use the Prometheus and Grafana web interfaces to view and analyze data that the monitoring service collects.

NOTE

  • If you already have a Prometheus-compatible monitoring system set up (for example, your own Prometheus server or VictoriaMetrics), you can use it to collect ADQM metrics. To do this, configure access to ADQM metrics in your monitoring system using ADQM’s Prometheus parameters specified on the Monitoring service page.

  • You can also use the Federation mechanism to migrate all metrics from the Prometheus server deployed in ADQM to your Prometheus.

Step 1. Add the service

  1. In the ADCM interface, open the Clusters page and click your ADQM cluster name. On the cluster page that opens, switch to the Services tab and click Add services.

    Switch to adding services
    Switch to adding services

  2. In the opened dialog, select the Monitoring service and click Add.

    Select the service
    Select the service

    As a result, the added service is displayed on the Services tab.

    The result of successful adding the service to the cluster
    The result of successful adding the service to the cluster

Step 2. Add components

  1. On the cluster page, open the Mapping tab to proceed to mapping service components to cluster hosts.

    Switch to mapping service components
    Switch to mapping service components

    All components of the Monitoring service are mandatory (highlighted in red). Each component should be installed on one host of the cluster.

    Monitoring service components
    Component Description

    Prometheus Server

    Deploys a Prometheus server that serves as a:

    • proxy channel for all collectors of metrics on a host;

    • storage for all metrics of a cluster;

    • generator of alerts based on collected metrics.

    Grafana

    Visualizes ADQM metrics as graphs and charts organized into dashboards

    Pushgateway

    Receives static metrics and sends them to Prometheus. In ADQM, it is used to pass a cluster structure to Prometheus

  2. Assign a host to each component of the Monitoring service — click Add hosts and select the desired host in the pop-up window.

    Select a host for a component
    Select a host for a component
    CAUTION
    It is not recommended to install the Prometheus Server and Pushgateway components on hosts with ADQM — use separate hosts for them. Otherwise, if an ADQM host fails and/or the load on it is critically high, information about the corresponding problems will not be saved.

  3. After the distribution of components is completed, click Save.

    Save mapping of components
    Save mapping of components

Step 3. Configure the service

  1. Open the Services tab on the cluster page and click the Monitoring service name in the Name column.

    Switch to the service configuration
    Switch to the service configuration

  2. In the page that opens, fill in the service’s configuration parameters — see the Monitoring section in the Configuration parameters article for parameter descriptions. Fields highlighted in red are required.

    Configure the Monitoring service
    Configure the Monitoring service

    After specifying all necessary parameters, click Save.

Step 4. Install the service

  1. On the Services tab, click the icon actions default dark actions default light for the Monitoring service in the Actions column and run the Install action.

    Switch to the service installation
    Switch to the service installation

  2. Confirm the action in the opened window.

    Confirm the action
    Confirm the action

  3. Wait until the installation is completed. Then check that the service state has changed from created to installed.

    Installation is complete
    Installation is complete

    To view the service installation process and analyze errors if they occur, select Jobs in the left navigation menu and click the Install job name in the Jobs list.

    Install service job page
    Install service job page

Step 5. View results

The Monitoring service starts automatically after installation. To ensure that monitoring works correctly, check the following:

  • Metrics are collected from all hosts of your cluster, not only from the hosts where components of the monitoring service are deployed.

  • Both system and ADQM service metrics are collected from the cluster hosts — see ADQM monitoring metrics.

To check both these points, you can view metrics in the Prometheus format in the browser, and also use the Prometheus or Grafana web interface.

Metrics in the Prometheus format

  1. In the address bar of your browser, enter an address of an ADQM host with the specified port and endpoint to listen for service or system metrics. Port numbers and endpoints are defined in the corresponding sections on the configuration page of the Monitoring service:

    • ADQM’s services metric settings — settings for access to monitoring metrics of ADQM services;

    • Node exporter settings — settings for access to system metrics of the ADQM cluster hosts.

    For example, http://10.92.40.107:9363/metrics is an address to view metrics of the ClickHouse server on a host with IP 10.92.40.107.

  2. The page that opens will display monitoring metrics from the specified host of the ADQM cluster in the Prometheus format.

ClickHouse server metrics in the Prometheus format
ClickHouse server metrics in the Prometheus format
ClickHouse server metrics in the Prometheus format
ClickHouse server metrics in the Prometheus format

Prometheus web interface

  1. In the address bar of your browser, enter an IP address of a host where the Prometheus Server component is installed. Specify also a port number from the listen_address parameter in the Prometheus settings section of the Monitoring service configuration (the default port is 9092). For example: http://10.92.40.107:9092. To log into the Prometheus interface, use a username and password that are also specified in the Prometheus settings section — the Prometheus users to login/logout to Prometheus setting.

  2. In the Expression field, you can enter a metric and click Execute — values of this metric on all hosts of the ADQM cluster will be shown in the interface.

Prometheus web interface
Prometheus web interface
Prometheus web interface
Prometheus web interface

Grafana web interface

  1. In the address bar of your browser, enter an address of a host on which Grafana is deployed and add a port number — a value of the Grafana listen port parameter located in the Grafana settings section of the Monitoring service configuration (the default value is 3000). For example, http://10.92.40.107:3000. To log in, use admin as a username, and the Grafana administrator’s password parameter value (also found in the Grafana settings section of service configuration parameters) as a password.

  2. In the window that opens, navigate to Home → Dashboards and expand the ADQM Dashboard <ADQM_cluster_name> section. In this section, you can select one of available dashboards to view service or system metrics coming from your ADQM cluster.

View ADQM metrics in Grafana
View ADQM metrics in Grafana
Found a mistake? Seleсt text and press Ctrl+Enter to report it