Install a cluster

Lisa Vaysman, Daria Barysheva

Contents

Step 1. Installation
- Whole cluster
- Single services
Step 2. View installation results
Step 3. Check the cluster state after the installation
Step 4. View metrics
- Grafana web interface
- VictoriaMetrics web interfaces

Step 1. Installation

There are two ways to install cluster services:

Whole cluster. In this method all services are installed automatically one by one after applying the Install action to a cluster.
Single services. In this method each service is installed manually after applying one or more actions to it. This way is also suitable for installing new services in the already deployed cluster.

In both cases each service starts automatically after installation.

IMPORTANT

The installation must be performed on clean hosts without prior installation of related or similar software.

Whole cluster

To install all cluster services within a single action, follow the steps:

Apply the Install action to the cluster selected on the Clusters page by clicking the icon in the Actions column.

Switch to the whole cluster installation
Verify the requested action in the opened window.

Verify the action
Wait until the job is completed.

Single services

To install one or several cluster services manually, add these services to the cluster and then perform the actions described below.

CAUTION

Notice that services are installed one by one. Do not install another service until the installation of the previous one is completed.

Open the Services tab on the cluster page. For each service that should be installed, click the icon in the Actions column and select the Install action.

Switch to installation of single services
Verify the action in the opened window.

Verify the action
Wait until the job is completed. Then check that the service state has changed from created to installed.
Repeat the previous actions for other services that should be installed.

Step 2. View installation results

ADCM starts a single job for installation process, as well as for any other task. You can find out about the status of jobs more specifically on the Jobs page.

The Jobs page

The successful completion of the service installation is determined by the transition of the job from the running status to the success status on the Jobs page. If the job fails, it switches to the failed status. In this case, it is possible to see more detailed information about the errors that occurred by clicking the failed job on the Jobs page. The window with the job’s inner steps is opened. Click the subjob’s name for additional information on the current step.

Switch to subjob details

A job page contains two sections: ansible [ stdout] and ansible [ stderr ]. These are technical logs of the job that include information from standard I/O streams stdout/stderr. These logs can help to understand what caused the problems.

There can also be the optional third section ansible [ check ]. These are the logs of the most frequent errors checks. The description of these errors is more simple and specific, than in two previous technical logs.

The contents of all three sections should be studied if errors occur.

Job details

Step 3. Check the cluster state after the installation

As a result of successful installation, the cluster and its services change their state according to the following rules:

The cluster state, displayed in the State field on the Clusters page, is being changed from created to installed.

The cluster state after successful installation
The state of cluster services, displayed on the Services tab of the cluster page, is being changed from created to installed.

The state of cluster services after successful installation

Step 4. View metrics

Grafana web interface

In the address bar of your browser, enter an address of the host on which Grafana is deployed and add a port number — the value of the Grafana port parameter is located in the Configuration section of the Grafana service (the default value is 3000). For example, http://10.92.40.107:3000.

To log in, use admin as a username, and the Admin password parameter value (also found in the Configuration section of the Grafana service) as a password.

Troubleshooting: empty dashboards in Grafana

If Grafana dashboards do not display metrics, perform the following steps.

Verify if Grafana can connect to the VictoriaMetrics Select component

Open the Grafana web interface (as described above).
Navigate to Connections → Data sources → VictoriaMetrics.
Scroll to the bottom of the page and click Test.
Analyze the result:
- Test successful. If you see a green checkmark with the message Data source is working, then the connection from Grafana to the VictoriaMetrics Select component is functional. However, this does not guarantee that the VictoriaMetrics Insert component can write data into the VictoriaMetrics Storage component.
- Error returned. If the test returns an error (e.g. request error), then the VictoriaMetrics Select component is unreachable from Grafana.

Check the health of the VictoriaMetrics components

Open the ADCM web interface. Select your ADM cluster on the Clusters page. To do this, click a cluster name in the Name column.
Go to the Services → VictoriaMetrics → Components tab.
Check that the status indicators next to all component names are green. If any component indicator is yellow, restart it using the Restart action.

Check the logs and network connectivity

To check the VictoriaMetrics Select component logs:
```
$ sudo docker logs --tail 50 vmselect
```
To check the VictoriaMetrics Storage component logs:
```
$ sudo docker logs --tail 50 vmstorage
```
To check the VictoriaMetrics Insert component logs:
```
$ sudo docker logs --tail 50 vminsert
```

When analyzing the logs, use the following table to identify the specific root causes and determine the required action.

VictoriaMetrics Docker log analysis
Error type in logs	Root cause	Required action
`no space left on device`	The VictoriaMetrics Storage component has run out of disk space and cannot accept new data	Free up disk space or expand the storage volume on the affected VictoriaMetrics Storage component host
`failed to reserve page summary memory`	A component (VictoriaMetrics Select or VictoriaMetrics Storage) crashed because the Docker container or the host ran out of RAM	Investigate memory utilization on the host and run the command for the required Docker container to check if its state is `OOMKilled`: `$ sudo docker inspect vmstorage $ sudo docker inspect vmselect` Increase the allocated memory limit if necessary.
Only service initialization messages (e.g. `successfully initialized netstorage` and `successfully dialed`) without data processing entries	Metrics from the data source cluster are not delivered: The data collection component (VictoriaMetrics Agent or Prometheus) on the data source cluster side is down or cannot connect to the VictoriaMetrics Insert component. Metrics scraping failed because the data source cluster applications or their metrics URLs are unreachable.	Log in to the data source cluster host and check the health of VictoriaMetrics Agent (or Prometheus), the scraping configuration and network access to the VictoriaMetrics Insert component

VictoriaMetrics web interfaces

VictoriaMetrics has three separate web interfaces:

VictoriaMetrics Insert
VictoriaMetrics Select
VictoriaMetrics Storage

To access VictoriaMetrics web interfaces, enter an address of the host where VictoriaMetrics is deployed with its respective port number into the browser address bar. Find specific port numbers on the Configuration → Ports Configuration tab (see Configure services):

VictoriaMetrics Insert: the vminsert HTTP Port parameter (the default value is 8480). Example: http://10.92.40.169:8480.
VictoriaMetrics Select: the vmselect HTTP Port parameter (the default value is 8481). Example: http://10.92.40.169:8481.
VictoriaMetrics Storage: the vmstorage HTTP Port parameter (the default value is 8482). Example: http://10.92.40.169:8482.

NOTE

Links to the Grafana and VictoriaMetrics web interfaces are also available on the Info tab of the corresponding services. To access them, go to your cluster page, select the required service on the Services tab, and open the Info tab.

Found a mistake? Seleсt text and press Ctrl+Enter to report it