Rolling upgrade
The Rolling upgrade functionality allows you to sequentially upgrade the ADQMDB service on separate ADQM hosts without a long shutdown of a cluster.
|
IMPORTANT
|
The process of ADQM rolling upgrade via the ADCM interface is described below.
Step 1. Prepare for the update
-
On the Bundles page, upload a new ADQM bundle — see the step Upload a cluster bundle to ADCM in the Create a cluster article.
-
Open the Clusters page, where the icon
becomes active for an ADQM cluster in the Actions column after a new bundle is loaded — click this icon.
Preparing a cluster for the upgradeIn the Upgrade Cluster window that opens, select a version you want to update ADQM to and click Upgrade.
Selecting a version to update ADQMIn the confirmation window that appears, click Run.
Confirming preparation for the updateWhen the action is completed, ADCM will change the cluster status to
upgrading— the cluster is ready to be upgraded.
Step 2. Update ADQMDB on host groups
You can update ADQMDB on individual hosts rather than on the entire cluster in one of the following ways:
-
Run the Upgrade action for the ADQMDB service and specify hosts on which the service should be upgraded in the action configuration. This action is available starting with the ADQM version 26.8.2.7.1.b1.
-
Use the Action hosts groups feature to create a group of hosts on which the ADQMDB version should be upgraded, and perform the Upgrade action for this group.
-
Run the Upgrade action for the ADQMDB service — Wizard opens, allowing you to configure the action. The following parameters are available in the first configuration step:
-
ADQMDB Hosts — list of hosts where the ADQMDB version should be updated.
-
Use compatibility mode with previous version — if enabled, default values of settings from the previous version will be used (the
compatibilityparameter of ClickHouse).
Settings for the "Upgrade" action of the ADQMDB service
-
-
On the next configuration step page, if necessary, you can adjust the following:
-
Disable the Raise blocking concern option to run the action without blocking the cluster.
-
Add a comment with a brief description of the action.
-
Enable Verbose to view additional information about the action execution on the Jobs page.
Final step of setting up the actionClick Run to confirm the action.
-
-
On the ADQMDB service configuration page (Clusters → <ADQM_cluster> → Services → ADQMDB), open the Action hosts groups tab and combine cluster hosts into groups to then upgrade the ADQMDB version for each host group sequentially. For example, you can create host groups so that each group contains one replica of each shard. The image below shows an example of dividing hosts into groups according to this principle for the following cluster topology:
-
shard 1 with two replicas — dev-adqm-01.ru-central1.internal and dev-adqm-02.ru-central1.internal;
-
shard 2 with two replicas — dev-adqm-03.ru-central1.internal and dev-adqm-04.ru-central1.internal.
Distributing hosts into groups for sequential ADQMDB service updates
-
-
Perform the ADQMDB service upgrade for one host group. To do this, in the host group row, click the icon
in the Actions column and run the Upgrade action.
Initiating ADQMDB update for a group of hostsIn Wizard that opens (see its description on the ADQMDB’s Upgrade action tab), confirm the upgrade.
After the ADQMDB service upgrade for a host group is complete, the cluster will remain in the upgrading state.
You can re-run the Upgrade action for a group of hosts if, for example, the upgrade failed on some hosts or new hosts were added to the group.
After the Upgrade action has been successfully completed, make sure that the new version of ClickHouse is installed on all hosts in the group (for example, you can run the select version() command in the clickhouse-client command-line client) and that databases work correctly (ClickHouse guarantees technical compatibility of two versions within a year, but business behavior may change, so it is recommended to check the applied result for each version separately). Further actions depend on the check result:
-
If the check is successful, you can continue upgrading ADQMDB on other hosts of the cluster (run the Upgrade action for the next group of hosts) or proceed to completing the cluster upgrade.
-
If you have detected any problems while testing the new version of ClickHouse, you can abort the cluster upgrade and roll back the previous version of the ADQMDB service on the previously upgraded hosts. To do this, switch to the Clusters page and run the Abort the upgrade action for the cluster.
Canceling the ADQMDB service update
After the update is canceled, the cluster bundle will revert to the previous version (and ClickHouse on all cluster hosts accordingly too), and the cluster state will be set to installed.
If you failed to cancel the upgrade (for example, because a file with information needed to roll back to the previous version is corrupted or lost, or the archive with the configuration files of the previous version of the ADQMDB service is missing), you can run the Abort the upgrade action with the Forced rollback option enabled. This will perform a forced rollback to the previous version of the ADQM bundle, after which you will need to run the Reconfig & restart action for the ADQMDB service.
Starting with ADQM 24.3.11.7.1.b1, the Abort the upgrade action with the Forced rollback flag set restores ADQM repositories and removes ADQM packages if they have been updated.
After you fill in the action parameters on the Configuration tab, click Next and confirm the action on the Confirmation tab by clicking Run.
Step 3. Complete the update
Once you have successfully upgraded the ADQMDB service to the new version on hosts where a preliminary upgrade was required, complete the cluster upgrade with the Complete the upgrade action (on the Clusters page) — it will upgrade ADQMDB on the remaining hosts, as well as all other ADQM services in the cluster. Once this action is performed, there is no option to revert to the previous version of ADQM.
When the upgrade process is complete, ADCM will change the cluster state to installed.
|
NOTE
The Complete the upgrade action runs the usual upgrade of the cluster, i.e. the upgrade of all ADQM services on all hosts (as the Upgrade action does for a cluster being upgraded to an ADQM version up to and including 24.3.2.23.1.b1). |