Home
Arenadata DB
How to
Manage a cluster
Cluster management via ADCM
Cluster actions

Cluster actions

Daria Barysheva

Collapse content Expand content

Overview

You can manage an ADB cluster on the Clusters page in the Arenadata Cluster Manager (ADCM) web interface.

The Clusters page

The Clusters page displays a table with the following columns:

Name — a name specified during the cluster creation.
State — a cluster status.
Product — a product name.
Version — a version of the bundle used to install or upgrade the cluster.
Description — a description specified during the cluster creation.
Concerns. If critical errors are detected in the cluster configuration or blocking actions are launched, the icon image is changed to — when you hover over it, a pop-up window appears showing an error description with a link that you can follow to perform the required configuration or view the detailed information.
Actions. The column shows icons for managing the cluster:
- — opens the drop-down list that offers actions to manage the cluster.
  
  Open a list of available cluster actions
- — indicates whether a new version of a bundle is available and allows you to upgrade the cluster.
- — deletes information about the cluster from ADCM (it does not remove ADB or make any changes to hosts that belong to the cluster).

You can click icons described above to perform the corresponding cluster actions. When you choose an action, ADCM displays a dialog window to confirm your choice. In this dialog window, you can select the Verbose checkbox to see additional execution details on the Jobs page. For some actions, you should first fill in additional options in a separate window.

Standard window to confirm all actions

When an action starts, ADCM displays its execution process and result on the Jobs page. From this page, you can navigate to a page with details on an individual job (by clicking a job name) to see inner steps of that job execution and analyze errors if they occur.

The job page

A set of cluster actions (available after clicking the icon actions default dark actions default light ) depends on the current ADB cluster status.

Available actions depending on the ADB status
Status	Condition	Available actions
created	The ADB cluster was created via ADCM, but not installed yet	Precheck Install
running	The ADB cluster was successfully installed via ADCM and started	Activate standby Check Expand Init Standby Master Reconfigure Vault integration Reconfigure parameter archiving Reinstall Reinstall statuschecker Stop
stopped	The ADB cluster was successfully installed via ADCM and stopped	Activate standby Check Reinstall Reinstall statuschecker Start
activating_standby	As a result of the Activate standby action, the Standby master was activated, but some errors occurred during the subsequent configuration process	Activate standby postprocess
expanding	The cluster expansion was performed via the Expand action. The redistribution of ADB tables is needed	Redistribute
ready to upgrade	The ADB cluster is prepared for upgrade — by running the action available after clicking the icon	Upgrade

For information on the Precheck and Install actions, see Install a cluster. Other actions available for an already installed cluster are listed below.

Activate standby

The Activate standby action activates the Standby master in case of failures on the current master.

IMPORTANT

You can activate the Standby master when a host of the current master is unavailable. To switch to Standby if there are no errors on the master host (it is available), you should first stop the ADB service and set the force activation flag in the Activate standby action parameters. Otherwise, the action fails.
Before running the Activate standby action, you should start the ADB Control and ADBM services since their reconfiguration is needed during the action launch. Otherwise, the action fails (but you can fix it by starting services and running the Activate standby postprocess action).

After you select the Activate standby action, a dialog box opens in which you can fill in the following options:

force activation — a flag that indicates whether to force switching to the Standby master when the host of the active master is available, but the master process inside the operating system is stopped;
run analyze — a flag that specifies whether to run ANALYZE after switching to Standby in all databases except template0, template1, and postgres.

The "Activate standby" action form

To edit any parameter, click its current value. In the window that opens, enter a new value and click Apply.

Enter a new parameter value

To run the Activate standby action, click Run in the action form. Then, confirm the action in the standard confirmation dialog.

While the action is running, the ADB cluster has the activating_standby status. If the action succeeds, the cluster status changes to running. If the Standby master is activated, but some errors occur during the subsequent configuration process — the cluster remains in the activating_standby status, and the Activate standby postprocess becomes available in the list of cluster actions. In this case, you need to determine the error cause on the Jobs page, make fixes, and run the Activate standby postprocess action.

Postprocessing

To create cron tasks on the new master, run the following actions after the Activate standby action is completed:
- Reconfigure parameter archiving for the ADB cluster — to have the ability to export ADB settings according to the schedule specified via the Parameter archiving/Parameter archiving job setting cluster parameters.
- Reconfigure for the ADB service — to have the ability to run scripts that interact with the arenadata_toolkit schema.
After the host of the previous master becomes available (or immediately after running the Activate standby action if switching to Standby was not caused by the master host unavailability), follow the steps on that host:
- Delete the cron tasks under the ADB administrator (gpadmin by default). This is necessary to prevent attempts to access the outdated master.
  $ sudo su - gpadmin $ crontab -r
- Rename or remove the master data catalog (/data1/master by default) to avoid accidentally running an outdated master and be able to place a new Standby master later.
After the Standby master is activated, an old master stops participating in the ADB cluster operations. The Standby master is no longer available, as it has become a master. To add a new Standby master, run the Init Standby Master action.

Activate standby postprocess

The Activate standby postprocess action becomes available if the Standby master is activated during the Activate standby action, but some errors occur during the subsequent configuration process. After fixing errors, run Activate standby postprocess to finish processing and switch the ADB status to running.

After you select the Activate standby postprocess action, a dialog box opens in which you can fill in the run analyze option that is described above for the Activate standby action.

The "Activate standby postprocess" action form

To run the Activate standby postprocess action, click Run in the action form. Then, confirm the action in the standard confirmation dialog.

Init Standby Master

The Init Standby Master action allows you to add or remove the Standby master. After you select the action, follow the steps:

On the Configuration tab of the window that opens, fill in the options:
- Reboot new servers after installation — a flag that indicates whether to reboot ADB hosts automatically after the Init Standby Master action is completed. If the flag is reset, you need to restart manually.
- Reboot timeout, sec — a timeout that is used to wait for the reboot of ADB hosts (in seconds).
  
  The "Init Standby Master" action form
  
  To edit any parameter, click its current value. In the window that opens, enter a new value and click Apply.
  
  Enter a new parameter value
Click Next.

On the Host - Component tab that opens, perform one of the following actions:

When adding Standby — place the ADB Standby component on a separate host. To do this, click Add hosts in the component section and select a host in the dialog that opens.

NOTE

You can place the Standby master on the host that was used for the previous master (before applying the Activate standby action). But you should first rename or delete the data catalog of the outdated master (/data1/master by default).

Add the Standby Master component

When removing Standby — click Remove for the host that is currently assigned to the ADB Standby component.

Remove the Standby Master component

Click Run.
Confirm the action in the standard confirmation window by clicking Run.

Expand

The Expand action allows you to add new segment hosts to the ADB cluster.

Prepare

Before starting the cluster expansion:

Create and configure new hosts on the Hosts page in ADCM. Use one of available hostproviders.
For each created host, apply the Check connection and Install statuschecker actions. Ensure that all actions are completed successfully and all hosts are available.
Add new hosts to the ADB cluster.

New hosts on the Hosts page

CAUTION

If the ADB cluster uses mirroring, the number of new hosts should be sufficient to apply one of the mirroring policies: spread or group. If the number of new hosts is greater than the number of segments per host in the current cluster configuration, the spread mirroring will be applied during the cluster expansion, otherwise — group. The group mirroring policy requires at least 2 new hosts, otherwise the action fails with the following error: Not enough hosts for grouped mirroring. You must have at least 2.

Run the action

After selecting the Expand action, follow the steps:

On the Configuration tab of the window that opens, switch on the Show advanced toggle and fill in the following options:
- Reboot new servers after installation — a flag that indicates whether to reboot new ADB hosts automatically after the Expand action is completed. A restart is required to apply some of the settings that are changed during the installation process. If the flag is reset, you need to restart manually.
- Reboot timeout, sec — a timeout that is used to wait for the reboot of ADB hosts (in seconds).
- Additional primary segments count — a number of primary segments that should be added to all segment hosts (including existing hosts). For example, if existing hosts have two primary segments, using the parameter value 2 will initialize two additional primary segments on existing hosts and four primary segments on new hosts. In addition, mirror segments will be added for these new primary segments if mirroring is enabled. The default parameter value is 0, which means that new hosts are configured with the same number of primary segments as existing hosts have.
- Attach new data catalog (optional) — a directory that will be used to store new segments. All directories used as data catalogs should contain an equal number of segments to avoid data skew. Expansion will fail if segments are unevenly distributed across data catalogs. If the parameter is empty, additional segments are evenly distributed across existing data catalogs.
- Storage for data catalog (optional) — a name of the block storage device that should be mounted to the directory specified via the Attach new data catalog (optional) parameter. For example, sdc (without the /dev prefix).
- Specify path for every non-default tablespace — paths to the directories that are used to store tablespace data. Paths should be specified if the Attach new data catalog (optional) parameter is filled in.
- Check array — a flag that indicates whether to check that all hosts have an equal number of primary and mirror segments and the same segment name suffixes in the -<n> format (-1, -2, -3, etc.).
- Directory for gpexpand tar file (optional) — a fully qualified path to the directory on segment hosts were the gpexpand utility copies a temporary TAR file. This archive contains database files that are used to create segment instances. The default directory is the user home directory.
  
  The Expand action form
  
  To edit any parameter (except the Specify path for every non-default tablespace field described below), click its current value. In the window that opens, enter a new value and click Apply.
  
  Enter a new parameter value
  
  To edit the Specify path for every non-default tablespace parameter value (like any other parameter that allows adding several values), expand a node with the parameter name and click Add property. In the dialog that opens, enter a tablespace name in the Enter field name field and a directory path in the Enter field value field. Click Apply to save changes.
  
  Go to fill in a path for the tablespace
  
  Fill in a path for the tablespace
Click Next.

On the Host - Component tab that opens, assign new segment hosts to the ADB Segment component. To do this, click Add hosts in the component section and select hosts in the dialog that opens.

Map hosts to the ADB Segment component

IMPORTANT

If you use PXF, Chrony, or Monitoring Clients in the ADB cluster, you should also map their components (PXF, NTP Slave, Monitoring Agents) to new segment hosts for correct cluster operating.

After the mapping of components is completed, click Run to launch the Expand action.
Confirm the action in the standard confirmation window by clicking Run.

During the Expand action, necessary packages are installed and configured on new hosts after some checks. Then, the gpexpand schema is created in the postgres database. Objects of this schema will be used to store metadata and statistics on the tables that require redistribution over segments after the cluster expansion. If all action steps are successfully finished, the cluster status changes to expanding, and reverting to a previous configuration becomes impossible.

In the expanding status, the Redistribute action becomes available for the ADB cluster. This action allows you to redistribute table data over all segments after the cluster expansion.

The ADB cluster has the expanding status

Synchronize PXF configurations

If you use PXF in the ADB cluster, you need to copy PXF profiles to new segment hosts after running the Expand action. Otherwise, foreign tables will stop working. To synchronize the PXF configuration on all hosts in the ADB cluster, follow the steps:

Login under the default user name gpadmin. All the commands listed in the subsequent steps should be performed on Master:
```
$ sudo su - gpadmin
```

Synchronize the PXF configuration on all hosts in the ADB cluster:

$ pxf cluster sync

The result:

Syncing PXF configuration files from master host to standby master host and 2 segment hosts...
PXF configs synced successfully on 3 out of 3 hosts

Restart PXF:

$ pxf cluster restart

The result:

Restarting PXF on master host, standby master host, and 2 segment hosts...
PXF restarted successfully on 4 out of 4 hosts

Redistribute

The Redistribute action redistributes tables to balance existing data over the newly expanded ADB cluster. The action uses the Greenplum utility gpexpand.

It is recommended to rank the database tables before starting the action, i.e. specify the order in which tables should be redistributed.

Rank tables for redistribution

A priority of the table processing during redistribution is determined by the rank column value in the gpexpand.status_detail table. Tables with the lowest rank are redistributed first. To rank tables, follow the steps:

Connect to the postgres database of the ADB cluster under the gpadmin user (e.g. via psql).
For all rows of the gpexpand.status_detail table, set the initial value for the rank column that will be considered as maximum (and will match the lowest processing priority), e.g. 100:
```
UPDATE gpexpand.status_detail SET rank = 100;
```
Decrease the rank value for those tables that should be redistributed first. For example, as a result of the following queries, the Redistribute action will process the public.test table first, then the public.test2 table, and then other tables in gpexpand.status_detail.
```
UPDATE gpexpand.status_detail SET rank = 10 WHERE fq_name = 'public.test';
UPDATE gpexpand.status_detail SET rank = 20 WHERE fq_name = 'public.test2';
```

TIP

Use a fully qualified table name (including a schema name) as the fq_name column value.
It is recommended to use the step 10 when ranking tables (instead of 1) — to provide the ability to quickly increase a table priority without affecting other tables.
Assign the highest priority (the smallest rank value) to those tables that are used more often than others.
To exclude any table from the redistribution process, you can remove it from the gpexpand.status_detail table.

Run redistribution

To run data redistribution, select the Redistribute cluster action. It is recommended to perform redistribution during low-use hours when table locks do not have a significant impact on the cluster operations.

IMPORTANT

To perform redistribution, segment hosts should have enough disk space to temporarily store a copy of the largest table/partition.
During redistribution, each table or partition is locked (with the ACCESS EXCLUSIVE lock type) and becomes unavailable for read and write operations. Once its redistribution is completed, normal operations for this particular table resume.
New tables and partitions are distributed across all segments like under normal operating conditions.

After you select the Redistribute action, a dialog box opens in which you can set the following fields:

Timeout for expanding — the maximum redistribution duration in hours, minutes, and seconds.
Number of parallel processes — the -n option value for the gpexpand utility. Defines the number of tables to redistribute simultaneously. Valid values are 1 — 96. Each table redistribution process requires two database connections: one to alter the table and another to update the table status in the gpexpand schema. Before increasing Number of parallel processes, check the current value of the server configuration parameter max_connections to ensure that the maximum connection limit is not exceeded.

The Redistribute action form

To run the Redistribute action, click Run in the action form. Then, confirm the action in the standard confirmation dialog. If redistribution of all tables is completed within the time specified in the Timeout for expanding field, the gpexpand schema is removed from the postgres database, and the ADB cluster status is changed to running. Otherwise, the cluster remains in the expanding status, and you can continue redistribution by running the Redistribute action again.

Monitor redistribution

To track the redistribution progress, you can run SQL queries against the following objects of the gpexpand schema in the postgres database:

gpexpand.status — the table that stores a history of redistribution status changes.

Structure

Column Description

status

A status of the ADB redistribution. Possible values:

SETUP — preparations for the cluster expansion are started during the Expand action.
SETUP DONE — preparations for the cluster expansion are completed during the Expand action.
EXPANSION STARTED — table redistribution is started during the Redistribute action.
EXPANSION STOPPED — table redistribution is stopped. The additional launch of the Redistribute action is required.
COMPLETED — redistribution of all tables is successfully completed after one or more launches of the Redistribute action.

updated

A timestamp of the status change

Query example

SELECT * FROM gpexpand.status;

The result:

      status       |          updated
-------------------+----------------------------
 EXPANSION STARTED | 2024-02-28 08:02:31.026412
 SETUP DONE        | 2024-02-27 17:40:55.160054
 SETUP             | 2024-02-27 17:40:50.830922
(3 rows)

gpexpand.expansion_progress — a view that shows the current redistribution status: how many tables/bytes have been successfully redistributed and how many are left. Also, the view provides calculations of the estimated redistribution rate and estimated time to completion. Calculations restart each time you run the Redistribute action — after the first table redistribution is completed. Statistics on tables/bytes are refreshed as they are redistributed.

NOTE

In ADB, the gpexpand utility is used with the --simple-progress option to improve performance by reducing the amount of progress information written to the gpexpand tables. Due to this, only the Tables Expanded and Tables Left metrics have values in the gpexpand.expansion_progress table.

Structure

Column Description

name

A metric name:

Bytes Left — amount of data that is not yet redistributed (in bytes).
Bytes Done — amount of data that is successfully redistributed (in bytes).
Estimated Expansion Rate — the estimated rate of data redistribution (with a measurement unit).
Estimated Time to Completion — the estimated time required for redistribution completion (in hours, minutes, and seconds).
Tables Expanded — a number of tables that are successfully redistributed.
Tables Left — a number of tables that are not yet redistributed.

value

A value of the metric that is specified in the name column

Query example

SELECT * FROM gpexpand.expansion_progress;

The result:

      name       | value
-----------------+-------
 Tables Expanded | 90
 Tables Left     | 7
(2 rows)

gpexpand.status_detail — the table that stores the current redistribution status for each table/partition.

Structure

Column Description

table_oid

A table object identifier (OID)

dbname

A name of the database to which the table belongs

fq_name

A fully qualified name of the table (with a schema name)

root_partition_oid

For a partitioned table, the OID of the root partition. Otherwise, None

rank

A rank that determines the priority of table processing during redistribution. Tables with the lowest rank are processed first

external_writable

Indicates whether the table is EXTERNAL WRITABLE (such tables require a different syntax when using gpexpand)

status

The current status of the table redistribution:

NOT STARTED;
IN PROGRESS;
COMPLETED;
NO LONGER EXISTS — the table does not exist anymore.

expansion_started

A timestamp of the table redistribution start. This column is only populated after the table is successfully redistributed

expansion_finished

A timestamp of the table redistribution completion

source_bytes

The disk size associated with the source table (in bytes). Due to the possible data bloat in heap tables and differing numbers of segments after expansion, it is not guaranteed that the final number of bytes will be equal to the source number. Values of the source_bytes column are used to estimate the redistribution duration.

In ADB, the source_bytes value is equal to 0 for all tables due to using the --simple-progress option

rel_storage

A table storage type. See pg_class.relstorage

Query example

SELECT dbname, fq_name, status, expansion_started, expansion_finished, source_bytes FROM gpexpand.status_detail;

The output fragment:

  dbname   |                        fq_name                        |   status    |     expansion_started      |     expansion_finished     | source_bytes
-----------+-------------------------------------------------------+-------------+----------------------------+----------------------------+--------------
 adb       | kadb.offsets                                          | NOT STARTED |                            |                            |            0
 adb       | arenadata_toolkit.db_files_history_1_prt_p202310      | NOT STARTED |                            |                            |            0
 gpperfmon | public.database_history_1_prt_r1013160842             | NOT STARTED |                            |                            |            0
 gpperfmon | public.log_alert_history_1_prt_r598663655             | NOT STARTED |                            |                            |            0
 gpperfmon | public.log_alert_history_1_prt_r1113542580            | NOT STARTED |                            |                            |            0
 adb       | public.kafka_ssl                                      | NOT STARTED |                            |                            |            0
 adb       | public.adb_to_kafka_table3                            | NOT STARTED |                            |                            |            0
 adb       | arenadata_toolkit.db_files_current                    | NOT STARTED |                            |                            |            0
 adb       | arenadata_toolkit.db_files_history_1_prt_p202312      | NOT STARTED |                            |                            |            0
 gpperfmon | public.database_history_1_prt_r2098142994             | NOT STARTED |                            |                            |            0
 gpperfmon | public.network_interface_history_1_prt_1              | NOT STARTED |                            |                            |            0
 adb       | public.ext_adb_to_kafka_sasl_gssapi                   | NOT STARTED |                            |                            |            0
 adb       | public.adb_to_kafka_table7                            | NOT STARTED |                            |                            |            0
 adb       | arenadata_toolkit.db_files_history_1_prt_default_part | NOT STARTED |                            |                            |            0
 gpperfmon | public.queries_history_1_prt_r81170841                | NOT STARTED |                            |                            |            0
 gpperfmon | public.queries_history_1_prt_r419258856               | NOT STARTED |                            |                            |            0
 gpperfmon | public.diskspace_history_1_prt_r526486693             | NOT STARTED |                            |                            |            0
 adb       | public.test2                                          | COMPLETED   | 2024-02-28 08:02:37.00728  | 2024-02-28 08:02:38.048592 |            0
 gpperfmon | public.system_history_1_prt_r1569528921               | COMPLETED   | 2024-02-28 08:02:41.066986 | 2024-02-28 08:02:44.182122 |            0
 gpperfmon | public.database_history_1_prt_r932188937              | COMPLETED   | 2024-02-28 08:02:44.305985 | 2024-02-28 08:02:44.697928 |            0
 gpperfmon | public.segment_history_1_prt_1                        | COMPLETED   | 2024-02-28 08:02:44.841786 | 2024-02-28 08:02:45.426392 |            0
 diskquota | arenadata_toolkit.db_files_history_1_prt_default_part | COMPLETED   | 2024-02-28 08:02:46.096083 | 2024-02-28 08:02:48.04711  |            0
 adb       | diskquota.target                                      | COMPLETED   | 2024-02-28 08:02:48.166249 | 2024-02-28 08:02:48.602075 |            0
 adb       | arenadata_toolkit.db_files_history_1_prt_p202402      | COMPLETED   | 2024-02-28 08:02:48.73606  | 2024-02-28 08:02:49.505108 |            0

Reinstall

The Reinstall action reinstalls the ADB cluster. After you select the action, a dialog box opens in which you can fill in the following options:

Reboot cluster servers after installation — a flag that indicates whether to reboot ADB hosts automatically after the Reinstall action is completed. If the flag is reset, you need to restart manually.
Reboot timeout, sec — a timeout that is used to wait for the reboot of ADB hosts (in seconds).

The Reinstall action form

To edit any parameter, click its current value. In the window that opens, enter a new value and click Apply.

Enter a new parameter value

To run the Reinstall action, click Run in the action form. Then, confirm the action in the standard confirmation dialog.

Reinstall statuschecker

The Reinstall statuschecker action reconfigures and restarts the statuschecker for all cluster services. Use this action when migrating a cluster to a new ADCM server.