Run actions in ADBM

Daria Barysheva

Overview

After you created a configuration, backups are launched and cleaned up according to the schedules that you defined. However, ADBM also allows you to manage backups manually via the cluster actions that are described below. To run the actions, follow the steps:

  1. Open the Backup Manager page via the ADB Control web interface.

  2. In the Clusters section of the page that opens, find a cluster for which you want to run actions and click the added actions dark added actions light icon in the row that corresponds to that cluster. Possible actions are listed in the drop-down list.

    adbm cluster action open1
    Actions in the Clusters section

  3. If you want to work with a specified cluster, click the cluster name in the table that is located in the Clusters section. The selected cluster page opens. The Backup button is designed to launch backups. To open a menu with other actions, click added actions dark added actions light.

    adbm cluster action open2
    Actions on the selected cluster page

Running actions is an exclusive operation. You cannot run another action until the previous one is completed (Terminate is exception). If you try to run several actions simultaneously, you get the following error.

adbm cluster action error409
The error when running actions simultaneously

Action types

Backup

Database backups are launched automatically on the schedules that are specified by the following parameters of the current configuration version (see the General configuration tab):

  • Full Backup schedule — full backups;

  • Differential backup schedule — differential backups;

  • Incremental backup schedule — incremental backups.

If you need to get an additional backup of any type without waiting for the next auto launch, do the following:

  1. Click Backup on the cluster page or select the Backup action for the specified cluster in the Clusters section.

    adbm cluster action backup
    Switch to the Backup action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Type — a backup type. Possible values:

      • Full — full;

      • Incr — incremental;

      • Diff — differential.

    • Configuration — the configuration version that will be applied. You can click it to check or edit the configuration parameters in the list of configurations.

    • Date — a current timestamp.

    • Delta — a flag that indicates the need to use checksums during the backup operation.

      adbm cluster action backup2
      The Backup action form

  2. Select the backup type in the Type field and change the Restore point value if necessary.

  3. Click Run.

  4. As a result, the Start 'Backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, the new backup becomes available in the list of backups on the Backups tab.

    adbm cluster action backup3
    The new backup is available on the Backups tab

Create restore point

Restore points are automatically created according to the schedule that is defined by the Restore point creation schedule parameter of the current configuration version (see the General configuration tab).

If you need to create a restore point without waiting for the next auto launch, follow the steps:

  1. Select the Create restore point action on the cluster page or in the Clusters section.

    adbm cluster action rp
    Switch to the "Create restore point" action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Configuration — the configuration version that will be applied. You can click it to check or edit the configuration parameters in the list of configurations.

    • Date — a current timestamp.

      adbm cluster action rp2
      The "Create restore point" action form

  2. Change the restore point name in the Restore point field if necessary.

  3. Click Run.

  4. As a result, the Start 'Create restore point' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, a new restore point becomes available. You can check it by running the Restore or the Cleanup actions.

    adbm cluster action rp3
    The new restore point is available

Restore

The Restore action allows you to restore databases at the moment of one of existing restore points. This is the only action that can be applied to both a running and a stopped ADB cluster. The action logic depends on the cluster status:

  • ADB cluster is running (marked as UP in the ADBM UI). In this case, before the action is started, additional checks are automatically performed to ensure that the selected backup can be applied to the cluster:

    • The topology that is saved in the backup is compared with the topology obtained from the ADB cluster (from the system table gp_segment_configuration).

    • The backup metadata are also checked (see Verify below).

    If the validation is successful, the cluster stops automatically. In addition, if any error is fixed during the Restore action, you can run Restore retry — this is a restart of the last restore process without any additional checks.

  • ADB cluster is stopped (marked as DOWN in the ADBM UI). In this case, any validation is skipped. However, it is still possible to start the restore process by setting the Skip Validation flag. This method is recommended only in case of fatal problems when the cluster does not even start.

To restore databases, follow the steps:

  1. Select the Restore action on the cluster page or in the Clusters section.

    adbm cluster action restore
    Switch to the Restore action

    The window that opens contains the following fields:

    • Time period — the timestamp range that is used to search restore points. The default value is the current date.

    • Restore point — the restore point at the moment of which you want to restore databases.

    • Databases — the databases that you can restore for the selected restore point.

    • Processes — a maximum number of processes to restore one segment.

    • Configuration — the configuration version that was applied to the selected restore point. You can click it to view the configuration parameters in the list of configurations.

    • Date created — a creation timestamp of the selected restore point.

    • Backup type — a type of the latest backup that was created before the selected restore point.

    • Backup date — a creation timestamp of the latest backup that was completed before the selected restore point.

    • Restore mirrors — a flag that indicates whether to restore data for mirrors and Standby if they exist in the cluster.

    • Delta — a flag that indicates the need to use checksums during the restore operation.

    • Force — a flag that indicates the need to completely overwrite data and tablespace paths in PostgreSQL.

    • Skip Validation — a flag that indicates the need to restore databases even though validation is not possible. Displayed only if the ADB cluster is stopped (DOWN).

      adbm cluster action restore2
      The Restore action form (ADB cluster is running)
      adbm cluster action backup force
      The Restore action form (ADB cluster is stopped)

  2. Fill in the necessary fields:

    • Select a restore point in the Restore point drop-down-list. Use the Time period filter if necessary.

    • In the Databases drop-down-list, select the databases that you want to restore.

    • If necessary, set the Skip Validation flag.

  3. Click Run.

  4. As a result, the Start 'Restore' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, the selected databases should be restored at the moment of the specified restore point.

For running ADB clusters (with the UP status), if any error occurs when running the Restore action — you can retry the latest restore process. To do this, select the Restore action again (see step 1 above) and click Retry in the window that opens. If you want to run a new Restore action from scratch (with all checks), click New restore.

adbm cluster action retry
Retry the last Restore action

Cleanup

The Cleanup action is automatically executed according to the schedule that is defined by the Cleanup schedule parameter of the current configuration version (see the General configuration tab). This action removes:

  • The oldest extra full backups — if the number of full backups exceeds the Number of full backups parameter defined for the current configuration.

  • The oldest extra differential backups — if the number of differential backups exceeds the Number of differential backups parameter defined for the current configuration.

  • All backups in the following statuses: Failed, Invalid.

  • Inactive configuration versions that have no related backups.

When you run the Cleanup action manually, it deletes all objects mentioned above but also allows you to remove restore points and backups related to the specified restore point. You define the restore point, and ADBM finds the latest full or differential backup that was completed before the selected point created. Then, all backups and restore points related to that backup are removed.

To run the Cleanup action manually, follow the steps:

  1. Select the Cleanup action on the cluster page or in the Clusters section.

    adbm cluster action cleanup
    Switch to the Cleanup action

    The window that opens contains the following fields:

    • Time period — the timestamp range that is used to search restore points. The default value is the current date.

    • Select restore point — the restore point at the moment of which you want to remove a set of related backups and restore points.

    • Configuration — the configuration version that was applied to the selected restore point. You can click it to view the configuration parameters in the list of configurations.

    • Date created — a creation timestamp of the selected restore point.

      adbm cluster action cleanup2
      The Cleanup action form

  2. Select a restore point in the Select restore point drop-down-list. Use the Time period filter if necessary.

  3. Click Run.

  4. As a result, the Start cleanup job for 'Verify backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, backups and restore points are removed according to the rules described above. You can check it in the list of backups on the Backups tab.

    For example, if you have two backups, full and incremental, and select the restore point that was created after the incremental backup — both backups will be deleted.

    adbm cluster action cleanup3
    List of backups before the Cleanup action
    adbm cluster action cleanup4
    List of backups after the Cleanup action

To better understand the logic of the Cleanup manual action, you can view the example listed below. The figure shows a timeline with a list of backups of different types (called full*, diff*, incr*) and restore points (called rp*). The table explains which objects will be deleted when any of restore points is selected in the Cleanup action form.

example cleanup dark
Example with a list of backups and restore points
example cleanup light
Example with a list of backups and restore points
Logic of the manual Cleanup action
Selected restore point Objects to be removed

rp1

All backups and restore points for the specified period

rp2

All backups and restore points for the specified period

rp3

All backups and restore points for the specified period

rp4

All backups and restore points for the specified period

rp5

diff1, incr2, rp5 — rp8

rp6

diff1, incr2, rp5 — rp8

rp7

diff1, incr2, rp5 — rp8

rp8

diff1, incr2, rp5 — rp8

Verify

The Verify action compares the backup metadata that are stored in ADBM with the backup metadata that are being received from pgbackrest (i.e. actual backups). If the difference is fixed for any backups, they acquire the Invalid status as they cannot be used to restore databases. Such backups will be removed during the next Cleanup action launch.

Before comparing metadata, ADBM additionally checks existence of configuration files (pgbackrest.conf) for all timelines on all segment hosts. If any of these files are not present, the last available version of the timeline configuration is taken, and a new configuration file is created on the basis of this configuration. Then this file is written to the segment host for the selected timeline. This is necessary to ensure that all configurations are present before the metadata for the backups are checked.

To run the Verify action, do the following:

  1. Select the Verify action on the cluster page or in the Clusters section.

    adbm cluster action verify
    Switch to the Verify action

  2. In the window that opens, click Run.

    adbm cluster action verify2
    The Verify action form

  3. As a result, the Start 'Verify backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  4. As a successful result of all actions, some of the backups can acquire the Invalid status. You can search such backups using the Status filter on the Backups tab.

Terminate

The Terminate action allows you to terminate some of the currently running actions. This is the only action type for which a separate distributed lock is used (in etcd), which allows you to run Terminate in parallel with other types of actions.

IMPORTANT

You can terminate actions of the following types:

Below is an example how to terminate the Backup action:

  1. Run the Backup action as shown above.

    adbm cluster action terminate0
    The new backup is being initialized

  2. Select the Terminate action on the cluster page or in the Clusters section.

    adbm cluster action terminate
    Switch to the Terminate action

  3. In the window that opens, click Run.

    adbm cluster action terminate2
    The Terminate action form

  4. As a result, the Start 'Terminate' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    The current backup acquires the Terminating status. You can check it in the list of backups on the Backups tab.

    adbm cluster action terminate3
    The backup has the Terminating status

  5. As a successful result of all actions, the backup acquires the Stopped status.

    adbm cluster action terminate4
    The backup status is switched to Stopped
NOTE

After applying the Terminate action, you can resume the latest terminated backup via the Resume action.

Resume

The Resume action allows you to resume the latest terminated backup launch. This action is available in the list of cluster actions only when backups with the Stopped status exist. Below is an example how to resume the backup that was terminated in the previous section:

  1. Select the Resume action on the cluster page or in the Clusters section.

    adbm cluster action resume
    Switch to the Resume action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Configuration — the configuration version that will be applied. You can click it to check the configuration parameters in the list of configurations.

    • Date — a current timestamp.

      adbm cluster action resume2
      The Resume action form

  2. Change the restore point name in the Restore point field if necessary.

  3. Click Run.

  4. As a result, the Start 'Resume backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    The backup acquires the Resume status. You can check it in the list of backups on the Backups tab.

    adbm cluster action resume3
    The backup has the Resume status

  5. As a successful result of all actions, the backup acquires the Done status.

    adbm cluster action resume4
    The backup status is switched to Done
Found a mistake? Seleсt text and press Ctrl+Enter to report it