Run actions in ADBM

Daria Barysheva

Overview

After you created a configuration, backups are launched and cleaned up according to the schedules that you defined. However, ADBM also allows you to manage backups manually via the cluster actions that are described below. To run the actions, follow the steps:

  1. Open the Backup Manager page via the ADB Control web interface.

  2. In the Clusters section of the page that opens, find a cluster for which you want to run actions and click the added actions dark added actions light icon in the row that corresponds to that cluster. Possible actions are listed in the drop-down list.

    adbm cluster action open1 dark
    Actions in the Clusters section
    adbm cluster action open1 light
    Actions in the Clusters section

  3. If you want to work with a specified cluster, click the cluster name in the table that is located in the Clusters section. The selected cluster page opens. The Backup button is designed to launch backups. To open a menu with other actions, click added actions dark added actions light.

    adbm cluster action open2 dark
    Actions on the selected cluster page
    adbm cluster action open2 light
    Actions on the selected cluster page

Running actions is an exclusive operation. You cannot run another action until the previous one is completed (Terminate is exception). If you try to run several actions simultaneously, you get the following error.

adbm cluster action error409 dark
The error when running actions simultaneously
adbm cluster action error409 light
The error when running actions simultaneously

Action types

Backup

Database backups are launched automatically on the schedules that are specified by the following parameters of the current configuration version (see the General configuration tab):

  • Full Backup schedule — full backups;

  • Differential backup schedule — differential backups;

  • Incremental backup schedule — incremental backups.

If you need to get an additional backup of any type without waiting for the next auto launch, do the following:

  1. Click Backup on the cluster page or select the Backup action for the specified cluster in the Clusters section.

    adbm cluster action backup dark
    Switch to the Backup action
    adbm cluster action backup light
    Switch to the Backup action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Type — a backup type. Possible values:

      • Full — full;

      • Incr — incremental;

      • Diff — differential.

    • Configuration — the configuration version that will be applied. You can click it to check or edit the configuration parameters in the list of configurations.

    • Date — a current timestamp.

    • Delta — a flag that indicates the need to use checksums during the backup operation.

      adbm cluster action backup2 dark
      The Backup action form
      adbm cluster action backup2 light
      The Backup action form

  2. Select the backup type in the Type field and change the Restore point value if necessary.

  3. Click Run.

  4. As a result, the Start 'Backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, the new backup becomes available in the list of backups on the Backups tab.

    adbm cluster action backup3 dark
    The new backup is available on the Backups tab
    adbm cluster action backup3 light
    The new backup is available on the Backups tab

Create restore point

Restore points are automatically created according to the schedule that is defined by the Restore point creation schedule parameter of the current configuration version (see the General configuration tab).

If you need to create a restore point without waiting for the next auto launch, follow the steps:

  1. Select the Create restore point action on the cluster page or in the Clusters section.

    adbm cluster action rp dark
    Switch to the "Create restore point" action
    adbm cluster action rp light
    Switch to the "Create restore point" action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Configuration — the configuration version that will be applied. You can click it to check or edit the configuration parameters in the list of configurations.

    • Date — a current timestamp.

      adbm cluster action rp2 dark
      The "Create restore point" action form
      adbm cluster action rp2 light
      The "Create restore point" action form

  2. Change the restore point name in the Restore point field if necessary.

  3. Click Run.

  4. As a result, the Start 'Create restore point' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, a new restore point becomes available. You can check it by running the Restore or the Cleanup actions.

    adbm cluster action rp3 dark
    The new restore point is available
    adbm cluster action rp3 light
    The new restore point is available

Restore

The Restore action allows you to restore databases at the moment of one of existing restore points. This is the only action that can be applied to both a running and a stopped ADB cluster. The action logic depends on the cluster status:

  • ADB cluster is running (marked as UP in the ADBM UI). In this case, before the action is started, additional checks are automatically performed to ensure that the selected backup can be applied to the cluster:

    • The topology that is saved in the backup is compared with the topology obtained from the ADB cluster (from the system table gp_segment_configuration).

    • The backup metadata are also checked (see Verify below).

    If the validation is successful, the cluster stops automatically. In addition, if any error is fixed during the Restore action, you can run Restore retry — this is a restart of the last restore process without any additional checks.

  • ADB cluster is stopped (marked as DOWN in the ADBM UI). In this case, any validation is skipped. However, it is still possible to start the restore process due to using the Skip Validation flag (that is set and disabled for users by default). This method is recommended only in case of fatal problems when the cluster does not even start.

To restore databases, follow the steps:

  1. Select the Restore action on the cluster page or in the Clusters section.

    adbm cluster action restore dark
    Switch to the Restore action
    adbm cluster action restore light
    Switch to the Restore action

    The window that opens contains the following fields:

    • Time period — the timestamp range that is used to search restore points. The default value is the current date.

    • Restore point — the restore point at the moment of which you want to restore databases.

    • Databases — the databases that you can restore for the selected restore point.

    • Processes — a maximum number of processes to restore one segment.

    • Configuration — the configuration version that was applied to the selected restore point. You can click it to view the configuration parameters in the list of configurations.

    • Date created — a creation timestamp of the selected restore point.

    • Backup type — a type of the latest backup that was created before the selected restore point.

    • Backup date — a creation timestamp of the latest backup that was completed before the selected restore point.

    • Restore mirrors — the drop-down list which value indicates whether to restore data for mirrors and Standby if they exist in the cluster:

      • No — do not restore mirrors.

      • In parallel — restore mirrors concurrently with primary segments.

      • After primary — restore mirrors after all primary segments are restored.

    • Delta — a flag that indicates the need to use checksums during the restore operation.

    • Force — a flag that indicates the need to completely overwrite data and tablespace paths in PostgreSQL.

    • Skip Validation — a flag that indicates the need to restore databases even though validation is not possible. Displayed only if the ADB cluster is stopped (DOWN). Set and disabled for users by default.

      adbm cluster action restore2 dark
      The Restore action form (ADB cluster is running)
      adbm cluster action restore2 light
      The Restore action form (ADB cluster is running)
      adbm cluster action backup force dark
      The Restore action form (ADB cluster is stopped)
      adbm cluster action backup force light
      The Restore action form (ADB cluster is stopped)

  2. Fill in the necessary fields:

    • Select a restore point in the Restore point drop-down-list. Use the Time period filter if necessary.

    • In the Databases drop-down-list, select the databases that you want to restore.

    • Select the Restore mirrors list value.

    • If necessary, set the Delta and Force flags.

  3. Click Run.

  4. As a result, the Start 'Restore' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, the selected databases should be restored at the moment of the specified restore point.

For running ADB clusters (with the UP status), if any error occurs when running the Restore action — you can retry the latest restore process. To do this, select the Restore action again (see step 1 above) and click Retry in the window that opens. If you want to run a new Restore action from scratch (with all checks), click New restore.

adbm cluster action retry dark
Retry the last Restore action
adbm cluster action retry light
Retry the last Restore action

Cleanup

The Cleanup action is automatically executed according to the schedule that is defined by the Cleanup schedule parameter of the current configuration version (see the General configuration tab). This action removes:

  • The oldest extra full backups — if the number of full backups exceeds the Number of full backups parameter defined for the current configuration.

  • The oldest extra differential backups — if the number of differential backups exceeds the Number of differential backups parameter defined for the current configuration.

  • All backups in the following statuses: Failed, Invalid.

  • Inactive configuration versions that have no related backups.

When you run the Cleanup action manually, it deletes all objects mentioned above but also allows you to remove restore points and backups related to the specified restore point. You define the restore point, and ADBM finds the latest full or differential backup that was completed before the selected point created. Then, all backups and restore points related to that backup are removed.

To run the Cleanup action manually, follow the steps:

  1. Select the Cleanup action on the cluster page or in the Clusters section.

    adbm cluster action cleanup dark
    Switch to the Cleanup action
    adbm cluster action cleanup light
    Switch to the Cleanup action

    The window that opens contains the following fields:

    • Time period — the timestamp range that is used to search restore points. The default value is the current date.

    • Select restore point — the restore point at the moment of which you want to remove a set of related backups and restore points.

    • Configuration — the configuration version that was applied to the selected restore point. You can click it to view the configuration parameters in the list of configurations.

    • Date created — a creation timestamp of the selected restore point.

      adbm cluster action cleanup2 dark
      The Cleanup action form
      adbm cluster action cleanup2 light
      The Cleanup action form

  2. Select a restore point in the Select restore point drop-down-list. Use the Time period filter if necessary.

  3. Click Run.

  4. As a result, the Start cleanup job for 'Verify backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  5. As a successful result of all actions, backups and restore points are removed according to the rules described above. You can check it in the list of backups on the Backups tab.

    For example, if you have two backups, full and incremental, and select the restore point that was created after the incremental backup — both backups will be deleted.

    adbm cluster action cleanup3 dark
    List of backups before the Cleanup action
    adbm cluster action cleanup3 light
    List of backups before the Cleanup action
    adbm cluster action cleanup4 dark
    List of backups after the Cleanup action
    adbm cluster action cleanup4 light
    List of backups after the Cleanup action

To better understand the logic of the Cleanup manual action, you can view the example listed below. The figure shows a timeline with a list of backups of different types (called full*, diff*, incr*) and restore points (called rp*). The table explains which objects will be deleted when any of restore points is selected in the Cleanup action form.

example cleanup dark
Example with a list of backups and restore points
example cleanup light
Example with a list of backups and restore points
Logic of the manual Cleanup action
Selected restore point Objects to be removed

rp1

All backups and restore points for the specified period

rp2

All backups and restore points for the specified period

rp3

All backups and restore points for the specified period

rp4

All backups and restore points for the specified period

rp5

diff1, incr2, rp5 — rp8

rp6

diff1, incr2, rp5 — rp8

rp7

diff1, incr2, rp5 — rp8

rp8

diff1, incr2, rp5 — rp8

Verify

The Verify action compares the backup metadata that are stored in ADBM with the backup metadata that are being received from pgbackrest (i.e. actual backups). If the difference is fixed for any backups, they acquire the Invalid status as they cannot be used to restore databases. Such backups will be removed during the next Cleanup action launch.

Before comparing metadata, ADBM additionally checks existence of configuration files (pgbackrest.conf) for all timelines on all segment hosts. If any of these files are not present, the last available version of the timeline configuration is taken, and a new configuration file is created on the basis of this configuration. Then this file is written to the segment host for the selected timeline. This is necessary to ensure that all configurations are present before the metadata for the backups are checked.

To run the Verify action, do the following:

  1. Select the Verify action on the cluster page or in the Clusters section.

    adbm cluster action verify dark
    Switch to the Verify action
    adbm cluster action verify light
    Switch to the Verify action

  2. In the window that opens, click Run.

    adbm cluster action verify2 dark
    The Verify action form
    adbm cluster action verify2 light
    The Verify action form

  3. As a result, the Start 'Verify backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  4. As a successful result of all actions, some of the backups can acquire the Invalid status. You can search such backups using the Status filter on the Backups tab.

Terminate

The Terminate action allows you to terminate some of the currently running actions. This is the only action type for which a separate distributed lock is used (in etcd), which allows you to run Terminate in parallel with other types of actions.

IMPORTANT

You can terminate actions of the following types:

Below is an example how to terminate the Backup action:

  1. Run the Backup action as shown above.

    adbm cluster action terminate0 dark
    The new backup is being initialized
    adbm cluster action terminate0 light
    The new backup is being initialized

  2. Select the Terminate action on the cluster page or in the Clusters section.

    adbm cluster action terminate dark
    Switch to the Terminate action
    adbm cluster action terminate light
    Switch to the Terminate action

  3. In the window that opens, click Run.

    adbm cluster action terminate2 dark
    The Terminate action form
    adbm cluster action terminate2 light
    The Terminate action form

  4. As a result, the Start 'Terminate' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    The current backup acquires the Terminating status. You can check it in the list of backups on the Backups tab.

    adbm cluster action terminate3 dark
    The backup has the Terminating status
    adbm cluster action terminate3 light
    The backup has the Terminating status

  5. As a successful result of all actions, the backup acquires the Stopped status.

    adbm cluster action terminate4 dark
    The backup status is switched to Stopped
    adbm cluster action terminate4 light
    The backup status is switched to Stopped
NOTE

After applying the Terminate action, you can resume the latest terminated backup via the Resume action.

Resume

The Resume action allows you to resume the latest terminated backup launch. This action is available in the list of cluster actions only when backups with the Stopped status exist. Below is an example how to resume the backup that was terminated in the previous section:

  1. Select the Resume action on the cluster page or in the Clusters section.

    adbm cluster action resume dark
    Switch to the Resume action
    adbm cluster action resume light
    Switch to the Resume action

    The window that opens contains the following fields:

    • Restore point — an automatically generated name for a new restore point.

    • Configuration — the configuration version that will be applied. You can click it to check the configuration parameters in the list of configurations.

    • Date — a current timestamp.

      adbm cluster action resume2 dark
      The Resume action form
      adbm cluster action resume2 light
      The Resume action form

  2. Change the restore point name in the Restore point field if necessary.

  3. Click Run.

  4. As a result, the Start 'Resume backup' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    The backup acquires the Resume status. You can check it in the list of backups on the Backups tab.

    adbm cluster action resume3 dark
    The backup has the Resume status
    adbm cluster action resume3 light
    The backup has the Resume status

  5. As a successful result of all actions, the backup acquires the Done status.

    adbm cluster action resume4 dark
    The backup status is switched to Done
    adbm cluster action resume4 light
    The backup status is switched to Done
Found a mistake? Seleсt text and press Ctrl+Enter to report it