Disaster recovery

Daria Barysheva

Configure and run

To configure Disaster recovery in ADBM, follow the steps:

  1. Using the ADB Control web interface, add a new cluster to which data from the source ADB cluster will be copied. Next, the added cluster will be called target or DR cluster. Note that both ADB Control and ADBM services should be added to the target cluster via ADCM.

  2. Open the Backup Manager page in the ADB Control web interface. Ensure that the source and target clusters are available on the page in the UP status.

    adbm dr dark
    Both clusters are available on the Backup Manager page
    adbm dr light
    Both clusters are available on the Backup Manager page
    CAUTION

    • To properly configure data copying, use two clusters with the same major version (the first two digits in the bundle number). For example, clusters with the 6.23.4 and 6.23.5 bundle versions can interact. But you cannot configure data copying between clusters with the 6.23.4 and 6.24.1 versions.

    • To restore data on a target ADB cluster from a backup created on another cluster, you need the same number of primary segments on both clusters. The rest of the cluster topology settings (such as mirroring, standby, number of host segments) do not affect the restore option.

  3. Add backup configurations for both clusters. Note that the source Repository section (which is defined for a source cluster) should describe the storage that is available for both clusters (for example, S3).

    Repository section for a source cluster
    adbm dr3 dark
    Example of the Repository section filling for a source cluster
    adbm dr3 light
    Example of the Repository section filling for a source cluster
    Repository section for a target cluster
    adbm dr2 dark
    Example of the Repository section filling for a target cluster
    adbm dr2 light
    Example of the Repository section filling for a target cluster

  4. To verify the successful result of data copy in the future, check which data is currently available in both clusters.

    Check a source cluster

    • List tables of the adb database via the \dt psql command:

      \dt

      Result:

                               List of relations
 Schema |      Name       | Type  |  Owner  |       Storage
--------+-----------------+-------+---------+----------------------
 public | best_books      | table | gpadmin | heap
 public | book_type       | table | gpadmin | heap
 public | spatial_ref_sys | table | gpadmin | heap
 public | test            | table | gpadmin | append only columnar
 public | test2           | table | gpadmin | append only columnar
(5 rows)

    • View the best_books table data:

      SELECT * FROM best_books;

      Result:

       id |   author   |         title         | public_year
----+------------+-----------------------+-------------
  1 | Harper Lee | To Kill a Mockingbird |        1960
(1 row)
    Check a target cluster

     
    View tables of the adb database via the \dt psql command:

    \dt

    Result:

                      List of relations
 Schema |      Name       | Type  |  Owner  | Storage
--------+-----------------+-------+---------+---------
 public | spatial_ref_sys | table | gpadmin | heap
(1 row)

  5. On the Copy → Configurations tab of the Backup Manager page, click Create to add a new DR configuration.

    adbm dr4 dark
    Switch to adding a DR configuration
    adbm dr4 light
    Switch to adding a DR configuration

  6. In the window that opens, fill in the following fields:

    • Source cluster — a source cluster name.

    • Target cluster — a target cluster name.

    • Autostart — a flag that indicates whether to create a data stream with the Copy restore type for a selected pair of clusters automatically each time a backup with the Backup type type is created in the source cluster.

    • Backup type — types of backups in the source cluster that will trigger copying data (stream) to the target cluster. The field becomes available if the Autostart flag is set. Possible values:

      • Full — full backups;

      • Diff — differential backups;

      • Incr — incremental backups.

    adbm dr5 dark
    Add a DR configuration
    adbm dr5 light
    Add a DR configuration

  7. Click Create. As a result, the cluster pair is displayed on the Backup Manager → Copy → Configurations tab.

    adbm dr6 dark
    Configuration is added
    adbm dr6 light
    Configuration is added

  8. Ensure that a source cluster has at least one restore point in ADBM. For this, a cluster should have a backup. Backups can be created either automatically according to the configured schedule or manually using the Backup action.

    adbm dr7 dark
    A Full backup exists for a source cluster
    adbm dr7 light
    A Full backup exists for a source cluster

  9. Click Create stream on the Backup Manager → Copy → Streams tab to switch to configuring a data stream for the selected cluster pair.

    adbm dr8 dark
    Switch to adding a stream
    adbm dr8 light
    Switch to adding a stream

  10. In the window that opens, fill in the following fields:

    • Configuration — a configuration that you created earlier (which defines a pair of clusters).

    • Restore type — a restore type. Possible values:

      • Copy — cold standby. The target cluster data is recovered only once based on the selected restore point.

      • Streaming — warm standby. The target cluster data is constantly recovered based on the restore points created in the source cluster. After the replication process is terminated, the target cluster data will be actual at the moment of the latest restore point applied from the source cluster.

    • Restore point — the restore point at the moment of which you want to restore the source cluster databases in the target cluster. If the Streaming restore type is selected, the latest restore point is always used, and the Restore point field is disabled.

    • Databases — a list of the source cluster databases to restore in the target cluster.

    • Repository path — a path to the repository that stores the source cluster backups. The default value is extracted from the current backup configuration.

    • Processes — a maximum number of processes to restore one segment.

    • Delta — a flag that indicates the need to use checksums during the restore operation.

    • Force — a flag that indicates the need to completely overwrite data and tablespace paths in PostgreSQL.

    • Restore mirrors — the drop-down list, which value indicates whether to restore data for mirrors and Standby if they exist in the cluster:

      • No — do not restore mirrors.

      • In parallel — restore mirrors concurrently with primary segments.

      • After primary — restore mirrors after all primary segments are restored.

        Stream configuration with the Copy type
        Stream configuration with the Copy type
        Stream configuration with the Copy type
        Stream configuration with the Copy type
        Stream configuration with the Streaming type
        Stream configuration with the Streaming type
        Stream configuration with the Streaming type
        Stream configuration with the Streaming type

  11. Click Run. As a result, the Disaster recovery start stream copy (for the Copy restore type) or Disaster recovery. Start streaming cluster (for the Streaming restore type) action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    The Disaster recovery. Start streaming cluster remains in the Started status until the stream is terminated (see the note below). During that action, all restore points that are created in the source cluster will be applied to the target cluster.

  12. As a successful result of all actions, the source cluster data is copied to the target cluster. To ensure this, you can run queries from the step 4.

    IMPORTANT

    For streams with the Streaming type:

    • Data restore completes only after terminating a stream — manually or automatically. Auto termination can be initiated by one of the following events:

      • Restore of the source cluster.

      • Running a new stream where the source cluster is selected as a target.

      • Creating a new backup configuration for the source cluster that changes the repository settings.

      Only after successful termination, you can run queries in the target cluster.

    • The target cluster database will contain the source cluster data at the time of the last restore point that was applied during the streaming process.
    Check a target cluster

    • List tables of the adb database via the \dt psql command:

      \dt

      Result:

                               List of relations
 Schema |      Name       | Type  |  Owner  |       Storage
--------+-----------------+-------+---------+----------------------
 public | best_books      | table | gpadmin | heap
 public | book_type       | table | gpadmin | heap
 public | spatial_ref_sys | table | gpadmin | heap
 public | test            | table | gpadmin | append only columnar
 public | test2           | table | gpadmin | append only columnar
(5 rows)

    • View the best_books table data:

      SELECT * FROM best_books;

      Result:

       id |   author   |         title         | public_year
----+------------+-----------------------+-------------
  1 | Harper Lee | To Kill a Mockingbird |        1960
(1 row)

Note that during each stream launch, the restore operation is created for the target cluster in ADBM. You can view it on the Backup Manager → Backup → <target cluster name> → Restores tab as well as a standard data restore that is launched via the Restore action. For restore operations made as a part of copying data between clusters, the Restore type column contains Copy or Streaming (depending on the selected restore type). For more information, see View restores in ADBM.

For streams with the Streaming restore type, a restore operation status is changed to Done only after terminating a stream — manually or automatically (see the note above). Until then, a restore operation has the Running status.

adbm dr22 dark
Data restore for the target cluster
adbm dr22 light
Data restore for the target cluster

DR configuration management

View a list of configurations

All cluster pairs between which DR is configured are displayed on the Copy → Configurations tab of the Backup Manager page.

adbm dr11 dark
DR configurations on the Backup Manager → Copy → Configurations tab
adbm dr11 light
DR configurations on the Backup Manager → Copy → Configurations tab

The tab displays the following information.

The "Backup Manager → Copy → Configurations" tab fields
Field Description

Source cluster

The source cluster from which the data is to be copied

Target cluster

The target cluster to which the data is to be copied

Username

A login of the user who created the current configuration via the ADB Control web interface

Stream ID

A unique identifier of the stream that is currently running for a pair of clusters defined in the current configuration

Status

The stream status. See Status below

Restore type

The restore type that was selected during the stream configuration

Above the table with DR configurations, there are filters that you can use to select specific data. Available filters are listed below:

  • Source cluster — filter by the source cluster name (see Source cluster above). Select a value from the drop-down list.

  • Target cluster — filter by the target cluster name (see Target cluster above). Select a value from the drop-down list.

  • Username — filter by the user name (see Username above). Enter a full value.

adbm dr23 dark
Filters on the Backup Manager → Copy → Configurations tab
adbm dr23 light
Filters on the Backup Manager → Copy → Configurations tab

Edit a configuration

To edit a DR configuration, follow the steps:

  1. Click the edit dark edit light icon in the Actions column on the Copy → Configurations tab of the Backup Manager page.

    adbm dr24 dark
    Switch to editing a configuration
    adbm dr24 light
    Switch to editing a configuration

  2. In the window that opens, edit necessary fields. The Source cluster and Target cluster values cannot be changed.

    adbm dr25 dark
    Edit a configuration
    adbm dr25 light
    Edit a configuration

  3. Click Save.

Delete a configuration

CAUTION
Do not delete a configuration of the currently running stream.

To delete an existing configuration, follow the steps:

  1. Click the delete dark delete light icon in the Actions column on the Copy → Configurations tab of the Backup Manager page.

    adbm dr12 dark
    Switch to deleting a configuration
    adbm dr12 light
    Switch to deleting a configuration

  2. In the window that opens, confirm the operation by clicking Delete.

    adbm dr13 dark
    Confirm the operation
    adbm dr13 light
    Confirm the operation

    As a result, the configuration data is removed from the system and no longer displayed on the Backup Manager → Copy → Configurations tab.

    adbm dr14 dark
    Configuration is removed
    adbm dr14 light
    Configuration is removed

Stream management

View active streams

All active streams that are created to copy data between two clusters are displayed on the Copy → Streams tab of the Backup Manager page. Active streams have the statuses different from Impossible, Down, and Restored (the Copy → History tab is designed to show streams in these three statuses).

adbm dr15 dark
The Backup Manager → Copy → Streams tab
adbm dr15 light
The Backup Manager → Copy → Streams tab

The tab displays the following information.

The "Backup Manager → Copy → Streams" tab fields
Field Description

Stream ID

A unique identifier of the stream

Source cluster

The source cluster from which the data is to be copied within the current stream

Target cluster

The target cluster to which the data is to be copied within the current stream

Username

A login of the user who created the current stream via the ADB Control web interface

Status

A stream status. Possible values:

  • Init — the stream is being initialized;

  • Ready — the stream is ready for data restore;

  • Restoring — the restore operation is running;

  • Streaming — the stream with the Streaming type is running;

  • Terminating — the stream launch is being terminated manually or automatically (se the note above).

Start time

A stream creation timestamp in the DD/MM/YYYY HH:mm:ss format

Restore type

The restore type that was selected during the stream configuration

Replication lag

How much the target cluster lags from the source cluster. Calculated as the difference between the current time and the time when the latest restore point was applied to the target cluster during the streaming process. Available only for streams with the Streaming type in the Streaming status

In the column headers of the table with a list of streams, there are filters that you can use to select specific data. To open a filter, click the openside dark openside light icon. For those columns where the set of possible values is limited (e.g. Status), you can select a value from the drop-down list. For some columns (e.g. Username), the search value should be entered. For columns that show date and time (e.g. Start time), the time range can be selected from the calendar.

The filtered dark filtered light icon means that a filter is defined for the column. To reset all filters, click Reset.

View finished streams

The Backup Manager → Copy → History tab is used to display completed streams, which have the following statuses:

  • Impossible — the stream launch is impossible.

  • Down — the stream launch is failed.

  • Restored — the stream launch is successful.

The content of the History tab is completely identical to the Streams tab described above, except that the former does not contain the Replication lag column. Also, there are no available actions for streams on the History tab.

IMPORTANT

By default, the table on the Backup Manager → Copy → History tab shows the current-day streams. To view all streams, use the time filters.
The Backup Manager - Copy - History tab
The Backup Manager → Copy → History tab
The Backup Manager - Copy - History tab
The Backup Manager → Copy → History tab

View the stream details

To view the stream details, click the stream identifier (Stream ID) in the table on the Backup Manager → Copy → Streams or Backup Manager → Copy → History tab.

adbm dr17 dark
Select a stream
adbm dr17 light
Select a stream

The next page contains the following sections:

  • Overview

  • Statuses

  • Restore points (shown only for the Streaming restore type)

The Overview section displays the general stream information.

adbm dr18 dark
The Overview section
adbm dr18 light
The Overview section
The "Overview" section fields
Field Description

Source cluster

A source cluster name

Target cluster

A target cluster name

Username

A login of the user who created the current stream via the ADB Control web interface

Date

A stream creation timestamp in the DD/MM/YYYY HH:mm:ss format

Restore ID

A unique identifier of the restore operation in the target cluster. To view the restore details, click the identifier

Database

A list of databases that were selected during the stream configuration

Restore type

The restore type that was selected during the stream configuration

Repository path

A repository path that was selected during the stream configuration

Processes

A number of processes that was selected during the stream configuration

Flags

The flags that were selected during the stream configuration (see the Restore mirrors, Delta, Force flags above)

Action name

A name of the action within which the stream was launched. To view the action details, click the name

Restore point

An identifier of the restore point that was selected during the stream configuration

Replication lag

How much the target cluster lags from the source cluster. Calculated as the difference between the current time and the time when the latest restore point was applied to the target cluster during the streaming process. Available only for streams with the Streaming type in the Streaming status

The Statuses section displays the history of changing the selected stream status.

The Statuses section
The Statuses section
The Statuses section
The Statuses section
The "Statuses" section fields
Field Description

Stream history ID

A unique identifier of the stream status change event

Status

A stream status at the time specified in the Date field

Date

A timestamp of the stream status change event in the DD/MM/YYYY HH:mm:ss format

The Restore points section displays a list of restore points that have been applied to the target cluster during the current stream process. The section is shown only for streams with the Streaming type.

The Restore points section
The "Restore points" section
The Restore points section
The "Restore points" section
The "Restore points" section fields
Field Description

Restore point name

A name that was assigned to the restore point in the source cluster

Time applied

A timestamp when the restore point was applied to the target cluster (in the DD/MM/YYYY HH:mm:ss format)

Terminate a stream

IMPORTANT
The Terminate action is available only for streams with the Streaming type in the Streaming status.

To terminate a running stream with the Streaming type, follow the steps:

  1. Apply the Terminate action to the specified stream on the Backup Manager → Copy → Streams tab. To open a drop-down list with possible actions, click added actions dark added actions light in the Action column.

    Go to terminating the stream
    Go to terminating the stream
    Go to terminating the stream
    Go to terminating the stream

  2. In the window that opens, confirm the action by clicking Run.

    Confirm the action
    Confirm the action
    Confirm the action
    Confirm the action

  3. As a result, the Disaster recovery. Start terminate action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    During the termination process:

    • A status of the previously created action Disaster recovery. Start streaming cluster is changed to Done.

    • A status of the restore operation for the target cluster is changed to Done.

    • A status of the selected stream is changed as follows: Streaming → Terminating → Restoring → Restored.

    • The selected stream is moved from the Backup Manager → Copy → Streams tab to the Backup Manager → Copy → History tab.

Found a mistake? Seleсt text and press Ctrl+Enter to report it