Disaster recovery

Daria Barysheva

Contents

Configure and run
DR configuration management
Stream management

Configure and run

To configure Disaster recovery in ADBM, follow the steps:

Using the ADB Control web interface, add a new cluster to which data from the source ADB cluster will be copied. Next, the added cluster will be called target or DR cluster. Note that both ADB Control and ADBM services should be added to the target cluster via ADCM.

Open the Backup Manager page in the ADB Control web interface. Ensure that the source and target clusters are available on the page in the UP status.

Both clusters are available on the Backup Manager page

CAUTION

To properly configure data copying, use two clusters with the same major version (the first two digits in the bundle number). For example, clusters with the 6.23.4 and 6.23.5 bundle versions can interact. But you cannot configure data copying between clusters with the 6.23.4 and 6.24.1 versions.
To restore data on a target ADB cluster from a backup created on another cluster, you need the same number of primary segments on both clusters. The rest of the cluster topology settings (such as mirroring, standby, number of host segments) do not affect the restore option.
The operating system versions and families should be compatible on the source and target cluster hosts.

Add backup configurations for both clusters. Note that the source Repository section (which is defined for a source cluster) should describe the storage that is available for both clusters (for example, S3).

Repository section for a source cluster

Example of the Repository section filling for a source cluster

Example of the Repository section filling for a source cluster

Repository section for a target cluster

Example of the Repository section filling for a target cluster

Example of the Repository section filling for a target cluster

To verify the successful result of data copy in the future, check which data is currently available in both clusters.

Check a source cluster

List tables of the adb database via the \dt psql command:

\dt

Result:

                         List of relations
 Schema |      Name       | Type  |  Owner  |       Storage
--------+-----------------+-------+---------+----------------------
 public | best_books      | table | gpadmin | heap
 public | book_type       | table | gpadmin | heap
 public | spatial_ref_sys | table | gpadmin | heap
 public | test            | table | gpadmin | append only columnar
 public | test2           | table | gpadmin | append only columnar
(5 rows)

View the best_books table data:

SELECT * FROM best_books;

Result:

 id |   author   |         title         | public_year
----+------------+-----------------------+-------------
  1 | Harper Lee | To Kill a Mockingbird |        1960
(1 row)

Check a target cluster

View tables of the adb database via the \dt psql command:

\dt

Result:

                  List of relations
 Schema |      Name       | Type  |  Owner  | Storage
--------+-----------------+-------+---------+---------
 public | spatial_ref_sys | table | gpadmin | heap
(1 row)

On the Copy → Configurations tab of the Backup Manager page, click Create to add a new DR configuration.

Switch to adding a DR configuration

Switch to adding a DR configuration
In the window that opens, fill in the following fields:
- Source cluster — a source cluster name.
- Target cluster — a target cluster name.
- Autostart — a flag that indicates whether to create a data stream with the Copy restore type for a selected pair of clusters automatically each time a backup with the Backup type type is created in the source cluster.
- Backup type — types of backups in the source cluster that will trigger copying data (stream) to the target cluster. The field becomes available if the Autostart flag is set. Possible values:
  
  Full — full backups;
  
  Diff — differential backups;
  
  Incr — incremental backups.
Add a DR configuration

Add a DR configuration
Click Create. As a result, the cluster pair is displayed on the Backup Manager → Copy → Configurations tab.

Configuration is added

Configuration is added
Ensure that a source cluster has at least one restore point in ADBM. For this, a cluster should have a backup. Backups can be created either automatically according to the configured schedule or manually using the Backup action.

A Full backup exists for a source cluster

A Full backup exists for a source cluster
Click Create stream on the Backup Manager → Copy → Streams tab to switch to configuring a data stream for the selected cluster pair.

Switch to adding a stream

Switch to adding a stream
In the window that opens, fill in the following fields:
- Configuration — a configuration that you created earlier (which defines a pair of clusters).
- Restore type — a restore type. Possible values:
  - Copy — cold standby. The target cluster data is recovered only once based on the selected restore point.
  - Streaming — warm standby. The target cluster data is constantly recovered based on the restore points created in the source cluster. After the replication process is terminated, the target cluster data will be actual at the moment of the latest restore point applied from the source cluster.
- Restore point — the restore point at the moment of which you want to restore the source cluster databases in the target cluster. If the Streaming restore type is selected, the latest restore point is always used, and the Restore point field is disabled.
- Databases — a list of the source cluster databases to restore in the target cluster.
- Repository path — a path to the repository that stores the source cluster backups. The default value is extracted from the current backup configuration.
- Processes — a maximum number of processes to restore one segment.
- On restore points streaming failure — when the Streaming restore type is selected, this field determines the behavior of the system in case of errors during the application of restore points to the target cluster (for example, damage of WAL archives, unplanned stream interruption, storage unavailability, etc.). The field is available starting with ADBM 2.4.0. Possible values:
  - Pause — transfer the stream to the Paused status for its temporary suspension. In this status, two actions become available for the stream: Resume (to resume the stream) and Cancel (to completely cancel the stream). During the Resume action execution, the application of restore points to the target cluster resumes from the state when processing failed last time; and the stream status returns to Streaming. The Cancel action, in turn, reverts the target cluster state to the previous restore point (preceding the one whose processing failed) and finishes the stream (the stream acquires the Cancelled status).
    
    This mode requires you to make manual corrections before resuming the stream. It is suitable when you need to keep the target cluster state as close to the current state of the source cluster as possible and do not require unlocking the target cluster (for running DB requests) while making corrections.
  - Bring online — revert the target cluster state to the previous restore point (preceding the one whose processing failed), restore data, and finish the stream (with unlocking the target cluster for running database queries and changing the stream status to Restored).
- Restore mirrors — the drop-down list, which value indicates whether to restore data for mirrors and Standby if they exist in the cluster:
  - No — do not restore mirrors.
  - Parallel — restore mirrors concurrently with primary segments.
  - Sequential — restore mirrors after all primary segments are restored.
- Flags:
  - Delta — a flag that indicates the need to use checksums during the restore operation.
  - Force — a flag that indicates the need to completely overwrite data and tablespace paths in PostgreSQL.
    
    Stream configuration with the Copy type
    
    Stream configuration with the Copy type
    
    Stream configuration with the Streaming type
    
    Stream configuration with the Streaming type
Click Run. As a result, the Disaster recovery start stream copy (for the Copy restore type) or Disaster recovery. Start streaming cluster (for the Streaming restore type) action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

The Disaster recovery. Start streaming cluster remains in the Started status until the stream is terminated (see the note below). During that action, all restore points that are created in the source cluster will be applied to the target cluster.

As a successful result of all actions, the source cluster data is copied to the target cluster. To ensure this, you can run queries from the step 4.

IMPORTANT

For streams with the Streaming type:

Data restore completes only after terminating a stream — manually or automatically. Auto termination can be initiated by one of the following events:
- Common restore of the source cluster.
- Running a new stream where the source cluster is selected as a target.
- Creating a new backup configuration for the source cluster that changes the repository settings.
Only after successful termination, you can run queries in the target cluster.
The target cluster database will contain the source cluster data at the time of the last restore point that was applied during the streaming process.

Check a target cluster

List tables of the adb database via the \dt psql command:

\dt

Result:

                         List of relations
 Schema |      Name       | Type  |  Owner  |       Storage
--------+-----------------+-------+---------+----------------------
 public | best_books      | table | gpadmin | heap
 public | book_type       | table | gpadmin | heap
 public | spatial_ref_sys | table | gpadmin | heap
 public | test            | table | gpadmin | append only columnar
 public | test2           | table | gpadmin | append only columnar
(5 rows)

View the best_books table data:

SELECT * FROM best_books;

Result:

 id |   author   |         title         | public_year
----+------------+-----------------------+-------------
  1 | Harper Lee | To Kill a Mockingbird |        1960
(1 row)

Note that during each stream launch, the restore operation is created for the target cluster in ADBM. You can view it on the Backup Manager → Backup → <target cluster name> → Restores tab as well as a standard data restore that is launched via the Common restore action. For restore operations made as a part of copying data between clusters, the Restore type column contains Copy or Streaming (depending on the selected restore type). For more information, see View restores in ADBM.

For streams with the Streaming restore type, a restore operation status is changed to Done only after terminating a stream — manually or automatically (see the note above). Until then, a restore operation has the Running status.

Data restore for the target cluster

DR configuration management

View a list of configurations

All cluster pairs between which DR is configured are displayed on the Copy → Configurations tab of the Backup Manager page.

DR configurations on the Backup Manager → Copy → Configurations tab

The tab displays the following information.

The "Backup Manager → Copy → Configurations" tab fields
Field	Description
Source cluster	The source cluster from which the data is to be copied
Target cluster	The target cluster to which the data is to be copied
Username	A login of the user who created the current configuration via the ADB Control web interface
Stream ID	A unique identifier of the latest stream that was launched for a pair of clusters defined in the current configuration
Status	A status of the *Stream ID* stream. See *Status* below
Restore type	The restore type that was selected during the stream configuration

Above the table with DR configurations, there are filters that you can use to select specific data. Available filters are listed below:

Source cluster — filter by the source cluster name (see Source cluster above). Select a value from the drop-down list.
Target cluster — filter by the target cluster name (see Target cluster above). Select a value from the drop-down list.
Username — filter by the user name (see Username above). Enter a full value.

Filters on the Backup Manager → Copy → Configurations tab

Edit a configuration

To edit a DR configuration, follow the steps:

Click the icon in the Actions column on the Copy → Configurations tab of the Backup Manager page.

Switch to editing a configuration

Switch to editing a configuration
In the window that opens, edit necessary fields. The Source cluster and Target cluster values cannot be changed.

Edit a configuration

Edit a configuration
Click Save.

Delete a configuration

CAUTION

Do not delete a configuration of the currently running stream.

To delete an existing configuration, follow the steps:

Click the icon in the Actions column on the Copy → Configurations tab of the Backup Manager page.

Switch to deleting a configuration

Switch to deleting a configuration
In the window that opens, confirm the operation by clicking Delete.

Confirm the operation

Confirm the operation

As a result, the configuration data is removed from the system and no longer displayed on the Backup Manager → Copy → Configurations tab.

Configuration is removed

Configuration is removed

Stream management

View active streams

All active streams that are created to copy data between two clusters are displayed on the Copy → Streams tab of the Backup Manager page. Active streams have the statuses different from Impossible, Down, and Restored (the Copy → History tab is designed to show streams in these three statuses).

The Backup Manager → Copy → Streams tab

The tab displays the following information.

The "Backup Manager → Copy → Streams" tab fields
Field	Description
Stream ID	A unique identifier of the stream
Source cluster	The source cluster from which the data is to be copied within the current stream
Target cluster	The target cluster to which the data is to be copied within the current stream
Username	A login of the user who created the current stream via the ADB Control web interface
Status	A stream status. Possible values: `Init` — the stream is being initialized; `Ready` — the stream is ready for data restore; `Paused` — the stream with the `Streaming` type is temporarily stopped (see *On restore points streaming failure* above); `Copying` — the backup is being copied; `Restoring` — the restore operation is running; `Streaming` — the stream with the `Streaming` type is running; `Terminating` — the stream launch is being terminated manually or automatically (see the note above). For more information on possible transitions between statuses, see the diagram below
Start time	A stream creation timestamp in the `DD/MM/YYYY HH:mm:ss` format
Restore type	The restore type that was selected during the stream configuration
Replication lag	How much the target cluster lags from the source cluster. Calculated as the difference between the current time and the time when the latest restore point was applied to the target cluster during the streaming process. Available only for streams with the `Streaming` type in the `Streaming` status

Stream statuses

In the column headers of the table with a list of streams, there are filters that you can use to select specific data. To open a filter, click the openside dark openside light icon. For those columns where the set of possible values is limited (e.g. Status), you can select a value from the drop-down list. For some columns (e.g. Username), the search value should be entered. For columns that show date and time (e.g. Start time), the time range can be selected from the calendar.

The filtered dark filtered light icon means that a filter is defined for the column. To reset all filters, click Reset.

View finished streams

The Backup Manager → Copy → History tab is used to display completed streams, which have the following statuses:

Impossible — the stream launch is impossible.
Down — the stream launch is failed.
Restored — the stream launch is successful.
Cancelled — the previously stopped stream with the Streaming type is cancelled (see On restore points streaming failure above).

The content of the History tab is completely identical to the Streams tab described above, except that the former does not contain the Replication lag column. Also, there are no available actions for streams on the History tab.

IMPORTANT

By default, the table on the Backup Manager → Copy → History tab shows the current-day streams. To view all streams, use the time filters.

The Backup Manager → Copy → History tab

View the stream details

To view the stream details, click the stream identifier (Stream ID) in the table on the Backup Manager → Copy → Streams or Backup Manager → Copy → History tab.

Select a stream

The next page contains the following sections:

Overview
Statuses
Restore points (shown only for the Streaming restore type)

The Overview section displays the general stream information.

The Overview section

The "Overview" section fields
Field	Description
Source cluster	A source cluster name
Target cluster	A target cluster name
Username	A login of the user who created the current stream via the ADB Control web interface
Date	A stream creation timestamp in the `DD/MM/YYYY HH:mm:ss` format
Restore ID	A unique identifier of the restore operation in the target cluster. To view the restore details, click the identifier
Database	A list of databases that were selected during the stream configuration
Restore type	The restore type that was selected during the stream configuration
Processes	A number of processes that was selected during the stream configuration
Flags	The flags that were selected during the stream configuration (see the *Delta, Force* flags above)
Replication lag	How much the target cluster lags from the source cluster. Calculated as the difference between the current time and the time when the latest restore point was applied to the target cluster during the streaming process. Available only for streams with the `Streaming` type in the `Streaming` status
Action name	A name of the action within which the stream was launched. To view the action details, click the name
Restore point	An identifier of the restore point that was selected during the stream configuration
Repository path	A repository path that was selected during the stream configuration
Mirrors restore mode	The restore mode selected for mirrors during the stream configuration (see *Restore mirrors* above)
Error handling mode	The error handling mode specified during the stream configuration (see *On restore points streaming failure* above)

The Statuses section displays the history of changing the selected stream status.

The Statuses section

The "Statuses" section fields
Field	Description
Stream history ID	A unique identifier of the stream status change event
Status	A stream status at the time specified in the *Date* field
Date	A timestamp of the stream status change event in the `DD/MM/YYYY HH:mm:ss` format

The Restore points section displays a list of restore points that have been applied to the target cluster during the current stream process. The section is shown only for streams with the Streaming type.

The "Restore points" section

The "Restore points" section fields
Field	Description
Restore point name	A name that was assigned to the restore point in the source cluster
Status	A status of applying the restore point to the target cluster: `Init` — initialized; `Done` — finished successfully; `Failed` — finished with an error.
Time applied	A timestamp when the restore point was applied to the target cluster (in the `DD/MM/YYYY HH:mm:ss` format)

Terminate a stream

IMPORTANT

The Terminate action is available only for streams with the Streaming type in the Streaming status.

To terminate a running stream with the Streaming type, follow the steps:

Apply the Terminate action to the specified stream on the Backup Manager → Copy → Streams tab. To open a drop-down list with possible actions, click in the Action column.

Go to terminating the stream

Go to terminating the stream
In the window that opens, confirm the action by clicking Run.

Confirm the action

Confirm the action
As a result, the Disaster recovery. Start terminate action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

During the termination process:
- A status of the previously created action Disaster recovery. Start streaming cluster is changed to Done.
- A status of the restore operation for the target cluster is changed to Done.
- A status of the selected stream is changed as follows: Streaming → Terminating → Restoring → Restored.
- The selected stream is moved from the Backup Manager → Copy → Streams tab to the Backup Manager → Copy → History tab.

Resume a stream

IMPORTANT

The Resume action is available only for streams with the Streaming type in the Paused status starting with ADBM 2.4.0.

To resume a stopped stream with the Streaming type (see On restore points streaming failure above), follow the steps:

Apply the Resume action to the specified stream on the Backup Manager → Copy → Streams tab. To open a drop-down list with possible actions, click in the Action column.

Go to resuming a stream

Go to resuming a stream
In the window that opens, you can change the stream error handling mode by selecting one of the values in the Handler field:
- No operations — leave the Pause stream handler without any changes (see On restore points streaming failure above).
- Set stream handler to Bring Online — change the stream error handling mode from Pause to Bring online (see On restore points streaming failure above).
- Immediate recovery — terminate the current stream immediately and restore the target cluster data at the moment of the last valid restore point.
  
  After you select the Handler value, confirm the action by clicking Run.
  
  Confirm the action
  
  Confirm the action
As a result, the Resume Stream action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

If the stream resumes successfully, it acquires the Streaming status. If you selected the target cluster restore in the action settings and it does not fail, the stream status changes to Restored.

Cancel a stream

IMPORTANT

The Cancel action is available only for streams with the Streaming type in the Paused status starting with ADBM 2.4.0.

To cancel a stopped stream with the Streaming type (see On restore points streaming failure above), follow the steps:

Apply the Cancel action to the specified stream on the Backup Manager → Copy → Streams tab. To open a drop-down list with possible actions, click in the Action column.

Go to cancelling a stream

Go to cancelling a stream
In the window that opens, confirm the action by clicking Run.

Confirm the action

Confirm the action
As a result, the Cancel Stream action starts for the target cluster. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

After all actions are successfully finished, the stream status changes to Cancelled.

Found a mistake? Seleсt text and press Ctrl+Enter to report it