How to speed up backup and save storage space: on the example of ArenadataDB ddboost and Dell EMC Data Domain

To obtain target metrics in the test environment, the ddpconnchk specialized utility was used. This utility allows performing a load test of the channel to the data storage system and, in fact, obtaining the expected speed of writing and reading data. The launch is parameterized by the number of threads that perform reading/writing, the final size of the data to be written, and a number of other parameters.

Several runs of this utility in the test environment allowed us to determine the target metrics of the write speed (see the table below).

The results of synthetic tests showed that the highest speed was achieved with a certain combination of parameters: a test file of 1 GB and 20 threads. With 30 threads, the speed is noticeably lower — at the level of 10 threads.

It is worth noting that this test, in terms of the write speed, does not present the picture that is typical for the process of transferring data with its deduplication. Also, in a real configuration, the speed can be limited from above by other subsystems, such as disk.

Thus, these metrics can be considered indicative for the case with a unique set of data in the absence of performance limitations of the client’s disk subsystem.

An isolated test is a test transfer of data directly to the data storage system through the plugin without processing and transferring data from gpbackup (such as preparing table data and then transferring it through standard I/O to the plugin, etc.).

The idea of the test is to check "pure" performance without the potential impact of operations performed by the gpbackup utility. The test stream for this test is randomly generated binary data (the total amount of data to be written is 200 GB in one file or the same size but divided into separate parts). This data is routed to the plugin directly via standard I/O. gpbackup and gprestore work with plugins in a similar way.

Several runs of this test allowed us to determine the following write speed metrics.

One server — one thread

The average speed of writing to one thread is ~150 MB/sec, the total time of transferring 200 GB is ~22 minutes.

 
One server — ten threads

From the graph above, we can see that when ten files were transferred in parallel from one host, the speed was on average slightly above 1 GB/sec and it decreased as individual parts were sent. The total time of transferring 200 GB was ~3.5 minutes.

In general, as sending of individual parts completes, the drop in the overall speed of sending is slightly greater than one connection in a single test provided. For example — for two files, one would expect a speed of about 300 MB/sec, but in fact we see ~235 MB/sec. Perhaps this is due to the internal thread pool of the DD Boost library and how bandwidth is shared between threads within the library.

 
One server — twenty threads

From the graph above, we can see that when transferring twenty files in parallel, the total time is even slightly longer than it was with ten threads.

As the speed slows down at the end, the issue of sharing bandwidth between open connections is also relevant. As sending individual parts of a file is completed and connections are released, one would expect to have the same speed for ten active connections (out of twenty at the start) as for ten ones. However, as graphs show, the drop in speed was proportional to the initial pool of active connections.

The write speed generally matches the speed to ten threads from one server that we saw in the results of previous tests (~1.1 GB/sec). On average, we received ~2 GB/sec for writing from two servers.

The total speed of 2.2 GB/sec is slightly higher than the speed for twenty connections from two servers, but it is already possible to assume that, starting with four servers, the maximum for writing from the storage and deduplication system is visible. The speed in peaks approximately corresponds to the maximum that is achieved from 1-2 servers, but on average it can be seen that the bandwidth is already shared between all forty connections from four servers — ~550 MB/sec per server.

A further drop in the write speed from one server with a similar overall write speed (2.2 GB/sec) indirectly confirms the hypothesis about the maximum write speed on the part of the storage and deduplication system. This hypothesis is confirmed by the fact that synthetic tests by the ddpconnchk utility with similar parameters (30 connections/threads) showed a generally similar result in terms of maximum write speed (2560.00 MB/sec).

Thus, we can conclude that the practical maximum achieved in isolated tests in a configuration with eight servers is comparable to the results of synthetic bandwidth tests.

For a series of final tests, the test environment configuration was the following:

A series of performance tests included:

* The amount of data actually transferred is 30-40% larger due to the need to convert to CSV format.

** The estimated speed was calculated based on the original size of the database, not data being actually copied.

We also performed several reference tests of backup to a local file system using the plugin (8) and without it (9), as well as a series of tests of local backups (10-17) with different settings of the queue length to prepare tables for backup (copy-queue-size).

Conclusions from the performed tests:

The detail of the full DB backup test runs is of the greatest interest, since it reflects the scenario that is closest to reality with tables of different sizes.

The image below shows a graph of copying data by one of the segment hosts.