Конференция Arenadata
Новое время — новый Greenplum
Мы приглашаем вас принять участие в конференции, посвященной будущему Open-Source Greenplum 19 сентября в 18:00:00 UTC +3. Встреча будет проходить в гибридном формате — и офлайн, и онлайн. Онлайн-трансляция будет доступна для всех желающих.
Внезапное закрытие Greenplum его владельцем — компанией Broadcom - стало неприятным сюрпризом для всех, кто использует или планирует начать использовать решения на базе этой технологии. Многие ожидают выхода стабильной версии Greenplum 7 и надеются на её дальнейшее активное развитие.
Arenadata не могла допустить, чтобы разрабатываемый годами Open-Source проект Greenplum прекратил своё существование, поэтому 19 сентября мы представим наш ответ на данное решение Broadcom, а участники сообщества получат исчерпывающие разъяснения на все вопросы о дальнейшей судьбе этой технологии.

На конференции вас ждёт обсуждение следующих тем:

  • План возрождения Greenplum;
  • Дорожная карта;
  • Экспертное обсуждение и консультации.
Осталось до события

Manage configurations in ADBM

Daria Barysheva

Overview

In ADBM, a configuration means a set of options that are used to manage binary database backups. These options include schedules for different backup types, paths for saving backups, log levels, timeouts, compression settings, etc. ADBM supports several configuration versions. But there can be only one active version, which is marked as current in the user interface.

Possible operations with configurations are described below. All operations are performed on the Configuration tab. To open this tab, follow the steps:

  1. Open the Backup Manager page via the ADB Control web interface.

  2. In the table that is located in the Clusters section of the opened page, click the ADB cluster name.

    Select the ADB cluster
    Select the ADB cluster
    Select the ADB cluster
    Select the ADB cluster

  3. Select the Configuration tab.

    The Configuration tab
    The Configuration tab
    The Configuration tab
    The Configuration tab

Initially, there are no configurations. To start work with backups in ADBM, add the first configuration.

Limitations

  • ADBM does not support topology of ADB clusters that consist of a single segment host with mirroring enabled. For these clusters, an error occurs when saving the configuration. Without this restriction, there would be duplicates in the pgBackRest configuration for each stanza.

  • If the cluster topology has changed before the configuration editing (via the Expand action in ADCM), the configuration will not apply as long as there is at least one valid backup in ADBM. This is due to the fact that after applying the configuration on the cluster with the changed topology, all previous backups automatically become invalid. Therefore, in such cases, it is necessary to explicitly remove all existing backups and then edit the configuration.

Add a new configuration

IMPORTANT

Remember that enabling any configuration (even without configured schedules) applies archive_command to your cluster and starts the WAL generating process.

  1. Open the Configuration tab as described above.

  2. Fill in the parameters that are located in the General configuration, Timeout, Repository, and Compression sections. When filling the fields, pay attention to the following:

    • Mandatory parameters are highlighted in red.

    • For those parameters that specify time values, you can use the following units:

      • ms — milliseconds;

      • sec — seconds;

      • min — minutes;

      • hr — hours;

      • d — days;

      • w — weeks.

    • For those parameters that specify data amount, you can use the following units:

      • b — bytes;

      • kb — kilobytes;

      • mb — megabytes;

      • gb — gigabytes.

    • For those parameters that specify time or data amount, use the <value><unit> format (without spaces). For example, 1min, 2hr. If you need a complex value, use a space (e.g. 30min 30sec).

    • By clicking the qm dark qm light icon (which appears to the right of some fields during their filling), you can view hints: available measurement units, minimum and maximum values, and so on.

    • The cron dark cron light icon indicates the ability to edit the field value in the separate form that you can open by clicking the icon. Currently, this icon is shown in the fields that define the backup and cleanup schedules. Click the icon to open the Cron expression generator window where you can set a schedule by selecting seconds, minutes, hours, days, months, and years. The selected values are automatically transformed into the cron expression that will be used as the field value. You can see it in the top part of the window along with its description. To save the current expression and return to the main form, click Save.

      Cron expression generator
      Cron expression generator
      Cron expression generator
      Cron expression generator
      General configuration
      Parameter Description Default value

      Number of full backups

      A maximum number of full backups. After this limit is reached, the oldest extra full backups will be removed during the next automatic or manual Cleanup action. For more details, see Cleanup

      3

      Number of differential backups

      A maximum number of differential backups. After this limit is reached, the oldest extra differential backups will be removed during the next automatic or manual Cleanup action. For more details, see Cleanup

      6

      Full Backup schedule

      A schedule for automatic creation of full backups. If you are going to create backups only manually, do not activate the switcher that is located next to the field

       — 

      Differential backup schedule

      A schedule for automatic creation of differential backups. If you are going to create backups only manually, do not activate the switcher that is located next to the field

       — 

      Incremental backup schedule

      A schedule for automatic creation of incremental backups. If you are going to create backups only manually, do not activate the switcher that is located next to the field

       — 

      Restore point creation schedule

      A schedule for automatic creation of restore points. If you are going to create restore points only manually, do not activate the switcher that is located next to the field

       — 

      Cleanup schedule

      A schedule for deleting extra backups (see Number of full backups and Number of differential backups above) and backups with the failed and invalid statuses. If you are going to run the cleanup action only manually, do not activate the switcher that is located next to the field

       — 

      File Log level

      A level for file logging.

      Possible values:

      • off — no logs;

      • error;

      • warn;

      • info;

      • detail;

      • debug;

      • trace.

      info

      Log path

      A path where log files are to be stored

      /home/gpadmin/gpAdminLogs

      Bundle size

      A target size for file bundles. A bundle means grouping single files into one file before the backup launch. It is not recommended to set the option too high because backup retries will need to redo the entire bundle

      100mb

      Bundle limit

      A limit for the files that are included in bundles (see Bundle size above). The files larger than this size are stored separately. Bundled files cannot be reused when a backup is resumed, so this option, in fact, controls the files that can be resumed

      2mb

      Use block incremental backup

      A switch that indicates whether to make more granular backups by splitting files into blocks that can be backed up independently

      false
      The General configuration section
      The "General configuration" section
      The General configuration section
      The "General configuration" section
      Timeout
      Parameter Description Default value

      Database query timeout

      Sets the timeout for queries against the database including the backup start/stop functions which can take a significant amount of time. The timeout should be kept high unless you know that these functions will return quickly. The Database query timeout value should be less than the Protocol timeout value

      30min

      Protocol timeout

      Sets the timeout that the local or the remote process can wait for a new message to be received on the protocol layer. The timeout prevents processes from the indefinite waiting. The Protocol timeout value should be greater than the Database query timeout value

      30min 30sec

      I/O timeout

      A timeout that is used for connections and read/write operations. The entire read/write operation does not need to complete within this timeout but some progress should be made, even if it is only a single byte

      1min

      Job wait timeout

      A timeout for a lock capture at the cluster level before the job starts

      15min
      The Timeout section
      The Timeout section
      The Timeout section
      The Timeout section
      Repository
      Parameter Description Default value

      Archive command

      The command that is to be executed when WAL (Write Ahead Log) files are transferred to the archive directory. If you change the Archive command field value, then the ADB cluster is automatically restarted when the configuration is applied (to change the value of the archive_command GUC).

       
      The stanza configuration name prefix for segments that is used in the Archive command should be equal to the Stanza template value (see below): --stanza=<Stanza template>%c

      		 
      PGOPTIONS=\"-c gp_session_role=utility\" /usr/lib/gpdb/bin/pgbackrest --stanza=seg%c archive-push %p

      Archive timeout

      Sets maximum time to wait for each WAL segment (required for backup consistency) to reach the archive repository

      1min

      Stanza template

      A stanza configuration name prefix for segments. Should be equal to the prefix that is used in the Archive command value (see above): --stanza=<Stanza template>%c

      seg

      Repository type

      A repository type.

      Possible values:

      • posix

      • s3

      • cifs

      posix

      Upload chunk size

      Size of the uploaded data chunk (part size in S3 terms). Displayed if s3 is selected in the Repository type field. Limitations shown in the tooltip (64kb-1tb) come from pgbackrest. Note that S3 storages may have their own restrictions

      5MB

      Exclude path

      Defines the paths and files that should be excluded from backups

      • pg_log/gp_era

      • backups

      • gpperfmon/data/

      • gpbackup_history.yaml

      Repository path

      The repository where backups and archived WAL segments are to be stored. The repository directory should be pre-created and available from Master and Segment hosts. You can use the NFS shared directory, which is connected to all cluster segments. Do not use local directories on segments if the mirroring is switched on — in that case the hosts with mirror segments won’t have the required stanzas

       — 

      URI Type

      An endpoint type (see Endpoint below). Displayed if s3 is selected in the Repository type field. Possible values:

      • host

      • path

       — 

      Endpoint

      An endpoint of the S3 storage. Displayed if s3 is selected in the Repository type field

       — 

      Bucket

      A bucket name in the S3 storage. A bucket name cannot contain dots. Displayed if s3 is selected in the Repository type field

       — 

      Key

      A key to access the S3 storage. Displayed if s3 is selected in the Repository type field

       — 

      Key secret

      A secret key to access the S3 storage. Displayed if s3 is selected in the Repository type field

       — 

      Region

      A region of the S3 storage. Displayed if s3 is selected in the Repository type field

       — 

      Repository storage CA file

      Certificate Authority (CA) PEM file to access the S3 storage. Displayed if s3 is selected in the Repository type field. To select the file, click Upload

       — 

      Certificate verification

      Defines whether to verify a repository storage certificate. Displayed if s3 is selected in the Repository type field. If you activate this switch, the Repository storage CA file becomes enabled

      true
      The Repository section
      The Repository section
      The Repository section
      The Repository section
      Compression
      Parameter Description Default value

      Compression type

      A compression type.

      Possible values:

      • none — no compression;

      • bz2;

      • gz;

      • lz4;

      • zst.

      gz

      Compression level

      A compression level. Possible values: 0 — 9

      0

      Max processes to use for compress/transfer

      The maximum number of processes to be used on every segment for compression/transferring

      1

      Buffer size

      A buffer size for I/O operations

      1mb
      The Compression section
      The Compression section
      The Compression section
      The Compression section

  3. Click Save. You can find this button both before and after the sections listed above.

    Save the configuration
    Save the configuration
    Save the configuration
    Save the configuration

  4. As a result, the Start 'Apply backup configuration' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

    Until all necessary actions are completed successfully, the configuration will have the Draft status.

  5. As a successful result of all actions, the configuration acquires the Current status. Along with status, the following information about the configuration is available:

    • The version number (starting with 1).

    • The user who created a configuration version.

    • The creation date in the DD/MM/YYYY format.

      The successful result
      The successful result
      The successful result
      The successful result

Edit a configuration

After you created the configuration, you cannot edit it. But it is possible to add a new configuration version based on one of the previously created. To do this, perform the following steps:

  1. Open the Configuration tab as described above.

  2. In the list of available configuration versions, click the version on base of which you want to create a new one. All parameters of the selected version are displayed under the list with available versions (in the read-only mode).

    NOTE

    You can create a configuration based on any previous version, not just the latest.
    Select a version
    Select a version
    Select a version
    Select a version

  3. Click Create new.

  4. Apply necessary changes to the parameters that are located in the General configuration, Timeout, Repository, and Compression sections. The parameters are the same as described above.

  5. Click Save to add a new configuration version. You can find this button both before and after the sections with parameters.

    Save a new configuration version
    Save a new configuration version
    Save a new configuration version
    Save a new configuration version

  6. As a result, the Start 'Apply backup configuration' action runs. This action, in turn, generates several subactions. You can see all of them on the Actions tab (for more details, see View actions in ADBM).

  7. After all necessary actions are completed successfully, the new configuration version becomes available on the Configuration tab and has the Current status.

    NOTE

    • ADBM uses the latest configuration version you added.

    • Inactive configuration versions that have no related backups will be removed during the next automatic or manual Cleanup action.
    The new configuration version
    The new configuration version
    The new configuration version
    The new configuration version

Draft configurations

Starting with the version 1.6.3, ADBM supports an initial status Draft for backup configurations. This status is assigned to all configurations when attempting to save them until all actions are successfully applied and the status is changed to Current. However, if a new configuration cannot be validated at the moment of its creation, it is saved as a draft. Thus, you can avoid re-entering all information during the next applying attempt. Configuration drafts can be modified (and re-applied) or removed by clicking Edit and Delete, respectively.

Starting with ADBM 1.7.3, the Verified status is also available for backup configurations. This status means that validation of the backup configuration is successfully completed, but some errors occurred during its application. Configurations in the Verified status can be edited or removed like drafts.

Configuration in the Draft status
Configuration in the Draft status
Configuration in the Draft status
Configuration in the Draft status
Found a mistake? Seleсt text and press Ctrl+Enter to report it