ADB known issues

Daria Barysheva
Issue From Fixed Workaround

In case of requests that have a slice executing on the coordinator, spill file entries in gp_toolkit.gp_workfile_entries and gp_toolkit.gp_workfile_usage_per_query might be duplicated (with different PIDs), resulting in incorrect total size calculation

 — 

 — 

Use SELECT DISTINCT ON (segid, sess_id, command_cnt, prefix) in queries against the mentioned views to remove duplicate entries. For example:

  • Get information about all spill files:

    SELECT DISTINCT ON (segid, sess_id, command_cnt, prefix)
    pid, sess_id, segid, command_cnt, size, numfiles, prefix
    FROM gp_toolkit.gp_workfile_entries
    ORDER BY segid, sess_id, command_cnt, prefix, pid;

  • Get total space usage for different sessions and segments:

    SELECT sess_id, segid, SUM(size) AS total_size, SUM(numfiles) AS total_files FROM
(
    SELECT DISTINCT ON (segid, sess_id, command_cnt, prefix)
    segid, size, numfiles
    FROM gp_toolkit.gp_workfile_entries
) t
GROUP BY segid, sess_id ORDER BY segid, sess_id;

There is an error in the description of the CPUSET option of the ALTER RESOURCE GROUP command in the VMWare documentation.

Actually, the segment_cores value comes first, and then the master_cores value

 — 

 — 

Use the correct description provided in the first table column

For the timestamp with time zone type, PXF does not always perform push-down

6.27.1.57

 — 

To ensure the correct handling of the timestamp with time zone type, an external table definition must include the date_wide_range=true parameter

Some ulimit parameters (e.g. nofile limit) may be not set when a cluster is started via the gpstart console utility under the gpadmin user (occurs in Astra Linux)

6.25.2.52.b1

 — 

When switching to gpadmin, do not use sudo -iu gpadmin. Use sudo su instead:

$ sudo su - gpadmin

If you made a backup via gpbackup for ADB cluster of a version preceding 6.25.2.52.b1 and try to restore a database from that backup in ADB 6.25.2.52.b1 via gprestore, you can get the following error:

[CRITICAL]:-ERROR: language "plpythonu" already exists (SQLSTATE 42710)

6.25.2.52.b1

 — 

Drop PROCEDURAL LANGUAGE plpythonu in a target database before running gprestore

If you made a backup via gpbackup for ADB cluster of a version preceding 6.25.2.52.b1 and try to restore a database from that backup in ADB 6.25.2.52.b1 via gprestore, you can get the following error:

[CRITICAL]:-ERROR: function public.dblink_connect_no_auth(text) does not exist (SQLSTATE 42883)

6.25.2.52.b1

 — 

  1. Run gprestore with the --metadata-only and --on-error-continue options.

  2. Check that there are not any critical errors in error log (if errors are present, add the necessary relations manually).

  3. Run gprestore with the --data-only option to restore data.

If you made a backup via gpbackup for ADB cluster of a version preceding 6.25.2.52.b1 and try to restore a database from that backup in ADB 6.25.2.52.b1 via gprestore, you can get the following errors:

[CRITICAL]:-ERROR: incompatible library "/usr/lib/gpdb/lib/postgresql/gptkh.so": version mismatch (dfmgr.c:367) (SQLSTATE XX000)

or

[CRITICAL]:-ERROR: required extension "pxf_fdw" is not installed (SQLSTATE 42704)

6.25.2.52.b1

 — 

Update the extension packages gptkh and pxf_fdw in each database that uses these extensions and then update the extensions.

To restore a backup, you only need to update the packages

The SELECT query to arenadata_toolkit.__db_files_current fails with the following error:

psql:/tmp/tmpLWiv6c:1: ERROR:  illegal rescan of motion node: invalid plan (nodeMotion.c:1712)  (seg105 slice5 10.183.104.153:10003 pid=22864) (nodeMotion.c:1712)
HINT:  Likely caused by bad NL-join, try setting enable_nestloop to off

6.25.1.49

 — 

  1. Via psql, connect to the database where you gather statistics (e.g. adb) under a superuser:

    $ psql adb -U <superuser>

  2. Run the following command to alter the function:

    ALTER FUNCTION arenadata_toolkit.adb_get_relfilenodes(tablespace_oid OID) ROWS 30000000;

The ADB bundle update action can fail with the following error: Failed: invalid syntax, for details look pg_hba.conf on master host

6.24.3.48

 — 

  1. Backup logs from the pg_log directory.

  2. Clear or compress log files.

  3. Restart ADB.

After updating ADB to version 6.24.3.48.b1 or higher, you can get the following error when using gpbackup/gprestore:

[CRITICAL]:-table backups has no column named segment_count

The error occurs due to the fact that in gpbackup 1.28.1 the segment_count field was added to the backups table of the service SQLite database backup_history.db.

This SQLite database is used to:

  • Find a previous completed backup with the same options to create an incremental backup if the --from-timestamp parameter is omitted.

  • During the restore process, find a plugin version (e.g. gpbackup_s3_plugin) that was used to create the specified backup.

Another side effect of the column absence is the incorrect value for a number of segments in backups that were already made. It leads to an error during a resize restore operation

6.24.3.48.b1

 — 

Use one of the following options (each one comes with certain limitations):

  • Option 1. Use the --from-timestamp and --no-history gpbackup parameters together for incremental backups. This option is not helpful when a plugin is used.

  • Option 2. Drop the old history database file backup_history.db after upgrade to gpbackup 1.28.1 or higher:

    $ rm $MASTER_DATA_DIR/gpseg-1/gpbackup_history.db

    You have to proceed with full backups to restore data from incremental backups and provide the backup plugin work.

  • Option 3. Alter the backups table schema in the backup_history.db SQLite database:

    $ sqlite3 $MASTER_DATA_DIR/gpseg-1/gpbackup_history.db
    ALTER TABLE backups ADD COLUMN segment_count INT;
pragma table_info('backups');

    You have to provide a valid number of segments if the resize restore operation is requested.

When data is selected from ClickHouse via ADB ClickHouse Connector, the following error occurs if the password for the default user is defined:

DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

6.24.3.47

 — 

Do not use a password for the default user

Due to technical limitations in the ADB DDBoost plugin, the restore_subset plugin option is automatically set to off when the on-error-continue option is specified

6.22.0.38

 — 

Do not use the on-error-continue option during data restore

Expanding a cluster may fail if fully qualified domain names (FQDN) are used for the cluster hosts and the Check array flag is set

6.16.1.20

 — 

Unset the Check array flag during the cluster expansion process if FQDNs are used in the cluster

When resource management based on resource groups is active, the memory allotted to a segment host is equally shared by active primary segments. Assigning of memory to primary segments happens when the segment takes the primary role. The initial memory allotment to a primary segment is calculated on starting a cluster (with gpstart) and does not change, even when number of primary segments increases during the failover situation. This may result in a segment host utilizing more memory than the gp_resource_group_memory_limit setting permits.

For more information, refer to the VMWare documentation

6.12.1.11

 — 

Restart your cluster to recalculate memory allotment in case of increased number of primary segments on hosts (e.g. after failover)

There is an error in the COPY command description in the VMWare documentation. The FILL MISSING FIELDS option should be replaced with FILL_MISSING_FIELDS

6.12.1.11

 — 

Use the correct option provided in the first table column

ADB Monitoring may raise an issue for metrics storage if RAID is used with the following error in /var/log/diamond/diamond.log:

error [MainThread] ATTRIBUTE_NAME not found in second column of smartctl output between lines 4 and 9

6.12.1.11

 — 

Set /etc/diamond/collectors/SmartCollector.conf parameter enabled = False

diskquota may lose table statistics when being paused during ADB cluster restart. As a result, quotas may be violated.

diskquota calculates table sizes and stores that information in the diskquota.table_size table periodically with a pause equal to diskquota.naptime (2 seconds by default). If you restart the ADB cluster during this interval of time, then diskquota will lose all changes that have occurred since the last data save to the diskquota.table_size table. For example, you could make a large insert into the tables, restart the cluster, and diskquota would not know about such a large insert. This happens because the diskquota.table_size table does not store information about data insert. Until these tables are active again, diskquota will not update their size. It can be useful to reinitialize diskquota by calling the diskquota.init_table_size_table function, but on large clusters this can take a significant amount of time

6.12.1.11

 — 

No workaround. There is no way to reliably persist in-memory changes from coordinator and segments when processes get a termination signal

The diskquota.pause() function description in the VMWare documentation is not correct. The right one is: diskquota.pause() makes bgworkers skip refreshing quotas entirely to avoid issues with wasting computation resources. Table sizes are updated correctly after resume

6.12.1.11

 — 

Use the correct description provided in the first table column

The formula for specifying the maximum number of table segments (shards) from the VMWare documentation is not correct. The right one is:

diskquota.max_table_segments = <maximum_number_of_tables> * ((<number_of_segments> mod 100) + 1) * 100

6.12.1.11

 — 

Use the correct formula provided in the first table column

When trying to insert a row into a leaf partition of a partitioned table, you get the following error:

SQL Error [23514]: ERROR: trying to insert row into wrong partition

The issue reveals itself when some columns are dropped from a partitioned table and then partitions are added via ADD PARTITION or EXCHANGE PARTITION

6.12.1.11

 — 

Recreate a table or, if possible, do not drop columns from a partitioned table if you are going to add new partitions after that

In CentOS and Astra Linux, you cannot install or update ADB to 6.27.1.58 without manual installation of Java 17

6.27.1.58

6.27.1.58

Before ADB installation/update, install Java 17 on cluster hosts and fill in the JAVA_HOME parameter on the cluster configuration page

ADB to ADB Connector installation fails due to the lack of extension files on segments

6.25.1.51.b1

6.25.2.52.b1

Install the adb-fdw package for all ADB segment nodes manually, e.g. install it en masse via running gpssh from master or run installation on each node

During installation of the ADB cluster (version < 6.23.3.44) with external ADB Control (version from 6.23.3.44 to 6.25.1.49), you get inconsistency of agents with server endpoint and the failed bundle task:

AssertionError: Action Install finished execution with unexpected result - 'failed'. Expected - 'success'
  TASK [adcc_client : Add ADB connection to ADCC]

6.23.3.44

6.25.1.49

No workaround. Only upgrade to the Fixed version resolves the issue

In ADB 6 starting with 6.22.1.41, Madlib uses a new path for its libraries — /usr/local/madlib-adb6. After upgrading from previous ADB versions, Madlib stops working

6.22.1.41

N/A

  1. Backup the madlib schema if necessary.

  2. Drop the madlib schema in the cascade mode.

  3. Run the Install Madlib action for ADB service via ADCM.

  4. Restore schema objects from a backup.

The shared library diskquota-2.0.so doesn’t appear in shared_preload_libraries during upgrade with the following pg_log error on master: FATAL: could not access file ""diskquota"": no such file or directory

6.22.1.41

6.24.3.48

  1. Ensure that the dry-run of the sed utility works correctly:

    $ sed 's/diskquota/diskquota-2.0/g' /data1/*/gpseg*/postgresql.conf

  2. Run the sed utility with the -i parameter to apply changes:

    $ sed -i 's/diskquota/diskquota-2.0/g' /data1/*/gpseg*/postgresql.conf

During installation of the encryption module, you get the following error: ERROR: Encryption library was not found in /var/lib/pxf/lib/ directory

6.22.1.40

6.22.1.41

Copy the necessary library from /usr/lib/pxf/lib/:

$ cp /usr/lib/pxf/lib/encryption-1.0.0-exec.jar /var/lib/pxf/lib/encryption-1.0.0-exec.jar

If you upgrade your cluster to 6.22.1.40, remember that during this process the arenadata_toolkit.db_files_history table is being recreated with loading partitions/compression options and all data into the new table. This process may take a long time for a huge table

6.22.1.40

6.22.1.41

If you do not want to include the arenadata_toolkit.db_files_history table migration into the upgrade process, rename this table before upgrade and return the source name after:

  1. Run the query:

    ALTER TABLE arenadata_toolkit.db_files_history RENAME TO db_files_history_skip;

  2. Run the Upgrade cluster action via ADCM and wait for a successful upgrade.

  3. Run the query:

    ALTER TABLE arenadata_toolkit.db_files_history_skip RENAME TO db_files_history;

If you upgrade your cluster to 6.22.1.40 with installed ADCC and the UI LDAP authentication parameter switched on, the newer ADB server is not registered in ADCC and its version on the Information page in ADCC remains the same as for the older ADB server. It does not affect any functionality

6.22.1.40

6.22.1.41

  1. Uninstall ADCC before upgrading the ADB cluster to 6.22.1.40.

  2. Run Upgrade.

  3. Reinstall ADCC after upgrade.

[6X issue 13067] Gradual memory leak on mirror segments

6.19.1.31

6.19.3.32

Increase monitoring for memory consumption in a cluster and restart a cluster during the maintenance if possible. This issue will be patched in the next release

madlib on Power8 (ppc64le arch) breaks a gpdb build

6.17.2.26

6.17.5.26

No workaround. madlib is not included into a Power8 (ppc64le arch) build

kafka-fdw of versions 0.11-0.12 (they were ported in 6.15.0.17) does not work correctly when a batch size exceeds 40000 - <msg_count> * 40 bytes (where <msg_count> is a count of messages). In this case, the SELECT operation from the KafkaToADB external table causes SEGFAULT. The fix on the issue is expected in the next ADB release

6.16.3.24

6.17.1.25

Do not use a batch size larger than 40000 - <msg_count> * 40 bytes if you read more than one message. There are no limits for reading a single message

The gpbackup utility of versions 1.20.1 - 1.20.4 does not work correctly when foreign tables are present in a database. Backup fails with the SQLSTATE 42809 error. The fix on the issue is expected in the next ADB release

6.16.1.20

6.16.2.21

Replace the current gpbackup with version 1.20.0 which is included in ADB 6.13.0.12

Attempt to upgrade a cluster with diskquota enabled and gp_default_storage_options='appendoptimized=true' results in the following error:

ERROR: "append-only tables do not support unique indexes"

6.12.1.11

6.25.1.49

  1. Set gp_default_storage_options='appendoptimized=false' before upgrade.

  2. Upgrade a cluster.

  3. Revert to gp_default_storage_options='appendoptimized=true'.

The Upgrade action fails if appendoptimized is set to true in the gp_default_storage_options parameter for any database

6.12.1.11

6.24.3.48.b1

Revert the gp_default_storage_options value to appendoptimized=false before the upgrade and set it to true after

gpperfmon can flood with logs to $MASTER_DATA_DIR/gpseg-1/gpperfmon/q*.txt even if it is not in use

6.12.1.11

6.23.5.45.b1

  1. Switch off the gp_enable_gpperfmon GUC:

    $ gpconfig -c gp_enable_gpperfmon -v off

  2. Purge the $MASTER_DATA_DIR/gpseg-1/gpperfmon/ directory.

When you expand ADB by running the Expand cluster action, you can get errors if cluster hosts were registered with full qualified domain names (FQDN). In that case, the hostname and address columns of the gp_segment_configuration table contain different values, which may cause errors during the Expand action

6.12.1.11

6.23.3.44

Before you expand an ADB cluster:

  1. Stop ADB, for example, via the following command:

    $ gpstop -M fast

  2. Start a master segment in the utility mode:

    $ gpstart -m

  3. Set the allow_system_table_mods GUC value to true:

    $ psql adb
    SET allow_system_table_mods = 'true';

  4. Fix gp_segment_configuration so that the hostname and address column values are the same:

    UPDATE gp_segment_configuration SET hostname = address;

  5. Stop and start ADB in a standard mode:

    $ gpstop -m
$ gpstart
Found a mistake? Seleсt text and press Ctrl+Enter to report it