ADB releases

Daria Barysheva

6.22.1

6.22.1.41

 
     Date: 15.02.2023

  • New features

  • Improvements

  • Bug fixes

  • Misc/Internal

  • Known issues

Upgraded pgbouncer to 1.18

Upgraded gpbackup to 1.27

Upgraded plcontainer to 2.2

Implemented a buffer parameter in the gpcheckperf utility

ADB bundle: added a checkbox to echo ADB-specific metrics to Monitoring Clients

Implemented ADBM 1.1.0

Implemented ADBCC 4.1.0

ADB bundle: removed the gppefmon dependency from ADBCC and refactored the ADBCC configs

ADB bundle: renamed Encryption Key to Alias in the PXF service

Added a precaution annotation to /home/gpadmin/arenadata_configs/postgresql.conf

Made migration of the arenadata_toolkit objects optional and unchecked by default

Components in ADB did not change their state to installed after installation

High memory utilization for queries with SQL subqueries

The service query collect_table_stats.sql periodically hung

The PXF encryption error: Encryption library was not found in /var/lib/pxf/lib/ directory

It was impossible to add the Chrony Secondary component on a new Standby host

PXF did not start with enabled option encryption: Implement check for keystore file is exist

Refactored arenadata_segments_monitor.sh to avoid crontab for a Standby node

EXPLAIN provided a wrong rows count for a WindowAgg node

The invalid byte sequence for encoding UTF8 in logs during accessing arenadata_toolkit.db_files

GPORCA produced a bogus plan for queries with CTE during handling distribution for Sequence children

The SIGSEGV error occurred when ADB read a PXF external table

Incorrect json type processing

The issue with changing an ADBCC service configuration file via ADCM

Fixed a system database name for ADBM

Fixed known issues for ADB 6.22.1.40

Placed the plcontainer Docker images (python2/python3) in our public registry

Issue

It is impossible to add external ADB clusters to the general-purpose ADBM from the ADBM UI.

 

Workaround

  1. Ensure the external ADB cluster with the installed ADBM service exists in ADCM.

  2. Connect to the Linux console. Generate the Basic Authorization hash with base64 for a user with access to Arenadata DB Backup Manager via the following command: echo -n <adbm_user>:<adbm_user_password> | base64. For example:

    $ echo -n adbm_user:Password | base64

    The result:

    YWRibV91c2VyOlBhc3N3b3Jk

  3. Send the following POST request to the general-purpose ADBM host:

    • Header parameters:

      • Accept.

      • Accept-Charset.

      • Authorization. It should have the value started with Basic  and then the base64 value from the step 1.

      • Content-Type.

      • X-ADBM_USER. It should have the adbm_user value from the step 1.

    • Body:

      {
"clusterName": "<name of external ADB cluster in ADCM>",
"clusterType": "<purpose of ADB cluster>",
"clusterVersion": "<version of ADB cluster>"
}

      The following is an example of the POST request with the Linux curl command:

      $ curl --request POST \
--url http://adbm-main:8890/backend/adbm/api/v1/clusters \
--header 'Accept: application/json' \
--header 'Accept-Charset: ISO-8859-1' \
--header 'Authorization: Basic YWRibV91c2VyOlBhc3N3b3Jk' \
--header 'Content-Type: application/json' \
--header 'X-ADBM-USER: adbm_user' \
--data '{
"clusterName": "adb-demo",
"clusterType": "ADB demo stand",
"clusterVersion": "6.22"}'

      A successful response contains a JSON with the generated identifier of an external cluster in ADBM. The cluster becomes visible in Arenadata DB Backup Manager → Clusters with Down status.

  4. In ADCM, go to the external ADB cluster and select Services → ADBM → Configuration.

  5. Set the Advanced flag and edit the ADBM agent service configuration parameter ADBM_EUREKA_CLIENT_SERV_URL_DEF_ZONE located in the bottom area. Replace its value with URL of the service registry for a general-purpose ADBM host, e.g. http://adbm-main:8761/eureka.

  6. Apply the Reconfigure & Restart action to ADBM. Upon success, the cluster status in Arenadata DB Backup Manager → Clusters is changed to Up.

6.22.1.40

 
     Date: 06.12.2022

  • New features

  • Improvements

  • Bug fixes

  • Known issues

Synced with upstream Greenplum Database 6.22.1

Upgraded gpbackup to 1.26.0.7

Upgraded PXF to 6.4.2.4

Upgraded kafka-adb to 0.16

Added encryption 1.0.0

Added adbm 1.0.0

Upgraded geospatial to 2.5.4.1

ADB bundle: implemented the pg_hba custom section in the ADB service configuration

ADB bundle: added the ADBM service and the ADBM configuration page

ADB bundle: added an option to the ADB configuration to store a Docker certificate for a trust self-signed registry

Obfuscated jdbc.password in the PXF jdbc-site.xml file

Added the ability to deserialize AVRO messages in kafka-adb

Implemented ADBCC 4.0.0

Added a new system for ADB binary backups management — Arenadata DB Backup Manager (ADBM)

Made optimizations in arenadata_toolkit.db_files_history

The error occurred during the postgis extension upgrade in ADB 6.22.0

The CREATE EXTENSION command in explicit transactions failed during assertion on executors

Fixed the cross-slice interaction detection for subplans

The error with Precheck and Check actions after ADB installation

The ADBCC client did not re-register on the ADBCC server when Reconfigure & Restart was being applied

PXF might not catch a malformed chunked encoded response

The postmaster process could exhaust all signal slots under the intensive connections pressure

PXF hung indefinitely when it executed queries using the maximumPoolSize = 1 parameter

Changed logic for ADB checkstatus
Issue Workaround

There is a potential memory leak with specific queries affected by the gpperfmon engine. If you use queries against huge tables with UDF that generate inner per-tuple queries, you may experience the significant RAM consumption, and even get OOM errors. The example is the information_schema.columns view, which can cause memory leak when being selected against a large system catalog

The possible solution is to switch off monitoring metrics (this operation requires cluster restart):

  1. Remove gpadcc from shared_preload_libraries, e.g.:

    $ gpconfig -c shared_preload_libraries -v ""

  2. Switch off the following parameters:

    $ gpconfig -c gp_enable_gpperfmon -v off
$ gpconfig -c adcc.explain_log_verbose -v off
$ gpconfig -c adcc.explain_log_analyze -v off

  3. Restart your cluster:

    $ gpstop -a -M fast
$ gpstart a

Remember that this workaround does not disable the ADBCC UI and Arenadata DB Backup Manager

During installation of the encryption module, you get the following error: ERROR: Encryption library was not found in /var/lib/pxf/lib/ directory

Copy the necessary library from /usr/lib/pxf/lib/:

$ cp /usr/lib/pxf/lib/encryption-1.0.0-exec.jar /var/lib/pxf/lib/encryption-1.0.0-exec.jar

If you upgrade your cluster to 6.22.1.40 with installed ADBCC and the ADCC-LDAP parameter switched on, the newer ADB server is not registered in ADBCC and its version on the Information page in ADBCC remains the same as for the older ADB server. It does not affect any functionality

  1. Uninstall ADBCC before upgrading the ADB cluster to 6.22.1.40.

  2. Run Upgrade.

  3. Reinstall ADBCC after upgrade.

If you upgrade your cluster to 6.22.1.40, remember that during this process the arenadata_toolkit.db_files_history table is being recreated with loading partitions/compression options and all data into the new table. This process may take a long time for a huge table

If you do not want to include the arenadata_toolkit.db_files_history table migration into the upgrade process, rename this table before upgrade and return the source name after:

  1. Run the query:

    ALTER TABLE arenadata_toolkit.db_files_history RENAME TO db_files_history_skip;

  2. Run the Upgrade cluster action via ADCM and wait for a successful upgrade.

  3. Run the query:

    ALTER TABLE arenadata_toolkit.db_files_history_skip RENAME TO db_files_history;

6.22.0

6.22.0.38

 
     Date: 20.10.2022

  • New features

  • Bug fixes

  • Misc/Internal

Synced with upstream Greenplum Database 6.22.0

Added support for AltLinux 8.4 SP. AltLinux 8.2 SP is now deprecated

Added Data Domain Boost 1.0.0 (adb_ddboost_plugin)

Upgraded gptkh to 0.14

Upgraded diskquota to 2.0.1

Upgraded PXF to 6.4.2.3

PXF: allowed setting of Oracle parallel instructions

PXF: enabled a filter pushdown status in PXF JDBC logs

Implemented ADBCC 3.7.0

PostgreSQL planner produced a bogus plan for queries against replicated tables with SIRV functions

The SIGSEGV error occurred when ADB read a PXF external table

Fixed the lost parameter delete in a cron log rotation script

Refactored Planchecker to use an external ADBCC database

The Postmaster process might exhaust all signal slots under the intensive connections pressure

gpbackup: fixed the metadata order so that now gprestore can restore functions after the tables that are used in functions as a returning type

Fixed the projection of boolean qualifiers which attributes are not in the SELECT list

PXF refactoring:

  • Fixed reading of configuration files after the PXF cluster sync.

  • Added the PXF bin to $PATH of the gpadmin user.

  • Installed PXF to Standby.

Applying the INSERT INTO command to a replicated table in combination with the SELECT FROM query against another replicated table with sequence generation failed

entrydb processed wrong slot resetting on moving a query to a resource group

A failure occurred when DynamicSeqScan had a subPlan

Failed to exchange a partition with a table having a dropped column that the table was distributed by

The SIGPIPE error occurred during fetching the external table data via the extended protocol

gptkh: fixed fetching of the actual system.tables columns in ClickHouse (according to the ClickHouse version)

Added a registry URL and arenadata-enterprise repository to the bundle configuration

6.21.1

6.21.1.36

 
     Date: 30.08.2022

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.21.1

Enabled core dump files for ADB processes

Implemented ADBCC 3.6.0

Tkhemali could not process text fields with a new line character

Read data stream is released now when a partition scan is over

The EvalPlanQual routine erroneously arose for the DELETE operation over a partitioned append-optimized relation

gp_toolkit.pg_resgroup_move_query could lose RG slots

auto_explain is not running now anywhere besides a dispatcher process

ALTER TABLE erased pg_appendonly values

Fixed a gang loss during running a transaction when gp_dist_wait_status was being called within a transaction

A database process entry initialized a whole plan tree

6.21.0

6.21.0.35

 
     Date: 21.07.2022

  • New features

  • Improvements

  • Bug fixes

Synced with upstream Greenplum Database 6.21.0

Implemented ADBCC 3.5.1

Optimized DML queries against partitioned tables to avoid further planning if a partition was pruned

Excluded the gpmon background process from the shared memory user list

Refactored the gpdb specific code for the dispatch result waiting

Implemented a fallback to PostgreSQL for an empty target list in CTE producer

Fixed the cross-slice interaction detection for subplans

Copying subplans for partitions by a planner broke the subplans order

gpinitsystem failed on a machine with a custom locale

gptkh could not parse engine settings with a dot symbol

kafka-adb: re-added kafka-client-2.5.0.jar to the kadb-fdw connector

6.20.4

6.20.4.34

 
     Date: 07.07.2022

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.20.4

ADB bundle: added the ability to specify a cluster network to distinguish a high-speed network for the interconnect purpose

ADBCC: the Container Memory Usage panel was blank under certain conditions

6.20.3

6.20.3.34

 
     Date: 19.05.2022

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.20.3

PXF: when the PXF service migrated, the rights to the directory were not being checked

PXF: the PXF_JVM_OPTS parameters could not be changed

The gpexpand_schema.tar archive was created in the directory /home/gpadmin and could overflow the space

A compatibility platform was incorrect for community bundles in AltLinux

6.20.1

6.20.1.33

 
     Date: 11.04.2022

  • New features

  • Improvements

  • Bug fixes

Synced with upstream Greenplum Database 6.20.1

PXF: added PXF 6.3.0 to the ADB bundle (with ability to upgrade from PXF 5.x)

PXF: activated a PXF cluster sync command

PXF: added the PXF bin directory to the PATH variable

PXF: added the ability to override data types mapping in external tables for PXF

Added Tkhemali connector 1.0.1 with a lot of refactoring

Added the Delete Tkhemali action

Added the Delete Kafka-to-ADB action

Added the Delete ADB-to-Kafka action

Added the -Dliquibase.releaseLocks=true parameter to the migration container for the Reconfigure & Restart action

Implemented ADBCC 3.4.0

Made editing of /etc/hosts optional

PXF: error with JDBC Hive when one of the fields was a reserved word

GPORCA failed on assert during transformation LEFT JOIN to INNER JOIN

The aggregate mode did not handle properly query finish commands from QD (Query Dispatcher)

kafka-adb: the storage option for an offset table is set explicitly now to avoid overriding

ADBCC installation failed if a cluster name consisted of more than one word

Interleaving of the append-optimized partitioned table removal and the pg_total_relation_size call for one of its partitions potentially caused SIGSERV

gptkh: fixed the tnx() function error that occurred when the data was being inserted into a partitioned table with the partitions number more than 1

Tkhemali: column names with spaces were not handled properly

GPORCA produced invalid plans for CTE (Common Table Expression) with replicated tables

GPORCA: the Gather motion merging is enabled now for singleton on a Segment

Monitor scripts were not added to crontab on the Activate Standby action

Backend server was displayed as IP while agent was displayed as FQDN on the Version page

6.19.3

6.19.3.32

 
     Date: 09.03.2022

  • New features

Synced with upstream Greenplum Database 6.19.3

Implemented ADBCC 3.3.1

6.19.1

6.19.1.31

 
     Date: 17.02.2022

  • New features

  • Improvements

  • Bug fixes

  • Misc/Internal

  • Known issues

Synced with upstream Greenplum Database 6.19.1

Hid a server name in the HTTPS group ADBCC service

Added ADB Loader tools for RHEL 8

Added the ability to deploy maintenance scripts for several databases

Enabled the backlog_lock_waits GUC

Added the Delete PXF action

Implemented ADBCC 3.3.0

Moved the Disable firewall option to the cluster level from the ADB service

Wrong column binding was used to extract values from a SharedScan node

gptkh: the distributed table test2_tmp_3054 looked at itself

gptkh: an error in parsing settings for distributed tables

Coordinator did not use quote escaping for GUC values when restored on QE

The sshd option MaxStartups had to be added before Match

Built the docker-compose RPM package 1.29.2
Issue Workaround

[6X issue 13067] Gradual memory leak on mirror segments

Increase monitoring for memory consumption in a cluster and restart a cluster during the maintenance if possible. This issue will be patched in the next release

6.18.2

6.18.2.30

 
     Date: 20.12.2021

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.18.2

[6X backport] Implemented archive_mode always

[6X backport] Zero fill pages skipped by a force WAL switch

Added Kafka ADB connector 1.0.4

Added plcontainer 2.1.5

Implemented ADBCC 3.2.5

Power: an unexpected subplans order caused subselect_gp test fails

6.18.0

6.18.0.29

 
     Date: 18.11.2021

  • New features

  • Bug fixes

Implemented ADBCC 3.2.4

When accessing from an append-optimized table, Index returned the wrong result

kafka-fdw: fixed garbage in output at least for the text format

gpbackup handled an empty tables set for an incremental data backup properly

gpbackup hung forever after it was interrupted
6.18.0.28

 
     Date: 12.11.2021

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.18.0

gpbackup: added an explicit order of tables by using pg_class.relpages

PXF: added the partitioning query support for Sybase

Implemented ADBCC 3.2.2

The SIGSEGV unexpected error on Reindex operation

madlib on ppc64le arch is now included into a build

gpbackup could dump the partitioned table data twice

The Not enough memory reserved for the statement error occurred when the PL/pgSQL function was called multiple times for CTE (Common Table Expression) SELECT statement against a table with a lot of partitions

6.17.5

6.17.5.26

 
     Date: 30.09.2021

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.17.5

ADB bundle: added the Delete service action for Monitoring Clients

ADB bundle: added the Ready to upgrade status for a bundle upgrade action

Fixed low CPU performance on Power with newly added CGLAGS build options

6.17.2

6.17.2.26

 
     Date: 07.09.2021

  • New features

  • Known issues

Synced with upstream Greenplum Database 6.17.2

Synced with PXF 5.16.3

Synced with gpbackup 1.21.0

Synced with gpbackup-s3-plugin 1.7.0

Added a build for Power8 LE platform (ppc64le arch)

Implemented ADBCC 3.1.3
Issue Workaround

madlib on Power8 (ppc64le arch) breaks a gpdb build

No workaround. madlib is not included into a Power8 (ppc64le arch) build

6.17.1

6.17.1.25

 
     Date: 17.08.2021

  • New features

  • Bug fixes

  • CVE

Synced with upstream Greenplum Database 6.17.1

Implemented the ZSTD compression support for gpbackup

adcc-extension: started to send a schema name in the Node message

ADB bundle: external database connection for ADBCC

Implemented ADBCC 3.1.0

kafka-fdw: segments failed after applying SELECT to a KafkaToADB external table with batch size more than 40000 - <msg_count> * 40 bytes (where <msg_count> is a count of messages)

PostgreSQL query optimizer built a bad plan for replicated tables with indexes

adcc-extension: the expected segment distributed transaction context was Segment Prepared

adcc-extension: there was no possibility to differ QueryStatus for slices executed on Master

Power: a bogus compiler error during pg_upgrade compilation

CVE-2020-25695

6.16.3

6.16.3.24

 
     Date: 03.08.2021

  • New features

  • Bug fixes

  • Known issues

Synced with upstream Greenplum Database 6.16.3

[6X backport] POLLHUP/POLLRDHUP are now detected while queries are running

adcc-extension: started to retrieve and send an error text

ADB bundle: defined adjustable parameters for ADBCC

Implemented ADBCC 3.0.3

Fixed ERROR: header checksum does not match

[6X backport] Restricted execution for multi-level correlated queries

adcc-extension: security_barrier views
Issue Workaround

kafka-fdw of versions 0.11-0.12 (they were ported in 6.15.0.17) does not work correctly when a batch size exceeds 40000 - <msg_count> * 40 bytes (where <msg_count> is a count of messages). In this case, the SELECT operation from the KafkaToADB external table causes SEGFAULT. The fix on the issue is expected in the next ADB release

Do not use a batch size larger than 40000 - <msg_count> * 40 bytes if you read more than one message. There are no limits for reading just one message

6.16.2

6.16.2.21

 
     Date: 24.06.2021

  • New features

  • Improvements

  • Bug fixes

Synced with upstream Greenplum Database 6.16.2

[6X Backport] Shrinking of relation segment files to zero on TRUNCATE and DELETE

ADBCC: added JVM arguments for logging

PXF: removed the tuple count check for JDBC queries INSERT

gpbackup failed when foreign tables were present in the database since it tried to acquire a lock on a foreign table

6.16.1

6.16.1.20

 
     Date: 03.06.2021

  • New features

  • Improvements

  • Bug fixes

  • Known issues

Synced with upstream Greenplum Database 6.16.1

Implemented ADBCC 2.1.1

Kafka ADB Connector: allowed users to set custom librdkafka options

[6X backport] GPORCA fails with SIGSEGV on queries to view against a table with dropped columns

[6X backport] The elog function ignores multibyte encoding when it truncates a long message

ADB bundle: set a netcat timeout for sending monitoring metrics in AltLinux

The PartitionSelector mechanism did not return tuples in case of the type mismatch

adcc-extension: unknown node with id=5 for a subplan in values

adcc-extension: a locally planned mark was lost in node status messages for queries with custom aggregate

Wrong CCNTs for plans dispatched after InitPlans were executed

Removed redundant Gpmon_Incr_Rows_Out() calls in internal sort functions

An invalid variable specified in the segment directory path check in ADCM
Issue Workaround

The gpbackup utility of versions 1.20.1 - 1.20.4 does not work correctly when foreign tables are present in a database. Backup fails with the SQLSTATE 42809 error. The fix on the issue is expected in the next ADB release

Replace the current gpbackup with version 1.20.0 which is included in ADB 6.13.0.12

6.16.0

6.16.0.19

 
     Date: 17.05.2021

  • New features

  • Improvements

Synced with upstream Greenplum Database 6.16.0

Added the gp_enable_gpperfmon=on parameter to Master and Segment servers for ADBCC

Implemented the diskquota extension update

6.15.0

6.15.0.18

 
     Date: 28.04.2021

  • New features

  • Improvements

  • Bug fixes

Added the $PXF_CONF and the $PXF_HOME environment variables to PXF hosts

Kafka ADB connector: implemented signal handlers to interrupt consuming

Ported ADB to Alt Linux 8.2

[6X backport PR 11625] pg_rewind: avoided removing files from the log directory

Invalid values in the prodataaccess attribute of the pg_proc relation after the cluster initialization/upgrade

Kafka ADB connector did not consider partitions empty when a query for their watermark offsets failed after a timeout

[6X backport] GPORCA produced an incorrect plan for the NOT IN clause
6.15.0.17

 
     Date: 13.04.2021

  • New features

  • Bug fixes

Implemented ADBCC 2.0

A broken URL in the ADB bundle service description
6.15.0.16

 
     Date: 24.03.2021

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.15.0

Implemented switchover from Master to Standby via ADCM

Kafka ADB connector: converting AVRO dates before UNIX epoch to the Greenplum Database representation

Kafka ADB connector: the Greenplum Database representation of AVRO double numbers that have more than 6 digits after a radix point

Kafka ADB connector: wrong interpretation for AVRO timestamps below zero

6.14.1

6.14.1.14

 
     Date: 03.03.2021

  • New features

  • Improvements

  • Bug fixes

Synced with upstream Greenplum Database 6.14.1

Added a link to the home ADBCC page, which is activated when a logo is clicked

Removed online loading of static resources from the ADBCC web interface

Resource group waiting queue corruption

The pg_aocsseg inconsistency after the column addition rollback

A cluster failed after the single segment host lost due to the FTS misbehavior

A multiline text in the CSV output from gplogfilter was incorrect

6.14.0

6.14.0.13.b2

 
     Date: 24.02.2021

  • Bug fixes

Fixed incorrect links in repositories of the 6.14.0.13.b1 release
6.14.0.13.b1

 
     Date: 15.02.2021

  • New features

Synced with upstream Greenplum Database 6.14.0

6.13.0

6.13.0.12

 
     Date: 21.01.2021

  • New features

  • Improvements

  • Bug fixes

  • Misc/Internal

Synced with upstream Greenplum Database 6.13.0

Supported auxiliary relations for append-optimized tables to be used by the pgstattuple extension to exactly estimate the bloat of those relations

Tkhemali-connector: avoided the intermediate conversion to a byte array

Removed the obsolete batching options in ADQM connector (since the TEXT is used now)

Reduced logging of gpstate

Greenplum Database failed after the developer GUCs of PostgreSQL being activated

GPORCA optimizer built inefficient query plans

Column-wise statistics target parameters did not apply to partitioning tables

Error with gpactivatestandby when running it too fast after the Master power off

Error in parsing decimal parts for time types

Scaling applied incorrectly to decimal values with a length less than a precision

Bumped to the latest version of additional components

6.12.1

6.12.1.11

 
     Date: 21.12.2020

  • New features

  • Bug fixes

Synced with upstream Greenplum Database 6.12.1

Added the offset function to Kafka connector

Added the rest committed function to Kafka ADB connector

Implemented the text format for Kafka ADB connector

Provided the JVM_OPTS setting for PXF

Supported AVRO logical types in Kafka ADB connector

Switched to the new way of postgis installation

Started to use rd_kafka_query_watermark_offsets to validate partition-offset pairs in Kafka ADB connector

ADBCC disk space usage troubles in case of core dumps

The missing data for column error in gprestore

The limit of distributed transactions has been reached error on Secondary Master

The WindowFunc with winref 1 assigned to WindowAgg with winref 2 (nodeWindowAgg.c:2264) error

Wrong behavior with a grouping function

An exclusive lock on any partitioned table locked the pg_partitions view

A server locked the kadb.offsets table
Found a mistake? Seleсt text and press Ctrl+Enter to report it