Release Notes ============= 6.22.0_arenadata38 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.22.0 * Added support for ``AltLinux 8.4 SP`` while ``AltLinux 8.2 SP`` was deprecated * Added Data Domain Boost ``adb_ddboost_plugin`` 1.0.0 * Added registry url and arenadata-enterprise repo to bundle config * Upgraded ``gptkh`` to 0.14 * Upgraded ``diskquota`` to 2.0.1 * Upgraded ``PXF`` to 6.4.2 * ``PXF``: Allow set ``Oracle`` parallel instructions * ``PXF``: Enable *filter pushdown* status in ``PXF-JDBC`` logs * [Docs] Updated ADCC Role/access level matrix * ADCC Release 3.7.0: * New performance metrics have been added for commands and transactions: Сpu usage total, Read bytes total, Write bytes total * Removed the possibility of stopping the execution of a separate command by canceling or terminating it * On the monitoring and history data page for commands and transactions, as well as for detailed pages, the *Workload* column has been renamed to *Resource Group* * The ability to change the resource group for command on the *Monitor Command* page has been removed * For commands and transactions, the rows are sorted by the *Status* column * Added the ability to repeatedly change the resource group for a transaction * Fixed: Data sorting has been corrected on the audit page, taking into account numbers, symbols, letters * Fixed a bug where it was impossible to enter data using the keyboard into the *Start Time* filter for the *Monitor/Transaction* pages * Fixed the error of multiple appearance of filters by cluster and database, if you add a filter by database from the filter list * Fixed: Removed the use of large and bold font for system metrics on detailed pages of commands and transactions * Fixed mapping of memory dimension from *MiB* to *Mb* * Fixed the count of the number of tuples affected in the request without updating the page * Fixed: Include *Planchecker* entities into migration Fixes ~~~~~ * Postgres planner produces bogus plan for query to replicated table with SIRV function * SIGSEGV while adb read pxf external table * Fixed lost parameter ``delete`` in cron log rotation script * Refactor *Planchecker* for use external adcc database * Postmaster process may exhaust all signal slots under intensive connections pressure * ``gpbackup``: Fix order of metadata so that gprestore restores functions after tables that used as returning type * Fix projection of boolean qualifiers whose attrs are not in SELECT list * ``PXF`` refactoring: fixed reading config files after ``pxf cluster sync``, added pxf bin to ``$PATH`` of user *gpadmin*, installed pxf to standby * Fix INSERT INTO replicated table SELECT FROM another replicated with sequence generation is broken * Fix entrydb process wrong slot resetting on moving query to resgroup * Fix failure when DynamicSeqScan has a subPlan * Failed to exchange partition with table with dropped column which was distributed by * SIGPIPE during fetching external table data by extended protocol * [gptkh] Fetch actual system.tables ``ClickHouse`` table columns for ``ClickHouse`` version 6.21.1_arenadata36 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.21.1 * Enabling core dump files for ADB processes * Update docs on appending echo to ``arenadata_segments_monitor.sh`` in case of *Zabbix* integration * Update docs on system requirement for ADCC server * ADCC Release 3.6.0: * Using a non-blocking socket to communicate with an agent * The display of the following metrics *Spill Files (skew), Row Skew in Node panel, Cpu Skew, Memory Skew, Read Skew, Write Skew* has been changed to a uniform format: `` Skew %`` * Added new column *Planner* * Added the ability to filter commands by planner * Added the ability to cancel a transaction on the detailed transaction page * Added a button to reset and delete all previously applied filters on the monitor and history command pages * Reassign queries to other resource group * ADCC UI. Add configuration for *planchecker* address * Add *planchecker* image to service ADCC (``docker-compose``) to ADB bundle * Fixed ADB unavailability error in case of ``OutOfMemoryError`` error on ADCC * Fixed actual tuples count for commands * Fixed the error of incorrect display of the command status on the detailed transaction page if the transaction was canceled Fixes ~~~~~ * Tkhemali can't process text fields with New line character * Release datum stream read when partition scan is over * ``EvalPlanQual`` routine erroneously arises for *DELETE* operation over partitioned AO relation * Fixed when ``gp_toolkit.pg_resgroup_move_query`` may lose RG slots * Avoid running ``auto_explain`` somewhere besides dispatcher process * Fixed when *ALTER TABLE* erase ``pg_appendonly`` values * Fixed gang loss during running transaction when we tried to call ``gp_dist_wait_status`` within a transaction * Fixed when entry db process initializes whole plan tree 6.21.0_arenadata35 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.21.0 * Optimize ``DML`` queries under partitioned tables to avoid further planning if partition was pruned * Exclude ``gpmon`` background process from shared memory user list * Refactor gpdb specific code for dispatch result waiting * Fallback to Postgres for empty target list in CTE producer * ADCC Release 3.5.1: * Using a non-blocking socket to communicate with an agent * Support processing of NodeStatusMetric messages * Added transaction pages: Transactions monitoring page, Transactions history page, Page with detailed information about the transaction and a list of requests made within this transaction * Added the ability to filter data by schema and relation on query history page * Added link *To transactions history* to navigate to queries from the audit page * Added a link to the parent transaction in ``Command Details`` tab * SQL statement groups ``DDL``, ``DML``, ``DCL``, and ``TCL`` are now displayed on Monitor and History pages * Added ``Spill`` and ``Spill Skew`` columns in table customization on Monitor/Command and History/Command pages * Fixed: History/Command: When generating long "Query ID" sizes, overlaps the next column with data * Fixed: Relation audit: When moving from statistics to a list of commands, the command is duplicated. * Fixed: Relation audit: Incorrect sorting according to the Scheme, with a heavy load * Fixed: Relation audit: When moving from statistics to a list of commands, the command is duplicated. * Fixed: Reset button tooltip missing * Fixed: ``NullPointerException`` when trying to get system metrics * Fixed: When sorting data, empty values are at the end of the list * Fixed: ``queuedTime`` displayed different values on the monitor page and the detailed command page * Fixed: Cache was not working properly, and the data was obtained directly from database * Fixed: After adding the *Cluster Name* column, the cluster value was not displayed in the table on Monitor/Command and History/Command pages * Fixed: the ``Duplicate key value violates unique constraint`` error * Fixed: bug with sorting on Monitor and History pages, when all the records became hidden Fixes ~~~~~ * Fix cross-slice interaction detection for subplans * Subplans ordering is broken when planner copy them for partitions * gpinitsystem fails on machine with custom locale * [gptkh] gptkh can't parse engine settings with dot symbol * [kafka-adb] Re-added kafka-client-2.5.0.jar to kadb-fdw connector 6.20.4_arenadata34 ------------------ Features and Improvements * Sync with upstream 6.20.4 * ADB Bundle: Possibility to specify cluster network to distinguish high-speed network for the interconnect purpose * Monitoring: Provide `instruction `_ on how to add database-specific metrics to Zabbix Fixes ~~~~~ * ADCC: Fixed when ADCC Container Memory Usage panel is blank under certain conditions 6.20.3_arenadata34 ------------------ Features and Improvements * Sync with upstream 6.20.3 Fixes ~~~~~ * PXF: When migrating a PXF service, the rights to the directory are not checked * PXF: Can't change PXF_JVM_OPTS parameters * Аrchive gpexpand_schema.tar is created in the directory /home/gpadmin and may overflow space * Compatibiliy platmorm is incorrect for Altlinux on community bundle 6.20.1_arenadata33 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.20.1 * PXF: Include PXF 6.3.0 into ADB Bundle with upgrade process from PXF 5.x * PXF: Activate pxf cluster sync command * PXF: Add pxf bin directory to `PATH` var * PXF: Add ability to override datatypes mapping in external table for PXF * Add Tkhemali connector ver 1.0.1 with a lot of refactoring * Make editing `/etc/hosts` optional * Add ADB Bundle Action `Delete Tkhemali` * Add ADB Bundle Action `Delete Kafka-to-ADB` * Add ADB Bundle Action `Delete ADB-to-Kafka` * Add parameter ``-Dliquibase.releaseLocks=true`` to migration containfer for reconfigure restart action * ADCC Release 3.4.0: * Process actual statistics from EXPLAIN ANALYZE for finished queries * Migrating `adcc-extension` to adbcc repository * Integrate Avg cluster query metrics * Hide `Plan&Progress` and `Text` panels if content is empty * Huge query compression * Added configuration in Registry Service to configure the addresses of backend servers * Fixed: ADCC exceeds the limit of connections to ADB * Fixed: Display relation name for Dynamic Seq Scan node * Fixed: Color-coding for plan node statuses is broken * Fixed: Login error with Cyrillic password Fixes ~~~~~ * PXF: JDBC Hive. Error when one of the fields is a reserved word * ORCA fails on assert during transformation left join to inner join * mode aggregate doesn't handle properly query finish command from QD * [kafka-adb] Set storage option for offset table explicitly to avoid overriding * ADCC installation fails if cluster name consists of more than one word * Interleaving of AO partitioned table removing and pg_total_relation_size call for one of its partition potentially causes SIGSERV * [gptkh] `tnx()` function error. Insert data into partition table and number of partitions more than 1 * Tkhemali: Column names with spaces isnt handled properly * ORCA produces invalid plan for CTE with replicated tables * ORCA: enable merging Gather motion for singleton on segment * Monitor scripts are not added to crontab on action `Activate Standby` * Backend server is displayed as IP while agent as FQDN in a Version page 6.19.3_arenadata32 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.19.3 Fixes ~~~~~ * ADCC Release 3.3.1: * Added filtering invalid metrics from gpperfmon 6.19.1_arenadata31 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.19.1 * Hidden servername in https group adcc service * Added ADB Loader tools for RHEL 8 * Added ability to deploy maintenance scripts for several databases * Enabled back log_lock_waits GUC * Built docker-compose rpm package 1.29.2 * Added Delete PXF Action * Moved "disable firewall" option on cluster level from ADB service * ADCC Release 3.3.0: * Added support data audit * Bumped log4j2 dependencies * Added support background jobs history * Added support virtual memory for process * Added support for the service load ratio * Changed sort order for metrics with NULLS LAST in Query Monitor/History * Added new system endpoints * Reworked top menu Fixes ~~~~~ * Wrong column binding is used to extract values from SharedScan node * [gptkh] Distributed table test2_tmp_3054 looks at itself * [gptkh] Error in parsing settings for Distributed tables * Coordinator doesn't use quote escaping for GUCs values when restore on QE * sshd option 'MaxStartups' must be added before 'Match' Known Issues and Workarounds ~~~~~ **Issue** Gradual memory leak on mirror segments: issue # `13067 `_ **Workaround** Increase monitoring for memory consumption in a cluster, and if possible restart cluster during the maintenance. Issue will be patched in next release. 6.18.2_arenadata30 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.18.2 * [6X backport] Implement archive_mode always * [6X backport] Zero fill pages skipped by a force WAL switch * Added kafka connector 1.0.4 * Added plcontainer 2.1.5 * ADCC Release 3.2.5: * Added supporting https in ADCC ssl * Added new parameter maxMessageKbSize to adcc-agent config file * Fixed agent stops processing query metrics due to RejectedExecutionException Fixes ~~~~~ * [power] Fixed unexpected subplans order leads to subselect_gp test fails 6.18.0_arenadata29 ------------------ Fixes ~~~~~ * Fixed when accessing from AO table Index returns the wrong result * `kafka-fdw`: fixed garbage in output at least for format text * `gpbackup`: fixed when it handles empty tables set for an incremental data backup properly * `gpbackup`: fixed when it hangs forever when interrupted * ADCC Release 3.2.4: * Added message size check * Fixed re-sending of incorrect messages * Added settings ``maxMessageKbSize`` to agent.properties * Fixed getting system metrics from clickhouse 6.18.0_arenadata28 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.18.0 * gpbackup: added explicit order of tables by ``pg_class.relpages`` during gpbackup * PXF: added support for partitioning query for Sybase * ADCC Release 3.2.2: * adcc-extension: added database and username to plan message * adcc-extension: disabling lock polling to avoid sending huge locks snapshots to agents * Added support for multi clusters * Added system metrics: CPU, RAM, IO * Added actualization for hanging queries * Added Clickhouse Database for saving draft system metrics * Added dynamically adding columns to history and monitoring forms * Added information about background processes on the system status information page Fixes ~~~~~ * Fixed unexpected SIGSEGV error on Reindex operation * `madlib` on ppc64le arch included into build * Fixed when `gpbackup` may dump partitioned table's data twice * Fixed ``Not enough memory reserved for the statement`` when calling the function multiple times for CTE SELECT statement in PL/pgSQL function on table with lot of partitions 6.17.5_arenadata26 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.17.5 * ADB Bundle: added `Delete` action for Monitoring Clients * ADB Bundle: added `Ready to upgrade` status for bundle upgrade Fixes ~~~~~ * Fixed low CPU performance on Power with new added CGLAGS build options 6.17.2_arenadata26 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.17.2 * Sync with pxf release-5.16.3 * Sync with gpbackup 1.21.0 release * Sync with gpbackup-s3-plugin 1.7.0 release * Added build for Power8 LE platform (ppc64le arch) * ADCC Release 3.1.3 Known Issues and Workarounds ~~~~~ **Issue** `madlib` on Power8 (ppc64le arch) breaks gpdb build **Workaround** No workaround. `madlib` is not included into Power8 (ppc64le arch) build. 6.17.1_arenadata25 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.17.1 * Implement ZSTD compression support for gpbackup * adcc-extension: send schema name in Node message * ADB Bundle: external database connection for ADCC Database * ADCC Release 3.1.0: * Time-based Retention Policy * Fine-grained access to multiple databases for Advanced users * Show extended information about error in ADCC * Add info on ADCC/ADB versions and docs from ADCC web interface * Add scaling to plan modal window * Add column search filters and sorting for ADCC pages Query Monitor/History * ADCC UI updates Fixes ~~~~~ * Security Patch for `cve-2020-25695 `_ (sandbox escape) * `kafka-fdw`: fixed when segments fails after SELECT operation from KafkaToADB external table with batch size more than ``40000 - * 40`` bytes * Fixed when Postgres optimizer produces bad plan for replicated table with index * adcc-extension: fixed when expected segment distributed transaction context was 'Segment Prepared' * adcc-extension: fixed when there is no possibility to differ QueryStatus for slices that executed on master * [Power] fixed bogus compiller error during pg_upgrade compilation 6.16.3_arenadata24 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.16.3 * [6X backport] Detect POLLHUP/POLLRDHUP while running queries * adcc-extension: Retrieve and send error text * ADB Bundle: Define adjustable parameters for ADCC * ADCC Release 3.0.3: * New UI Backend Server * Query Details and Monitoring: canceling and terminate * Query Details: Locks and Blocks * Support displaying information for the ADCC agent * Query plan details are truncated * Support LDAP authentication Fixes ~~~~~ * ERROR: header checksum does not match * [6X] Restrict execution for multilevel correlated queries * adcc-extension: view with security barrier Known Issues and Workarounds ~~~~~ **Issue** `kafka-fdw` of versions 0.11-0.12 (they were ported since *6.15.0_arenadata17*) is known to have a bug when batch size exceeds ``40000 - * 40`` bytes. SELECT operation from KafkaToADB external causes SEGFAULT. The fix on the issue is expected in a next ADB release. **Workaround** Do not use batch size larger than ``40000 - * 40`` bytes if you read more that one message (when reading just one message no any limits exist). 6.16.2_arenadata21 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.16.2 * PXF: remove tuple count check for jdbc INSERT * [6X Backport] Shrink to zero relation's segment files on truncate and delete * ADCC: Add jvm arguments for logging Fixes ~~~~~ * Fixed gpbackup failure when foreign tables are present in a backuping database: gpbackup tries to acquire lock on foreign table .. important:: For installing or upgrading to 6.16.2_arenadata21 use ADCM version 2021.05.26.12 or higher 6.16.1_arenadata20 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.16.1 * ADCC Release 2.1.1 * [6X Backport] ORCA fails with SIGSEGV on queries to view under table with dropped columns * [6X backport] elog function ignore multibyte encoding when truncate long messages * ADB Bundle: Set netcat timeout for send metrics to monitoring in Altlinux * Kafka-ADB Connector: Allow user to set custom librdkafka options Fixes ~~~~~ * Fixed when PartitionSelector mechanism doesn't return tuples in case of type mismatch * Fixed adcc-extension: unknown node with id==5 for subplan in values * Fixed adcc-extension: lost locally planed mark for node status messages for query with custom aggregate * Fixed wrong CCNTs for plans dispatched after InitPlans were executed * Removed redundant Gpmon_Incr_Rows_Out() calls from internal sort functions * Fixed invalid variable specified in segment directory path check in ADCM Known Issues and Workarounds ~~~~~ **Issue** `gpbackup` utility of versions 1.20.1-1.20.4 is known to have a bug when foreign tables are present in a backuping database. Backup fails with error SQLSTATE 42809. The fix on the issue is expected in a next ADB release. **Workaround** Replace current `gpbackup` with version 1.20.0 which included in ADB 6.13.0_arenadata12. 6.16.0_arenadata19 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.16.0 * Implement diskquota extension update * Add gp_enable_gpperfmon=on parameter on master and segment server for ADCC 6.15.0_arenadata18 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * [6X backport gpdb PR `11625 `_] pg_rewind: avoid removing log files * Add $PXF_CONF and $PXF_HOME env variable on PXF hosts * Kafka-ADB Connector: Implement signal handlers to interrupt consuming * Port ADB to Alt Linux 8.2 Fixes ~~~~~ * Fixed invalid values in `prodataaccess` attribute of `pg_proc` relation after cluster initialization/upgrade * Fixed when Kafka-ADB Connector do not consider partitions empty when a query for their watermark offsets fails on timeout * [6X Backport] ORCA produces incorrect plan for NOT IN clause 6.15.0_arenadata17 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Implement Arenadata Command Center 2.0 Fixes ~~~~~ * Broken URL in ADB bundle service description 6.15.0_arenadata16 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.15.0 * Implement switchover from master to standby via ADCM Fixes ~~~~~ * Kafka-ADB-connector: issue with converting AVRO dates before UNIX epoch to GPDB representation * Kafka-ADB-connector: issue with GPDB representation of AVRO double number that has more than 6 digits after radix point * Kafka-ADB-connector: issue with wrong interpretation for AVRO timestamps that below zero 6.14.1_arenadata14 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.14.1 * Add link to main page in ADCC logo Fixes ~~~~~ * Fix resource group waiting queue corruption * pg_aocsseg inconsistency after column addition rollback * Remove online loading of static resources from the ADCC web interface * Cluster failed after single segment-host lost due to FTS misbehavior * Multiline text in CSV output from gplogfilter is incorrect 6.14.0_arenadata13_b2 --------------------- Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Fixup incorrect links in repos of b1 release 6.14.0_arenadata13 ------------------ Sync only release Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Sync with upstream 6.14.1 6.13.0_arenadata12 ------------------ Features and Improvements ~~~~~~~~~~~~~~~~~~~~~~~~~ * Tkhemali-connector: Avoid intermediate conversion to byte array * Remove obsolete batching options in ADQM Connector (sinse we are using TEXT now) * Sync with upstream 6.13.0 codebase * Bump to latest version of additional components * Reduce noise in logs about gpstate * Support auxilliary relations that belongs to AO table to be income to pgstattupe extension to exactly estimate bloat of these relations Fixes ~~~~~ * GPDB fails after activation developer gucs of PostgreSQL * Orca optimizer builds inefficient query plan * Column-wise statistics target parameter doesn't apply to partitioning tables * Error gpactivatestandby when run it too fast after power off master * Mistake in the parsing of decimal part for time logical types * Scale is applied incorrectly for decimal values what has a length less than precision 6.12.1_arenadata11 ------------------ New Features ~~~~~~~~~~~~ * Sync code base with upstream GPDB 6.12.1 * Add ofset function to Kafka (ADS) connector * Add rest commited function to Kafka (ADS) connector * Implement 'text' format for kafka connector * Provide JVM_OPTS setting for PXF * Support AVRO logical types in Kafka-ADB connector * Switch to new way of postgis installation * Kafka-ADB: Use rd_kafka_query_watermark_offsets to validate partition-offset pairs Fixes ~~~~~ * ADCC disk space usage troubles in case of coredumps * gprestore - ERROR: missing data for column * "The limit of distributed transactions has been reached" error on secondary * WindowFunc with winref 1 assigned to WindowAgg with winref 2 (nodeWindowAgg.c:2264) * Wrong behavior with grouping function * Exclusive lock on any partitioned table locks the pg_partitions view. * Server locks kadb.offsets table