ADO releases

ADO includes the DuckDB service — a high-performance analytical database system for local data analysis. DuckDB installation and configuration are centrally-managed through ADCM

The Airflow Migration Helper utility is now available and allows pre-checking and preparing user DAGs for the transition to Airflow 3

An Airflow provider for DuckDB has been developed with support for Connection, Hook, SQL Operator, and SQL Sensor. The provider allows running DuckDB SQL directly from a DAG using standard Airflow mechanisms

DuckDB integration with HDFS, Ozone, and Hive Metastore. The ADO Enterprise Edition now includes a DuckDB extension for working with HDFS, Apache Ozone, and Hive Metastore, including Apache Paimon and Apache Iceberg tables

DuckDB Integration with Vault/OpenBao. The DuckDB extension has been added to ADO Enterprise Edition to store persistent secrets in HashiCorp Vault/OpenBao instead of local storage. Secrets can be used by multiple DuckDB processes

HA for Airflow API Server. A HAProxy component has been added to ADO, providing a single endpoint, request load balancing, and a failover mechanism for Airflow API Server instances

Integrated ADO with ADM 5.0.0, based on VictoriaMetrics. VMAgent is used to collect and transmit metrics

Upgraded Apache Airflow to 3.2.1_arenadata1

Adapted the ADO providers for Apache HBase and Apache Ozone to work with Airflow 3

Improvements to the GitSync service. GitSync synchronization scenarios have been extended: different Python environments can be used for different repositories, and a single repository can be synchronized across multiple branches or directories. Monitoring for the GitSync service has also been added

The DBT adapter for Greengage DB now supports Greengage DB 7. An incremental strategy has been added for partitioned tables, allowing individual partitions to be updated

Apache Ozone provider: enhanced Kerberos authentication capabilities and configuration of the example DAG for Apache Ozone

Discontinued support for Astra Linux SE 1.7.6 (Orel/Voronezh)

Added dbt (1.11.3.1) service for data transformation workflows with support for dbt Greengage adapter included

New Git Sync service for automatic synchronization of DAGs and dbt projects from Git, including syntax checks, file filtering, dependency management, and observability

Added support for upgrading from Community Edition (CE) to Enterprise Edition (EE) within the same version

Implemented rollback after upgrade functionality for ADO versions

Added support for Astra Linux SE 1.8.4 Orel and Astra Linux SE 1.8.4 Voronezh

Added Airflow provider for Apache HBase with support for Thrift2 protocol for data operations (put, get, scan, batch) and HBase CLI for administrative tasks such as backup and restore, enabling table management, row-level operations, including batch and pooled (parallel) processing, and cluster maintenance within DAGs

Added Airflow provider for Apache Ozone based on native Ozone and Hadoop CLI tools, enabling management of volumes, buckets, objects, and data transfer operations. It is focused on native Ozone operations and does not replace or rely on S3-compatible APIs

Upgraded the Airflow version to 2.10.5_arenadata1

Upgraded the Monitoring version to 1.3

Upgraded the duckdb version to 1.4.4

Added support for integration with services requiring client access on workers (for example, ADH Spark) via shared hosts concept

Added support for connection strings to Metastore HA clusters

Added proxy server support for package repositories

Implemented support for constraints when managing Python dependencies

The Info tab now takes custom configuration settings into account

Added support for ALT Linux 10.2 SP

Added support for Ansible 2.16

Implemented automatic removal of status check artifacts during host-component mapping changes to prevent false alerts

Airflow and Redis no longer fail to start on new hosts when moving components from hosts in maintenance mode

Fixed failure of the Sync requirements action on AltLinux

Minimum required ADCM version is now 2.9.0

Prometheus 1.2_arenadata1

Implemented support for maintenance mode for services

Added the ability to manage Python dependencies via ADCM

Added support for Astra Linux SE 1.7.6 Voronezh

Manage SSL action now also manages monitoring options

Monitoring configurations can now be set granularly using scrape_config_files

Internal refactoring has been performed for optimization and safe operation

Flower no longer fails to switch to HTTPS after enabling SSL

Fixed missing airflow.flower group check in the flower_scraper.yml template

Minimum required ADCM version is now 2.6.0

The Airflow bundle and package were refactored to add the ADO version rollback feature. This will be available starting with the next release

Redis is now a cluster service for ease of configuration and deployment

A new monitoring service based on Prometheus and Grafana has been implemented

Added support for Node Exporter out of the box

Added support for statsd-exporter out of the box

Added dashboards in Grafana that display host metrics, Airflow cluster metrics (servers and workers), running DAGs, and Celery queues

DAG Processor and Triggerer are divided into separate Airflow components for granular cluster setup

Flower component configuration became more detailed and offers more options

Added the ability to natively configure a secret backend using integration with HashiCorp Vault

The link to the Flower web interface is now displayed in the info block

The service configuration now explicitly displays all settings with their descriptions. Additional parameters can be set through the custom section

Introduced a single action for changing the service topology (Add, remove, move components, etc). You can run it both for a service and for the entire cluster

Fixed errors that occurred when parsing Airflow config with special characters in secret_key

Fixed an error in the ulimits configuration

The minimum ADCM version now is 2.5.0