ADH Cloud releases
1.4.0
Added YuniKorn as a resource scheduler for Spark on Kubernetes. It enables management of task queues, fair resource allocation between users and teams, avoiding conflicts during concurrent execution of heavy Spark applications. The following enhancements were implemented:
|
Added Celeborn as a remote shuffle service for Spark on Kubernetes. It offloads shuffle operations from local node disks to a dedicated cluster, allowing Spark to be used in Kubernetes clusters where Kubernetes workers lack local disks for writing shuffle data. The following enhancements were implemented:
|
Added the Spark History Server service for storing and viewing Spark application execution history. It allows analyzing logs, metrics, and task statuses even after the application itself has finished, without needing access to containers or logs of individual pods. The following enhancements were implemented:
|
Added the Spark Kubernetes operator which covers a full application lifecycle: deployment, scaling, status monitoring, and automatic cleanup of completed tasks |
Added the ability to configure the Impala resource pools via Arenadata Cloud CLI |
Implemented the ability to manage the LDAP settings for Impala via Arenadata Cloud CLI |
Implemented the ability to manage the LDAP settings for Trino via Arenadata Cloud CLI |
Added support for Kerberos and LDAP authentication for YuniKorn Web |
Implemented the configuration of Kerberos authentication and SSL encryption for Spark History Server |
Added the ability to authorize Spark users via Apache Ranger |
Added the ability to configure SSL encryption and Kerberos authentication for Spark operator via Arenadata Cloud CLI |
Added the ability to configure SSL encryption and Kerberos authentication for Spark History Server via Arenadata Cloud CLI |
Added the adc list Arenadata Cloud CLI command, which allows quick access to the current infrastructure state without switching between |
Added the adc export dashboards and adc export hadoop-config Arenadata Cloud CLI commands for exporting configurations of created objects |
Added the |
Implemented a mechanism to use pre-created Kubernetes secrets stored centrally for Trino, Impala, Spark, YuniKorn, and Celeborn |
Implemented a unified system to export metrics in the Prometheus format for Trino, Impala, Spark, Spark History Server, YuniKorn, and Celeborn |
Implemented a mechanism for configuring security contexts for Trino, Impala, Spark History Server, YuniKorn, and Celeborn |
Implemented the VictoriaMetrics/Prometheus integration for collecting the Kerberos operator status metrics. It allows tracking the health and performance of the Kerberos infrastructure in a single dashboard alongside other platform components |
Added support for PodMonitor, ServiceMonitor, VMPodScrape, and VMServiceScrape for collecting metrics from components. The administrator can choose the most suitable connection method depending on the monitoring system used (Prometheus or VictoriaMetrics) without requiring modifications to the components |
Refactored the Arenadata Cloud CLI commands, eliminating duplicates and rarely used options |
Implemented a flexible configuration application policy with the ability to adjust it via Arenadata Cloud CLI for Trino, Impala, Spark History Server, YuniKorn, and Celeborn |
Upgraded Spark3 to 3.5.4.4 with the following patches: Spark3 updates
|
NGSOK-1130: Spark executor used to hang during integration with Ranger |
NGSOK-1481: Spark History Server allowed downloading objects without having the necessary permissions |
NGSOK-1876-1, NGSOK-1876-2: Spark History Server did not receive the SIGTERM signal |
NGSOK-1831-1, NGSOK-1831-2, NGSOK-1831-3: The Hadoop configuration was read incorrectly when Spark was integrated with Ranger |
NGSOK-1401: User options were saved to the Spark configuration file |
1.3.1
Upgraded Impala to 4.5.0.2 |
Upgraded Trino to 481.1 |
Upgraded Spark3 to 3.5.4.4 |
Upgraded Spark4 to 4.2.0.1 |
Keytabs and secret caches were not limited to configured namespaces |
Trino pods were terminating for a long time |
Metrics endpoint authentication and the related cluster-wide RBAC were mandatory |
Kerberos operator generated |
There was no support for container environment variables in Trino operator and CLI |
Various CVEs |
CVE-2026-27145. This issue is still open in the Trino product image |
CVE-2026-39822. This issue is still open in the Trino product image |
CVE-2026-42504. This issue is still open in the Trino product image |
Developed the Impala Kubernetes operator |
Developed the Trino Kubernetes operator |
Developed the |
Implemented support for running Spark from ADH in Kubernetes with Kerberos, SSL, and Ranger integration |
Implemented Impala installation/uninstallation in Kubernetes via Helm with:
|
Implemented Trino installation/uninstallation in Kubernetes via Helm with:
|
Implemented Impala installation/uninstallation in Kubernetes via CLI with:
|
Implemented Trino installation/uninstallation in Kubernetes via CLI with:
|