Arenadata Orchestrator
The flexible capabilities of the scheduler, combined with its reliability, fault tolerance and scalability, make the platform indispensable for planning and orchestrating processes of any complexity.
Machine learning applications require careful data organization to manage the entire lifecycle of models – from their creation to deployment and monitoring.
ADO is a powerful data management platform built on Apache Airflow. It provides robust tools to streamline MLOps workflows by simplifying model development, deployment, and maintenance.
Data pipelines written in Python make it easy to turn custom functions into tasks and interact with any API, making it a great tool for managing your infrastructure, such as Kubernetes clusters.
ADO has built-in integration with a variety of data and analytics platforms and tools, and supports all the popular Python libraries used by data scientists, data engineers, and other professionals. This provides teams with an easy-to-configure orchestration framework that can be integrated into their preferred tools. In this way, ADO makes it easy to collaborate on designing, debugging, and maintaining data pipelines as code, accelerating the development process and making deployment and maintenance easier.
Extract-Transform-Load (ETL) and Extract-Load-Transform (ELT) data pipelines are the most common use cases for Apache Airflow due to the following features:
- Tool agnostic. Airflow can be used to orchestrate ETL/ELT pipelines for any data source or destination.
- Extensions. Airflow supports a variety of modules and also allows you to create your own operators and hooks for specific use cases.
- Dynamics. The platform allows dynamic creation of new data pipelines based on input parameters/metadata.
- Scalability. Airflow can scale to handle an infinite number of tasks and workflows given enough computing power.
Native integration with dbt Core with support for all necessary adapters for Arenadata EDP
Integration with the Git version control system for easy deployment of workflows
- Alt Linux 8.4 SP is supported
- Alt Linux 10 SP is supported
- Astra Linux SE 1.7 Orel is supported
- Astra Linux SE 1.7 Voronezh is supported
- Enhanced Python dependencies management
- Support for HA Metastore and repository proxying
- Integration with external services via shared hosts
- Upgraded Airflow and monitoring components
- Support for AltLinux 10 and Ansible 2.16
- Improved stability and user experience
- Python dependencies management via ADCM
- Support for Astra Linux "Voronezh"
- Enhanced monitoring configuration and SSL management
- Support for service maintenance mode
- Stability improvements and internal optimizations
- First release maintaining compatibility with Airflow (ADH)
- Advanced service management capabilities
- Additional features available out of the box
- Security improvements