Arenadata Orchestrator
The flexible capabilities of the scheduler, combined with its reliability, fault tolerance and scalability, make the platform indispensable for planning and orchestrating processes of any complexity.
Machine learning applications require careful data organization to manage the entire lifecycle of models – from their creation to deployment and monitoring.
ADO is a powerful data management platform built on Apache Airflow. It provides robust tools to streamline MLOps workflows by simplifying model development, deployment, and maintenance.
Data pipelines written in Python make it easy to turn custom functions into tasks and interact with any API, making it a great tool for managing your infrastructure, such as Kubernetes clusters.
ADO has built-in integration with a variety of data and analytics platforms and tools, and supports all the popular Python libraries used by data scientists, data engineers, and other professionals. This provides teams with an easy-to-configure orchestration framework that can be integrated into their preferred tools. In this way, ADO makes it easy to collaborate on designing, debugging, and maintaining data pipelines as code, accelerating the development process and making deployment and maintenance easier.
Extract-Transform-Load (ETL) and Extract-Load-Transform (ELT) data pipelines are the most common use cases for Apache Airflow due to the following features:
- Tool agnostic. Airflow can be used to orchestrate ETL/ELT pipelines for any data source or destination.
- Extensions. Airflow supports a variety of modules and also allows you to create your own operators and hooks for specific use cases.
- Dynamics. The platform allows dynamic creation of new data pipelines based on input parameters/metadata.
- Scalability. Airflow can scale to handle an infinite number of tasks and workflows given enough computing power.
- Alt Linux 8.4 SP is supported
- Alt Linux 10 SP is supported
- Astra Linux SE 1.7 Orel is supported
- Astra Linux SE 1.7 Voronezh is supported
- Astra Linux SE 1.8 Orel is supported
- Astra Linux SE 1.8 Voronezh is supported
Apache Airflow is the core orchestration engine of ADO, used to programmatically define, schedule, and monitor data workflows. It enables building complex data pipelines as code (DAGs) and provides rich capabilities for dependency management, retries, scheduling, and observability. Airflow serves as the central control plane for executing data pipelines across the platform.
dbt (data build tool) is a transformation framework used to implement ELT workflows directly in the data warehouse. It allows data teams to define transformations using SQL and manage them as version-controlled code. In ADO, dbt is integrated with Airflow, enabling seamless orchestration of transformation pipelines alongside other data processing tasks.
DuckDB is an in-process analytical database optimized for fast execution of analytical queries. It is designed for local data processing and lightweight workloads, making it ideal for intermediate transformations, data exploration, and embedded analytics scenarios within ADO workflows.
