Arenadata Orchestrator
The flexible capabilities of the scheduler, combined with its reliability, fault tolerance and scalability, make the platform indispensable for planning and orchestrating processes of any complexity.
Machine learning applications require careful data organization to manage the entire lifecycle of models – from their creation to deployment and monitoring.
ADO is a powerful data management platform built on Apache Airflow. It provides robust tools to streamline MLOps workflows by simplifying model development, deployment, and maintenance.
Data pipelines written in Python make it easy to turn custom functions into tasks and interact with any API, making it a great tool for managing your infrastructure, such as Kubernetes clusters.
ADO has built-in integration with a variety of data and analytics platforms and tools, and supports all the popular Python libraries used by data scientists, data engineers, and other professionals. This provides teams with an easy-to-configure orchestration framework that can be integrated into their preferred tools. In this way, ADO makes it easy to collaborate on designing, debugging, and maintaining data pipelines as code, accelerating the development process and making deployment and maintenance easier.
Extract-Transform-Load (ETL) and Extract-Load-Transform (ELT) data pipelines are the most common use cases for Apache Airflow due to the following features:
- Tool agnostic. Airflow can be used to orchestrate ETL/ELT pipelines for any data source or destination.
- Extensions. Airflow supports a variety of modules and also allows you to create your own operators and hooks for specific use cases.
- Dynamics. The platform allows dynamic creation of new data pipelines based on input parameters/metadata.
- Scalability. Airflow can scale to handle an infinite number of tasks and workflows given enough computing power.
- ALT Linux 10.2 SP is supported
- Astra Linux SE 1.8.6 Orel is supported
- Astra Linux SE 1.8.6 Voronezh is supported
- Ubuntu 22.04 LTS is supported
- Ubuntu 24.04 LTS is supported
- RED OS 7.3 Certified edition is supported
- RED OS 8.0 Certified edition is supported
- Ubuntu 22.04 LTS is supported
- Ubuntu 24.04 LTS is supported
Apache Airflow is the core orchestration engine of ADO, used to programmatically define, schedule, and monitor data workflows. It enables building complex data pipelines as code (DAGs) and provides rich capabilities for dependency management, retries, scheduling, and observability. Airflow serves as the central control plane for executing data pipelines across the platform.
dbt (data build tool) is a transformation framework used to implement ELT workflows directly in the data warehouse. It allows data teams to define transformations using SQL and manage them as version-controlled code. In ADO, dbt is integrated with Airflow, enabling seamless orchestration of transformation pipelines alongside other data processing tasks.
DuckDB is a lightweight analytical database delivered as a built-in service in ADO. It enables fast analytical processing, intermediate data transformations, and data exploration directly within ADO workflows. DuckDB can be used from Airflow pipelines and extended with additional capabilities for accessing external data and storage systems.
