Overview of Airflow 3 changes
This article describes the main changes introduced in Airflow 3 and their impact on the Airflow service in ADO 3.0.0.
ADO provides an upgrade path to Airflow 3, including Airflow package delivery, metadata DB migration, configuration and runtime migration, upgrade validation, and rollback support.
DAG compatibility and provider migration require additional changes and are described below.
Architecture changes
Airflow 3 introduces a service-oriented architecture with stricter isolation between task execution and the Airflow metadata database.
Web Server is replaced by API Server, and the DAG Processor component became mandatory. Workers communicate with API Server instead of accessing the metadata database directly.
For more information about the new architecture and database access model, see Airflow architecture.
DAG compatibility
Before upgrading, review DAGs for Airflow 3 compatibility. Airflow provides Ruff rules for detecting incompatible DAG code.
For more information on how to update your DAGs for working with Airflow 3, see the Migration helper article.
Airflow SDK imports
Airflow 3 provides airflow.sdk as the stable interface for DAGs. Update imports from internal Airflow modules to the corresponding airflow.sdk paths from the table below.
| Airflow 2.x | Airflow 3.x |
|---|---|
airflow.decorators.dag |
airflow.sdk.dag |
airflow.decorators.task |
airflow.sdk.task |
airflow.models.dag.DAG |
airflow.sdk.DAG |
airflow.models.baseoperator.BaseOperator |
airflow.sdk.BaseOperator |
airflow.models.param.Param |
airflow.sdk.Param |
airflow.sensors.base.BaseSensorOperator |
airflow.sdk.BaseSensorOperator |
airflow.hooks.base.BaseHook |
airflow.sdk.BaseHook |
airflow.utils.task_group.TaskGroup |
airflow.sdk.TaskGroup |
airflow.utils.context.Context |
airflow.sdk.Context |
airflow.datasets.Dataset |
airflow.sdk.Asset |
airflow.datasets.DatasetAlias |
airflow.sdk.AssetAlias |
airflow.models.connection.Connection |
airflow.sdk.Connection |
airflow.models.variable.Variable |
airflow.sdk.Variable |
Standard provider
Several operators, sensors, and triggers that were previously included in Airflow core have been moved to the Standard provider. This includes commonly used components such as BashOperator, PythonOperator, ExternalTaskSensor, and FileSensor.
Direct metadata database access
Task code and custom operators can no longer use Airflow database sessions to access the metadata database directly.
If custom code requires Airflow metadata, use the Airflow API instead. This approach preserves task isolation and does not require database credentials or drivers in the worker environment.
Direct database access through database hooks is only a workaround for cases that cannot be handled through the API. It is not recommended because the metadata database schema is not a public API and can change in future Airflow versions.
Removed features and context variables
Airflow 3 removes several features deprecated in Airflow 2.x. Review DAGs that use:
-
SubDAGs;
-
SequentialExecutor; -
CeleryKubernetesExecutororLocalKubernetesExecutor; -
SLAs;
-
the
--subdir(-S) CLI option; -
the
/api/v1REST API.
Use TaskGroups instead of SubDAGs, Deadline Alerts instead of SLAs, Multiple Executor Configuration instead of the removed Kubernetes executors, and /api/v2 instead of /api/v1.
The following context variables are also removed:
tomorrow_ds tomorrow_ds_nodash yesterday_ds yesterday_ds_nodash prev_ds prev_ds_nodash prev_execution_date prev_execution_date_success next_execution_date next_ds_nodash next_ds execution_date
Update DAGs that use these variables before upgrading.
DAG scheduling changes
The default value of catchup_by_default is now False.
The default value of create_cron_data_intervals is also False. As a result, DAGs that use a bare cron expression use CronTriggerTimetable instead of CronDataIntervalTimetable.
If a DAG depends on data_interval_start or data_interval_end, set the create_cron_data_intervals option to True before the upgrade.
For manually triggered DAG runs, do not assume that the data interval is derived from the supplied logical_date. Use logical_date when the DAG requires the date specified when the run was triggered.
For example:
from airflow.sdk import dag, task
@dag
def process_data():
@task
def process():
from airflow.sdk import get_current_context
context = get_current_context()
processing_date = context["logical_date"]
return f"Processing data for {processing_date}"
process()
process_data()Continue using data_interval_start and data_interval_end when the DAG requires the resolved data interval.
Authentication changes
In ADO, FAB is used as a default authentication manager, and the FAB provider comes with an ADO bundle.
Authentication routes are now prefixed with /auth. For example, an OAuth redirect URL changes from https://<your-airflow-url.com>/oauth-authorized/google to https://<your-airflow-url.com>/auth/oauth-authorized/google.
If you use OAuth, OIDC, or LDAP, verify authentication after the upgrade.
Plugins
Plugins using Flask-AppBuilder views or menu items, or Flask blueprints, require additional migration.
Affected plugins can either be migrated to the Airflow 3 plugin interfaces or use the FAB provider as a compatibility layer.
The preferred Airflow 3 interfaces include:
-
external_views -
fastapi_apps -
fastapi_root_middlewares
Upgrade in ADO 3.0.0
ADO 3.0.0 supports an in-place upgrade of existing ADO clusters from Airflow 2.11.1 to Airflow 3.2.1_arenadata1.
The upgrade procedure includes:
-
Airflow 3 package delivery;
-
metadata DB migration;
-
configuration migration;
-
runtime and service migration;
-
upgrade validation;
-
rollback support.
Before starting the upgrade, make sure that DAG compatibility, providers, custom operators, plugins, authentication, and external API integrations have been reviewed.
After the upgrade, verify that the Airflow services are running, DAGs are parsed and scheduled correctly, tasks execute successfully, authentication works, and required providers and plugins are available.