Airflow performance tuning

For better performance, run schedulers on CPU-optimized nodes. For high scale, run multiple schedulers in HA mode with use_row_level_locking=True and tune max_dagruns_to_create_per_loop so creation of DAG runs is spread across schedulers.

Increase scheduler CPU and memory if you parse many DAGs or use dynamic DAG generation, but first refactor DAGs to minimize top-level work.

When writing DAG files in Airflow, it is important to avoid heavy imports or blocking calls at the module’s top level (outside of task functions or operators). This is because the scheduler and Web Server need to continuously parse and load DAG files to keep track of available workflows.

Every time a DAG is parsed, all the top-level code in the file is executed. If the file contains expensive imports, database queries, or network calls at the top level, this significantly increases the parsing time and can put unnecessary strain on the scheduler and Web Server. Under high load with many DAGs, these delays compound, leading to slower scheduling loops, missed SLAs, and reduced cluster throughput.

Another risk is that blocking calls or large dependencies at the module level can introduce instability. For example, if a DAG file imports a library that takes several seconds to load or attempts to fetch data from an external service, the scheduler may hang or even fail to parse the DAG, marking it as broken. This can prevent critical workflows from being scheduled at all.

To avoid these issues, heavy operations should be moved inside task functions or operators, where they will only execute when the task runs. Keeping DAG files lightweight ensures that parsing remains fast, predictable, and resilient even in environments with thousands of workflows and frequent code updates.

An example of a problematic DAG:

An example of how to fix that DAG:

Removing top-level work in this way reduces parse latency and scheduler CPU spikes.

Dynamic task mapping enables optimizing Airflow performance in environments with many parameterized or repetitive tasks through a DAG design. It enables creating tasks at runtime rather than generating task instances at parse time.

When generating DAGs this way, you define a single mapped task and supply it with a list of inputs instead of creating all task objects manually. When the DAG runs, Airflow automatically expands that task into multiple parallel instances, then it creates the individual task instances in the metadata database. This reduces scheduler load, speeds up DAG parsing, and keeps the DAG lightweight.

It’s not recommended to push large payloads to XCom (metadata DB). For large artifacts it’s better to use object storage (for example, S3 or HDFS) and push only pointers, like paths and URIs, via XCom.

If you need structured small messages, use XCom JSON serialization or a remote XCom backend designed for larger payloads.

Set SQLAlchemy pool values so connections are available for scheduler, Web Server, workers, and external tooling.

Increasing sql_alchemy_pool_size reduces wait time for DB connections. Ensure your Postgres max_connections is sized to accommodate total Airflow connections plus other DB users.

By default, Airflow will schedule as many tasks as allowed by the Executor’s capacity and worker concurrency. While this maximizes throughput, it can easily lead to resource contention. For example, too many tasks hammering a database, API rate limits being exceeded, or I/O-heavy jobs monopolizing a worker. Pools act as a concurrency throttle: they group tasks into resource categories and limit how many of them can run at once across the entire Airflow cluster.

One of the better ways to utilize this feature is to create pools for resource-bounded operations (e.g. hadoop_jobs, db_ingest, api_calls), set explicit slot counts, and assign tasks to pools via pool argument in operators.

You can configure pools using Airflow CLI or UI.

Use priority_weight on tasks to influence slot allocation when pools are saturated. Combine with pools to ensure high-priority jobs get scheduled sooner.

Using DAG-level caps such as max_active_runs_per_dag together with pools helps to smooth out bursts of task executions and prevents resource exhaustion. By default, if a DAG is triggered frequently or has a large backlog of catchup runs, Airflow may attempt to schedule many DAG runs at once. Setting max_active_runs_per_dag limits how many runs of that DAG can execute in parallel, ensuring that execution is paced and system resources are not overwhelmed. When combined with pools, DAG-level settings control how a single workflow consumes cluster capacity, while pools coordinate concurrency across different workflows and resource types.

You can also implement a short task-rate limiter via an upstream DAG or queue consumer that writes to an intermediate queue (e.g. Kafka) for some cases.