Connect to Airflow via REST API

Overview

Along with the web interfaces, Airflow provides access to its REST API. This article contains a few examples of how to work with DAGs via the API using curl. The full list of endpoints is available in the Airflow REST API specification.

A URL for Airflow API requests has the following format:

http://<server-address>:<port>/api/v1/<endpoint>

Where:

  • <server-address> — the IP address or the host name of the Airflow server;

  • <port> — the port of the Airflow server;

  • <endpoint> — the endpoint of the desired method.

Most endpoints accept JSON as input and return JSON in response. Add the following header when making requests:

Content-Type: application/json

The example request:

$ curl -L 'http://localhost:8080/api/v1/dags' -H 'Content-Type: application/json' --user "admin:admin"

This command requests information about existing DAGs in the JSON format.

Output sample:

{
      "dag_id": "example_params_ui_tutorial",
      "default_view": "grid",
      "description": "DAG demonstrating various options for a trigger form generated by DAG params",
      "file_token": ".eJw9yjEOgCAMAMC_sEs1PohURWwE29AS5fdujJecAxYDpHpmfiHTBtLt4mf1ywxKFifB_cYUdaT4YZEcw4FJBwQrFg2NgjXjSpi9dPcDbOcjlA.9dagyc8E47YZAC_P1pUK1AnuBgA",
      "fileloc": "/opt/airflow/lib/python3.10/site-packages/airflow/example_dags/example_params_ui_tutorial.py",
      "has_import_errors": false,
      "has_task_concurrency_limits": false,
      "is_active": true,
      "is_paused": true,
      "is_subdag": false,
      "last_expired": null,
      "last_parsed_time": "2024-08-29T13:43:21.591051+00:00",
      "last_pickled": null,
      "max_active_runs": 16,
      "max_active_tasks": 16,
      "next_dagrun": null,
      "next_dagrun_create_after": null,
      "next_dagrun_data_interval_end": null,
      "next_dagrun_data_interval_start": null,
      "owners": [
        "airflow"
      ],
      "pickle_id": null,
      "root_dag_id": null,
      "schedule_interval": null,
      "scheduler_lock": null,
      "tags": [
        {
          "name": "example_ui"
        }
      ],
      "timetable_description": "Never, external triggers only"
    },

Example requests

Here are the examples of Airflow REST API requests. The examples that contain the <dag-id> placeholder require a DAG ID. Provide the ID of the required DAG instead of the placeholder.

For authorization, examples use the default credentials: the admin user with the password admin.

List pools

To list the existing Airflow pools, execute a GET request to the pools endpoint.

For example:

$ curl -X GET 'http://localhost:8080/api/v1/pools' \
     -H 'Content-Type: application/json' --user "admin:admin"

The output:

{
  "pools": [
    {
      "description": "Default pool",
      "name": "default_pool",
      "occupied_slots": 0,
      "open_slots": 128,
      "queued_slots": 0,
      "running_slots": 0,
      "scheduled_slots": 7,
      "slots": 128
    }
  ],
  "total_entries": 1
}

Check import issues

To check if there are any DAG import issues, send a request to the importErrors endpoint.

For example:

$ curl --verbose 'http://localhost:8080/api/v1/importErrors' -H 'Content-Type: application/json' --user "admin:admin"

The output:

{
  "import_errors": [
    {
      "filename": "/opt/airflow/dags/second_example.py",
      "import_error_id": 7,
      "stack_trace": "Traceback (most recent call last):\n  File \"<frozen importlib._bootstrap_external>\", line 947, in source_to_code\n  File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n  File \"/opt/airflow/dags/second_example.py\", line 1\n    mport pandas as pd\n          ^^^^^^\nSyntaxError: invalid syntax\n",
      "timestamp": "2024-09-03T13:24:12.808983+00:00"
    }
  ],
  "total_entries": 1
}

Get information about a DAG

To retrieve information about a specific DAG, send a GET request with the DAG’s ID to the details endpoint.

For example:

$ curl -v -X GET http://localhost:8080/api/v1/dags/<dag-id>/details \
  -H 'Content-Type: application/json' \
  --user "admin:admin"

The output:

{
  "catchup": false,
  "concurrency": 16,
  "dag_id": "tutorial_dag",
  "dag_run_timeout": null,
  "default_view": "grid",
  "description": "DAG tutorial",
  "doc_md": "\n### DAG Tutorial Documentation\nThis DAG is demonstrating an Extract -> Transform -> Load pipeline\n",
  "end_date": null,
  "file_token": ".eJw9ysEOQDAMANB_2d1KfJAUtTVKm63C_l5cHF_yAqg5IJdN9AbhGax51nOMQw-VnTrDZcdE9U_04GFC04qpgl-uhVE-RWvhBZv_HbQ.xzpxxi3Ra3GD87EJmfl6rDYC8ME",
  "fileloc": "/opt/airflow/lib/python3.10/site-packages/airflow/example_dags/tutorial_dag.py",
  "is_active": true,
  "is_paused": true,
  "is_paused_upon_creation": null,
  "is_subdag": false,
  "last_parsed": "2024-09-03T13:20:29.979033+00:00",
  "max_active_runs": 16,
  "max_active_tasks": 16,
  "orientation": "LR",
  "owners": [
    "airflow"
  ],
  "params": {},
  "pickle_id": null,
  "render_template_as_native_obj": false,
  "schedule_interval": null,
  "start_date": "2021-01-01T00:00:00+00:00",
  "tags": [
    {
      "name": "example"
    }
  ],
  "timezone": "Timezone('UTC')"
}

Trigger a DAG run

To trigger a DAG, send a POST request that contains the DAG run parameters to the dagRuns endpoint.

For example:

$ curl -v -X POST http://localhost:8080/api/v1/dags/<dag-id>/dagRuns \
  -H 'Content-Type: application/json' \
  -d '<json>' \
  --user "admin:admin"

Where <json> is the parameters of the DAG run in the JSON format.

For example:

{
  "dag_run_id": "string",
  "logical_date": "2019-08-24T14:15:22Z",
  "execution_date": "2019-08-24T14:15:22Z",
  "conf": {},
  "note": "string"
}

To set a specific execution date (logical_date) for your DAG, pass a timestamp in the YYYY-MM-DDTHH:mm:S format.

The output:

{
  "conf": {},
  "dag_id": "tutorial_dag",
  "dag_run_id": "string",
  "data_interval_end": "2019-08-24T14:15:22+00:00",
  "data_interval_start": "2019-08-24T14:15:22+00:00",
  "end_date": null,
  "execution_date": "2019-08-24T14:15:22+00:00",
  "external_trigger": true,
  "last_scheduling_decision": null,
  "logical_date": "2019-08-24T14:15:22+00:00",
  "note": "string",
  "run_type": "manual",
  "start_date": null,
  "state": "queued"
}

Get tasks

To see the list of tasks for a DAG, send a GET request to the DAG’s tasks endpoint.

The example request:

$ curl -v -X GET http://localhost:8080/api/v1/dags/<dag-id>/tasks  \
      -H 'Content-Type: application/json'  \
      --user "admin:admin"
The output
{
  "tasks": [
    {
      "class_ref": {
        "class_name": "PythonOperator",
        "module_path": "airflow.operators.python"
      },
      "depends_on_past": false,
      "downstream_task_ids": [
        "transform_people"
      ],
      "end_date": null,
      "execution_timeout": null,
      "extra_links": [],
      "is_mapped": false,
      "operator_name": "PythonOperator",
      "owner": "airflow",
      "params": {},
      "pool": "default_pool",
      "pool_slots": 1.0,
      "priority_weight": 1.0,
      "queue": "default",
      "retries": 0.0,
      "retry_delay": {
        "__type": "TimeDelta",
        "days": 0,
        "microseconds": 0,
        "seconds": 300
      },
      "retry_exponential_backoff": false,
      "start_date": "2021-12-17T00:00:00+00:00",
      "task_id": "extract_people",
      "template_fields": [
        "templates_dict",
        "op_args",
        "op_kwargs"
      ],
      "trigger_rule": "all_success",
      "ui_color": "#ffefeb",
      "ui_fgcolor": "#000",
      "wait_for_downstream": false,
      "weight_rule": "downstream"
    },
    {
      "class_ref": {
        "class_name": "PythonOperator",
        "module_path": "airflow.operators.python"
      },
      "depends_on_past": false,
      "downstream_task_ids": [],
      "end_date": null,
      "execution_timeout": null,
      "extra_links": [],
      "is_mapped": false,
      "operator_name": "PythonOperator",
      "owner": "airflow",
      "params": {},
      "pool": "default_pool",
      "pool_slots": 1.0,
      "priority_weight": 1.0,
      "queue": "default",
      "retries": 0.0,
      "retry_delay": {
        "__type": "TimeDelta",
        "days": 0,
        "microseconds": 0,
        "seconds": 300
      },
      "retry_exponential_backoff": false,
      "start_date": "2021-12-17T00:00:00+00:00",
      "task_id": "load_people",
      "template_fields": [
        "templates_dict",
        "op_args",
        "op_kwargs"
      ],
      "trigger_rule": "all_success",
      "ui_color": "#ffefeb",
      "ui_fgcolor": "#000",
      "wait_for_downstream": false,
      "weight_rule": "downstream"
    },
    {
      "class_ref": {
        "class_name": "PythonOperator",
        "module_path": "airflow.operators.python"
      },
      "depends_on_past": false,
      "downstream_task_ids": [
        "load_people"
      ],
      "end_date": null,
      "execution_timeout": null,
      "extra_links": [],
      "is_mapped": false,
      "operator_name": "PythonOperator",
      "owner": "airflow",
      "params": {},
      "pool": "default_pool",
      "pool_slots": 1.0,
      "priority_weight": 1.0,
      "queue": "default",
      "retries": 0.0,
      "retry_delay": {
        "__type": "TimeDelta",
        "days": 0,
        "microseconds": 0,
        "seconds": 300
      },
      "retry_exponential_backoff": false,
      "start_date": "2021-12-17T00:00:00+00:00",
      "task_id": "transform_people",
      "template_fields": [
        "templates_dict",
        "op_args",
        "op_kwargs"
      ],
      "trigger_rule": "all_success",
      "ui_color": "#ffefeb",
      "ui_fgcolor": "#000",
      "wait_for_downstream": false,
      "weight_rule": "downstream"
    }
  ],
  "total_entries": 3
}
Found a mistake? Seleсt text and press Ctrl+Enter to report it