Connect to Airflow via REST API
Overview
Along with the web interfaces, Airflow provides access to its REST API. This article contains a few examples of how to work with DAGs via the API using curl. The full list of endpoints is available in the Airflow REST API specification.
A URL for Airflow API requests has the following format:
Where:
-
<server-address>
— the IP address or the host name of the Airflow server; -
<port>
— the port of the Airflow server; -
<endpoint>
— the endpoint of the desired method.
Most endpoints accept JSON as input and return JSON in response. Add the following header when making requests:
Content-Type: application/json
The example request:
$ curl -L 'http://localhost:8080/api/v1/dags' -H 'Content-Type: application/json' --user "admin:admin"
This command requests information about existing DAGs in the JSON format.
Output sample:
{
"dag_id": "example_params_ui_tutorial",
"default_view": "grid",
"description": "DAG demonstrating various options for a trigger form generated by DAG params",
"file_token": ".eJw9yjEOgCAMAMC_sEs1PohURWwE29AS5fdujJecAxYDpHpmfiHTBtLt4mf1ywxKFifB_cYUdaT4YZEcw4FJBwQrFg2NgjXjSpi9dPcDbOcjlA.9dagyc8E47YZAC_P1pUK1AnuBgA",
"fileloc": "/opt/airflow/lib/python3.10/site-packages/airflow/example_dags/example_params_ui_tutorial.py",
"has_import_errors": false,
"has_task_concurrency_limits": false,
"is_active": true,
"is_paused": true,
"is_subdag": false,
"last_expired": null,
"last_parsed_time": "2024-08-29T13:43:21.591051+00:00",
"last_pickled": null,
"max_active_runs": 16,
"max_active_tasks": 16,
"next_dagrun": null,
"next_dagrun_create_after": null,
"next_dagrun_data_interval_end": null,
"next_dagrun_data_interval_start": null,
"owners": [
"airflow"
],
"pickle_id": null,
"root_dag_id": null,
"schedule_interval": null,
"scheduler_lock": null,
"tags": [
{
"name": "example_ui"
}
],
"timetable_description": "Never, external triggers only"
},
Example requests
Here are the examples of Airflow REST API requests. The examples that contain the <dag-id>
placeholder require a DAG ID. Provide the ID of the required DAG instead of the placeholder.
For authorization, examples use the default credentials: the admin
user with the password admin
.
List pools
To list the existing Airflow pools, execute a GET
request to the pools
endpoint.
For example:
$ curl -X GET 'http://localhost:8080/api/v1/pools' \
-H 'Content-Type: application/json' --user "admin:admin"
The output:
{
"pools": [
{
"description": "Default pool",
"name": "default_pool",
"occupied_slots": 0,
"open_slots": 128,
"queued_slots": 0,
"running_slots": 0,
"scheduled_slots": 7,
"slots": 128
}
],
"total_entries": 1
}
Check import issues
To check if there are any DAG import issues, send a request to the importErrors
endpoint.
For example:
$ curl --verbose 'http://localhost:8080/api/v1/importErrors' -H 'Content-Type: application/json' --user "admin:admin"
The output:
{
"import_errors": [
{
"filename": "/opt/airflow/dags/second_example.py",
"import_error_id": 7,
"stack_trace": "Traceback (most recent call last):\n File \"<frozen importlib._bootstrap_external>\", line 947, in source_to_code\n File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n File \"/opt/airflow/dags/second_example.py\", line 1\n mport pandas as pd\n ^^^^^^\nSyntaxError: invalid syntax\n",
"timestamp": "2024-09-03T13:24:12.808983+00:00"
}
],
"total_entries": 1
}
Get information about a DAG
To retrieve information about a specific DAG, send a GET
request with the DAG’s ID to the details
endpoint.
For example:
$ curl -v -X GET http://localhost:8080/api/v1/dags/<dag-id>/details \
-H 'Content-Type: application/json' \
--user "admin:admin"
The output:
{
"catchup": false,
"concurrency": 16,
"dag_id": "tutorial_dag",
"dag_run_timeout": null,
"default_view": "grid",
"description": "DAG tutorial",
"doc_md": "\n### DAG Tutorial Documentation\nThis DAG is demonstrating an Extract -> Transform -> Load pipeline\n",
"end_date": null,
"file_token": ".eJw9ysEOQDAMANB_2d1KfJAUtTVKm63C_l5cHF_yAqg5IJdN9AbhGax51nOMQw-VnTrDZcdE9U_04GFC04qpgl-uhVE-RWvhBZv_HbQ.xzpxxi3Ra3GD87EJmfl6rDYC8ME",
"fileloc": "/opt/airflow/lib/python3.10/site-packages/airflow/example_dags/tutorial_dag.py",
"is_active": true,
"is_paused": true,
"is_paused_upon_creation": null,
"is_subdag": false,
"last_parsed": "2024-09-03T13:20:29.979033+00:00",
"max_active_runs": 16,
"max_active_tasks": 16,
"orientation": "LR",
"owners": [
"airflow"
],
"params": {},
"pickle_id": null,
"render_template_as_native_obj": false,
"schedule_interval": null,
"start_date": "2021-01-01T00:00:00+00:00",
"tags": [
{
"name": "example"
}
],
"timezone": "Timezone('UTC')"
}
Trigger a DAG run
To trigger a DAG, send a POST
request that contains the DAG run parameters to the dagRuns
endpoint.
For example:
$ curl -v -X POST http://localhost:8080/api/v1/dags/<dag-id>/dagRuns \
-H 'Content-Type: application/json' \
-d '<json>' \
--user "admin:admin"
Where <json>
is the parameters of the DAG run in the JSON format.
For example:
{
"dag_run_id": "string",
"logical_date": "2019-08-24T14:15:22Z",
"execution_date": "2019-08-24T14:15:22Z",
"conf": {},
"note": "string"
}
To set a specific execution date (logical_date
) for your DAG, pass a timestamp in the YYYY-MM-DDTHH:mm:S
format.
The output:
{
"conf": {},
"dag_id": "tutorial_dag",
"dag_run_id": "string",
"data_interval_end": "2019-08-24T14:15:22+00:00",
"data_interval_start": "2019-08-24T14:15:22+00:00",
"end_date": null,
"execution_date": "2019-08-24T14:15:22+00:00",
"external_trigger": true,
"last_scheduling_decision": null,
"logical_date": "2019-08-24T14:15:22+00:00",
"note": "string",
"run_type": "manual",
"start_date": null,
"state": "queued"
}
Get tasks
To see the list of tasks for a DAG, send a GET
request to the DAG’s tasks
endpoint.
The example request:
$ curl -v -X GET http://localhost:8080/api/v1/dags/<dag-id>/tasks \
-H 'Content-Type: application/json' \
--user "admin:admin"
{
"tasks": [
{
"class_ref": {
"class_name": "PythonOperator",
"module_path": "airflow.operators.python"
},
"depends_on_past": false,
"downstream_task_ids": [
"transform_people"
],
"end_date": null,
"execution_timeout": null,
"extra_links": [],
"is_mapped": false,
"operator_name": "PythonOperator",
"owner": "airflow",
"params": {},
"pool": "default_pool",
"pool_slots": 1.0,
"priority_weight": 1.0,
"queue": "default",
"retries": 0.0,
"retry_delay": {
"__type": "TimeDelta",
"days": 0,
"microseconds": 0,
"seconds": 300
},
"retry_exponential_backoff": false,
"start_date": "2021-12-17T00:00:00+00:00",
"task_id": "extract_people",
"template_fields": [
"templates_dict",
"op_args",
"op_kwargs"
],
"trigger_rule": "all_success",
"ui_color": "#ffefeb",
"ui_fgcolor": "#000",
"wait_for_downstream": false,
"weight_rule": "downstream"
},
{
"class_ref": {
"class_name": "PythonOperator",
"module_path": "airflow.operators.python"
},
"depends_on_past": false,
"downstream_task_ids": [],
"end_date": null,
"execution_timeout": null,
"extra_links": [],
"is_mapped": false,
"operator_name": "PythonOperator",
"owner": "airflow",
"params": {},
"pool": "default_pool",
"pool_slots": 1.0,
"priority_weight": 1.0,
"queue": "default",
"retries": 0.0,
"retry_delay": {
"__type": "TimeDelta",
"days": 0,
"microseconds": 0,
"seconds": 300
},
"retry_exponential_backoff": false,
"start_date": "2021-12-17T00:00:00+00:00",
"task_id": "load_people",
"template_fields": [
"templates_dict",
"op_args",
"op_kwargs"
],
"trigger_rule": "all_success",
"ui_color": "#ffefeb",
"ui_fgcolor": "#000",
"wait_for_downstream": false,
"weight_rule": "downstream"
},
{
"class_ref": {
"class_name": "PythonOperator",
"module_path": "airflow.operators.python"
},
"depends_on_past": false,
"downstream_task_ids": [
"load_people"
],
"end_date": null,
"execution_timeout": null,
"extra_links": [],
"is_mapped": false,
"operator_name": "PythonOperator",
"owner": "airflow",
"params": {},
"pool": "default_pool",
"pool_slots": 1.0,
"priority_weight": 1.0,
"queue": "default",
"retries": 0.0,
"retry_delay": {
"__type": "TimeDelta",
"days": 0,
"microseconds": 0,
"seconds": 300
},
"retry_exponential_backoff": false,
"start_date": "2021-12-17T00:00:00+00:00",
"task_id": "transform_people",
"template_fields": [
"templates_dict",
"op_args",
"op_kwargs"
],
"trigger_rule": "all_success",
"ui_color": "#ffefeb",
"ui_fgcolor": "#000",
"wait_for_downstream": false,
"weight_rule": "downstream"
}
],
"total_entries": 3
}