Logging in Airflow

Daria Barysheva

Airflow writes text logs used for analyzing errors that can occur while running DAGs. These logs are located in the logs subfolder of the Airflow home directory.

Log paths

 
The full path to each log file looks like this:

/srv/airflow/home/logs/<DAG_id>/<Task_id>/<DAG_Run_timestamp>/<log_number>.log

where:

  • <DAG_id> is the DAG identifier.

  • <Task_id> is the task identifier.

  • <DAG_Run_timestamp> is the timestamp of the DAG run.

  • <log_number> is the sequence number of the log (begins with 1).

To view Airflow logs, follow these steps:

  1. Connect to the Airflow server via SSH and run the following command:

    $ ls -la /srv/airflow/home/logs

    The output looks similar to this:

    total 0
drwxr-xr-x. 8 50000 50000 137 Dec 18 14:12 .
drwxr-xr-x. 6 50000 50000 126 Dec 18 13:58 ..
drwxrwxrwx. 3 50000 50000  24 Dec 16 07:20 adcm_check
drwxr-xr-x. 2 50000 50000  39 Dec 16 07:19 dag_processor_manager
drwxrwxrwx  5 50000 50000  71 Dec 18 14:12 ETL_test
drwxrwxrwx. 4 50000 50000  43 Dec 17 11:13 phoenix_count_books
drwxr-xr-x. 5 50000 50000  74 Dec 18 11:06 scheduler
drwxrwxrwx. 5 50000 50000  71 Dec 17 17:47 transform_people

  2. List the contents of the directory corresponding to the desired DAG:

    $ ls -la /srv/airflow/home/logs/ETL_test

    The output looks similar to this:

    total 0
drwxrwxrwx  5 50000 50000  71 Dec 18 14:12 .
drwxr-xr-x. 8 50000 50000 137 Dec 18 14:12 ..
drwxrwxrwx  9 50000 50000 279 Dec 18 15:25 extract_people
drwxrwxrwx  9 50000 50000 279 Dec 18 15:25 load_people
drwxrwxrwx  9 50000 50000 279 Dec 18 15:25 transform_people

  3. List the contents of the directory corresponding to the specified task of the DAG:

    $ ls -la /srv/airflow/home/logs/ETL_test/load_people

    The output looks similar to this:

    total 0
drwxrwxrwx 9 50000 50000 279 Dec 18 15:25 .
drwxrwxrwx 5 50000 50000  71 Dec 18 14:12 ..
drwxrwxrwx 2 50000 50000  19 Dec 18 14:12 2021-12-17T00:00:00+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:05 2021-12-18T15:05:31.504198+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:07 2021-12-18T15:06:51.360475+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:15 2021-12-18T15:15:07.501886+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:20 2021-12-18T15:20:24.033598+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:23 2021-12-18T15:23:45.565043+00:00
drwxrwxrwx 2 50000 50000  19 Dec 18 15:25 2021-12-18T15:25:09.775256+00:00

  4. View the log file corresponding to the specified DAG run timestamp:

    NOTE
    The default file name is 1.log. In case of large logs, Airflow creates more log files. 
    $ cat /srv/airflow/home/logs/ETL_test/load_people/2021-12-18T15:25:09.775256+00:00/1.log

    The output is similar to this:

    [2021-12-18 15:25:21,625] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: ETL_test.load_people 2021-12-18T15:25:09.775256+00:00 [queued]>
[2021-12-18 15:25:21,640] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: ETL_test.load_people 2021-12-18T15:25:09.775256+00:00 [queued]>
[2021-12-18 15:25:21,641] {taskinstance.py:880} INFO -
--------------------------------------------------------------------------------
[2021-12-18 15:25:21,641] {taskinstance.py:881} INFO - Starting attempt 1 of 1
[2021-12-18 15:25:21,641] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2021-12-18 15:25:21,651] {taskinstance.py:901} INFO - Executing <Task(PythonOperator): load_people> on 2021-12-18T15:25:09.775256+00:00
[2021-12-18 15:25:21,655] {standard_task_runner.py:54} INFO - Started process 362 to run task
[2021-12-18 15:25:21,675] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'ETL_test', 'load_people', '2021-12-18T15:25:09.775256+00:00', '--job_id', '100', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/ETL_test.py', '--cfg_path', '/tmp/tmp54p0gtml']
[2021-12-18 15:25:21,677] {standard_task_runner.py:78} INFO - Job 100: Subtask load_people
[2021-12-18 15:25:21,708] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: ETL_test.load_people 2021-12-18T15:25:09.775256+00:00 [running]> 8a89c1304a37
[2021-12-18 15:25:21,738] {python_operator.py:114} INFO - Done. Returned value was: None
[2021-12-18 15:25:21,744] {taskinstance.py:1070} INFO - Marking task as SUCCESS.dag_id=ETL_test, task_id=load_people, execution_date=20211218T152509, start_date=20211218T152521, end_date=20211218T152521
[2021-12-18 15:25:26,614] {local_task_job.py:102} INFO - Task exited with return code 0
Found a mistake? Seleсt text and press Ctrl+Enter to report it