Identify logging statements from Airflow versus top level code

[wolfier]

Description

I want to filter out logs generated by Airflow workers during the files parsing process before task execution. Logs are printed when the file contains top level logging statements or module imports that contain logging statements.

Use case/motivation

In an effort to reduce cost from log ingestion, I want to be able to identify these top level logging statements to filter them.

I am able to determine the logs for a common successful celery task execution.

[2023-02-07 18:48:39,101: INFO/MainProcess] Task airflow.executors.celery_executor.execute_command[5589d08f-4b4c-4a1b-9227-7c124abe0d24] received
[2023-02-07 18:48:39,141: INFO/ForkPoolWorker-9] [5589d08f-4b4c-4a1b-9227-7c124abe0d24] Executing command in Celery: ['airflow', 'tasks', 'run', 'ae_cancellation_flows', 'dbt_run.f_loan_cancellation_requests_co_logs.model.lake_modeling.f_loan_cancellation_requests_co_logs', 'scheduled__2023-02-07T16:45:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/lake_modeling/templated_dags.py']
[2023-02-07 18:48:39,252: INFO/ForkPoolWorker-9] Filling up the DagBag from /usr/local/airflow/dags/lake_modeling/templated_dags.py
[2023-02-07 18:51:16,557: INFO/ForkPoolWorker-9] Running <TaskInstance: ae_cancellation_flows.dbt_run.f_loan_cancellation_requests_co_logs.model.lake_modeling.f_loan_cancellation_requests_co_logs scheduled__2023-02-07T16:45:00+00:00 [queued]> on host 172.20.19.199
[2023-02-07 18:51:32,965: INFO/ForkPoolWorker-9] Using connection ID 'astro_s3_logging' for task execution.
[2023-02-07 18:51:33,155: INFO/ForkPoolWorker-9] Using connection ID 'astro_s3_logging' for task execution.
[2023-02-07 18:51:33,287: INFO/ForkPoolWorker-9] Task airflow.executors.celery_executor.execute_command[5589d08f-4b4c-4a1b-9227-7c124abe0d24] succeeded in 174.15662495000288s: None

Users may include arbitrary logging statements that is hard to identify there are no distinguish markers.

[2023-02-07 18:48:56,754: INFO/ForkPoolWorker-9] Model: superduperjellymonster
[2023-02-07 18:48:56,756: INFO/ForkPoolWorker-9] Model: slipperybones

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I think it should be done via convention. In your DAGs you can add specific logger "filtered.out" for example and configue it in the way you want in LOGGING_CONFIG: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#advanced-configuration - you can remove log propagation and make sure they are not sent to root logger.

Try it this way. Let us know if it solves your issue. I am converting this one to a discussion and you if you test it @wolfier.