Add a log callback option within Task Instances

[JamesTreap]

Description

Within the Task instances context, there is a way to obtain a log URL. However, there is no way to directly access the log via the Task Instance context. Would be nice to have an API endpoint or callback that retrieves it, e.g outputlog = context.get_log.

Bonus points if additional query params could be added, such as ?lines=5 (yielding the last 5 lines) or so on, but is optional.

Use case/motivation

Use cases would be alert notifications where you may want the last ~nth lines of output or so, or DAG summaries per step. Users could call upon the log object, from there they could pipe the last nth lines or so to output (and/or any other manipulations they choose), depending on how they see fit.

Related issues

N/A - could not find any.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

You can Use: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_log - generally, it's OK to use the REST API of Airflow from the task. It's just a matter of installing Python Client and setting up the authentication. This has the advantage that it will provide you the logs in unified way - no matter what task handler you are using.

Does it solve your problem?

[JamesTreap]

It does but to a degree - it also forces me to expose API GET endpoints by setting the api.auth.backend to basic_auth or rolling my own API authentication. I'd rather have it block all API requests if possible for security purposes, instead having Airflow obtain the details from itself and sending that out via POST requests.

[potiuk]

It does but to a degree - it also forces me to expose API GET endpoints by setting the api.auth.backend to basic_auth or rolling my own API authentication. I'd rather have it block all API requests if possible for security purposes, instead having Airflow obtain the details from itself and sending that out via POST requests.

Well. I think you are describing EXACTLY what remote logging option we already have does? You have a number of logging handlers available https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/logging.html and you can roll your loggging handler https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#implementing-a-custom-file-task-handler

Converting it to a discussion in case more is needed