Config variable execute_tasks_new_python_interpreter not sufficiently documented

[robinaly]

What do you see as an issue?

The documentation of execute_tasks_new_python_interpreter does not mention that the run_as_user option has to be also specified (e.g. via core.default_impersonation).

That's at least the case when run using the StandardTaskRunner

Solving the problem

Specify that run_as_user needs to be set

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

Have you considered raising a PR and contributing an improvement?

[potiuk]

You are wrong. The line pointed out simply mentions that fork option does not work when run_as_user is specified (which is obvious when you know how fork works (which is more or less opposite what you wrote)

And As @jscheffl mentioned - if you think it's worth documenting, feel free to submit a PR with better documentation. there is no neeed nor even expectation to open issue about it, you are absolutely welcome to submit a change to the docs as a PR.

[robinaly]

Thanks @potiuk for your feedback. I am happy to submit a PR, once I think I understand the code well enough:

Maybe first some background: We needed to use exec because we use the s3fs library, which doesn't work with fork(). After setting execute_tasks_new_python_interpreter=True, we still found that airflow was forking. Only after setting default_impersination=<user>, airflow used exec(). That led us to believe we needed to use impersonation and the submission of the original issue.

Looking at this line in StandardTaskRunner it does not depend on execute_tasks_new_python_interpreter. It always gives preference to fork(), and we only (improperly?) deactivated it by setting impersonation. What I need to understand is why the code didn't pass this line in LocalExecutor and ignored impersonation all together.

Could someone clarify this to me? Thanks!

[potiuk]

These two are unrelated. There are two forks done in the process:

1. the first one is LocalExecutor forking to start multiple processes that generally run in the loop and pick tasks to execute from the processs (from multiprocessing queue in case of executor). Note, that this is specific to executor chosen - with local executor, this is fork (or iin case you use impersonation starting new interpreter). In case of Celery executor, Celery uses the right method of spawning N workers (and you can look it up here for all the different options https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#pool ), in case of K8S executor, there are no long running processes and you always start a new POD (it runs LocaExecuto with single process under the hood).

2. then Runner is the way how particular tasks are run for such "worker" - once tasks is picked up, such "local executor process" either forks or starts new interpreter to make sure that each task is executed in isolation from the executor "worker" that picks tasks sequentially from the queue

So yes - by default you have two forks in the process when you run Local Executor - the first one is to start long running worker process and the second is starting separate process (either fork or spawned process) for each individual task.

So if your goal was not to do fork whatsoever, then likely yes - you have to disable both , but it's only in Local Executor case. It would be a different thing in Celery, and entirely different thing in Kubernetes etc.