Speedup CLI by pre-collecting `get_cli_commands` of heavy module

[jason810496]

TL;DR:

The core concept is to use a pre-commit hook to precompute and serialize the results of get_cli_commands into _generated_commands.py.

After conducting some benchmarking ( check last part of issue ) and a proof of concept (POC), I found that the Airflow CLI can be sped up by 3 to 4 times for non-custom AuthManagers and Executors.

Why

While exploring airflow/cli/cli_parser.py, I noticed that get_auth_manager_cls is imported from airflow.api_fastapi.app. Since FastAPI is unrelated to the CLI, I investigated further.

The primary reasons why the current CLI is slow:

airflow/airflow/cli/cli_parser.py

Lines 39 to 87 in 4c031cb

	from airflow.api_fastapi.app import get_auth_manager_cls
	from airflow.cli.cli_config import (
	DAG_CLI_DICT,
	ActionCommand,
	DefaultHelpParser,
	GroupCommand,
	core_commands,
	)
	from airflow.cli.utils import CliConflictError
	from airflow.exceptions import AirflowException
	from airflow.executors.executor_loader import ExecutorLoader
	from airflow.utils.helpers import partition
	if TYPE_CHECKING:
	from airflow.cli.cli_config import (
	Arg,
	CLICommand,
	)
	airflow_commands = core_commands.copy() # make a copy to prevent bad interactions in tests
	log = logging.getLogger(__name__)
	for executor_name in ExecutorLoader.get_executor_names():
	try:
	executor, _ = ExecutorLoader.import_executor_cls(executor_name)
	airflow_commands.extend(executor.get_cli_commands())
	except Exception:
	log.exception("Failed to load CLI commands from executor: %s", executor_name)
	log.error(
	"Ensure all dependencies are met and try again. If using a Celery based executor install "
	"a 3.3.0+ version of the Celery provider. If using a Kubernetes executor, install a "
	"7.4.0+ version of the CNCF provider"
	)
	# Do not re-raise the exception since we want the CLI to still function for
	# other commands.
	try:
	auth_mgr = get_auth_manager_cls()
	airflow_commands.extend(auth_mgr.get_cli_commands())
	except Exception as e:
	log.warning("cannot load CLI commands from auth manager: %s", e)
	log.warning("Authentication manager is not configured and webserver will not be able to start.")
	# do not re-raise for the same reason as above
	if len(sys.argv) > 1 and sys.argv[1] == "webserver":
	log.exception(e)
	sys.exit(1)

1. It loads the entire executor module.
2. It loads the entire auth manager module.

Although these modules are dynamically loaded using import_string, there is still significant overhead when loading CeleryKubernetesExecutor, as it requires the celery and kubernetes packages.

How

Instead of fully decoupling the CLI interface from AuthManagers and Executors—by separating get_cli_commands from Base<AuthManager/Executor> interfaces, which could introduce breaking changes and require significant migration effort.

By adding a pre-commit hook to collect all CLI commands and store them precomputed and statically in airflow.cli.commands._generated_commands.py which can significantly speed up CLI response times.

Note:
For custom AuthManagers or Executors, the original approach will still be used, meaning get_cli_commands will be dynamically loaded and called as before.

Steps

Step 3 is the key focus.

1. Decouple auth_manager from heavy dependencies like FastAPI and Flask by only importing them within methods.

   Since the auth_manager implementation itself does not import FastAPI directly (only uses it for type hints), this ensures lighter module loading.

2. Introduce auth_manager_constants.py as a single source of truth for available AuthManagers, similar to how executor_constants.py works for Executors.
3. Add a pre-commit hook to generate EXECUTORS_CLI_COMMANDS and AUTH_MANAGERS_CLI_COMMANDS by precomputing CLI commands for all supported Executors and AuthManagers.

What

This change is expected to speed up the CLI by 3–4× for non-custom AuthManagers and Executors.

It could be a significant performance improvement in either Airflow 3.0 or a 2.10.x release!

Details of the Pre-Commit Script

POC branch

Example of _generated_commands.py

EXECUTORS_CLI_COMMANDS = {
    "KubernetesExecutor": [
        GroupCommand(
            name="kubernetes",
            help="Tools to help run the KubernetesExecutor",
            subcommands=[
                ActionCommand(
                    name="cleanup-pods",
                    help="Clean up Kubernetes pods (created by KubernetesExecutor/KubernetesPodOperator) in evicted/failed/succeeded/pending states",
                    func=lazy_load_command(
                        "airflow.providers.cncf.kubernetes.cli.kubernetes_command.cleanup_pods"
                    ),
                    args=[
                        Arg(
                            flags=("--namespace",),
                            help="Kubernetes Namespace. Default value is `[kubernetes] namespace` in configuration.",
                            default=conf.get("kubernetes_executor", "namespace"),
                        ),
                        Arg(
                            flags=("--min-pending-minutes",),
                            help="Pending pods created before the time interval are to be cleaned up, measured in minutes. Default value is 30(m). The minimum value is 5(m).",
                            default=30,
                            type=positive_int(allow_zero=False),
                        ),
                        Arg(
                            flags=("-v", "--verbose"),
                            help="Make logging output more verbose",
                            action="store_true",
                        ),
                    ],
                    description=None,
                    epilog=None,
                    hide=False,
                ),
                # ...

Key Considerations for Serialization

• The func: Callable field in ActionCommand

  • The function reference must match exactly how lazy_load_command is called for the func field.
  • This is handled using closures and free variables.

• The type: type | Callable field in Arg

  • The type field can be a built-in type such as int, str, or bool, or a callable like positive_int from airflow.cli.cli_config or parse from airflow.utils.timezone.
  • This is resolved by comparing the __module__ attribute and mapping it to the correct callable.

• The default field in Arg

  • The default field can either be a hardcoded value or retrieved using conf.get*(section, key).
  • If it comes from conf.get, conf.getboolean, or conf.getint, it should be serialized as a conf.get* string instead of a precomputed result.
    • This ensures that configuration values respect changes in airflow.cfg or AIRFLOW__* environment variables.
  • This is addressed using a monkey patch pattern + wrapper type:
    • The conf.get* methods are monkey-patched by adding a decorator that returns a wrapper type with a custom __conf_source__ attribute.
    • By checking for the presence of __conf_source__, determine whether to serialize it as a conf.get* string or a hardcoded constant.

Benchmark

MacOS, M2 Chip, 16G RAB, In breeze container.
Terminate breeze container and start a new session every benchmark.

breeze shell --answer n  
/files/timer.sh airflow --help

benchmark script: timer.sh

With LocalExecutor

Configuration	Run 1	Run 2	Run 3	Run 4	Run 5	Average
Original	3.0536	3.0363	3.1202	3.2155	3.1527	3.1157
Remove ExecutorLoader	3.5628	3.0111	3.0560	3.0268	3.2067	3.1727
Remove get_auth_manager_cls	1.1342	0.9723	1.1694	1.1146	0.9813	1.0744
Remove ExecutorLoader & get_auth_manager_cls	1.0761	1.0334	0.9444	0.9392	0.9404	0.9867
My POC	1.1256	0.9538	0.9181	0.9486	0.9374	0.9767

3.1157 / 0.9767 = 3.19002764, 3.19x faster !

With CeleryKubernetesExecutor

Configuration	Run 1	Run 2	Run 3	Run 4	Run 5	Average
Original	4.0599	3.9483	3.7732	3.5978	3.6112	3.7980
Remove ExecutorLoader	3.1474	2.8519	2.8384	2.8141	2.7976	2.8899
Remove get_auth_manager_cls	2.0906	1.7065	1.7159	1.7245	1.7101	1.7895
Remove ExecutorLoader & get_auth_manager_cls	1.1131	0.9080	0.9181	0.9265	0.9177	0.9567
My POC	1.0236	0.9300	0.9308	0.9221	0.9177	0.9448

3.7980 / 0.9448 = 4.01989839, 4x faster !

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I think this one shoukd be revised after final packaging/airlfow implicit initialization decisions for Airflow 3.

We have plans to change the way airlfow is packaged and how "import airflow" works - with only explicit initializaiton of specific CLI commands likely - and that might significantly change the stats.

See https://lists.apache.org/thread/k50s33fgnrxrvgj0msx2313kb3rmyh58

[jason810496]

Thanks, @potiuk!

I've gone through the discussion on restructuring the providers. We could wait until that is completed and then revisit this approach.

However, from my perspective, restructuring the providers won't necessarily speed up the CLI. The main root cause lies here:

airflow/airflow/cli/cli_parser.py

Lines 39 to 87 in 4c031cb

	from airflow.api_fastapi.app import get_auth_manager_cls
	from airflow.cli.cli_config import (
	DAG_CLI_DICT,
	ActionCommand,
	DefaultHelpParser,
	GroupCommand,
	core_commands,
	)
	from airflow.cli.utils import CliConflictError
	from airflow.exceptions import AirflowException
	from airflow.executors.executor_loader import ExecutorLoader
	from airflow.utils.helpers import partition
	if TYPE_CHECKING:
	from airflow.cli.cli_config import (
	Arg,
	CLICommand,
	)
	airflow_commands = core_commands.copy() # make a copy to prevent bad interactions in tests
	log = logging.getLogger(__name__)
	for executor_name in ExecutorLoader.get_executor_names():
	try:
	executor, _ = ExecutorLoader.import_executor_cls(executor_name)
	airflow_commands.extend(executor.get_cli_commands())
	except Exception:
	log.exception("Failed to load CLI commands from executor: %s", executor_name)
	log.error(
	"Ensure all dependencies are met and try again. If using a Celery based executor install "
	"a 3.3.0+ version of the Celery provider. If using a Kubernetes executor, install a "
	"7.4.0+ version of the CNCF provider"
	)
	# Do not re-raise the exception since we want the CLI to still function for
	# other commands.
	try:
	auth_mgr = get_auth_manager_cls()
	airflow_commands.extend(auth_mgr.get_cli_commands())
	except Exception as e:
	log.warning("cannot load CLI commands from auth manager: %s", e)
	log.warning("Authentication manager is not configured and webserver will not be able to start.")
	# do not re-raise for the same reason as above
	if len(sys.argv) > 1 and sys.argv[1] == "webserver":
	log.exception(e)
	sys.exit(1)

There is significant overhead when loading executors and auth_managers, as get_cli_commands is tightly coupled with those base classes. I believe this issue will persist even after the providers restructuring.

[potiuk]

It's not the provider's restructuring that might change it. It's the planned removal of code imported in __init__,py of airflow (and other packages) and replacing it with explicit, initialization of what is needed. That might allow us to only intialize what is needed, when needed. This will mean that likely we will be able to make CLI to load hierarchically only what is needed - i.e. instead of importing a lot of things by import airflow and implementing some lazy-loading in it, we will be able to separate CLI modules to completely independently imported modules and only load them when needed - for example initially we might only load list of commands first and all the parameters of that command only when that command is invoked.

I am not saying that your proposal is wrong, some parts of it (or all) is likely to be implemented as part of that restructuring - what I am saying, is that we should first solve the slow "import airflow" project, and then look how we can speedup all other pieces that follow from that.

[potiuk]

To say it simply: currently when we do "import airfow" we load everything. And some of things we load are "lazy loaded", some of them (even if they take a long time) are not lazy loaded.

What I think is the right solution is not to load everything when you do import airflow but to load thigs as needed by initializing specific things explicitly in the commands we run (that includes displaying cli commands and autocomplete of those).

[bugraoz93]

Thanks for preparing this! I agree with what has been said, it is a good idea to wait. Additionally, for commands, soon 🤞, most dependencies will be reduced and will be well-defined. This will help us with the package management part which will eventually reduce the dependencies between modules. I think we are close to achieving all of these.

[jason810496]

Thanks for clarifying the context of the discussion!

What I think is the right solution is not to load everything when you do import airflow but to load thigs as needed by initializing specific things explicitly in the commands we run (that includes displaying cli commands and autocomplete of those).

Yes, indeed , this solution is kind of workaround to optimize on the CLI.
If we have already decided to refactor on the import hierarchy, then lets wait until the refactor is complete to decide whether the need to adapt this solution.

[jason810496]

This issue should be resolved by #59805.
In 3.2.0, we are allow to provide "provider-level" CLI.