Speed up breeze start by caching Python bytecode in a docker volume

[Andrushika]

Why

Breeze sets PYTHONDONTWRITEBYTECODE=true in its containers, so every airflow process recompiles all mounted sources on import, through the bind mount. On macOS this is about 1s per CLI call and several seconds per component start. Writing .pyc next to the sources was disabled on purpose, it would leave root-owned files in the checkout.

What

Set PYTHONPYCACHEPREFIX=/root/.cache/airflow-pycache and mount an external docker volume airflow-pycache-volume there, same pattern as mypy-cache-volume. Only breeze shell and breeze start-airflow mount it and enable the cache. Other containers (breeze testing, CI) keep PYTHONDONTWRITEBYTECODE=true, so CI behaviour does not change. breeze down --cleanup-pycache removes the volume.

Benchmark

Measured inside the breeze container on macOS (Docker Desktop 28.4, 16 cores, --backend postgres --dev-mode, CI image python3.10). Time is until the api-server answers its first HTTP request, with scheduler, triggerer and dag-processor starting at the same time.

step	before	after, first run	after, second run
airflow db migrate	3.8s	4.1s	2.2s
airflow api-server -d first response	10.8s	6.7s	6.6s
airflow version	2.1s	2.4s	1.2s
start-airflow critical path in container	16.8s	10.8s	8.8s

The first run is slightly slower because the prefix also replaces the pre-built .pyc of site-packages, they are compiled once into the volume.

One known limit, same as any normal Python setup: .pyc freshness is checked by source mtime and size, so an edit within the same second that keeps the file size would reuse the old bytecode. That is why the cleanup flag exists.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Fable 5.1)

Generated-by: Claude Code (Fable 5.1) following the guidelines

[potiuk]

We need different volume per Python version

[potiuk]

And rebase is needed now.

[Andrushika]

Rebased. Let's wait for all CI passes.

[Andrushika]

Unrelated CI fails, seems it could be addressed by #72701. Let's wait for that PR merge and rerun checks.

[Andrushika]

Hi @potiuk all CI passed, I think we may merge this. Thanks!

[potiuk]

LGTM — my earlier objection about needing a separate volume per Python version doesn't hold, and I checked rather than taking it on trust. Nice change: careful, well-benchmarked, and it keeps CI behaviour untouched.

On the per-Python-version volume

You were right about PEP 3147, and it's worth recording why rather than just resolving the thread. The interpreter tag stays in the filename when PYTHONPYCACHEPREFIX is set — the prefix only relocates the directory tree, it doesn't replace the __pycache__-style naming:

$ PYTHONPYCACHEPREFIX=/tmp/cache python3 -c "import sys; sys.path.insert(0,'src'); import mod_under_test"
$ find /tmp/cache -name '*.pyc'
/tmp/cache/tmp/pycachetest/src/mod_under_test.cpython-313.pyc

So a 3.10 and a 3.12 container writing into the same volume produce foo.cpython-310.pyc and foo.cpython-312.pyc side by side and can never read each other's bytecode. One shared volume is correct. Thanks for pushing back with the reasoning instead of just splitting it.

The other things I checked

• PYTHONPYCACHEPREFIX actually reaches the container. base.yml forwards only PYTHONWARNINGS explicitly, everything else rides on env_file: _generated_docker_compose.env — so the new key arrives. Good that you added it to the generated env rather than to the explicit list.
• Empty string really does mean "unset". Both PYTHONDONTWRITEBYTECODE and PYTHONPYCACHEPREFIX go through CPython's _Py_GetEnv(), which returns NULL for a set-but-empty variable — so always emitting both keys is safe and your comment on that is accurate. That was the bit I most expected to be subtly wrong; it isn't.
• start-airflow creates the volume too. It routes through enter_shell(), so create_pycache_volume_if_needed() runs before compose sees external: true. No missing-volume failure on that path.
• The dropped PYTHONDONTWRITEBYTECODE = True constant in global_constants.py had no remaining readers — the check_docker_resources() occurrence is a separate hardcoded string on a one-off docker run with no source mount, so it's unaffected. Good cleanup.

One doc suggestion (non-blocking)

The note reads well, but two facts that reviewers will otherwise re-derive are missing — the Python-version safety above, and that the volume is shared across worktrees. Suggestion inline. On the second point: because breeze always mounts sources at /opt/airflow, two worktrees map onto the same cache paths, and since invalidation is source mtime + size, alternating between worktrees will keep invalidating each other's entries. Correct, just not free — worth a reader knowing before they wonder why the cache feels cold.

No newsfragment needed here and none added — dev tooling isn't user-facing, so that's right.

---

This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 8843de3 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.