Skip to content

Fix env var names in airflow config list for dotted and team sections - #73136

Closed
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-config-list-env-var-names-dotted-team-sections
Closed

FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-config-list-env-var-names-dotted-team-sections

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

Why

  • airflow config list prints a # Variable: line for each option when --include-env-vars or --defaults is set.
  • For sections with a dot (like providers.odbc), Airflow reads AIRFLOW__PROVIDERS_ODBC__ALLOW_DRIVER_IN_EXTRA, but the line showed AIRFLOW__PROVIDERS.ODBC__ALLOW_DRIVER_IN_EXTRA. Setting that variable has no effect. The Setting Configuration Options page says to replace a dot in the section name with an underscore.
  • For team-specific sections such as [team_a=celery], Airflow reads AIRFLOW__TEAM_A___CELERY__WORKER_CONCURRENCY, but the line showed AIRFLOW__TEAM_A=CELERY__WORKER_CONCURRENCY. The Multi-Team page documents the AIRFLOW__{TEAM}___{SECTION}__{KEY} format.

How

  • _write_option_header now gets the name from _env_var_name, the method the parser uses to read environment variables.
  • For a team-specific section, the team name is split off the section name at the last = and passed as team_name.

Verification

  • Unit test: uv run --frozen --project shared/configuration pytest shared/configuration/tests
  • Integration test:
  1. Setup
export AIRFLOW_HOME="${TMPDIR:-/tmp}/airflow-config-list-demo/airflow-home"
mkdir -p "$AIRFLOW_HOME"
cat > "$AIRFLOW_HOME/airflow.cfg" <<'EOF'
[team_a=celery]
worker_concurrency = 8
EOF
uv run --frozen --project providers/odbc airflow config get-value providers.odbc allow_driver_in_extra | tail -n 1
uv run --frozen --project providers/odbc python -c 'from airflow.configuration import conf; print(conf.get("celery", "worker_concurrency", team_name="team_a"))'

Its output is:

False
8
  1. Check airflow config list --defaults prints and set it for the command that reads the option back.
export AIRFLOW_HOME="${TMPDIR:-/tmp}/airflow-config-list-demo/airflow-home"
ODBC_VAR=$(uv run --frozen --project providers/odbc airflow config list --defaults --show-values --section providers.odbc | sed -n 's/^# Variable: //p')
echo "$ODBC_VAR"
env "$ODBC_VAR=True" uv run --frozen --project providers/odbc airflow config get-value providers.odbc allow_driver_in_extra | tail -n 1
TEAM_VAR=$(uv run --frozen --project providers/odbc airflow config list --defaults --show-values --section team_a=celery | sed -n 's/^# Variable: //p')
echo "$TEAM_VAR"
env "$TEAM_VAR=16" uv run --frozen --project providers/odbc python -c 'from airflow.configuration import conf; print(conf.get("celery", "worker_concurrency", team_name="team_a"))'

On the main branch, the setting doesn't take effect, so reading configuration fallbacks to default value:

AIRFLOW__PROVIDERS.ODBC__ALLOW_DRIVER_IN_EXTRA
False
AIRFLOW__TEAM_A=CELERY__WORKER_CONCURRENCY
8

On this branch, it shows correct variable and the value takes effect:

AIRFLOW__PROVIDERS_ODBC__ALLOW_DRIVER_IN_EXTRA
True
AIRFLOW__TEAM_A___CELERY__WORKER_CONCURRENCY
16

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@FrankYang0529
FrankYang0529 marked this pull request as ready for review September 15, 2026 00:42
This was referenced Sep 25, 2026
@potiuk potiuk added the closed because of open PR limit Closed as a one-time step of introducing the open pull request limit label Sep 25, 2026
@potiuk

potiuk commented Sep 25, 2026

Copy link
Copy Markdown
Member

Hello @FrankYang0529 - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 30 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

closed because of open PR limit Closed as a one-time step of introducing the open pull request limit

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants