Fix Google IdP credentials failing when project id cannot be resolved

[FrankYang0529]

Why

• Fix mypy type checking issues after upgrade to github-auth 2.46.0 #60146 made mypy pass after google-auth 2.46.0 added type hints and google.auth.default() started returning str | None. It did so by raising AirflowException whenever the resolved project id is None, in three branches of credentials_provider.py. That raise fires inside the provider, before the caller can apply a project id it was given explicitly.
• CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) #61217 reported the consequence on the ADC branch: CloudSecretManagerBackend failed even when configured with an explicit project_id. Fix CloudSecretManagerBackend regression with explicit project_id #61654 fixed that branch by returning an empty string instead of raising, and moved the check to the consumer. The backend now raises ValueError only when neither its explicit project_id nor ADC provides one.
• The IdP branch kept the raise. It fires before GoogleBaseHook can apply the connection's project extra, so every Google hook on an IdP connection fails once google-auth cannot resolve a project id. The message also asks for key_secret_project_id, which only the Secret Manager key path reads.

How

• Return an empty project id from the IdP branch. No new check is needed: the only consumer of this branch is GoogleBaseHook, which already overrides the value with the connection's project extra and, through fallback_to_default_project_id, raises a message naming the fields that work when nothing provides a project id.

Verification

• Unit test: uv run --frozen --project providers/google pytest providers/google/tests/unit/google/cloud/utils/test_credentials_provider.py
• Integration test:

1. Setup

export AIRFLOW_HOME=/tmp/airflow-idp-check
mkdir -p $AIRFLOW_HOME/dags
export AIRFLOW__CORE__DAGS_FOLDER=$AIRFLOW_HOME/dags
export AIRFLOW__CORE__LOAD_EXAMPLES=False
uv run --frozen airflow db migrate

2. Create Dag

cat > $AIRFLOW_HOME/dags/gcp_idp_check.py <<'EOF'
from airflow.sdk import dag, task


@dag(schedule=None, catchup=False)
def gcp_idp_check():
    @task
    def resolve_project():
        from airflow.providers.google.cloud.hooks.gcs import GCSHook

        hook = GCSHook(gcp_conn_id="gcp_idp")
        credentials, project_id = hook.get_credentials_and_project_id()
        print(f"credentials={type(credentials).__module__}.{type(credentials).__name__}")
        print(f"project_id={project_id!r}")
        print(f"storage_client_project={hook.get_conn().project!r}")

    resolve_project()


gcp_idp_check()
EOF

3. Add conn id

uv run --frozen airflow connections add gcp_idp --conn-type google_cloud_platform --conn-extra '{"credential_config_file": {"type": "external_account", "audience": "//iam.googleapis.com/locations/global/workforcePools/airflow-pool/providers/okta", "subject_token_type": "urn:ietf:params:oauth:token-type:id_token", "token_url": "https://sts.googleapis.com/v1/token"}, "idp_issuer_url": "https://login.example.com/oauth2/token", "client_id": "airflow-prod", "client_secret": "not-used-offline", "project": "my-explicit-project"}'

4. Run Dag

uv run --frozen airflow dags test gcp_idp_check

On main branch, this fails with airflow.sdk.exceptions.AirflowException: Project ID could not be determined from default credentials. Please provide key_secret_project_id parameter..

On this PR, this runs successfully.

credentials=google.auth.identity_pool.Credentials
project_id='my-explicit-project'
storage_client_project='my-explicit-project'

---

Was generative AI tooling used to co-author this PR?

• Yes - Claude Code

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[potiuk]

Verified small fix: the IdP branch now returns an empty project id (like the ADC branch after #61654) so GoogleBaseHook can apply the connection's project extra, with a regression test that fails on main.

I checked the claims against the code: GoogleBaseHook.get_credentials_and_project_id overrides project_id with the project extra only after get_credentials_and_project_id(...) returns, so the old in-provider raise really did pre-empt an explicitly configured project; the IdP branch's only consumer is GoogleBaseHook, and fallback_to_default_project_id still raises (naming the right fields) when nothing provides a project. The known_airflow_exceptions.txt count (16 → 15) matches the removed raise, and no existing test was changed.

Smaller observations

• test_get_credentials_using_idp_no_project_id: mock_credentials = mock.MagicMock() is unspec'd; mock.create_autospec(Credentials, instance=True) would be slightly stricter. The neighbouring ADC test uses the same pattern, so this is just a nit.
• Optional follow-up, not for this PR: the plain credential_config_file branch can also hand back None for the project id despite its str annotation; normalising it to "" like the ADC and IdP branches would make the three paths consistent.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.