Allow passing airflow params as job parameter in databricks job

[SubhamSinghal]

Description

Currently if we need to pass any parameter to spark job we need to hardcode parameters in json config. But we want to pass parameters dynamically through airflow.
This feature allows to pass airflow config paramaters as job parameters in spark job.

Use case/motivation

Dynamically update job parameter in spark job

Related issues

no

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

@SubhamSinghal We are unassigning you from this issue as part of our updated assignment policy.

This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.

We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.

[galafis]

I'd like to take this on. Noticed the previous PR #39007 was closed, and the issue was recently re-labeled as good first issue.

From what I understand, the goal is to allow Airflow DAG params (e.g., from dag_run.conf) to be passed dynamically as job parameters to Databricks spark jobs, instead of having them hardcoded in the JSON config.

I'll look into the DatabricksSubmitRunOperator and DatabricksRunNowOperator to understand how json / spark_submit_params are currently handled, and figure out the best way to inject Airflow params via template_fields or an explicit mapping parameter. Will also add tests covering the new behavior.

I'll open a PR soon.

[laurenhall1526-netizen]

Stale

…

On Mon, Mar 16, 2026, 2:59 PM Gabriel Demetrios Lafis < ***@***.***> wrote: *galafis* left a comment (apache/airflow#39002) <#39002 (comment)> I'd like to take this on. Noticed the previous PR #39007 <#39007> was closed, and the issue was recently re-labeled as good first issue. From what I understand, the goal is to allow Airflow DAG params (e.g., from dag_run.conf) to be passed dynamically as job parameters to Databricks spark jobs, instead of having them hardcoded in the JSON config. I'll look into the DatabricksSubmitRunOperator and DatabricksRunNowOperator to understand how json / spark_submit_params are currently handled, and figure out the best way to inject Airflow params via template_fields or an explicit mapping parameter. Will also add tests covering the new behavior. I'll open a PR soon. — Reply to this email directly, view it on GitHub <#39002 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B7IHA42LZ6IOA3W5PYUZJG34RAJG7AVCNFSM6AAAAACWETLN4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRYGMYDKMRTGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[eladkal]

cc @moomindani not sure if this one is relevant but if is sounds like a very easy fix?

[moomindani]

Hi @eladkal — taking this one on. Opened #66613 which builds on @SubhamSinghal's earlier work in #39007 (closed as stale). Picks up @dirrao's and @Lee-W's review feedback (list-comprehension refactor + use of params.dump() to resolve Param objects to their values) and extends the auto-injection from DatabricksCreateJobsOperator to DatabricksRunNowOperator (job_parameters) and DatabricksSubmitRunOperator (per-task dict-shaped slots) per @galafis's suggestion. Verified end-to-end against a real Databricks workspace.