var.json.get not working correctly with nested JSON objects

[sirVir]

Apache Airflow version

2.7.1

What happened

According to Airflow Documentation on Airflow Variables in Templates there are two ways of accessing the JSON variables in templates:

• using the direct {{ var.json.my_dict_var.key1 }}
• using a getter with a default fallback value {{{{ var.json.get('my.dict.var', {'key1': 'val1'}) }}

However, only the first approach works for nested variables, as demonstrated in the example.

Or is it me not understanding this documentation? Alternatively it could get updated to make it clearer.

What you think should happen instead

No response

How to reproduce

The following test demonstrates the issue:

from datetime import datetime, timezone

import pytest
from airflow import DAG
from airflow.models.baseoperator import BaseOperator
from airflow.models.dag import DAG
from airflow.utils import db
from airflow.utils.state import DagRunState
from airflow.utils.types import DagRunType
from pytest import MonkeyPatch
from airflow.models import Variable

class TemplatedBaseOperator(BaseOperator):
    template_fields = BaseOperator.template_fields + (
        "templated_direct", "templated_getter")
    
    def __init__(
        self,
        *args,
        **kwargs,
    ):
        self.templated_direct = "{{ var.json.somekey.somecontent }}"
        self.templated_getter = "{{ var.json.get('somekey.somecontent', false) }}"

        super().__init__(
                *args,
                **kwargs,
            )

@pytest.fixture()
def reset_db():
    db.resetdb()
    yield

@pytest.fixture
def dag() -> DAG:
    with DAG(
        dag_id="templating_dag",
        schedule="@daily",
        start_date=datetime(2023, 1, 1, tzinfo=timezone.utc),
        render_template_as_native_obj=True,
    ) as dag:
        TemplatedBaseOperator(
            task_id="templating_task"
        )
    return dag


def test_templating(dag: DAG, reset_db: None, monkeypatch: MonkeyPatch):
    """Test if the templated values get intialized from environment variables when rendered"""

    # setting env variables
    monkeypatch.setenv(
        "AIRFLOW_VAR_SOMEKEY",
        '{"somecontent": true}',
    )

    dagrun = dag.create_dagrun(
        state=DagRunState.RUNNING,
        execution_date=datetime(2023, 1, 1, tzinfo=timezone.utc),
        data_interval=(datetime(2023, 1, 1, tzinfo=timezone.utc), datetime(2023, 1, 7, tzinfo=timezone.utc)),
        start_date=datetime(2023, 1, 7, tzinfo=timezone.utc),
        run_type=DagRunType.MANUAL,
    )

    ti = dagrun.get_task_instance(task_id="templating_task")
    ti.task = dag.get_task(task_id="templating_task")

    rendered_template = ti.render_templates()

    assert {'somecontent': True} == Variable.get("somekey", deserialize_json=True)
    assert getattr(rendered_template, "templated_direct") == True
    # the following test is failing, getting default "False" instead of Variable 'True'
    assert getattr(rendered_template, "templated_getter") == True

Operating System

Ubuntu 22.04.3 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

In fact, the initial use case of this method was fetching the variable using the provided key as it is, and it's very helpful if your variable key contains a dot:

The other method will not work, because it will try to get the nested variable b from the variable a.

I created #34411 to add a small comment about this need.

[sirVir]

@hussein-awala I appreciate the PR with change of documentation, but I still see it as an inconsistency that
{{ var.json.somekey.somecontent } works
and
{{ var.json.get('somekey.somecontent', false)
Doesn’t. The difference between those is not documented, apart from mention that one can assign the default value with the json.get.

[hussein-awala]

They are two different paragraphs:

The var template variable allows you to access Airflow Variables. You can access them as either plain-text or JSON. If you use JSON, you are also able to walk nested structures, such as dictionaries like: {{ var.json.my_dict_var.key1 }}.

Here we explain that you can walk nested structures using var.json.level1.level2.....

In the second paragraph, we explain the you have another option to fetch the variable by key.

It is also possible to fetch a variable by string if needed with {{ var.value.get('my.var', 'fallback') }} or {{ var.json.get('my.dict.var', {'key1': 'val1'}) }}. Defaults can be supplied in case the variable does not exist.

So here my.var and my.dict.var are the keys names, and not a nested structures.

In my PR I explain that the second one it helpful if your key contains a dot, because you cannot access it by the first method.

{{ var.json.get('somekey.somecontent', false) doesn't work because you don't have a variable with the key somekey.somecontent. I will try to update the doc and make it clearer.

[Taragolis]

I appreciate the PR with change of documentation, but I still see it as an inconsistency that
{{ var.json.somekey.somecontent } works
and
{{ var.json.get('somekey.somecontent', false)
Doesn’t.

In this cases var.json.my_var is references to object (dictionary) and other part it is the way how you could access __getitem__ in Jinja Variable.

Some native Jinja templates sample

import jinja2
environment = jinja2.Environment()

var = {"a": {"b": {"c": 1 }}}

template = environment.from_string("Sample a.b.c = {{ awesome_var.a.b.c }}")
print(template.render(awesome_var=var))
# Sample a.b.c = 1

template = environment.from_string("Sample ['a']['b']['c'] = {{ awesome_var['a']['b']['c'] }}")
print(template.render(awesome_var=var))
# Sample ['a']['b']['c'] = 1

template = environment.from_string("Sample mixin = {{ awesome_var.get('a')['b'].c }}")
print(template.render(awesome_var=var))
# Sample mixin = 1

template = environment.from_string("Sample by key = {{ awesome_var.get('a.b.c', 'Foo') }}")
print(template.render(awesome_var=var))
# Sample by key = Foo

template = environment.from_string("Sample by pseudo json-path = {{ awesome_var.get('a.b.c') }}")
print(template.render(awesome_var=var))
# Sample by pseudo json-path = None

AFAIK var.json.name it is specific class with own get method, which close to regular nested dict and it is work by the same way as dictionary works in Jinja Templates, so this behaviour pretty consistent with Jinja

[sirVir]

@Taragolis thank you for the clarification. So the getter with default value works only on the top level and it looks like it is currently not possible to use default/fallback variable values for missing parts of JSON content in Airflow?

I [can do with if the variable is missing completely by providing the dictionary as a fallback (as demonstrated in the unit tests):
"{{ var.json.get('somekey', { 'somecontent': false } ).somecontent }}"

But if any somekey is available, even if it doesn't contain somecontent I will get an error.

[sirVir]

For the future reference. The way to get the "getter" functionality with a nested json variables seems to be to use the full JINJA conditional syntax, checking for each level with a fallback value.
"{% if var.json.get('somekey', { 'somecontent': false } ).somecontent is defined %}{{var.json.get('somekey', { 'somecontent': false } ).somecontent}}{% else %}{{false}}{% endif %}"

This works, but is not really straightforward.

[Taragolis]

This works, but is not really straightforward.

That how Jinja works with StrictUndefined, I've found my old presentation for internal trainings, it could help or maybe not. At least there should be examples how to deal with undefined values

[sirVir]

I think that the confusing is that the missing Variables (first level key) results in KeyError, which is an Airflow exception which stops the DAG from rendering and the deeper levels result in jinja2.exceptions.UndefinedError, but the DAG still renders the template (only with Undefined as value).