Meaning of '.' in expr and special $input and $context variables

[fjtirado]

I will describe my current understanding
There are currently two special variables

• $input is a read only variable which contains the workflow input (as provided when the workflow is started)
• $context is a global variable for the whole workflow instance (what happens with subflows?) instance that is updated in the output section of task (so it is initially empty) and can be accessed by any expression

I understood that (please note thats is just my understanding)

• the output of the last task is the workflow instance output.
• When using . in input: keyword expression, . refers to the previously executed task result. In the case of the first task, . is equivalent to $input
• When using . in with: payload: expression, . refers to the result of the input: keyword (if no input: then the result of the previous task)
• When using . in output: keyword expression, . refers to the result of the current task (this means that, at this stage, the only way to acccess the previous task result, in case you want to merge current and prev result, is to have stored the result of the previous task into the context)

@cdavernas @JBBianchi @matthias-pichler-warrify @ricardozanini if we agree on that meaning, which will be the right place to document that?

[fjtirado]

There are some open questions that come to my mind:

• Is context accesible to children workflows instances?
• which is the last task in a concurrent execution? (this is relevant if the assumption that the output of the last task is the output of the workflow is true)
• Is context the right word? We might have a context shared by several workflow instance to share information. I feel workflowData o workflowContext might be more accurated (although longer)

[JBBianchi]

Is context accesible to children workflows instances?

I don't think it is. A subflow will take an input like any other workflow. For a parent workflow, when calling run.workflow, you can build an input based on the parent context if you want to pass it down.

which is the last task in a concurrent execution? (this is relevant if the assumption that the output of the last task is the output of the workflow is true)

IMO, the output of a concurrent execution should be an array of task outputs. e.g.:

document:
  dsl: '1.0.0-alpha1'
  namespace: test
  name: for-example
  version: '0.1.0'
do:
  - mathOperations
      fork:
        branches:
          - double:
              set: ${ . * 2 }
          - triple:
              set: ${ . * 3 }

With an input of 2, the output should be [4, 6]

Is context the right word? We might have a context shared by several workflow instance to share information. I feel workflowData o workflowContext might be more accurated (although longer)

IMO, prefixing the concept with workflow might be redundant, workflow.context sounds better than workflow.workflowContext

[fjtirado]

@JBBianchi
Since we have $input for task input and $workflow.input for workflow input, I would say that we should remove $context and just have $workflow.context (or make both equivalent)

[JBBianchi]

Well, I think there is a little bit of subtlety with $context we didn't approach yet. The context might not always be "workflow-wide" but scoped at some level of the execution (e.g. in a loop, you might want to share only a slice of the workflow context as the context for the tasks to be run).

[cdavernas]

we should remove $context and just have $workflow.context

Your suggestion would imply that the context is specific to the workflow, which is IMHO not necessarily the case.

As a matter of fact, I think we should be able to "scope" the context, to provide authors with the ability not to leak senstive data to child tasks. The feature is planned, even though not yet implemented. I think you and/or @matthias-pichler-warrify and me had a talk about it, but can't find the issue...

Furthermore, scoping of the context is IMHO crucial to keep branch/iteration data to a minimum.

[fjtirado]

@cdavernas Precisely because I feel that the context is not specific to the workflow, I think we might remove $context. Right now is implicitly implied that $context is the workflow context (so, strictly speaking we should use $workflow.context), or document which is the scope of $context in every scenario. I think we are saying the same with different words.

[JBBianchi]

The documentation should be improved as some parts can be confusing.

I will describe my current understanding There are currently two special variables

• $input is a read only variable which contains the workflow input (as provided when the workflow is started)
• $context is a global variable for the whole workflow instance (what happens with subflows?) instance that is updated in the output section of task (so it is initially empty) and can be accessed by any expression

From the dsl doc, I understand there are 4 (5 actually, there is at least one missing) special variables:

• $input: the tasks input
• $context: the context available to the task
• $task: the current task
• $workflow: the workflow instance
• (?undocumented?)$output: the task output accessible in export ?

Therefore, from my understanding, to access the workflow input, it's not $input but $workflow.input

I understood that (please note thats is just my understanding)

• the output of the last task is the workflow instance output.

Agreed

• When using . in input: keyword expression, . refers to the previously executed task result. In the case of the first task, . is equivalent to $input

It's the output of the previous task (or what you call result), whether it has been transformed via the output directive or not.

• When using . in with: payload: expression, . refers to the result of the input: keyword (if no input: then the result of the previous task)

Within a task, the root input value of an expression* (aka identity filter, . in jq) represents the input of the task, whether it has been transformed via the input directive or not.

• When using . in output: keyword expression, . refers to the result of the current task (this means that, at this stage, the only way to acccess the previous task result, in case you want to merge current and prev result, is to have stored the result of the previous task into the context)

In the output directive, the root input value of an expression* is the "result" of the related task. If you want to merge it with another task that ran before, you indeed will have to rely on the context via export.

* You may notice I introduced a notion of root input value for an expression. I think it's also important to note that . is a symbol specific to jq. For instance, if we'd like to use a JavaScript expression instead, . wouldn't be valid. We'd need to use another symbol, for instance $.

As a general concept, the data flow follows the concept of piping, the output of a previous step is the input of the next step:

input | task 1 | task 2 ... > output

And you can use transformers using the input and output directives to manipulate data before or after a task is executed:

input | input transform | task 1 | output transform | input transform | task 2 ... > ouput

[cdavernas]

Well @fjtirado , I don't know if that answers your questions, but @JBBianchi's explanations are perfectly on spot, about everything.

[fjtirado]

sure, thanks a lot.
maybe we should incorporate the answer somehow into the documentation?

[fjtirado]

@JBBianchi Which will be The "root input value" ($ in Js, . in jq, nothing in other languages ;)) for the first task of the workflow?

[JBBianchi]

The input of the workflow, maybe transformed using the input directive.

Given the workflow

do:
  - sum:
      input:
        from: ${ .number}
      set: ${ . * 2 }
      export:
        as: 
          answer: ${ $output }
  - greet:
      input:
        from: ${ $workflow.input.greet }
      set: ${ "Hello " + . }
      output:
        as: ${ $context + { "greetings": $output } }

With the given workflow input:

{
  "number": 21,
  "greet": "Francisco"
}

step	raw input	transformed input	raw output	transformed output	context
sum	{ "number": 21, "greet": "Francisco" }	21	42	42	{ "answer": 42 }
greet	42	"Francisco"	"Hello Francisco"	{ "answer": 42, "greetings": "Hello Francisco" }	{ "answer": 42 }

The workflow output is:

{
  "answer": 42, 
  "greetings": "Hello Francisco" 
}

The same could have been achieved with an over complexified workflow (which serves as an example for "using the output of previous tasks):

do:
  - sum:
      input:
        from: ${ .number}
      set: ${ . * 2 }
      export:
        as: 
          answer: ${ $output }
  - greet:
      input:
        from: ${ $workflow.input.greet }
      set: ${ "Hello " + . }
      export:
        as: ${ . + { "greetings": $output } }
          
  - merge:
      input:
        from: ${ .context}
      set: ${ . }

step	raw input	transformed input	output	context
sum	{ "number": 21, "greet": "Francisco" }	21	42	{ "answer": 42 }
greet	42	"Francisco"	"Hello Francisco"	{ "answer": 42, "greetings": "Hello Francisco" }
merge	"Hello Francisco"	{ "answer": 42, "greetings": "Hello Francisco" }	{ "answer": 42, "greetings": "Hello Francisco" }	{ "answer": 42, "greetings": "Hello Francisco" }

[fjtirado]

Ok, thanks for confirming my initial understanding that input of the first task is the workflow input.