Allow using python functions instead of operators (e.g in pre-processing pipeline)

[elronbandel]

If you cannot find operators fit to your needs simply use function to modify every instance in the data:

        def my_function(instance, stream_name=None):
            instance["x"] += 42
            return instance

Or a function that modify the entire stream:

        def my_other_function(stream, stream_name=None):
            for instance in stream:
                instance["x"] += 42
                yield instance

Both functions can be plugged in every place in unitxt requires operators, e.g pre-processing pipeline.

[dafnapension]

very nice, this is how the result looks at the catalog:

[dafnapension]

this very dataset is gated, I suggest to use an example that is accessible to all:

Dataset 'Salesforce/xlam-function-calling-60k' is a gated dataset on the Hub. You must be authenticated to access it.

[elronbandel]

@dafnapension it is very easy to access it with huggingface-login, it almost became a standard this days

[dafnapension]

So I easily generated a token for myself and read the dataset, and bumped into a schema validation error, that I suggest to look into. (This is independently of the logical bug I pointed at below; I actually intended to prove that error along the dataset, but bumped into this error before being able to read along the whole dataset):

[dafnapension]

This preprocessing is wrong (was so also before this PR):
Here is what happens in case of two tools, as is with the single example I found in https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k.
Step Set(fields={"tools/*/parameters": {"type": "object"}}, use_deepcopy=True), applies deep copy only once -- before invoking dict_utils . The latter processes the * component of the query, by definition, employing a single value to assign to all satisfying the *. Since this value is a dict, the very same dict is assigned to all the paths complying with the query.
Thus, all the tools are modified such that the subfield parameters of each is assigned the very same dict (the very same instance of the dict) {"type": "object"}.
Then, in Step Copy(field="properties", to_field="tools/*/parameters/properties", set_every_value=True, )
dict utils goes down the first tool, tracing path tools/0/parameters, reaching the above said dictionary, where it sees that it does not have a field named properties, and it adds that fields (by default, not-exist-ok==True), assigning the first property to it, and now that above said dict has two fields: "type" and "properties". Then (continuing to process the * in the to_field) dict utils goes down the second tool, tools/1/parameters, reaching the very same said dict, and now overwrites the properties field with the second property.

The end result is both tools pointing to same properties, the properties of the second tool.

To fix: simply remove Step Set(fields={"tools/*/parameters": {"type": "object"}}, use_deepcopy=True)
If you need this type field as a sibling of properties, add the following Step, after the above Copy Step:
Set(fields={"tools/*/parameters/type": "object"})

[dafnapension]

Please see detailed comment above

[dafnapension]

Hi @elronbandel , I think I now managed to correctly fix the jsonschema issue (rather than translate all to 'type': 'object' which I erroneously suggested Friday).
Here is a notebook that runs through the whole recipe output, emphasizing instance 15 that has this special case of 'str, optional' which failed the runs before.

[dafnapension]

[dafnapension]

bfcl on main:

and now fixed, with same new Operator that is used for xlam: