New Step idea - get data from plugin function

[aspark21]

Maybe it's a daft idea but we're building some complex assessment reports which need to pull data not just through SQL but enriching data through a bunch of other core functions, so we thought we could create an external function (slow running) within that plugin that would allow this data to then be exported via dataflows.

e.g.

use report_feedback_tracker\local\admin as feedback_tracker;

feedback_tracker::get_institutional_data_export();

but I guess this is usually hardcoded so we could make a bespoke step for that one function but that's a bit clunky and not maintainable.

Do you think that's a sensible pattern? Has that come up with other users of dataflows?

[brendanheywood]

Doing it via webservice would be the simplest and we've done that for a few clients. The biggest downside is working with lots of data. But if you have an sql or csv stream of records and are just enriching it row by row then it will streams well and we've used that pattern a lot.

If the ws returns a giant chunk of data then it would be better to make it return an iterable and a wrap it into a step which should be around the same amount of work and would have the advantage of being able to stream in fixed memory.

I don't like the security aspect of allowing a generic step which can call any function, that's almost as permissive as having an 'eval' step. We could add it with some config.php allow list.

[aspark21]

My best guess was that that function will take a very long time to run -> 1-2h+ so was best suited to run as a scheduled task.

To give an idea of kind of load - this would be pulling every single assessment activity (assign, quiz, turnitin, workshop and a bunch of other activities) in all courses in the site, for every activity it looks at each student's overrides to see if they submitted on time and then calculates if the feedback was released on time for that student by evaluating the number of working days at our institution between submit and release dates, in addition to pulling a bunch of metadata about the course. We will probably only include summative assessments though so it will reduce the number of rows returned but it's that need to crunch a lot of data for each assessment that's the big multiplier.

It's not so much about quantity of the output (though it is significant) but lengthy computing per row.

Team had been suggesting the plugin should run the task itself and place the exported file in a defined location on disk for the other existing steps to pick up.

Now that I think about it, we could emit an event once the data has been exported from the other plugin and use an event trigger for dataflows to pickup the export and do the transfer with existing steps.

[brendanheywood]

Well most dataflows are already run in an adhoc task anyway so that's not a big issue either way.

But it sounds to me like this has cross the threshold of what I'd want in a dataflow and I think that should be in version control and then like you said it triggers another flow via file dump