We should provide a higher level summary for each task. This can be coordinated under a single JobSummary class for instance. The goal is to address two issues:
- Provide an indication of job health and any assumptions/logic branches taken
- Perform necessary cleanup functionality in the event of timeout/failure
For instance this will allow us to address issues like job timeout. E.g. if a find_peaks job times out on a long run, it has processed a fair amount of data but will not produce a lst file. As a result the index task cannot begin. If the functionality of lst production was decoupled from the find_peaks job a workflow would be able to at least produce partial statistics. The job summary could also include information to the effect that the job timed out and data was not fully processed.
We should provide a higher level summary for each task. This can be coordinated under a single
JobSummaryclass for instance. The goal is to address two issues:For instance this will allow us to address issues like job timeout. E.g. if a
find_peaksjob times out on a long run, it has processed a fair amount of data but will not produce alstfile. As a result theindextask cannot begin. If the functionality oflstproduction was decoupled from thefind_peaksjob a workflow would be able to at least produce partial statistics. The job summary could also include information to the effect that the job timed out and data was not fully processed.