Add checkpointing in the pipeline to resume after timeout on the HPC

[m4gpi]

For longer running jobs, we should reload state (based on processed files) and resume from that point:

Checkpointing is the process of adding break points into long running code, where the current state of your workflow is dumped. At the start of your job script, you check for if there is an existing checkpoint for you job, and then continue from where you left off.

An example in Python would be to write out results every 100 steps, and write a temporary file to note how far along you were in your loop. And then simply check for that number and continue the loop from there on job resubmit.

If your application takes longer than 2 hours to run but you still want to run on more than 400 CPUs you can checkpoint your application (i.e, save a snapshot of your application’s state) so that it can stop at least every two hours, then you can resume from where it was.