Pipelining idea

[tashrifbillah]

Hi Habib,

Until I make a pipeline automating things, you can run your jobs in chain in one step:

dplocate/dplocate0-read/parse_gps_mc.py            --phoenix-dir dplocate-phoenix --study BWH --consent-dir dplocate-phoenix/GENERAL/ --matlab-dir dplocate/dplocate0-read/ && \
dplocate/dplocate1-preprocess/preprocess_gps_mc.py --phoenix-dir dplocate-phoenix --study BWH --consent-dir dplocate-phoenix/GENERAL/ --matlab-dir dplocate/dplocate1-preprocess/ && \
dplocate/dplocate2-process/process_gps_mc.py       --phoenix-dir dplocate-phoenix --study BWH --consent-dir dplocate-phoenix/GENERAL/ --matlab-dir dplocate/dplocate2-process/ && \
dplocate/dplocate3-aggregate/aggregate_gps_mc.py   --phoenix-dir dplocate-phoenix --study BWH --consent-dir dplocate-phoenix/GENERAL/ --matlab-dir dplocate/dplocate3-aggregate/ && \
dplocate/dplocate4-plot/phone_gps_mc.py            --phoenix-dir dplocate-phoenix --study BWH --consent-dir dplocate-phoenix/GENERAL/ --matlab-dir dplocate/dplocate4-plot/

The &&s essentially do the trick i.e. wait for completion of previous steps.

@hre80

[hre80]

This is great! I like it. Thank you very much!

[tashrifbillah]

Hi @hre80 , did you get a chance to test it for your research? Although unlikely, I want to confirm if &&s adequately wait for any background process to complete before initiating the next one.

[tashrifbillah]

Ping Habib ...

[hre80]

I did not. I used the same command for the prerequisites that are not very time consuming. I can check it for one subject with different modules and let you know.

[hre80]

It did not work with my first attempt. The error I got implies that the results of the first module was not present at the operation time of the second.

[tashrifbillah]

Can you provide the full log within triple backticks:

```

insert your log here

```

[tashrifbillah]

I hope you pulled the latest master and adjusted the values of arguments for your use case.

Here is the clone if you need it:

/data/predict/dplocate

[hre80]

I did not run it for this pipeline. I needed to run the dpsleep which is modular in the same way on a subject. I tried to run them with && instead of running the module commands one by one, and it failed.

[hre80]

While running the modules manually perfectly works.

[tashrifbillah]

Hmm .... Sorry, I missed the ending slash in my command. Edited it. Now each line ends in && \. Feel free to retry, thank you.

[hre80]

It did not work this way. Submits 5-6 jobs simultaneously without any dependency.

[tashrifbillah]

It should submit one job comprising 5 steps. I am not sure how you are running/submitting things. Anyways, feel free not to work on it anymore. I shall 100% confirm it when I get real data. Thank you for your time so far.