[CI] Made Nightly Tests Continue on Error

[AlexandreSinger]

Currently, the Nightly Tests will stop whenever any of the Nightly Tests fail (and will not run the following tests). So for example, if Nightly Test 1 has a QoR failure, all other Nightly tests will not be run.

Made each step in the Nightly Test manual job continue on error. This will run the following steps even if one of them fail.

I found this while running the Nightly Tests on Haydar's recent PR (PR #2897 ):

I do not think this is the intended behavior we want based on previous discussions.

[AlexandreSinger]

I ran the current Nightly Tests (which I know have QoR failures) and found the following problem with my solution:

According to the annotations, Nightly Tests 1 and 3 failed; however the overall workflow did not fail! This is because we marked them continue on failure. This is not desired behaviour. We want a big red X on the worflow saying something went wrong.

@AmirhosseinPoolad Can you look into this? I think as a work-around we can add a step at the end which will produce a "summary" of the nightly tests run and that one can fail. I think its possible to have each nightly test step produce some variable (pass / fail) which the summary step can consume at the end. What do you think?

[AmirhosseinPoolad]

Thanks, I'll look into it. If that's okay with you, I'm going to just make another PR in another branch to fix this issue. Just to keep the git history clean.

[AlexandreSinger]

Thanks, I'll look into it. If that's okay with you, I'm going to just make another PR in another branch to fix this issue. Just to keep the git history clean.

Up to you! You can also just modify this branch directly. feature-ci-nightly-test is on origin. You are free to commit your changes with amend to update the commit and push-force over the original commit.

[AmirhosseinPoolad]

Decided to make a new PR since this one had a merge commit and I don't feel confident enough with my gitfu to change stuff from 2 commits ago. Closing this.