Skip to content

Update checks for MOM6 restarts when performing a re-run on failure #4179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Oct 30, 2025
Merged

Update checks for MOM6 restarts when performing a re-run on failure #4179

merged 7 commits into from
Oct 30, 2025

Conversation

@aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Oct 24, 2025 •
edited
Loading

Description

This PR:

This PR also:

  • has a minor fix in ush/preamble.sh that prints the Begin date time in human readable format similar to End date time message.

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Tests 1, 2, 3 below have FHMAX_GFS=384 and restart_interval_gfs=48

  1. Ran C48_S2SW case forecast out to FHMAX_GFS without stopping
  2. Start running C48_S2SW forecast
    a. kill forecast job at fhr=126. Restarts are available at hrs 48, 96
    b. re-submit forecast job. The job picks the restarts at hr 96 and completes
  3. Start running C48_S2SW forecast
    a. kill forecast job at fhr = 102. Restarts are available at hrs 48, 96
    b. re-submit forecast job. The job picks restarts at hr 48 (ocean forecast history file at 105 is not available, rendering restart at hr 96 invalid) and completes.
  4. Ran C48_S2SW test start to end.
  5. Ran C48_S2SWA_gefs test to test the segment functionality.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

JessicaMeixner-NOAA
JessicaMeixner-NOAA reviewed Oct 27, 2025
View reviewed changes
ush/forecast_det.sh Outdated
else
# Also check for MOM6 history file availability
# TODO: Need to adapt for SFS where averaging period may be different, need to generalize
hdate=$(date -u -d "${rdate:0:8} ${rdate:8:2}:00:00 + 3 hours" +"%Y_%m_%d_%H") # MOM6 history is averaged 6 hrs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be generalized now with FHOUT_OCN ?

Also - This should have some sort of if not gdas/enkfgdas because the output is different for those runs. Should those runs have different checks here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generalized to use FHOUT_OCN. I had committed what I had on Friday before leaving work, and a TODO to remind me about it when I pick it up on Monday.
2f1650f addresses the first part of this comment.

RERUN is not available for RUN = gdas|enkfgdas as their forecast lengths are less than 12 hours (15 if you count IAU).
https://github.com/aerorahul/global-workflow/blob/2f1650f2eb7237ab95373e50cb081097545781d0/ush/forecast_det.sh#L38-L42
enkfgfs does not have a forecast segment.
I'll test this for a RUN=gefs with segments to ensure this change does not break that.

@aerorahul
Copy link
Contributor Author

GEFS test passed.
Merged develop after renaming PR.
Opening up for review.

@aerorahul aerorahul marked this pull request as ready for review October 28, 2025 13:56
JessicaMeixner-NOAA
JessicaMeixner-NOAA previously approved these changes Oct 28, 2025
View reviewed changes
Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves the open issue.

jiandewang
jiandewang previously approved these changes Oct 28, 2025
View reviewed changes
@emcbot emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Oct 28, 2025
@emcbot emcbot added CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Oct 28, 2025
@emcbot
Copy link

emcbot commented Oct 28, 2025

C96mx100_S2S FAILED on Gaeac6 (pipeline ID: 5409)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/EXPDIR/C96mx100_S2S_87175e4c-5409

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem000_seg1.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem001_seg1.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem002_seg1.log

View Error Logs: (sfs_fcst_mem000_seg1.log) (sfs_fcst_mem001_seg1.log) (sfs_fcst_mem002_seg1.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress labels Oct 28, 2025
@aerorahul
Copy link
Contributor Author

I did not test the SFS case in my testing. I will do that and report back. SFS has a restart_interval_gfs=12 and does 24hr averaging of the ocean output. The checks for restarts are probably either not enough or too long.

@aerorahul aerorahul mentioned this pull request Oct 29, 2025
Closed
@aerorahul
Copy link
Contributor Author

See comment in the issue #3822 and advise if the approach there is acceptable.

@aerorahul aerorahul requested a review from jiandewang October 30, 2025 15:26
@aerorahul aerorahul removed the CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed label Oct 30, 2025
DavidHuber-NOAA
DavidHuber-NOAA reviewed Oct 30, 2025
View reviewed changes
DavidHuber-NOAA
DavidHuber-NOAA reviewed Oct 30, 2025
View reviewed changes
DavidHuber-NOAA
DavidHuber-NOAA approved these changes Oct 30, 2025
View reviewed changes
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a nitpick on variable name.

@DavidHuber-NOAA
Copy link
Contributor

Launching tests on C6.

@aerorahul aerorahul mentioned this pull request Oct 30, 2025
Open
@emcbot emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Oct 30, 2025
@emcbot emcbot added CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress labels Oct 30, 2025
@DavidHuber-NOAA
Copy link
Contributor

All tests passed on C6. Merging.

@DavidHuber-NOAA DavidHuber-NOAA merged commit 5f0cfee into NOAA-EMC:develop Oct 30, 2025
5 checks passed
@aerorahul aerorahul mentioned this pull request Oct 31, 2025
Closed
@aerorahul aerorahul deleted the feature/issue-3822b branch October 31, 2025 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@jiandewang jiandewang Awaiting requested review from jiandewang

@JessicaMeixner-NOAA JessicaMeixner-NOAA Awaiting requested review from JessicaMeixner-NOAA

Assignees

No one assigned

Labels

CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Restart runs are not ensuring MOM6 output has completed writing.

5 participants

@aerorahul @emcbot @DavidHuber-NOAA @jiandewang @JessicaMeixner-NOAA