Skip to content

Have the release script warn if a workflow is queued for >15m #19084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

Have the release script warn if a workflow is queued for >15m #19084

wants to merge 4 commits into from

Conversation

@anoadragon453
Copy link
Member

@anoadragon453 anoadragon453 commented Oct 21, 2025 •
edited
Loading

Currently the script will just emit "Workflows failed" if a workflow is queued for over 15m (which has been happening recently due to MacOS runners being slow to get).

After this change, the release manager will be warned that a workflow is still queued, prompting them to go and do something about it.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct (run the linters)

@anoadragon453 anoadragon453 marked this pull request as ready for review October 21, 2025 13:31
@anoadragon453 anoadragon453 requested a review from a team as a code owner October 21, 2025 13:31
MadLittleMods
MadLittleMods reviewed Oct 23, 2025
View reviewed changes
scripts-dev/release.py
Comment on lines 599 to 602
# Warn the user if any workflows are still queued. They might need to fix something.
if any(workflow["status"] == "queued" for workflow in resp["workflow_runs"]):
_notify("Warning: at least one release workflow is still queued...")
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prevents someone from continuing the release if one of the workflows is queued. It looks like previously we allowed people to continue even if some workflows failed so I don't think we would want to block forever in the queued case.

We should probably integrate this into the logic below and give people the option to continue anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I've done so in b5c66de.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, this allows you to continue even without the in_progress workflows being done yet. I think we at-least want to wait for those.

@anoadragon453 anoadragon453 force-pushed the anoa/release_script_queued_workflows branch from 7f3a836 to 9cd8166 Compare October 24, 2025 10:12
MadLittleMods
MadLittleMods reviewed Oct 24, 2025
View reviewed changes
scripts-dev/release.py
Comment on lines +600 to +601
if any(workflow["status"] == "queued" for workflow in resp["workflow_runs"]):
_notify("Warning: at least one release workflow is still queued...")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of the macOS runners not working for that whole day, we'd see this warning every 5 minutes and asked to continue.

I assume that's fine as at that point you would be monitoring the actions and continue once you see them done and not pressing continue on this script over and over.

scripts-dev/release.py
Comment on lines 599 to 602
# Warn the user if any workflows are still queued. They might need to fix something.
if any(workflow["status"] == "queued" for workflow in resp["workflow_runs"]):
_notify("Warning: at least one release workflow is still queued...")
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, this allows you to continue even without the in_progress workflows being done yet. I think we at-least want to wait for those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MadLittleMods MadLittleMods MadLittleMods left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@anoadragon453 @MadLittleMods