Skip to content

OCPBUGS-58178: Bump promtail version to 3.4.3 #66518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vrutkovs
Copy link
Member

Lets see if bumping promtail helps with sigkills on GCP

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 27, 2025
@openshift-ci-robot
Copy link
Contributor

@vrutkovs: This pull request references Jira Issue OCPBUGS-58178, which is invalid:

  • expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Lets see if bumping promtail helps with sigkills on GCP

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Contributor

openshift-ci bot commented Jun 27, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2025
@openshift-ci openshift-ci bot requested review from smg247 and stbenjam June 27, 2025 11:49
@vrutkovs
Copy link
Member Author

/pj-rehearse 5

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@vrutkovs: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.5-e2e-operator openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.14-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.13-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.12-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.11-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.10-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.9-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.8-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.7-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.6-e2e-aws-csi openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.14-e2e-aws-csi-extended openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.11-e2e-aws-csi-migration openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.10-e2e-aws-csi-migration openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.9-e2e-aws-csi-migration openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.8-e2e-aws-csi-migration openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-aws-ebs-csi-driver-operator-release-4.14-e2e-aws-ovn-upgrade openshift/aws-ebs-csi-driver-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-main-okd-scos-e2e-aws-ovn openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.21-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.20-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.19-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.19-okd-scos-e2e-aws-ovn openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.18-okd-scos-e2e-aws-ovn openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.18-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed
pull-ci-openshift-cluster-version-operator-release-4.17-e2e-aws-ovn-techpreview openshift/cluster-version-operator presubmit Registry content changed

A total of 28272 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: job(s): 5 either don't exist or were not found to be affected, and cannot be rehearsed

@vrutkovs
Copy link
Member Author

/pj-rehearse more

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@vrutkovs
Copy link
Member Author

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-cgroupsv2 periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-serial

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@yuqi-zhang
Copy link
Contributor

The passing AWS journals don't show any issues with promtail or any sigkills, hopefully that indicates this is helpful (Not sure if this issue is more prevalent on GCP, though, or if AWS is representative as well, since it's a bit racy)

@vrutkovs
Copy link
Member Author

lets run a few GCP tests to be sure
/pj-rehearse periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn periodic-ci-openshift-release-master-nightly-4.18-e2e-gcp-ovn-upgrade periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-upgrade

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@vrutkovs
Copy link
Member Author

/pj-rehearse ack

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 27, 2025
@vrutkovs
Copy link
Member Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 27, 2025
@openshift-ci-robot
Copy link
Contributor

@vrutkovs: This pull request references Jira Issue OCPBUGS-58178, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@vrutkovs
Copy link
Member Author

/pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op pull-ci-openshift-machine-config-operator-release-4.20-e2e-gcp-op

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@vrutkovs: job(s): either don't exist or were not found to be affected, and cannot be rehearsed

Copy link
Contributor

openshift-ci bot commented Jun 27, 2025

@vrutkovs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/kubevirt/kubevirt-tekton-tasks/release-v0.12/e2e-tests-namespace-scope 94b1031 link unknown /pj-rehearse pull-ci-kubevirt-kubevirt-tekton-tasks-release-v0.12-e2e-tests-namespace-scope
ci/rehearse/kubevirt/kubevirt-tekton-tasks/release-v0.15/e2e-tests 94b1031 link unknown /pj-rehearse pull-ci-kubevirt-kubevirt-tekton-tasks-release-v0.15-e2e-tests
ci/rehearse/kubevirt/kubevirt-tekton-tasks/release-v0.12/e2e-tests-cluster-scope 94b1031 link unknown /pj-rehearse pull-ci-kubevirt-kubevirt-tekton-tasks-release-v0.12-e2e-tests-cluster-scope
ci/rehearse/kubevirt/kubevirt-tekton-tasks/release-v0.9.0/e2e-tests-cluster-scope 94b1031 link unknown /pj-rehearse pull-ci-kubevirt-kubevirt-tekton-tasks-release-v0.9.0-e2e-tests-cluster-scope
ci/rehearse/openshift/machine-config-operator/main/e2e-gcp-op 94b1031 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op
ci/rehearse/kubevirt/kubevirt-tekton-tasks/release-v0.9.0/e2e-tests-namespace-scope 94b1031 link unknown /pj-rehearse pull-ci-kubevirt-kubevirt-tekton-tasks-release-v0.9.0-e2e-tests-namespace-scope
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-upgrade 94b1031 link unknown /pj-rehearse periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-upgrade
ci/rehearse/openshift/machine-config-operator/release-4.20/e2e-gcp-op 94b1031 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-release-4.20-e2e-gcp-op

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@yuqi-zhang
Copy link
Contributor

Hmm, I see the same problem:

Jun 27 20:59:07.508572 ci-op-ymqk39bh-6c778-8569j-worker-a-r75zl systemd[1]: crio-3b3085b5e26b22ccf410dd8efa2fd3bbe44718061cbbb6b786fdef2db5c15b7b.scope: Stopping timed out. Killing.
Jun 27 20:59:07.509292 ci-op-ymqk39bh-6c778-8569j-worker-a-r75zl systemd[1]: crio-3b3085b5e26b22ccf410dd8efa2fd3bbe44718061cbbb6b786fdef2db5c15b7b.scope: Killing process 4159 (promtail) with signal SIGKILL.

on the latest failed e2e-gcp-op run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. rehearsals-ack Signifies that rehearsal jobs have been acknowledged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants