OCPBUGS-90135: Add more oc-mirror retries#730
Conversation
Because oc-mirror currently fails if there is a single dropped connection while copying any image, we have to run it multiple times. 3 is not proving enough to reliably get all of the images mirrored. Increase to 5. Restore the number of parallel images to 4, since that is likely not having any effect other than slowing down the whole process.
openshift-ci-robot
added
the
jira/valid-reference
|
@zaneb: This pull request references Jira Issue OCPBUGS-90135, which is valid. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-bug
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: zaneb The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/lgtm |
|
@zaneb: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@zaneb: Jira Issue OCPBUGS-90135: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged:
All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-90135 has not been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
* AGENT-1193: Add --mirror-path flag to support pre-mirrored images (#634) * Add mirror-path support and fix custom registry handling Add support for using pre-mirrored images from oc-mirror workspace by specifying the mirror path in appliance-config.yaml. When mirrorPath is configured, the appliance skips running oc-mirror and uses the pre-mirrored registry data directly. Also fixes issues when using custom registries (non-quay.io) where oc adm release info may return incomplete image references missing port and repository path. Adds IDMS entries to ensure the cluster redirects pulls from custom registries to the appliance's internal registry. Changes: - Add MirrorPath field to top level of ApplianceConfig in types - Update appliance-config.yaml template to document the new field - Update code to read mirrorPath from ApplianceConfig.Config.MirrorPath - Add validateMirrorPath() to ApplianceConfig with comprehensive validation - Use pre-mirrored images when mirrorPath is provided - Add fixImageReference() to repair incomplete image refs from oc commands Example: registry.example.com@sha256:... becomes registry.example.com:5000/repo/image@sha256:... - Add addLocalRegistryIDMS() to generate IDMS for custom registry mirrors - Fix release image tag parsing to preserve port in registry URLs The mirror-path can be specified in appliance-config.yaml as: mirrorPath: /path/to/oc-mirror/workspace Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Add debug logging for IsLiveISO tracking * Add --registry-config to oc commands for custom registry authentication Pass --registry-config ~/.docker/config.json to all oc commands (release info, release extract, oc mirror) and --authfile to skopeo. This allows the appliance builder to authenticate against private registries such as the one used in dev-scripts (virthost.ostest.test.metalkube.org:5000). The pull secret from appliance-config is written to ~/.docker/config.json (existing behavior) so no separate credential management is needed. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix tag stripping for release image URLs with registry port strings.Split(image, ":")[0] incorrectly strips the registry port when the release image URL contains one. For example, with the dev-scripts local registry virthost.ostest.test.metalkube.org:5000/openshift/release-images:tag, Split on ":" takes only "virthost.ostest.test.metalkube.org", losing the port and path entirely. Use LastIndex to locate the tag colon after the last "/" so the port is preserved, producing the correct digest reference: virthost.ostest.test.metalkube.org:5000/openshift/release-images@sha256:... Extract the logic into appendDigest() with a unit test. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Refactor mirror-path handling in mirrorImages and DataISO - Swap mirrorPath condition in mirrorImages() so normal flow comes first - Extract mirror-path registry data copy logic into copyMirrorRegistryData() Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Remove explicit --registry-config and --authfile flags from oc and skopeo oc and skopeo automatically pick up credentials from ~/.docker/config.json, which is already written by storePullSecret(). Passing the flags explicitly is redundant and was only introduced when the pull secret was temporarily stored in a separate temp file. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix mirror-path registry data path and missing release-bundles Two related bugs in the --mirror-path flow: 1. copyMirrorRegistryData was called after bundle.Push(), causing the mirror-path docker data to overwrite the release-bundles image that was just pushed to the registry. Move the copy before starting the registry so bundle.Push() adds release-bundles on top of the pre-populated mirror data. 2. Moving the copy earlier exposed a path mismatch: registryDir from GetRegistryDataPath() was mangled (e.g. /assetstemp/data instead of /assets/temp/data). Switch to dataDirPath (filepath.Join(envConfig.TempDir, dataDir)) throughout, consistent with how CopyRegistryImageIfNeeded computes paths. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> * OCPBUGS-90135: Reduced oc-mirror parallel images; update version In order to help reduce mirroring download issues reduce the parallel-images to 2. The oc version is also being updated. * Set default grub menu entry Since 4/27/2026, the agent presubmit tests have been failing as "UEFI Firmware Settings" is the first GRUB menu entry, and since no default was test, GRUB always booted into firmware setup instead of the installer. This change sets the installer as the default Grub entry. * OCPBUGS-85255: Ensure that installed image is booted For the agent installer #725 properly set the Agent ISO to boot instead of the Grub menu. However, this change needs to be conditional so that the installed image will be booted after the first reboot. Fixing to make it conditional. * OCPBUGS-90135: Reduce parallelism for oc mirror The fix #722 increased the retry times to help allievate the oc mirror failures; this helped but didn't resolve the issue. Decrease the parallel images being used to attempt to increase reliability of this command. * Add more oc-mirror retries (#730) Because oc-mirror currently fails if there is a single dropped connection while copying any image, we have to run it multiple times. 3 is not proving enough to reliably get all of the images mirrored. Increase to 5. Restore the number of parallel images to 4, since that is likely not having any effect other than slowing down the whole process. --------- Co-authored-by: Richard Su <rwsu@redhat.com> Co-authored-by: Zane Bitter <zbitter@redhat.com>
3 participants
Because oc-mirror currently fails if there is a single dropped connection while copying any image, we have to run it multiple times. 3 is not proving enough to reliably get all of the images mirrored. Increase to 5. Restore the number of parallel images to 4, since that is likely not having any effect other than slowing down the whole process.