chore: remove 8 deprecated KFTO v1 training image sources and Tekton … - #979
Conversation
📝 WalkthroughWalkthroughThe diff removes legacy CUDA and ROCm training image definitions, including Dockerfiles, dependency manifests, licenses, READMEs, and associated Tekton PipelineRuns. It removes older runtime and training image build pipelines with image publishing, scanning, signing, and tagging tasks. It updates the architecture inventory and training runtime documentation to reference CUDA 13.0/Torch 2.10 and ROCm 6.4/Torch 2.9 OpenMPI 4.1 images. Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
Comment |
d6b4baf to
b0124a2
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ARCHITECTURE.md`:
- Around line 130-132: Update the stale training-image references in the Tekton
pull-request pipelines and benchmarks/osu-benchmarks/Dockerfile.cuda to use the
existing training/py312-cuda130-torch210-openmpi41 and
training/py312-rocm64-torch29-openmpi41 paths, or remove the retired pipelines
if they are no longer supported.
In `@images/runtime/training/README.md`:
- Line 37: Update the OpenMPI image dependency workflow instructions to use each
image’s pyproject.toml and requirements.txt with uv, replacing
Pipfile/Pipfile.lock edits and Pipenv commands. Regenerate requirements.txt
after updating dependencies, and change the flash-attn checks to validate
flash-attn==2.8.3 in both OpenMPI image directories so the documented CVE
workflow matches the Dockerfiles’ built dependency set.
- Around line 53-58: Update the py312-cuda130-torch210-openmpi41 lock-file
workflow to reference pyproject.toml and requirements.txt, replacing the
Pipfile/Pipfile.lock and pipenv lock steps with the Dockerfile’s uv pip compile
workflow. Remove the unnecessary openmpi41 qualifier from Python dependency
resolution while keeping OpenMPI handling separate.
- Line 57: Update the training image reference in the README command to use the
approved `@sha256` digest instead of the mutable :odh-stable tag. Before
documenting the container execution, verify the image signature and provenance
so only the approved image can generate the lock file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 4a078480-e27b-4982-841d-7bddb0bea879
⛔ Files ignored due to path filters (8)
images/runtime/training/py311-cuda121-torch241/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-cuda124-torch251/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-rocm62-torch241/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-rocm62-torch251/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-cuda128-torch280/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-cuda128-torch290/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-rocm64-torch280/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-rocm64-torch290/Pipfile.lockis excluded by!**/*.lock
📒 Files selected for processing (58)
.tekton/odh-training-cuda121-torch24-py311-pull-request.yaml.tekton/odh-training-cuda121-torch24-py311-push.yaml.tekton/odh-training-cuda124-torch25-py311-pull-request.yaml.tekton/odh-training-cuda124-torch25-py311-push.yaml.tekton/odh-training-cuda128-torch28-py312-pull-request.yaml.tekton/odh-training-cuda128-torch28-py312-push.yaml.tekton/odh-training-cuda128-torch29-py312-pull-request.yaml.tekton/odh-training-cuda128-torch29-py312-push.yaml.tekton/odh-training-rocm62-torch24-py311-pull-request.yaml.tekton/odh-training-rocm62-torch24-py311-push.yaml.tekton/odh-training-rocm62-torch25-py311-pull-request.yaml.tekton/odh-training-rocm62-torch25-py311-push.yaml.tekton/odh-training-rocm64-torch28-py312-pull-request.yaml.tekton/odh-training-rocm64-torch28-py312-push.yaml.tekton/odh-training-rocm64-torch29-py312-pull-request.yaml.tekton/odh-training-rocm64-torch29-py312-push.yaml.tekton/training-push-py311-cuda121-torch241.yaml.tekton/training-push-py311-cuda124-torch251.yaml.tekton/training-push-py311-rocm62-torch241.yaml.tekton/training-push-py311-rocm62-torch251.yaml.tekton/training-runtime-py312-cuda128-torch290-pull-request.yaml.tekton/training-runtime-py312-cuda128-torch290-push.yaml.tekton/training-runtime-py312-rocm64-torch290-pull-request.yaml.tekton/training-runtime-py312-rocm64-torch290-push.yamlARCHITECTURE.mdimages/runtime/training/README.mdimages/runtime/training/py311-cuda121-torch241/Dockerfileimages/runtime/training/py311-cuda121-torch241/LICENSE.mdimages/runtime/training/py311-cuda121-torch241/Pipfileimages/runtime/training/py311-cuda121-torch241/README.mdimages/runtime/training/py311-cuda124-torch251/Dockerfileimages/runtime/training/py311-cuda124-torch251/LICENSE.mdimages/runtime/training/py311-cuda124-torch251/Pipfileimages/runtime/training/py311-cuda124-torch251/README.mdimages/runtime/training/py311-rocm62-torch241/Dockerfileimages/runtime/training/py311-rocm62-torch241/LICENSE.mdimages/runtime/training/py311-rocm62-torch241/Pipfileimages/runtime/training/py311-rocm62-torch241/README.mdimages/runtime/training/py311-rocm62-torch251/Dockerfileimages/runtime/training/py311-rocm62-torch251/LICENSE.mdimages/runtime/training/py311-rocm62-torch251/Pipfileimages/runtime/training/py311-rocm62-torch251/README.mdimages/runtime/training/py312-cuda128-torch280/Dockerfileimages/runtime/training/py312-cuda128-torch280/LICENSE.mdimages/runtime/training/py312-cuda128-torch280/Pipfileimages/runtime/training/py312-cuda128-torch280/README.mdimages/runtime/training/py312-cuda128-torch290/Dockerfileimages/runtime/training/py312-cuda128-torch290/LICENSE.mdimages/runtime/training/py312-cuda128-torch290/Pipfileimages/runtime/training/py312-cuda128-torch290/README.mdimages/runtime/training/py312-rocm64-torch280/Dockerfileimages/runtime/training/py312-rocm64-torch280/LICENSE.mdimages/runtime/training/py312-rocm64-torch280/Pipfileimages/runtime/training/py312-rocm64-torch280/README.mdimages/runtime/training/py312-rocm64-torch290/Dockerfileimages/runtime/training/py312-rocm64-torch290/LICENSE.mdimages/runtime/training/py312-rocm64-torch290/Pipfileimages/runtime/training/py312-rocm64-torch290/README.md
💤 Files with no reviewable changes (56)
- images/runtime/training/py311-cuda124-torch251/README.md
- images/runtime/training/py312-rocm64-torch280/LICENSE.md
- images/runtime/training/py312-rocm64-torch280/README.md
- images/runtime/training/py311-rocm62-torch251/LICENSE.md
- images/runtime/training/py311-rocm62-torch241/LICENSE.md
- images/runtime/training/py312-cuda128-torch280/README.md
- images/runtime/training/py311-rocm62-torch251/README.md
- images/runtime/training/py311-cuda124-torch251/LICENSE.md
- .tekton/odh-training-cuda128-torch28-py312-pull-request.yaml
- images/runtime/training/py311-cuda121-torch241/LICENSE.md
- images/runtime/training/py312-cuda128-torch290/Dockerfile
- images/runtime/training/py312-cuda128-torch280/Dockerfile
- images/runtime/training/py312-rocm64-torch290/Pipfile
- images/runtime/training/py312-rocm64-torch290/README.md
- images/runtime/training/py311-rocm62-torch241/Dockerfile
- images/runtime/training/py311-cuda121-torch241/Dockerfile
- images/runtime/training/py311-rocm62-torch251/Dockerfile
- images/runtime/training/py311-rocm62-torch241/README.md
- images/runtime/training/py312-cuda128-torch290/Pipfile
- images/runtime/training/py311-cuda121-torch241/README.md
- images/runtime/training/py312-cuda128-torch280/Pipfile
- .tekton/odh-training-rocm62-torch24-py311-push.yaml
- images/runtime/training/py311-cuda124-torch251/Dockerfile
- images/runtime/training/py311-cuda124-torch251/Pipfile
- .tekton/odh-training-rocm62-torch25-py311-push.yaml
- .tekton/training-push-py311-rocm62-torch251.yaml
- .tekton/odh-training-cuda121-torch24-py311-pull-request.yaml
- .tekton/training-runtime-py312-cuda128-torch290-pull-request.yaml
- images/runtime/training/py312-cuda128-torch280/LICENSE.md
- images/runtime/training/py311-rocm62-torch251/Pipfile
- .tekton/training-runtime-py312-rocm64-torch290-push.yaml
- images/runtime/training/py312-rocm64-torch290/Dockerfile
- .tekton/odh-training-cuda121-torch24-py311-push.yaml
- .tekton/training-runtime-py312-rocm64-torch290-pull-request.yaml
- .tekton/odh-training-cuda128-torch28-py312-push.yaml
- images/runtime/training/py312-rocm64-torch280/Pipfile
- .tekton/odh-training-cuda124-torch25-py311-pull-request.yaml
- .tekton/training-push-py311-cuda124-torch251.yaml
- images/runtime/training/py312-cuda128-torch290/LICENSE.md
- .tekton/odh-training-cuda128-torch29-py312-push.yaml
- .tekton/odh-training-rocm62-torch25-py311-pull-request.yaml
- .tekton/odh-training-cuda128-torch29-py312-pull-request.yaml
- images/runtime/training/py311-cuda121-torch241/Pipfile
- .tekton/odh-training-rocm62-torch24-py311-pull-request.yaml
- images/runtime/training/py312-rocm64-torch290/LICENSE.md
- images/runtime/training/py311-rocm62-torch241/Pipfile
- .tekton/odh-training-cuda124-torch25-py311-push.yaml
- .tekton/odh-training-rocm64-torch29-py312-push.yaml
- .tekton/training-runtime-py312-cuda128-torch290-push.yaml
- .tekton/odh-training-rocm64-torch28-py312-pull-request.yaml
- images/runtime/training/py312-cuda128-torch290/README.md
- .tekton/odh-training-rocm64-torch29-py312-pull-request.yaml
- images/runtime/training/py312-rocm64-torch280/Dockerfile
- .tekton/training-push-py311-cuda121-torch241.yaml
- .tekton/training-push-py311-rocm62-torch241.yaml
- .tekton/odh-training-rocm64-torch28-py312-push.yaml
| # 1. Start container | ||
| podman run --rm -d --name pipenv-lock \ | ||
| quay.io/opendatahub/odh-training-cuda128-torch28-py312-rhel9:odh-stable \ | ||
| quay.io/opendatahub/odh-training-cuda130-torch210-py312-rhel9:odh-stable \ |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
printf '%s\n' '--- README context ---'
sed -n '45,70p' images/runtime/training/README.md
printf '%s\n' '--- related image references ---'
rg -n --glob '!*node_modules*' 'odh-training-cuda130-torch210-py312-rhel9|odh-stable|cosign|signature|digest' images .github 2>/dev/null | head -200Repository: opendatahub-io/distributed-workloads
Length of output: 1227
🏁 Script executed:
set -eu
image='opendatahub/odh-training-cuda130-torch210-py312-rhel9'
tag='odh-stable'
printf '%s\n' '--- Quay tag metadata ---'
curl -fsSL "https://quay.io/api/v1/repository/${image}/tag/?specificTag=${tag}" |
python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d, indent=2, sort_keys=True))' |
sed -n '1,160p'
printf '%s\n' '--- registry manifest headers ---'
curl -sSI -H 'Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://quay.io/v2/${image}/manifests/${tag}" |
sed -n '1,80p'
printf '%s\n' '--- local guidance for image trust ---'
rg -n -i --glob '*.md' --glob '*.yaml' --glob '*.yml' --glob '*.sh' \
'quay.io|cosign|sigstore|signature|provenance|slsa|digest|image pin' . 2>/dev/null | head -240Repository: opendatahub-io/distributed-workloads
Length of output: 7108
🏁 Script executed:
set -eu
image='opendatahub/odh-training-cuda130-torch210-py312-rhel9'
tag='odh-stable'
scope="repository:${image}:pull"
auth_json="$(curl -fsSLG \
--data-urlencode 'service=quay.io' \
--data-urlencode "scope=${scope}" \
'https://quay.io/v2/auth')"
token="$(printf '%s' "$auth_json" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"
printf '%s\n' '--- resolved manifest digest ---'
curl -fsSI \
-H "Authorization: Bearer ${token}" \
-H 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
"https://quay.io/v2/${image}/manifests/${tag}" |
tr -d '\r' |
grep -Ei '^(HTTP/|docker-content-digest:|content-type:)' || true
printf '%s\n' '--- README image notes ---'
sed -n '88,108p' images/runtime/training/README.mdRepository: opendatahub-io/distributed-workloads
Length of output: 1010
Pin the training image to an approved digest and verify its provenance.
Replace :odh-stable with @sha256:<approved-digest> before running the container. Verify the image signature and provenance to prevent an unreviewed image from generating a poisoned lock file. This creates CWE-494 risk.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@images/runtime/training/README.md` at line 57, Update the training image
reference in the README command to use the approved `@sha256` digest instead of
the mutable :odh-stable tag. Before documenting the container execution, verify
the image signature and provenance so only the approved image can generate the
lock file.
Source: Path instructions
|
Keep this PR on hold until new images are onboarded |
robert-bell
left a comment
robert-bell
left a comment
There was a problem hiding this comment.
/lgtm
Do we need to ask devops to decommission anything? E.g. konflux offboarding?
@ChughShilpa - ptal if you can.
|
|
||
| # 2. Copy Pipfile | ||
| podman cp py312-cuda128-torch280/Pipfile pipenv-lock:/opt/app-root/src/ | ||
| podman cp py312-cuda130-torch210-openmpi41/Pipfile pipenv-lock:/opt/app-root/src/ |
There was a problem hiding this comment.
nit: needs updating to uv
|
|
||
| # 4. Copy lock file back | ||
| podman cp pipenv-lock:/opt/app-root/src/Pipfile.lock py312-cuda128-torch280/ | ||
| podman cp pipenv-lock:/opt/app-root/src/Pipfile.lock py312-cuda130-torch210-openmpi41/ |
There was a problem hiding this comment.
nit: needs updating to uv
Yes devops will do the konflux offboarding |
@robert-bell |
…pipelines Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
b0124a2 to
d8c33de
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
benchmarks/osu-benchmarks/Dockerfile.cuda (2)
1-5: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy liftUse a builder stage for OSU compilation.
This Dockerfile installs compilers and CUDA development packages, builds OSU, and removes them in the same final stage. The required Dockerfile policy is multi-stage. Move compilation to a builder stage and copy only
/usr/local/osu-benchmarksinto a clean runtime stage.dnf removeis not a substitute for stage separation.As per path instructions: “Use multi-stage builds.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/osu-benchmarks/Dockerfile.cuda` around lines 1 - 5, Convert the Dockerfile into a multi-stage build: use a builder stage based on TRAINING_BASE_IMAGE to install compilers and CUDA development packages and compile OSU, then create a clean runtime stage from the same base image and copy only /usr/local/osu-benchmarks from the builder. Remove the same-stage cleanup approach and preserve the existing TRAINING_BASE_IMAGE build argument.Source: Path instructions
1-5: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winUse the published training image as the default base.
CI builds
Dockerfile.cudawithout aTRAINING_BASE_IMAGEoverride. The build therefore pullsquay.io/ksuta/odh-mpi-cuda:0.0.14instead of the repository’s published training image. Set the default toquay.io/opendatahub/odh-training-cuda130-torch210-py312-openmpi41:odh-stablewith an approved digest. This prevents an unreviewed base image from entering the benchmark supply chain (CWE-829).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/osu-benchmarks/Dockerfile.cuda` around lines 1 - 5, Update the TRAINING_BASE_IMAGE default in Dockerfile.cuda to quay.io/opendatahub/odh-training-cuda130-torch210-py312-openmpi41:odh-stable and pin it to the approved image digest, while preserving the existing build-time override behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@images/runtime/training/README.md`:
- Around line 59-62: Update the uv pip compile command in the README to remove
the unsafe-best-match index strategy and use first-index with the curated RHAI
index prioritized before PyPI, then regenerate requirements.txt using the
corrected resolver configuration.
- Line 11: Update the runtime training README package-index documentation to
describe each build path accurately: CUDA uses the curated Red Hat RHAI index,
ROCm prioritizes RHAI with PyPI as fallback, and Konflux Dockerfiles use the
hermetic PIP_FIND_LINKS wheel set with --no-index. Align each documented
requirements.txt generation command with its corresponding Dockerfile path.
---
Outside diff comments:
In `@benchmarks/osu-benchmarks/Dockerfile.cuda`:
- Around line 1-5: Convert the Dockerfile into a multi-stage build: use a
builder stage based on TRAINING_BASE_IMAGE to install compilers and CUDA
development packages and compile OSU, then create a clean runtime stage from the
same base image and copy only /usr/local/osu-benchmarks from the builder. Remove
the same-stage cleanup approach and preserve the existing TRAINING_BASE_IMAGE
build argument.
- Around line 1-5: Update the TRAINING_BASE_IMAGE default in Dockerfile.cuda to
quay.io/opendatahub/odh-training-cuda130-torch210-py312-openmpi41:odh-stable and
pin it to the approved image digest, while preserving the existing build-time
override behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 19584549-a187-4d3d-be42-622ad95368ab
⛔ Files ignored due to path filters (8)
images/runtime/training/py311-cuda121-torch241/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-cuda124-torch251/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-rocm62-torch241/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py311-rocm62-torch251/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-cuda128-torch280/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-cuda128-torch290/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-rocm64-torch280/Pipfile.lockis excluded by!**/*.lockimages/runtime/training/py312-rocm64-torch290/Pipfile.lockis excluded by!**/*.lock
📒 Files selected for processing (61)
.tekton/odh-training-cuda121-torch24-py311-pull-request.yaml.tekton/odh-training-cuda121-torch24-py311-push.yaml.tekton/odh-training-cuda124-torch25-py311-pull-request.yaml.tekton/odh-training-cuda124-torch25-py311-push.yaml.tekton/odh-training-cuda128-torch28-py312-pull-request.yaml.tekton/odh-training-cuda128-torch28-py312-push.yaml.tekton/odh-training-cuda128-torch29-py312-pull-request.yaml.tekton/odh-training-cuda128-torch29-py312-push.yaml.tekton/odh-training-rocm62-torch24-py311-pull-request.yaml.tekton/odh-training-rocm62-torch24-py311-push.yaml.tekton/odh-training-rocm62-torch25-py311-pull-request.yaml.tekton/odh-training-rocm62-torch25-py311-push.yaml.tekton/odh-training-rocm64-torch28-py312-pull-request.yaml.tekton/odh-training-rocm64-torch28-py312-push.yaml.tekton/odh-training-rocm64-torch29-py312-pull-request.yaml.tekton/odh-training-rocm64-torch29-py312-push.yaml.tekton/training-cuda-pull-request.yaml.tekton/training-push-py311-cuda121-torch241.yaml.tekton/training-push-py311-cuda124-torch251.yaml.tekton/training-push-py311-rocm62-torch241.yaml.tekton/training-push-py311-rocm62-torch251.yaml.tekton/training-rocm-pull-request.yaml.tekton/training-runtime-py312-cuda128-torch290-pull-request.yaml.tekton/training-runtime-py312-cuda128-torch290-push.yaml.tekton/training-runtime-py312-rocm64-torch290-pull-request.yaml.tekton/training-runtime-py312-rocm64-torch290-push.yamlARCHITECTURE.mdbenchmarks/osu-benchmarks/Dockerfile.cudaimages/runtime/training/README.mdimages/runtime/training/py311-cuda121-torch241/Dockerfileimages/runtime/training/py311-cuda121-torch241/LICENSE.mdimages/runtime/training/py311-cuda121-torch241/Pipfileimages/runtime/training/py311-cuda121-torch241/README.mdimages/runtime/training/py311-cuda124-torch251/Dockerfileimages/runtime/training/py311-cuda124-torch251/LICENSE.mdimages/runtime/training/py311-cuda124-torch251/Pipfileimages/runtime/training/py311-cuda124-torch251/README.mdimages/runtime/training/py311-rocm62-torch241/Dockerfileimages/runtime/training/py311-rocm62-torch241/LICENSE.mdimages/runtime/training/py311-rocm62-torch241/Pipfileimages/runtime/training/py311-rocm62-torch241/README.mdimages/runtime/training/py311-rocm62-torch251/Dockerfileimages/runtime/training/py311-rocm62-torch251/LICENSE.mdimages/runtime/training/py311-rocm62-torch251/Pipfileimages/runtime/training/py311-rocm62-torch251/README.mdimages/runtime/training/py312-cuda128-torch280/Dockerfileimages/runtime/training/py312-cuda128-torch280/LICENSE.mdimages/runtime/training/py312-cuda128-torch280/Pipfileimages/runtime/training/py312-cuda128-torch280/README.mdimages/runtime/training/py312-cuda128-torch290/Dockerfileimages/runtime/training/py312-cuda128-torch290/LICENSE.mdimages/runtime/training/py312-cuda128-torch290/Pipfileimages/runtime/training/py312-cuda128-torch290/README.mdimages/runtime/training/py312-rocm64-torch280/Dockerfileimages/runtime/training/py312-rocm64-torch280/LICENSE.mdimages/runtime/training/py312-rocm64-torch280/Pipfileimages/runtime/training/py312-rocm64-torch280/README.mdimages/runtime/training/py312-rocm64-torch290/Dockerfileimages/runtime/training/py312-rocm64-torch290/LICENSE.mdimages/runtime/training/py312-rocm64-torch290/Pipfileimages/runtime/training/py312-rocm64-torch290/README.md
💤 Files with no reviewable changes (58)
- images/runtime/training/py311-rocm62-torch241/README.md
- images/runtime/training/py311-cuda121-torch241/Pipfile
- .tekton/training-push-py311-cuda124-torch251.yaml
- images/runtime/training/py312-rocm64-torch280/LICENSE.md
- images/runtime/training/py312-cuda128-torch280/LICENSE.md
- images/runtime/training/py311-cuda121-torch241/README.md
- images/runtime/training/py312-cuda128-torch290/Pipfile
- .tekton/odh-training-cuda121-torch24-py311-push.yaml
- .tekton/odh-training-cuda121-torch24-py311-pull-request.yaml
- images/runtime/training/py311-rocm62-torch241/Dockerfile
- .tekton/odh-training-cuda128-torch28-py312-pull-request.yaml
- images/runtime/training/py311-rocm62-torch241/Pipfile
- images/runtime/training/py312-rocm64-torch290/Dockerfile
- .tekton/odh-training-rocm62-torch25-py311-pull-request.yaml
- images/runtime/training/py311-cuda124-torch251/Dockerfile
- images/runtime/training/py312-cuda128-torch280/Dockerfile
- .tekton/odh-training-cuda124-torch25-py311-push.yaml
- images/runtime/training/py312-cuda128-torch280/Pipfile
- images/runtime/training/py311-cuda124-torch251/README.md
- .tekton/training-push-py311-cuda121-torch241.yaml
- images/runtime/training/py312-rocm64-torch290/README.md
- images/runtime/training/py312-rocm64-torch280/README.md
- .tekton/training-cuda-pull-request.yaml
- .tekton/odh-training-rocm64-torch29-py312-pull-request.yaml
- .tekton/odh-training-rocm62-torch24-py311-pull-request.yaml
- .tekton/training-runtime-py312-cuda128-torch290-pull-request.yaml
- .tekton/odh-training-cuda128-torch29-py312-pull-request.yaml
- images/runtime/training/py312-rocm64-torch290/Pipfile
- images/runtime/training/py312-rocm64-torch280/Pipfile
- .tekton/odh-training-rocm64-torch28-py312-pull-request.yaml
- images/runtime/training/py312-cuda128-torch290/README.md
- images/runtime/training/py311-rocm62-torch251/Pipfile
- .tekton/training-runtime-py312-rocm64-torch290-push.yaml
- .tekton/odh-training-rocm64-torch28-py312-push.yaml
- .tekton/training-rocm-pull-request.yaml
- images/runtime/training/py312-cuda128-torch290/LICENSE.md
- images/runtime/training/py311-cuda124-torch251/LICENSE.md
- images/runtime/training/py312-rocm64-torch280/Dockerfile
- images/runtime/training/py312-rocm64-torch290/LICENSE.md
- .tekton/training-runtime-py312-cuda128-torch290-push.yaml
- .tekton/odh-training-rocm62-torch24-py311-push.yaml
- .tekton/odh-training-cuda128-torch29-py312-push.yaml
- images/runtime/training/py312-cuda128-torch290/Dockerfile
- .tekton/training-push-py311-rocm62-torch241.yaml
- images/runtime/training/py311-rocm62-torch251/Dockerfile
- images/runtime/training/py311-rocm62-torch251/LICENSE.md
- .tekton/odh-training-rocm62-torch25-py311-push.yaml
- .tekton/odh-training-cuda128-torch28-py312-push.yaml
- images/runtime/training/py311-rocm62-torch241/LICENSE.md
- images/runtime/training/py311-cuda121-torch241/LICENSE.md
- images/runtime/training/py311-cuda124-torch251/Pipfile
- .tekton/training-runtime-py312-rocm64-torch290-pull-request.yaml
- images/runtime/training/py311-rocm62-torch251/README.md
- images/runtime/training/py312-cuda128-torch280/README.md
- .tekton/odh-training-rocm64-torch29-py312-push.yaml
- images/runtime/training/py311-cuda121-torch241/Dockerfile
- .tekton/odh-training-cuda124-torch25-py311-pull-request.yaml
- .tekton/training-push-py311-rocm62-torch251.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
- ARCHITECTURE.md
| ## CVE Fixes — Python dependency updates | ||
|
|
||
| The runtime training images install Python packages from **public PyPI** (unlike universal training images which use the private AIPCC index). Each image uses `Pipfile` + `Pipfile.lock` (pipenv). | ||
| The runtime training images install Python packages from **public PyPI** (unlike universal training images which use the private AIPCC index). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 2 -- \
'public PyPI|index-url|extra-index-url|uv pip install|requirements.txt' \
images/runtime/training/README.md \
images/runtime/training/py312-cuda130-torch210-openmpi41 \
images/runtime/training/py312-rocm64-torch29-openmpi41Repository: opendatahub-io/distributed-workloads
Length of output: 13829
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
for f in \
images/runtime/training/README.md \
images/runtime/training/py312-cuda130-torch210-openmpi41/pyproject.toml \
images/runtime/training/py312-cuda130-torch210-openmpi41/requirements.txt \
images/runtime/training/py312-cuda130-torch210-openmpi41/Dockerfile \
images/runtime/training/py312-cuda130-torch210-openmpi41/Dockerfile.konflux \
images/runtime/training/py312-rocm64-torch29-openmpi41/pyproject.toml \
images/runtime/training/py312-rocm64-torch29-openmpi41/requirements.txt \
images/runtime/training/py312-rocm64-torch29-openmpi41/Dockerfile \
images/runtime/training/py312-rocm64-torch29-openmpi41/Dockerfile.konflux
do
if [ -f "$f" ]; then
echo "===== $f ====="
cat -n "$f" | sed -n \
-e '1,12p' \
-e '/index-url/,+4p' \
-e '/uv pip install/,+8p' \
-e '/PIP_FIND_LINKS/,+4p'
fi
done
echo "===== all relevant source references ====="
rg -n -C 2 --glob '!**/requirements.txt' --glob '!**/README.md' \
'public-rhai|AIPCC|PIP_FIND_LINKS|pypi\.org|index-url|extra-index-url' \
images/runtime/trainingRepository: opendatahub-io/distributed-workloads
Length of output: 17177
🌐 Web query:
What is console.redhat.com/api/pypi/public-rhai/rhoai/3.4 and is it a public PyPI mirror or a Red Hat RHAI package index?
💡 Result:
The URL console.redhat.com/api/pypi/public-rhai/rhoai/3.4 is part of the Red Hat AI (RHAI) Python package index system [1][2][3]. It is not a general-purpose public PyPI mirror, but rather a curated, maintained repository of Python packages that have been built, vetted, and shipped by Red Hat specifically for use with Red Hat OpenShift AI (RHOAI) workloads [4][5][3]. Key characteristics of this index include: 1. Purpose: It provides a secure, reliable software supply chain for RHOAI environments [4][3]. By using packages from this index, users benefit from Red Hat's testing and vetting processes, which are designed to reduce risks associated with untrusted dependencies [3]. 2. Scope: It is not a complete mirror of the public PyPI registry [3]. It contains a specific, limited collection of packages—primarily those focused on machine learning, data processing, and LLM-related workloads—that are supported for use with RHOAI [3]. 3. Usage: Workbench images in OpenShift AI are typically pre-configured to use this index [4][5][3]. When installing packages in these environments, pip and uv are pre-configured to pull from this specific index to ensure secure and consistent deployments [4][5]. 4. Disconnected Environments: The index supports disconnected environments, where organizations may mirror this content locally to maintain their own internal hosting services for air-gapped deployments [4][6][5]. In summary, it is a specialized, curated package index provided by Red Hat for its AI platform, rather than a general public PyPI mirror [3].
Citations:
- 1: https://github.com/vllm-project/vllm/blob/main/build_vllm_ppc64le.sh
- 2: Security: address dependencies CVEs trustyai-explainability/llama-stack-provider-trustyai-garak#166
- 3: https://ai-on-openshift.io/odh-rhoai/workbench-python-packages/
- 4: https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html-single/customize_models_to_build_gen_ai_applications/index
- 5: https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.5/html/customize_models_for_gen_ai_and_agentic_ai_applications/set-up-your-working-environment_custom-models
- 6: https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.3/html/customize_models_to_build_gen_ai_applications/set-up-your-working-environment_custom-models
Document the package index for each build path.
Replace the “public PyPI” statement. CUDA uses the curated Red Hat RHAI index. ROCm uses RHAI first and PyPI only as a fallback. Konflux Dockerfiles install from the hermetic PIP_FIND_LINKS wheel set with --no-index. Document these sources and keep the requirements.txt generation commands aligned with each Dockerfile path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@images/runtime/training/README.md` at line 11, Update the runtime training
README package-index documentation to describe each build path accurately: CUDA
uses the curated Red Hat RHAI index, ROCm prioritizes RHAI with PyPI as
fallback, and Konflux Dockerfiles use the hermetic PIP_FIND_LINKS wheel set with
--no-index. Align each documented requirements.txt generation command with its
corresponding Dockerfile path.
| uv pip compile --python-platform=linux --python-version=3.12 \ | ||
| --index-url=https://console.redhat.com/api/pypi/public-rhai/rhoai/3.4/rocm6.4-ubi9-test/simple/ \ | ||
| --extra-index-url=https://pypi.org/simple --index-strategy=unsafe-best-match \ | ||
| -o requirements.txt pyproject.toml |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
uv pip compile index-strategy unsafe-best-match extra-index-url index priority first-index official docs
💡 Result:
In uv, the index strategy determines how the tool resolves dependencies when multiple package indexes are configured. You can configure this via the --index-strategy command-line flag or the UV_INDEX_STRATEGY environment variable [1][2]. The available strategies are: first-index (Default): uv searches for each package across all configured indexes and limits candidate versions to those found in the first index that contains the package [3][1]. This strategy is designed to prevent "dependency confusion" attacks by stopping at the first index that provides a match [4][5]. unsafe-first-match: uv searches all indexes for a package but prefers the first index that provides a compatible version, even if newer versions are available on subsequent indexes [4][1]. unsafe-best-match: This strategy most closely aligns with standard pip behavior [4][2]. It searches all configured indexes for a package, considers all versions from the combined set of candidates, and selects the "best" (typically the highest compatible) version [4][1]. Note that while this provides the most flexibility, it exposes users to "dependency confusion" risks where malicious actors may publish packages with the same names to public indexes [4][2]. Regarding index priority and configuration: Index Priority: Indexes are consulted in the order they are defined [1]. Indexes provided via command-line arguments take precedence over those defined in configuration files [1]. Among indexes, those defined via --extra-index-url or the [[tool.uv.index]] configuration are given higher priority than the default index (often PyPI) [3][1]. Configuration: While --index-url and --extra-index-url are supported for pip compatibility, the recommended approach is to use [[tool.uv.index]] in your pyproject.toml [1]. When using multiple indexes, the order in which they appear determines the resolution precedence [1][6]. For uv pip compile specifically, the tool does not write index URLs to the output file by default [2]. If you need the index URLs to be included in your generated requirements.txt, you must pass the --emit-index-url flag [2]. Unlike pip-compile, if this flag is used, uv will include all configured index URLs in the output [2].
Citations:
- 1: https://docs.astral.sh/uv/concepts/indexes/
- 2: https://docs.astral.sh/uv/pip/compatibility/
- 3: https://docs.astral.sh/uv/reference/settings/
- 4: https://docs.rs/uv-configuration/latest/uv_configuration/enum.IndexStrategy.html
- 5: https://docs.rs/uv-settings/latest/uv_settings/struct.PipOptions.html
- 6: tweak the order of index priority astral-sh/uv#2083
Security Misconfiguration (CWE-427)
Reachability: External
Block the unsafe multi-index resolver configuration.
unsafe-best-match selects the best compatible candidate across both indexes. A public PyPI package can replace a curated RHAI package or satisfy an unpinned transitive dependency. Use first-index with the RHAI index first, or define explicit package-to-index mappings, before regenerating requirements.txt.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@images/runtime/training/README.md` around lines 59 - 62, Update the uv pip
compile command in the README to remove the unsafe-best-match index strategy and
use first-index with the curated RHAI index prioritized before PyPI, then
regenerate requirements.txt using the corrected resolver configuration.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ChughShilpa, robert-bell The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
3 participants
Description
Remove 8 deprecated KFTO v1 training image sources and their 24 associated
Tekton pipelines (pull-request + push). Conforma exceptions for these images
have expired and they are decommissioned for RHOAI 3.6.
Ref: RHOAIENG-78708
Removed image directories
py311-cuda121-torch241py311-cuda124-torch251py311-rocm62-torch241py311-rocm62-torch251py312-cuda128-torch280py312-cuda128-torch290py312-rocm64-torch280py312-rocm64-torch290Retained (Trainer v2)
py312-cuda130-torch210-openmpi41py312-rocm64-torch29-openmpi41Docs updated
ARCHITECTURE.mdtraining image listingimages/runtime/training/README.mdexamples and flash-attn statusHow Has This Been Tested?
.tekton/,ARCHITECTURE.md, andREADME.mdquay.io/modh/training:py311-*references are out of scope (tracked under RHOAIENG-78705)Summary by CodeRabbit
Removed
Documentation
pyproject.toml,requirements.txt, anduv.