Skip to content

Remove old th06 universal training images #978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChughShilpa merged 1 commit into from
Aug 6, 2026
Merged

Remove old th06 universal training images #978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,9 @@ images/
│ └── examples/ Example-specific runtime images
├── universal/
│ └── training/ Universal training images (3 variants)
│ ├── th06-cpu-torch210-py312/
│ ├── th06-cuda130-torch210-py312/
│ └── th06-rocm64-torch291-py312/
│ ├── th-torch-cpu-py312/
│ ├── th-torch-cuda-py312/
│ └── th-torch-rocm-py312/
├── tests/ Test runner image
└── util/
└── mc-cli/ MinIO client utility image
Expand Down
16 changes: 8 additions & 8 deletions images/universal/training/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,14 @@ When `DOWNSTREAM=true`:

| Flavor | Directory | Use Case |
|--------|-----------|----------|
| **CUDA** | `th<VERSION>-cuda<VERSION>-torch<VERSION>-py<VERSION>/` | NVIDIA GPU training |
| **ROCm** | `th<VERSION>-rocm<VERSION>-torch<VERSION>-py<VERSION>/` | AMD GPU training |
| **CPU** | `th<VERSION>-cpu-torch<VERSION>-py<VERSION>/` | CPU-only training |
| **CUDA** | `th-torch-cuda-py312/` | NVIDIA GPU training |
| **ROCm** | `th-torch-rocm-py312/` | AMD GPU training |
| **CPU** | `th-torch-cpu-py312/` | CPU-only training |

All flavors include:
- **Training Hub** with LoRA support
- **PyTorch <VERSION>**
- **Python <VERSION>**
- **PyTorch**
- **Python 3.12**
Comment on lines +36 to +43

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

new_dirs=(
  images/universal/training/th-torch-cpu-py312
  images/universal/training/th-torch-cuda-py312
  images/universal/training/th-torch-rocm-py312
)

failed=0

for dir in "${new_dirs[@]}"; do
  if [[ ! -d "$dir" ]]; then
    echo "Missing new image directory: $dir" >&2
    failed=1
  fi
done

if rg -n --hidden --glob '!.git/**' \
  'th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)' .; then
  echo "Stale references to removed image directories found." >&2
  failed=1
fi

exit "$failed"

Repository: opendatahub-io/distributed-workloads

Length of output: 2165

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '--- new image directories ---\n'
for dir in images/universal/training/th-torch-cpu-py312 images/universal/training/th-torch-cuda-py312 images/universal/training/th-torch-rocm-py312; do
  if [[ -d "$dir" ]]; then
    echo "present: $dir"
  else
    echo "missing: $dir"
  fi
done

printf '\n--- all deleted th06 training directory references ---\n'
rg -n --hidden --glob '!.git/**' \
  'images/universal/training/(th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|th-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312))' . || true

printf '\n--- all th06 runtime/image references ---\n'
rg -n --hidden --glob '!.git/**' \
  'training-hub-th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|odh-th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|"th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)"' . || true

Repository: opendatahub-io/distributed-workloads

Length of output: 265

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- new image directories ---'
for dir in images/universal/training/th-torch-cpu-py312 images/universal/training/th-torch-cuda-py312 images/universal/training/th-torch-rocm-py312; do
  if [[ -d "$dir" ]]; then
    echo "present: $dir"
  else
    echo "missing: $dir"
  fi
done

printf '%s\n' ''
printf '%s\n' '--- old image directory references ---'
rg -n --hidden --glob '!.git/**' \
  'images/universal/training/(th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|th-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312))' . || true

printf '%s\n' ''
printf '%s\n' '--- th06 runtime/image references ---'
rg -n --hidden --glob '!.git/**' \
  'training-hub-th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|odh-th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)|"th06-(cpu-torch210-py312|cuda130-torch210-py312|rocm64-torch291-py312)"' . || true

Repository: opendatahub-io/distributed-workloads

Length of output: 2385

Update the stale runtime image references in tests/trainer/utils/utils_runtimes.go.

The renamed th-torch-* image directories exist, but this file still maps runtimes to deleted th06-* image names. CWE-1104 (Cross-module Inconsistency).

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@images/universal/training/README.md` around lines 36 - 43, Update the
runtime-to-image mappings in the relevant definitions in utils_runtimes.go to
use the renamed th-torch-cuda-py312, th-torch-rocm-py312, and th-torch-cpu-py312
image references instead of the deleted th06-* names, preserving the existing
CUDA, ROCm, and CPU runtime associations.

Source: Path instructions

- **JupyterLab** (workbench mode)

### CUDA Image
Expand Down Expand Up @@ -163,9 +163,9 @@ Update to dependencies **other than** CUDA, ROCm, PyTorch, or Training Hub.
2. Coordinate with AIPCC team for the new index with updated major packages
3. Coordinate with Notebooks team for the new base image
4. Create new directory following naming convention:
- CUDA: `th<TH_VERSION>-cuda<CUDA_VERSION>-torch<TORCH_VERSION>-py<PYTHON_VERSION>/`
- ROCm: `th<TH_VERSION>-rocm<ROCM_VERSION>-torch<TORCH_VERSION>-py<PYTHON_VERSION>/`
- CPU: `th<TH_VERSION>-cpu-torch<TORCH_VERSION>-py<PYTHON_VERSION>/`
- CUDA: `th-torch-cuda-py<PYTHON_VERSION>/`
- ROCm: `th-torch-rocm-py<PYTHON_VERSION>/`
- CPU: `th-torch-cpu-py<PYTHON_VERSION>/`
Comment on lines +166 to +168

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Resolve the major-update naming conflict.

Scenario 2 requires a new directory for major CUDA, ROCm, PyTorch, or Training Hub updates. These names vary only by Python version. A major dependency update with Python 3.12 would reuse th-torch-*-py312/, so the workflow cannot preserve the previous GA image or support deterministic rollback.

Either add the relevant dependency version to the directory name, or change Scenario 2 to update the existing stable directory and define immutable image tags, digests, and rollback rules. Update ARCHITECTURE.md and downstream build references with the selected contract.

As per path instructions, architectural issues and bug-prone patterns are review priorities.

Possible documentation fix for stable directories 
-**New folder required:**
+**Existing flavor directory update:**
...
-4. Create new directory following naming convention:
+4. Update the existing `th-torch-<flavor>-py<PYTHON_VERSION>/` directory in place.
+   Publish an immutable image tag or digest for each major dependency update.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@images/universal/training/README.md` around lines 166 - 168, Resolve the
Scenario 2 image-directory naming collision around the CUDA, ROCm, and CPU paths
in the training image documentation by including the relevant dependency version
in each directory name, or explicitly define the stable-directory update
contract with immutable tags, digests, and rollback rules. Update
ARCHITECTURE.md and all downstream build references to consistently use the
selected naming and rollback contract.

Source: Path instructions

5. Copy files from previous version and update:
- `Dockerfile` - update base image, package versions
- `pyproject.toml` - update dependencies to match `training-hub` requirements
Expand Down
109 changes: 0 additions & 109 deletions images/universal/training/th06-cpu-torch210-py312/Dockerfile
View file
Open in desktop

This file was deleted.

21 changes: 0 additions & 21 deletions images/universal/training/th06-cpu-torch210-py312/LICENSE.md
View file
Open in desktop

This file was deleted.

18 changes: 0 additions & 18 deletions images/universal/training/th06-cpu-torch210-py312/entrypoint-universal.sh
View file
Open in desktop

This file was deleted.

92 changes: 0 additions & 92 deletions images/universal/training/th06-cpu-torch210-py312/pyproject.toml
View file
Open in desktop

This file was deleted.

Loading