fix(customizer): include models SDK package in task images

[mckornfield]

Summary

Customizer task and training images were still missing generated SDK alias packages. After those image builds began passing, the Docker customizer E2E path exposed follow-on runtime issues in CPU-build smoke imports, deployment container networking, and deployments-plugin Docker endpoint readiness from inside a containerized Platform API.

Changes

• Install packages/models editably alongside the SDK in customizer, automodel, unsloth, and RL task/training images.
• Add packages/models to the Unsloth and RL reduced workspace slices.
• Broaden Docker workspace regression coverage to require all SDK alias packages used by the editable SDK install.
• Keep the Automodel smoke test CPU-build-safe by hiding Transformer Engine during the PEFT/torchao dispatcher import check.
• Keep deployment weight puller containers CPU-only while preserving GPU access for vLLM serving containers.
• Rewrite Docker loopback Files HF URLs so deployment containers can reach the platform API.
• Honor an executor-level Docker network for deployments-plugin containers, with E2E wiring via NEMO_DEPLOYMENTS_DOCKER_NETWORK and compatibility with MODELS_DOCKER_NETWORK.
• Add Docker deployments endpoint_mode: network support so containerized Platform APIs probe/report deployment endpoints as http://<container-name>:<container-port> instead of loopback host ports.
• Configure quickstart and the Docker E2E harness to use deployments-plugin network endpoint mode.
• Add a customizer task image smoke test that touches SDK files and models lazy resources so missing generated alias packages fail during image build.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior - justification:
• Tests not applicable - justification:
• Documentation updated for user-visible behavior
• Documentation not applicable - justification: Docker packaging, backend wiring, and smoke-test fixes only; no user-visible docs change.

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation:

• uv run --frozen pytest tests/unit/test_docker_workspace_slices.py -v - 23 passed
• uv run ruff check tests/smoke_gpu/test_customizer_automodel.py tests/unit/test_docker_workspace_slices.py - passed
• make docker-print TARGET=nmp-customizer - passed
• uv run --frozen pytest services/core/models/tests/unit/controllers/backends/deployments_plugin/test_resolve.py services/core/models/tests/unit/controllers/backends/deployments_plugin/test_compiler.py services/core/models/tests/unit/controllers/backends/deployments_plugin/test_backend.py services/core/models/tests/unit/controllers/backends/deployments_plugin/test_status.py -q - 53 passed
• uv run --frozen pytest plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py -q - 66 passed
• uv run --frozen pytest packages/nmp_testing/tests/unit/test_e2e_docker_backend.py packages/nemo_platform_ext/tests/quickstart/test_container.py -q - 56 passed
• uv run --frozen pytest tests/smoke_gpu/test_customizer_tasks.py::test_sdk_alias_resources_importable -q - 1 passed
• uv run --frozen ty check plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/config.py plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py - passed
• uv run ruff check tests/smoke_gpu/test_customizer_tasks.py - passed
• uv run ruff check plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/config.py plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py packages/nmp_testing/src/nmp/testing/e2e/docker.py packages/nmp_testing/tests/unit/test_e2e_docker_backend.py packages/nemo_platform_ext/src/nemo_platform_ext/quickstart/container.py packages/nemo_platform_ext/tests/quickstart/test_container.py - passed
• uv run ruff format --check plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/config.py plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py packages/nmp_testing/src/nmp/testing/e2e/docker.py packages/nmp_testing/tests/unit/test_e2e_docker_backend.py packages/nemo_platform_ext/src/nemo_platform_ext/quickstart/container.py packages/nemo_platform_ext/tests/quickstart/test_container.py - passed
• git diff --check - passed
• flox -q activate -- uv run pre-commit run -a - passed

Targeted Platform-Deploy validation:

• https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32973302926 targets this PR at 98787062c91c371d1b73203089e36b7c5e666dde using NVIDIA-NeMo/Platform-Deploy#209 at 44b7e20ee896ed4f2eebf62a0fafe52f93d3a277.

[mckornfield]

Started Platform-Deploy customizer-focused CI against this PR head (ebd8aa7): https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32911839430

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bfd374b6-0b3a-445c-be7a-01d829e4e7d5

📥 Commits

Reviewing files that changed from the base of the PR and between c4d9a6f and a314dbf.

📒 Files selected for processing (6)

• packages/nmp_testing/src/nmp/testing/e2e/docker.py
• packages/nmp_testing/tests/unit/test_e2e_docker_backend.py
• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/config.py
• plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py
• plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR adds packages/models to Docker workspaces and runtime images, expands workspace-slice tests, isolates transformer_engine in a GPU smoke test, centralizes Docker loopback URL rewriting, adds Docker network selection, and removes GPU resources from weighted model puller containers.

Changes

Models workspace integration

Layer / File(s)	Summary
Workspace registration and source wiring docker/rl/Dockerfile.platform-workspace, docker/rl/pyproject.workspace.toml, docker/unsloth/Dockerfile.platform-workspace, docker/unsloth/pyproject.workspace.toml	Docker build contexts copy packages/models. Both uv workspaces register the package and map the models dependency to it.
Runtime image installation docker/Dockerfile.nmp-customizer-tasks, docker/Dockerfile.nmp-unsloth-training, docker/automodel/Dockerfile.nmp-automodel-training, docker/rl/Dockerfile.nmp-rl-training	Runtime and training images install packages/models as an editable package.
Workspace test coverage tests/unit/test_docker_workspace_slices.py	Parameterized tests cover editable installation and workspace slices for filesets and models.

Deployment resolution

Layer / File(s)	Summary
Docker loopback URL rewriting services/core/models/src/nmp/core/models/controllers/backends/deployments_plugin/resolve.py, services/core/models/src/nmp/core/models/controllers/backends/deployments_plugin/compiler.py, services/core/models/tests/unit/controllers/backends/deployments_plugin/test_resolve.py, services/core/models/tests/unit/controllers/backends/deployments_plugin/test_compiler.py	Docker loopback URLs use the configured override or host.docker.internal. URL credentials, ports, and IPv6 formatting are preserved. Non-Docker and non-loopback URLs remain unchanged.
Puller resource mapping services/core/models/src/nmp/core/models/controllers/backends/deployments_plugin/compiler.py, services/core/models/tests/unit/controllers/backends/deployments_plugin/test_compiler.py	The weighted model puller no longer receives GPU requests or limits. GPU allocation remains on the model server.

Docker network resolution

Layer / File(s)	Summary
Executor network configuration plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/config.py, plugins/nemo-deployments/tests/unit/backends/docker/test_executor_config.py	DockerExecutorConfig.network uses NEMO_DEPLOYMENTS_DOCKER_NETWORK first and MODELS_DOCKER_NETWORK as a fallback.
Deployment network selection plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py, plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py	Deployment-specific network settings override the executor network. The executor network applies when no deployment network is configured.
E2E network wiring packages/nmp_testing/src/nmp/testing/e2e/docker.py, packages/nmp_testing/tests/unit/test_e2e_docker_backend.py	The E2E API container receives the test Docker network through NEMO_DEPLOYMENTS_DOCKER_NETWORK.

GPU smoke-test isolation

Layer / File(s)	Summary
Module resolution isolation tests/smoke_gpu/test_customizer_automodel.py	The PEFT torchao dispatch test masks transformer_engine and its submodules during module resolution.

Suggested reviewers: a2bondar

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 35 functions across 12 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: adding the models SDK package to customizer task images.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-customizer-models-alias/mck

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	35426/45218	78.3%	62.9%
Integration Tests	21113/42993	49.1%	22.3%

[mckornfield]

Started fresh Platform-Deploy customizer-focused CI after the smoke-test update, against PR head 349da23: https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32913311704

[mckornfield]

Pushed 0b87fcc10 to keep deployments-plugin weight pullers CPU-only; serving containers still request GPU. Local validation: uv run --frozen pytest services/core/models/tests/unit/controllers/backends/deployments_plugin/test_compiler.py services/core/models/tests/unit/controllers/backends/deployments_plugin/test_backend.py services/core/models/tests/unit/controllers/backends/deployments_plugin/test_status.py -q, uv run ruff check ..., and flox -q activate -- uv run pre-commit run -a all passed. Kicked off targeted Platform-Deploy customizer run: https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32917930741

[mckornfield]

Pushed c4d9a6ffab6ac3ba9dd6d726ccdd1237c9da3a2a with Docker deployment loopback rewriting for Files HF endpoints. This should address the Docker customizer vLLM puller failures where puller containers received a host-side localhost:<port> HF_ENDPOINT.

Started a new Platform-Deploy customizer-targeted run: https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32922161305

Note: I also cancelled https://github.com/NVIDIA-NeMo/Platform-Deploy/actions/runs/32922109380 after catching an invalid full SHA dispatch with the same short prefix.