Skip to content

fix: third-party RDMA unload env var misdescribed as list #219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andrewwhitecdw wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: third-party RDMA unload env var misdescribed as list #219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion skills/k8s-launch-kit-config/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ networkNamespaces: ["my-namespace"]
- Use `l8k schema` to discover the Network Operator release keys supported by
the installed l8k version.
- `nvIpam` subnets are auto-generated if not specified — one per rail using non-routable ranges.
- `docaDriver.unloadThirdPartyRDMAModules: true` auto-populates `UNLOAD_THIRD_PARTY_RDMA_MODULES` from discovered OFED-dependent modules.
- `docaDriver.unloadThirdPartyRDMAModules: true` sets the `UNLOAD_THIRD_PARTY_RDMA_MODULES` environment variable to `"true"` (a boolean flag) in generated manifests. Discovered OFED-dependent modules are saved as `thirdPartyRDMAModules` for visibility and warnings, not passed to that environment variable.
- For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
- Updating an existing release to the generated requestor-mode Helm values requires `--overwrite-existing`.

Expand Down
2 changes: 1 addition & 1 deletion skills/k8s-launch-kit-config/references/config-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ docaDriver:
enableNFSRDMA: false

# bool | default: false, auto-enabled by discovery when third-party RDMA modules are found
# When true, adds UNLOAD_THIRD_PARTY_RDMA_MODULES env var to the ofedDriver container.
# When true, sets UNLOAD_THIRD_PARTY_RDMA_MODULES env var to "true" in the ofedDriver container.
# Third-party RDMA modules are blacklisted and unloaded before OFED driver reload.
# Discovery automatically sets this to true when third-party RDMA modules are detected.
unloadThirdPartyRDMAModules: true
Expand Down
8 changes: 5 additions & 3 deletions skills/k8s-launch-kit-discover/references/discovery-internals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,9 +150,11 @@ directory to find modules that depend on OFED. Common dependents include:
- `nvidia_peermem` -- modern GPUDirect RDMA peer memory module.
- `mlx5_vdpa` -- vDPA offload module.

The discovered dependents are saved per group as `thirdPartyRDMAModules`. These are used
during manifest generation to configure the NicClusterPolicy's `ofedDriver` section
with the correct secondary module list.
The discovered dependents are saved per group as `thirdPartyRDMAModules` for visibility
and warnings. When `unloadThirdPartyRDMAModules` is true, manifest generation sets the
`UNLOAD_THIRD_PARTY_RDMA_MODULES` environment variable to `"true"` (a boolean flag) in
the NicClusterPolicy's `ofedDriver` section; the module names themselves are not passed
to that variable.

---

Expand Down
4 changes: 2 additions & 2 deletions skills/k8s-launch-kit-troubleshoot/references/common-failures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ failures or dependency errors.
1. Check `thirdPartyRDMAModules` in your cluster config -- these modules must be unloaded.
2. Set `docaDriver.unloadThirdPartyRDMAModules: true` in your l8k config.
3. Re-run discovery (`--discover-cluster-config`) to refresh the dependent modules list.
4. Redeploy. The generated NicClusterPolicy will include `UNLOAD_THIRD_PARTY_RDMA_MODULES` env var
in the ofedDriver section.
4. Redeploy. The generated NicClusterPolicy will include `UNLOAD_THIRD_PARTY_RDMA_MODULES: "true"`
(a boolean flag) in the ofedDriver section.
5. If the issue persists, exec into the OFED pod and check `/sys/module/*/holders/` to
find additional unlisted dependents.

Expand Down
8 changes: 5 additions & 3 deletions skills/k8s-network-engineer/references/config-schema.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@ Controls DOCA/OFED driver deployment in the NicClusterPolicy.
| `enableNFSRDMA` | bool | `false` | Enable NFS over RDMA support |
| `unloadThirdPartyRDMAModules` | bool | `true` | Unload kernel modules that depend on MLX/OFED drivers |

When `unloadThirdPartyRDMAModules` is true and dependent modules are discovered,
the generated NicClusterPolicy includes `UNLOAD_THIRD_PARTY_RDMA_MODULES` env var
(space-separated module names) in the ofedDriver section.
When `unloadThirdPartyRDMAModules` is true, the generated NicClusterPolicy
sets the `UNLOAD_THIRD_PARTY_RDMA_MODULES` environment variable to `"true"`
(a boolean flag) in the ofedDriver section. Dependent modules discovered on each
group are saved as `thirdPartyRDMAModules` for visibility and warnings, but are
not passed as module names to that environment variable.
Comment on lines +35 to +39

@greptile-apps greptile-apps Bot Aug 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Synchronize the related skill references

This correction now conflicts with skills/k8s-launch-kit-config/SKILL.md, which says discovered modules populate this environment variable, and skills/k8s-launch-kit-discover/references/discovery-internals.md, which says they configure the OFED secondary-module list. Agents consulting these complementary references receive contradictory configuration and troubleshooting guidance, so update those references alongside this correction.

Knowledge Base Used: Agent Skills for k8s-launch-kit

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@andrewwhitecdw andrewwhitecdw Aug 18, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed — config-schema.md now describes UNLOAD_THIRD_PARTY_RDMA_MODULES as a boolean flag and notes that discovered modules are saved as thirdPartyRDMAModules for visibility, consistent with k8s-launch-kit-config/SKILL.md and k8s-launch-kit-discover/references/discovery-internals.md.


## maintenance

Expand Down