diff --git a/README.md b/README.md index 2a69864..b513664 100644 --- a/README.md +++ b/README.md @@ -1067,7 +1067,8 @@ nicConfigurationOperator: rdmaPrefix: "rdma_r%rail%" # RDMA device name template (%rail% substituted per rail) netdevPrefix: "eth_r%rail%" # Network interface name template (%rail% substituted per rail) spectrumX: - # nicSelector.nicType is derived from the shared east-west PF deviceID. + # NicConfigurationTemplate nicType and pciAddresses are derived from the + # east-west PF inventory of each source group. overlay: none singlePlane: netdevPrefix: "eth_r%rail_id%" @@ -1331,7 +1332,7 @@ clusterConfig: During cluster discovery, the tool automatically identifies BlueField DPU devices (as opposed to SuperNICs or ConnectX NICs) by matching each device's `partNumber` against a known list of DPU product codes in [pkg/networkoperatorplugin/discovery/ns-product-ids](pkg/networkoperatorplugin/discovery/ns-product-ids). Devices matching a DPU product code are classified as **north-south** traffic (management/external), while all other devices are classified as **east-west** traffic (GPU interconnect). -North-south PFs are included in the saved cluster configuration for visibility, but are **automatically filtered out** during template rendering so that only east-west PFs appear in the generated manifests. Each east-west PF is assigned a sequential rail number (rail-0, rail-1, rail-2, ...) used for naming resources like SriovNetworkNodePolicy and IPPool entries. +North-south PFs are included in the saved cluster configuration for visibility, but are **automatically filtered out** during template rendering so that only east-west PFs appear in the generated manifests. Spectrum-X renders one `NicConfigurationTemplate` per source hardware group and selects the intersection of that group's east-west NIC type and PCI addresses. This prevents a north-south DPU from receiving Spectrum-X settings when it reports the same device ID as an east-west SuperNIC. Each east-west PF is assigned a sequential rail number (rail-0, rail-1, rail-2, ...) used for naming resources like SriovNetworkNodePolicy and IPPool entries. Example of mixed traffic types in the config: ```yaml diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index b79842b..1677325 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -245,11 +245,13 @@ spectrumX: | `spectrumX.hwplb` | Prefix block selected by `hwplb`; defaults RDMA to rail-only and NET to rail-plane names. | | `spectrumX.swplb` | Prefix block selected by `swplb`; defaults both device types to rail-plane names. | -Spectrum-X `NicConfigurationTemplate.spec.nicSelector.nicType` is not a -separate configuration input. Launch Kit derives it for each generated group -from `clusterConfig[].pfs[].deviceID` entries whose `traffic` is `east-west`. -North-south PFs are ignored; every selected east-west PF must have the same -non-empty device ID or generation fails. +Spectrum-X `NicConfigurationTemplate.spec.nicSelector` is not a separate +configuration input. Launch Kit renders one template per source hardware group +and derives both `nicType` and `pciAddresses` from `clusterConfig[].pfs[]` +entries whose `traffic` is `east-west`. The NIC Configuration Operator matches +the intersection of those fields, so a north-south DPU with the same device ID +as an east-west SuperNIC is not selected. Every selected east-west PF must have +the same non-empty device ID and a non-empty PCI address or generation fails. ## Profile diff --git a/docs/user/spectrum-x.md b/docs/user/spectrum-x.md index b2da3ab..16a3318 100644 --- a/docs/user/spectrum-x.md +++ b/docs/user/spectrum-x.md @@ -50,11 +50,21 @@ The platform is read from `clusterConfig[].gpuType`, with `machineType` as a fallback. `--for` presets participate in the same resolution before manifests are rendered. -The generated `NicConfigurationTemplate` also derives -`spec.nicSelector.nicType` from the target group's east-west PF device IDs. -There is no separate `spectrumX.nicType` setting. North-south PFs are ignored, -and generation requires every selected east-west PF to have the same non-empty -`deviceID`. +The generated `NicConfigurationTemplate` derives both +`spec.nicSelector.nicType` and `spec.nicSelector.pciAddresses` from the target +source group's east-west PFs. There is no separate `spectrumX.nicType` setting. +Each source hardware group gets its own template, and the operator matches the +intersection of NIC type and PCI address. A north-south DPU is therefore not +selected even when it reports the same device ID as an east-west SuperNIC. +Generation requires every selected east-west PF to have the same non-empty +`deviceID` and a non-empty `pciAddress`. + +Template names remain source-group based in both full and strict-subset +renders, so changing `--groups` does not rename retained templates. When +upgrading from a Launch Kit version that generated one merged, type-only +template, deploy preflight reports that legacy template as a stray resource. +Review the report, then use `--overwrite-existing` to delete the broad template +before applying the new PCI-scoped templates. B300 and GB300 support both `swplb` and `hwplb`, so platform type does not identify which load-balancing mechanism the fabric uses. l8k defaults to the diff --git a/pkg/networkoperatorplugin/grouping_test.go b/pkg/networkoperatorplugin/grouping_test.go index f9d6bc1..59b2493 100644 --- a/pkg/networkoperatorplugin/grouping_test.go +++ b/pkg/networkoperatorplugin/grouping_test.go @@ -238,9 +238,10 @@ func TestSpectrumXGrouping(t *testing.T) { // north-south DPU PCIs. Merge keys on (gpuType, east-west rail count) // — both match — and hasRailPciConflict only looks at east-west PFs, so // no conflict. The merged bucket emits one rail-pool config and one - // CIDRPool per bucket. NicInterfaceNameTemplate is ScopePerSource — - // always one per source group regardless of whether PCIs agree, so each - // node gets its own udev-rule template even when bodies happen to match. + // CIDRPool per bucket. NicInterfaceNameTemplate and + // NicConfigurationTemplate are ScopePerSource — always one per source + // group regardless of whether PCIs agree, so each node gets selectors + // built from its own hardware inventory. name: "same east-west PCIs different north-south: merges fully, per-source name templates", configFile: "same-ew-different-ns.yaml", planes: 1, @@ -252,7 +253,8 @@ func TestSpectrumXGrouping(t *testing.T) { }, wantCidrPools: []string{"60-cidrpool-gpu-model-x.yaml"}, wantNicCfgTmpls: []string{ - "30-nicconfigurationtemplate-gpu-model-x.yaml", + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", }, wantNicType: "1023", }, @@ -276,7 +278,9 @@ func TestSpectrumXGrouping(t *testing.T) { }, wantCidrPools: []string{"60-cidrpool-gpu-model-y.yaml"}, wantNicCfgTmpls: []string{ - "30-nicconfigurationtemplate-gpu-model-y.yaml", + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", + "30-nicconfigurationtemplate-group-2.yaml", }, wantNicType: "a2dc", }, @@ -306,7 +310,8 @@ func TestSpectrumXGrouping(t *testing.T) { "60-cidrpool-group-2.yaml", }, wantNicCfgTmpls: []string{ - "30-nicconfigurationtemplate-gpu-model-y.yaml", + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", "30-nicconfigurationtemplate-group-2.yaml", }, wantNicType: "a2dc", @@ -472,7 +477,7 @@ func TestSpectrumXRA23RendersProfileConfigMap(t *testing.T) { require.Contains(t, cm, `docaCCVersion: "example"`) require.NotContains(t, cm, "\r") - nct := rendered["30-nicconfigurationtemplate-gpu-model-y.yaml"] + nct := rendered["30-nicconfigurationtemplate-group-0.yaml"] require.Contains(t, nct, `version: "site-ra23-profile"`) } @@ -592,9 +597,10 @@ func TestPCIeRoot(t *testing.T) { // - ScopeBucketed (CIDRPool): one CR per bucket, named with bucket id // - ScopeAggregate (DaemonSet): one CR per bucket; nodeAffinity // emits an `In` list of source machine labels -// - ScopeSimpleSelect (SpectrumXRailPoolConfig, NicConfigurationTemplate): +// - ScopeSimpleSelect (SpectrumXRailPoolConfig): // N CRs per bucket — one per source group -// - ScopePerSource (NicInterfaceNameTemplate): N CRs per bucket +// - ScopePerSource (NicInterfaceNameTemplate, NicConfigurationTemplate): +// N CRs per bucket func TestSpectrumXModeBSubsetFilter(t *testing.T) { ctrllog.SetLogger(zap.New(zap.UseDevMode(true))) cfg, err := config.LoadFullConfig( @@ -617,6 +623,9 @@ func TestSpectrumXModeBSubsetFilter(t *testing.T) { }, } + fullRendered, err := (&NetworkOperatorPlugin{}).GenerateProfileDeploymentFiles(loadSpectrumXProfile(t), cfg) + require.NoError(t, err) + plugin := &NetworkOperatorPlugin{Groups: []string{"group-0", "group-1"}} rendered, err := plugin.GenerateProfileDeploymentFiles(loadSpectrumXProfile(t), cfg) require.NoError(t, err) @@ -642,6 +651,24 @@ func TestSpectrumXModeBSubsetFilter(t *testing.T) { "25-nicinterfacenametemplate-group-1.yaml", }, ninTmpls) + // NicConfigurationTemplate (PerSource): source-based names are stable + // between full Mode A and strict-subset Mode B renders. A retained source + // must never be renamed just because the selection mode changed. + nicConfigTmpls := fileNamesMatching(rendered, "30-nicconfigurationtemplate") + require.Equal(t, []string{ + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", + }, nicConfigTmpls) + require.Equal(t, []string{ + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", + "30-nicconfigurationtemplate-group-2.yaml", + }, fileNamesMatching(fullRendered, "30-nicconfigurationtemplate")) + for _, fileName := range nicConfigTmpls { + require.Contains(t, fullRendered, fileName, + "retained NicConfigurationTemplate %s must keep its identity across render modes", fileName) + } + // Example DaemonSet (Aggregate): one CR per bucket, with In selector // listing source machine labels. dsFiles := fileNamesMatching(rendered, "90-example-daemonset") diff --git a/pkg/networkoperatorplugin/nic_configuration_template_test.go b/pkg/networkoperatorplugin/nic_configuration_template_test.go new file mode 100644 index 0000000..b057b9b --- /dev/null +++ b/pkg/networkoperatorplugin/nic_configuration_template_test.go @@ -0,0 +1,154 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// SPDX-License-Identifier: Apache-2.0 + +package networkoperatorplugin + +import ( + "fmt" + "path/filepath" + "testing" + + "github.com/nvidia/k8s-launch-kit/pkg/config" + "github.com/stretchr/testify/require" + ctrllog "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/yaml" +) + +func TestSpectrumXPCIAddresses(t *testing.T) { + group := &config.ClusterConfig{ + Identifier: "machine-a", + PFs: []config.PFConfig{ + {PciAddress: " 000A:19:00.0 ", Traffic: "east-west"}, + {PciAddress: "000a:19:00.0", Traffic: "east-west"}, + {PciAddress: "0000:03:00.0", Traffic: "north-south"}, + }, + } + + addresses, err := spectrumXPCIAddresses(group) + require.NoError(t, err) + require.Equal(t, []string{"000a:19:00.0"}, addresses, + "the selector must normalize, deduplicate, and exclude north-south PCI addresses") + + group.PFs[0].PciAddress = "" + group.PFs = group.PFs[:1] + _, err = spectrumXPCIAddresses(group) + require.ErrorContains(t, err, `group "machine-a" has an east-west PF without a pciAddress`) +} + +func TestSpectrumXNicConfigurationTemplateExcludesSameTypeNorthSouthDevices(t *testing.T) { + tests := []struct { + name string + profileDir string + spcxVersion string + }{ + {name: "RA2.1", profileDir: "spectrum-x-ra2.1", spcxVersion: "RA2.1"}, + {name: "RA2.2", profileDir: "spectrum-x-ra2.2", spcxVersion: "RA2.2"}, + {name: "RA2.3", profileDir: "spectrum-x", spcxVersion: "RA2.3"}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctrllog.SetLogger(zap.New(zap.UseDevMode(true))) + cfg, err := config.LoadFullConfig( + filepath.Join("testdata", "grouping", "same-ew-different-ns.yaml"), + ctrllog.Log, + ) + require.NoError(t, err) + + for groupIndex := range cfg.ClusterConfig { + machineLabel := "machine-a-gpu-model-x" + if groupIndex == 1 { + machineLabel = "machine-b-gpu-model-x" + } + cfg.ClusterConfig[groupIndex].NodeSelector = map[string]string{ + config.MachineLabelKey: machineLabel, + } + for pfIndex := range cfg.ClusterConfig[groupIndex].PFs { + // Model the reported failure: both the east-west SuperNIC and + // north-south DPU expose the same BlueField-3 device ID. + cfg.ClusterConfig[groupIndex].PFs[pfIndex].DeviceID = "a2dc" + } + } + cfg.Profile = &config.Profile{ + Fabric: "ethernet", + Deployment: "sriov", + Multirail: true, + SpectrumX: &config.ProfileSpectrumX{ + Enable: true, + SPCXVersion: test.spcxVersion, + MultiplaneMode: "none", + NumberOfPlanes: 1, + ConfigMapName: "site-ra23-profile", + }, + } + + profile := loadProfileFromDir(t, test.profileDir) + var nicConfigurationTemplate string + for _, templatePath := range profile.Templates { + if filepath.Base(templatePath) == "30-nicconfigurationtemplate.yaml" { + nicConfigurationTemplate = templatePath + break + } + } + require.NotEmpty(t, nicConfigurationTemplate) + profile.Templates = []string{nicConfigurationTemplate} + + rendered, err := (&NetworkOperatorPlugin{}).GenerateProfileDeploymentFiles(profile, cfg) + require.NoError(t, err) + require.Equal(t, []string{ + "30-nicconfigurationtemplate-group-0.yaml", + "30-nicconfigurationtemplate-group-1.yaml", + }, fileNamesMatching(rendered, "30-nicconfigurationtemplate"), + "a merged hardware bucket must still emit one PCI-scoped template per source group") + + for groupIndex, group := range cfg.ClusterConfig { + fileName := fmt.Sprintf("30-nicconfigurationtemplate-group-%d.yaml", groupIndex) + manifest := rendered[fileName] + require.NotEmpty(t, manifest) + + var object struct { + Spec struct { + NodeSelector map[string]string `yaml:"nodeSelector"` + NicSelector struct { + NicType string `yaml:"nicType"` + PCIAddresses []string `yaml:"pciAddresses"` + } `yaml:"nicSelector"` + } `yaml:"spec"` + } + require.NoError(t, yaml.Unmarshal([]byte(manifest), &object)) + require.Equal(t, "a2dc", object.Spec.NicSelector.NicType) + require.Equal(t, group.NodeSelector, object.Spec.NodeSelector) + + var eastWestPCIs []string + var northSouthPCIs []string + for _, pf := range group.PFs { + switch pf.Traffic { + case "east-west": + eastWestPCIs = append(eastWestPCIs, pf.PciAddress) + case "north-south": + northSouthPCIs = append(northSouthPCIs, pf.PciAddress) + } + } + require.Equal(t, eastWestPCIs, object.Spec.NicSelector.PCIAddresses) + for _, pciAddress := range northSouthPCIs { + require.NotContains(t, object.Spec.NicSelector.PCIAddresses, pciAddress, + "north-south PCI address %s must not be selected", pciAddress) + } + } + }) + } +} diff --git a/pkg/networkoperatorplugin/scopes.go b/pkg/networkoperatorplugin/scopes.go index 04cc6f4..22c4dfe 100644 --- a/pkg/networkoperatorplugin/scopes.go +++ b/pkg/networkoperatorplugin/scopes.go @@ -46,13 +46,14 @@ const ( // poolName-equivalent) so companion ScopeBucketed / ScopeAggregate // CRs can reference one stable name. Examples: NicNodePolicy, // SriovNetworkNodePolicy, SriovNetworkPoolConfig, - // SpectrumXRailPoolConfig, NicConfigurationTemplate. + // SpectrumXRailPoolConfig. ScopeSimpleSelect // ScopePerSource: always one CR per source group, regardless of // merge. Used for Kinds whose body depends on machine-specific data // (e.g. PCI addresses) that cannot meaningfully merge across - // source machineTypes. Example: NicInterfaceNameTemplate. + // source machineTypes. Examples: NicInterfaceNameTemplate, + // NicConfigurationTemplate. ScopePerSource ) @@ -106,15 +107,15 @@ var crScopeByKind = map[string]CRScope{ // once per source under Mode B. Shared resources (resourceName, // poolName, etc.) reference the merged bucket identifier so all // per-source CRs register the same kubelet resource. - "NicNodePolicy": ScopeSimpleSelect, - "SriovNetworkNodePolicy": ScopeSimpleSelect, - "SriovNetworkPoolConfig": ScopeSimpleSelect, - "SpectrumXRailPoolConfig": ScopeSimpleSelect, - "NicConfigurationTemplate": ScopeSimpleSelect, + "NicNodePolicy": ScopeSimpleSelect, + "SriovNetworkNodePolicy": ScopeSimpleSelect, + "SriovNetworkPoolConfig": ScopeSimpleSelect, + "SpectrumXRailPoolConfig": ScopeSimpleSelect, // Per-source — body depends on machine-specific data like PCI // addresses; never merged across source groups. "NicInterfaceNameTemplate": ScopePerSource, + "NicConfigurationTemplate": ScopePerSource, } // ScopeForKind returns the registered scope for a Kubernetes Kind. diff --git a/pkg/networkoperatorplugin/templates.go b/pkg/networkoperatorplugin/templates.go index 891f881..da3629a 100644 --- a/pkg/networkoperatorplugin/templates.go +++ b/pkg/networkoperatorplugin/templates.go @@ -134,6 +134,7 @@ var templateFuncs = template.FuncMap{ return netdevPrefix }, "spectrumXNicType": spectrumXNicType, + "spectrumXPCIAddresses": spectrumXPCIAddresses, "spectrumXProfileConfigRequired": config.SpectrumXProfileConfigRequired, "spectrumXCIDRPools": func(root any, clusterConfig *config.ClusterConfig) ([]spectrumxaddressing.CIDRPool, error) { var cfg *config.LaunchKitConfig @@ -228,6 +229,34 @@ func spectrumXNicType(group *config.ClusterConfig) (string, error) { return deviceID, nil } +// spectrumXPCIAddresses returns the exact east-west PCI inventory used to +// constrain a Spectrum-X NicConfigurationTemplate. nicType alone cannot +// distinguish a BlueField DPU from a SuperNIC when both devices report the +// same PCI device ID, so the generated selector must include both fields. +func spectrumXPCIAddresses(group *config.ClusterConfig) ([]string, error) { + if group == nil { + return nil, fmt.Errorf("Spectrum-X PCI selector requires a clusterConfig group") + } + + addresses := make([]string, 0, len(group.PFs)) + seen := make(map[string]struct{}, len(group.PFs)) + for _, pf := range pfutil.FilterEastWestPFs(group.PFs) { + address := strings.ToLower(strings.TrimSpace(pf.PciAddress)) + if address == "" { + return nil, fmt.Errorf("group %q has an east-west PF without a pciAddress", group.Identifier) + } + if _, exists := seen[address]; exists { + continue + } + seen[address] = struct{}{} + addresses = append(addresses, address) + } + if len(addresses) == 0 { + return nil, fmt.Errorf("group %q has no east-west PFs", group.Identifier) + } + return addresses, nil +} + func secondaryNetworkMetaPlugins(profile *config.Profile) string { if profile == nil || (profile.SpectrumX != nil && profile.SpectrumX.Enable) { return "" diff --git a/profiles/spectrum-x-ra2.1/30-nicconfigurationtemplate.yaml b/profiles/spectrum-x-ra2.1/30-nicconfigurationtemplate.yaml index 7c4a958..eb5cd19 100644 --- a/profiles/spectrum-x-ra2.1/30-nicconfigurationtemplate.yaml +++ b/profiles/spectrum-x-ra2.1/30-nicconfigurationtemplate.yaml @@ -12,6 +12,7 @@ spec: {{- end }} nicSelector: nicType: "{{spectrumXNicType .ClusterConfig}}" + pciAddresses: [{{- $first := true }}{{- range $addr := spectrumXPCIAddresses .ClusterConfig }}{{- if not $first }}, {{ end }}{{- $first = false }}"{{$addr}}"{{- end }}] template: numVfs: 1 linkType: Ethernet diff --git a/profiles/spectrum-x-ra2.1/README.md b/profiles/spectrum-x-ra2.1/README.md index b64dec6..97ea60e 100644 --- a/profiles/spectrum-x-ra2.1/README.md +++ b/profiles/spectrum-x-ra2.1/README.md @@ -61,6 +61,11 @@ Same set as the RA2.2 profile: `none`, `swplb`, `hwplb`. | `80-spectrumxrailpoolconfig.yaml` | SpectrumXRailPoolConfig (v1alpha1) | matches 50- | | `90-example-daemonset.yaml` | DaemonSet (example workload) | 1 per group | +Each `NicConfigurationTemplate` is rendered per source hardware group and +selects both the NIC type and PCI addresses derived from east-west PFs. The +intersection prevents a north-south DPU from receiving Spectrum-X settings +when it reports the same device ID as an east-west SuperNIC. + `60-cidrpool.yaml` derives complete IPv4 or IPv6 CIDRPools from the topology file. IPv4 uses a `/31` per node with gateway index `0`. IPv6 uses the standard Spectrum-X `/64` per-node layout with leaf gateway `::2`, gateway index `2`, diff --git a/profiles/spectrum-x-ra2.2/30-nicconfigurationtemplate.yaml b/profiles/spectrum-x-ra2.2/30-nicconfigurationtemplate.yaml index 7c4a958..eb5cd19 100644 --- a/profiles/spectrum-x-ra2.2/30-nicconfigurationtemplate.yaml +++ b/profiles/spectrum-x-ra2.2/30-nicconfigurationtemplate.yaml @@ -12,6 +12,7 @@ spec: {{- end }} nicSelector: nicType: "{{spectrumXNicType .ClusterConfig}}" + pciAddresses: [{{- $first := true }}{{- range $addr := spectrumXPCIAddresses .ClusterConfig }}{{- if not $first }}, {{ end }}{{- $first = false }}"{{$addr}}"{{- end }}] template: numVfs: 1 linkType: Ethernet diff --git a/profiles/spectrum-x-ra2.2/README.md b/profiles/spectrum-x-ra2.2/README.md index 159fdac..3efd859 100644 --- a/profiles/spectrum-x-ra2.2/README.md +++ b/profiles/spectrum-x-ra2.2/README.md @@ -31,10 +31,11 @@ nodeCapabilities: ### NIC Configuration -- **nicSelector.nicType**: Derived in each generated - `NicConfigurationTemplate` from the device ID shared by the selected - east-west PFs. North-south PFs are ignored; missing or mixed east-west - device IDs fail generation. +- **nicSelector.nicType / pciAddresses**: Derived in each per-source + `NicConfigurationTemplate` from the selected east-west PFs. The combined + selector excludes a north-south DPU even when it has the same device ID as + the SuperNIC. Missing or mixed east-west device IDs and missing east-west PCI + addresses fail generation. - **firmwareVersion**: Spectrum-X firmware version (e.g., `"RA2.2"`) - **multiplaneMode**: Multiplane configuration - `none`: Single plane @@ -101,6 +102,8 @@ The profile generates the following Kubernetes Custom Resources: 3. **NicConfigurationTemplate** (`30-nicconfigurationtemplate.yaml`) - Configures Spectrum-X optimized firmware settings (RA2.2). + - Rendered once per source hardware group with east-west-only NIC type and + PCI-address selectors. 4. **CIDRPool** (`60-cidrpool.yaml`) - One topology-derived IPv4 or IPv6 pool per rail (non-swplb) or per diff --git a/profiles/spectrum-x/30-nicconfigurationtemplate.yaml b/profiles/spectrum-x/30-nicconfigurationtemplate.yaml index b2dac09..c11da01 100644 --- a/profiles/spectrum-x/30-nicconfigurationtemplate.yaml +++ b/profiles/spectrum-x/30-nicconfigurationtemplate.yaml @@ -12,6 +12,7 @@ spec: {{- end }} nicSelector: nicType: "{{spectrumXNicType .ClusterConfig}}" + pciAddresses: [{{- $first := true }}{{- range $addr := spectrumXPCIAddresses .ClusterConfig }}{{- if not $first }}, {{ end }}{{- $first = false }}"{{$addr}}"{{- end }}] template: numVfs: 1 linkType: Ethernet diff --git a/profiles/spectrum-x/README.md b/profiles/spectrum-x/README.md index 8a0a98e..5d98da9 100644 --- a/profiles/spectrum-x/README.md +++ b/profiles/spectrum-x/README.md @@ -31,10 +31,11 @@ nodeCapabilities: ### NIC Configuration -- **nicSelector.nicType**: Derived in each generated - `NicConfigurationTemplate` from the device ID shared by the selected - east-west PFs. North-south PFs are ignored; missing or mixed east-west - device IDs fail generation. +- **nicSelector.nicType / pciAddresses**: Derived in each per-source + `NicConfigurationTemplate` from the selected east-west PFs. The combined + selector excludes a north-south DPU even when it has the same device ID as + the SuperNIC. Missing or mixed east-west device IDs and missing east-west PCI + addresses fail generation. - **firmwareVersion**: Spectrum-X firmware version (e.g., `"RA2.3"`) - **multiplaneMode**: Multiplane configuration - `none`: Single plane @@ -114,6 +115,8 @@ The profile generates the following Kubernetes Custom Resources: 4. **NicConfigurationTemplate** (`30-nicconfigurationtemplate.yaml`) - Configures Spectrum-X optimized firmware settings. For RA2.3, `spectrumXOptimized.version` is the generated profile ConfigMap name. + - Rendered once per source hardware group with east-west-only NIC type and + PCI-address selectors. 5. **CIDRPool** (`60-cidrpool.yaml`) - One topology-derived IPv4 or IPv6 pool per rail (non-swplb) or per diff --git a/skills/k8s-launch-kit-config/references/config-reference.md b/skills/k8s-launch-kit-config/references/config-reference.md index f518ac1..06a9e53 100644 --- a/skills/k8s-launch-kit-config/references/config-reference.md +++ b/skills/k8s-launch-kit-config/references/config-reference.md @@ -278,9 +278,9 @@ spectrumX: netdevPrefix: "eth_r%rail_id%_p%plane_id%" rdmaPrefix: "roce_r%rail_id%_p%plane_id%" -# Generated Spectrum-X NicConfigurationTemplate resources derive -# spec.nicSelector.nicType from the non-empty deviceID shared by every -# selected east-west PF. North-south PFs are ignored. +# Generated Spectrum-X NicConfigurationTemplate resources are per source group +# and derive spec.nicSelector.nicType plus pciAddresses from selected east-west +# PFs. North-south PFs are ignored, including same-device-ID DPUs. # ============================================================================ # Profile Selection diff --git a/skills/k8s-launch-kit-generate/SKILL.md b/skills/k8s-launch-kit-generate/SKILL.md index 7ab3746..84db568 100644 --- a/skills/k8s-launch-kit-generate/SKILL.md +++ b/skills/k8s-launch-kit-generate/SKILL.md @@ -170,10 +170,12 @@ win when a one-off override is needed. B300/GB300 platform type does not distinguish SWPLB from HWPLB. l8k defaults to SWPLB; use an explicit HWPLB override when the site topology requires it. Read `references/spectrum-x-modes.md`. -- Spectrum-X `NicConfigurationTemplate.spec.nicSelector.nicType` is derived - from the device ID shared by the selected east-west PFs; do not ask users to - configure `spectrumX.nicType`. Missing or mixed east-west device IDs are a - generation error. +- Spectrum-X renders one `NicConfigurationTemplate` per source group and + derives `spec.nicSelector.nicType` plus `pciAddresses` from that group's + east-west PFs. The intersection prevents same-device-ID north-south DPUs from + matching. Do not ask users to configure `spectrumX.nicType`; missing or mixed + east-west device IDs and missing east-west PCI addresses are generation + errors. - NVIDIA AIR topology support requires the documented one-based node/interface naming contract (`su`, `h`, `leaf-p

`, `r`, `railp

`, and `pod` for 3-tier). See `docs/user/spectrum-x.md` in the l8k repository. diff --git a/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md b/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md index ae6bbc9..88b67c4 100644 --- a/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md +++ b/skills/k8s-launch-kit-generate/references/spectrum-x-modes.md @@ -13,11 +13,12 @@ All Spectrum-X deployments require: - `multirail=true` - `spectrumX.enable=true` -Launch Kit derives each generated -`NicConfigurationTemplate.spec.nicSelector.nicType` from the selected -east-west PF `deviceID` values. The selector is not configured under -`spectrumX`; north-south PFs are ignored, and the selected east-west IDs must -be non-empty and unanimous. +Launch Kit renders one `NicConfigurationTemplate` per source hardware group and +derives `spec.nicSelector.nicType` plus `pciAddresses` from the selected +east-west PF inventory. The selector is not configured under `spectrumX`; +north-south PFs are ignored, and the east-west IDs must be non-empty and +unanimous while every east-west PF must have a PCI address. Combining type and +PCI selectors prevents a same-device-ID DPU from receiving the SuperNIC config. ## Mode: none diff --git a/skills/k8s-network-engineer/references/config-schema.md b/skills/k8s-network-engineer/references/config-schema.md index d795273..9eaee49 100644 --- a/skills/k8s-network-engineer/references/config-schema.md +++ b/skills/k8s-network-engineer/references/config-schema.md @@ -197,10 +197,11 @@ Spectrum-X specific NIC and overlay configuration. | `hwplb` | object | rail-only RDMA, rail-plane NET | Prefix block selected by `hwplb` | | `swplb` | object | rail-plane RDMA and NET | Prefix block selected by `swplb` | -Each mode object contains user-overridable `rdmaPrefix` and `netdevPrefix` strings. -The generated `NicConfigurationTemplate.spec.nicSelector.nicType` comes from -the unanimous, non-empty `deviceID` of the selected east-west PFs; -north-south PFs do not participate. +Each mode object contains user-overridable `rdmaPrefix` and `netdevPrefix` +strings. Launch Kit renders one `NicConfigurationTemplate` per source group; +its `spec.nicSelector.nicType` and `pciAddresses` come from that group's +east-west PF inventory. North-south PFs do not participate, so a DPU with the +same device ID as a SuperNIC is excluded by the PCI selector. ## profile diff --git a/skills/k8s-network-engineer/references/spectrum-x-guide.md b/skills/k8s-network-engineer/references/spectrum-x-guide.md index 7ac21eb..34f8e02 100644 --- a/skills/k8s-network-engineer/references/spectrum-x-guide.md +++ b/skills/k8s-network-engineer/references/spectrum-x-guide.md @@ -139,7 +139,7 @@ The Spectrum-X profile generates these Kubernetes Custom Resources: | Order | CRD | File | Purpose | |-------|----------------------------|-----------------------------------|-----------------------------------------------| | 1 | NicClusterPolicy | 10-nicclusterpolicy.yaml | Network Operator, NV-IPAM, Spectrum-X Operator| -| 2 | NicConfigurationTemplate | 30-nicconfigurationtemplate.yaml | Spectrum-X firmware settings per NIC type | +| 2 | NicConfigurationTemplate | 30-nicconfigurationtemplate.yaml | Per-source Spectrum-X settings for east-west NIC PCIs | | 3 | NICInterfaceNameTemplate | 35-nicinterfacenametemplate.yaml | Interface naming for multi-rail topology | | 4 | SriovNetworkPoolConfig | 40-sriovnetworkpoolconfig.yaml | RDMA mode and OVS hardware offload per rail | | 5 | SriovNetworkNodePolicy | 50-sriovnetworknodepolicy.yaml | SR-IOV VF policies per rail (per plane for swplb)|