Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1388,15 +1388,21 @@ During discovery, each node group's `machineType` and `gpuType` are populated fr
Values are sanitized to match the GPU operator label format (spaces replaced with dashes). If either probe fails (e.g., `nvidia-smi` not installed, DMI not readable), the corresponding field is left empty and discovery continues without error.

When both values are available, discovery derives the group `identifier` from
`<machineType>-<gpuType>`. Generated identifiers are limited to 40 bytes; long
values keep a readable prefix plus an 8-character deterministic hash. This
keeps resource names and label values that append the identifier below their
Kubernetes size limits without making similar hardware identities collide.
`<machineType>-<gpuType>`, lowercases it, and removes complete `NVIDIA`
segments. Common machine segments are shortened in the same pass:
`ThinkSystem` becomes `ts` and `PowerEdge` becomes `pe`. The
`nvidia.kubernetes-launch-kit.machine` node label uses that same identifier;
the separate `.gpu` label retains the discovered GPU value.
Generated identifiers are limited to 30 bytes. Long identities keep balanced
machine/GPU prefixes plus a 6-character deterministic hash, with unused prefix
space reassigned when one component is short. This keeps resource names and
label values that append the identifier below their Kubernetes size limits
without making similar hardware identities collide.

Example of discovered hardware types in the config:
```yaml
clusterConfig:
- identifier: group-0
- identifier: ts-sr680a-v3-h100-nvl
machineType: ThinkSystem-SR680a-V3
gpuType: NVIDIA-H100-NVL
workerNodes:
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ l8k generate \
# An explicit source-group subset
l8k generate \
--user-config ./cluster-config.yaml \
--groups poweredge-xe9680-h200,thinksystem-sr680a-v3-h200 \
--groups pe-xe9680-h200,ts-sr680a-v3-h200 \
--save-deployment-files ./deployment-stage
```

Expand Down
8 changes: 4 additions & 4 deletions docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ Each `clusterConfig` entry represents one source group:

```yaml
clusterConfig:
- identifier: poweredge-xe9680-h200
- identifier: pe-xe9680-h200
machineType: PowerEdge-XE9680
gpuType: NVIDIA-H200
capabilities:
Expand All @@ -307,7 +307,7 @@ clusterConfig:
rdma: true
ib: false
nodeSelector:
nvidia.kubernetes-launch-kit.machine: PowerEdge-XE9680-NVIDIA-H200
nvidia.kubernetes-launch-kit.machine: pe-xe9680-h200
pfs:
- deviceID: a2dc
pciAddress: 0000:1a:00.0
Expand All @@ -321,14 +321,14 @@ Fresh discovery may merge compatible source groups during generation, but it kee

| Field | Meaning |
| --- | --- |
| `identifier` | Lowercase resource-name form of machine/GPU identity, bounded to 40 bytes with a deterministic hash suffix when needed, or `group-N` fallback. |
| `identifier` | Lowercase resource-name form of machine/GPU identity with complete `NVIDIA` segments removed and common machine segments shortened (`ThinkSystem` → `ts`, `PowerEdge` → `pe`), bounded to 30 bytes with balanced machine/GPU prefixes and a 6-character deterministic hash when needed, or `group-N` fallback. The Launch Kit machine node label uses the same value. |
| `machineType` / `gpuType` | Discovered hardware identity. |
| `linkType` | Per-group `Ethernet` or `InfiniBand` result when fabric probes agree. |
| `presetApplied` | An exact topology preset was applied. |
| `presetDeviation` | PF count, PCI address, or device ID drift from a matched preset. |
| `capabilities.nodes` | Group-level SR-IOV, RDMA, and InfiniBand capability flags. |
| `workerNodes` | Kubernetes node names in the source group. |
| `nodeSelector` | Deployment selector, normally the Launch Kit machine label. |
| `nodeSelector` | Deployment selector, normally the Launch Kit machine label whose value matches `identifier`. |
| `storageModules` / `thirdPartyRDMAModules` | Optional site-supplied dependent module lists. |
| `pfs` | Physical-function inventory. |

Expand Down
18 changes: 10 additions & 8 deletions docs/user/discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,16 +90,18 @@ Discovery also writes up to two Launch Kit-owned labels:

| Label | Purpose |
| --- | --- |
| `nvidia.kubernetes-launch-kit.machine: <machineType>-<gpuType>` | Selects one source hardware group. |
| `nvidia.kubernetes-launch-kit.machine: <identifier>` | Selects one source hardware group using the same value persisted in `clusterConfig[].identifier`. |
| `nvidia.kubernetes-launch-kit.gpu: <gpuType>` | Selects compatible source groups that share a GPU type. |

Label values preserve the discovered case and normalize spaces for Kubernetes.
The combined machine/GPU label and its lowercase group `identifier` are bounded
to 40 bytes; long values keep a readable prefix plus an 8-character
deterministic hash. GPU-only labels use the Kubernetes 63-byte limit and the
same shortening rule. If the machine or GPU type cannot be resolved, discovery
uses a fallback `group-N` identifier and writes only the labels it can
construct.
The machine label and group `identifier` share one lowercase, vendor-free value
bounded to 30 bytes. The same normalization shortens common machine segments:
`ThinkSystem` becomes `ts` and `PowerEdge` becomes `pe`. Long identities keep
balanced machine/GPU prefixes plus a 6-character deterministic hash; unused
prefix space moves to the longer component. GPU-only labels preserve the
discovered case and `NVIDIA` segment, using the Kubernetes 63-byte limit and the
existing 8-character shortening hash. If the machine or GPU type cannot be
resolved, discovery uses a fallback `group-N` identifier and writes only the
labels it can construct.

See [Heterogeneous Clusters](heterogeneous-clusters.md) for group merging and targeted generation.

Expand Down
17 changes: 11 additions & 6 deletions docs/user/heterogeneous-clusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,15 @@ Each `clusterConfig` entry is a source group with its own:
- East-west PF inventory, rail assignments, and capabilities.
- Storage and third-party RDMA kernel modules.

Fresh discovery derives `identifier` from the machine/GPU identity and bounds
it to 40 bytes with a deterministic hash suffix when the natural value is
longer. Use the persisted identifier shown in `cluster-config.yaml` with
`--groups`.
Fresh discovery derives `identifier` from the machine/GPU identity, lowercases
it, and removes complete `NVIDIA` segments. For example,
`DGX-B200-NVIDIA-H200` becomes `dgx-b200-h200`; the Launch Kit machine node
label uses the same value. Common machine segments are shortened in the same
pass (`ThinkSystem` → `ts`, `PowerEdge` → `pe`). Identifiers are bounded to 30
bytes. Long values use balanced machine/GPU prefixes plus a 6-character
deterministic hash, with unused prefix space reassigned to the longer component.
The separate GPU label still retains its discovered value such as `NVIDIA-H200`.
Use the persisted identifier shown in `cluster-config.yaml` with `--groups`.

Inspect the available groups before filtering:

Expand Down Expand Up @@ -67,7 +72,7 @@ Use `--groups` for an exact set of source identifiers. Identifier matching is ca
```bash
l8k generate \
--user-config ./cluster-config.yaml \
--groups poweredge-xe9680-h200,thinksystem-sr680a-v3-h200 \
--groups pe-xe9680-h200,ts-sr680a-v3-h200 \
--save-deployment-files ./deployment
```

Expand Down Expand Up @@ -165,7 +170,7 @@ l8k generate \
# Stage two of the three source groups
l8k generate \
--user-config ./cluster-config.yaml \
--groups dgx-b200-nvidia-h200,thinksystem-sr680a-v3-nvidia-h200 \
--groups dgx-b200-h200,ts-sr680a-v3-h200 \
--save-deployment-files ./deployment-stage1
```

Expand Down
4 changes: 2 additions & 2 deletions docs/user/profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,13 +82,13 @@ Discovery writes l8k-owned node labels:

| Label | Meaning |
| --- | --- |
| `nvidia.kubernetes-launch-kit.machine` | One source group, value `<machineType>-<gpuType>` |
| `nvidia.kubernetes-launch-kit.machine` | One source group, value matches the generated `clusterConfig[].identifier` |
| `nvidia.kubernetes-launch-kit.gpu` | All source groups sharing the same GPU type |

Use `--groups` when named source groups require different outputs:

```bash
l8k generate --groups poweredge-xe9680-h200,thinksystem-sr680a-v3-h200
l8k generate --groups pe-xe9680-h200,ts-sr680a-v3-h200
```

Use `--gpu-type` when all groups with the same GPU type can share a generated bundle:
Expand Down
138 changes: 97 additions & 41 deletions pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,10 @@ func DefaultLaunchKitConfig() (*LaunchKitConfig, error) {
return &cfg, nil
}

// MachineLabelKey is the node label `l8k discover` writes onto every
// node whose group has both machineType and gpuType resolved. The value
// is the literal `<machineType>-<gpuType>` (e.g. `DGX-B200-NVIDIA-H100-NVL`)
// — upstream discovery already trims whitespace and converts spaces to
// hyphens to match GPU operator label format. Per-source-group
// `NodeSelector` keys on this label.
// MachineLabelKey is the node label `l8k discover` writes onto every node
// whose group has both machineType and gpuType resolved. Its value is the
// group's generated identifier, so per-source-group NodeSelectors and
// generated names use the same stable identity.
const MachineLabelKey = "nvidia.kubernetes-launch-kit.machine"

// GPULabelKey is the node label `l8k discover` writes onto every node
Expand All @@ -101,7 +99,13 @@ const (
MaxLabelValueLength = 63
// MaxGeneratedIdentifierLength leaves room for prefixes added to generated
// resource names and label values while retaining a collision-resistant hash.
MaxGeneratedIdentifierLength = 40
MaxGeneratedIdentifierLength = 30
// generatedIdentifierHashLength is the number of FNV-32a hex digits kept
// in shortened generated identities. The leading dash is additional.
generatedIdentifierHashLength = 6
// labelValueHashLength retains the existing hash width for GPU label values,
// which continue to use Kubernetes' full label-value budget.
labelValueHashLength = 8
)

const (
Expand All @@ -116,23 +120,52 @@ const (
SpectrumXIPVersionIPv6 = "ipv6"
)

// MachineLabelValue returns the per-source-group machine label value:
// `<machineType>-<gpuType>` literal when it fits the 40-byte generated-group
// identity limit, or a deterministic shortened form for long names
// (truncated prefix + 8-hex FNV-32a suffix). Returns the empty string
// only when either input is empty.
//
// The shortened form looks like
// `HPE-ProLiant-Compute-DL380-Gen1-0885f134`
// and is reproducible: identical inputs always produce the same
// value, so the label discover writes onto nodes always matches what
// `MachineLabelValue` returns at filter time.
func MachineLabelValue(machineType, gpuType string) string {
// GeneratedGroupIdentifier returns the shared identifier used by a source
// group and its MachineLabelKey node label. Normalization removes complete
// "nvidia" segments and shortens common machine segments ("thinksystem" to
// "ts", "poweredge" to "pe"). Long identities retain balanced prefixes from
// machineType and gpuType plus a deterministic 6-hex FNV-32a hash. If one
// component is shorter than half of the readable budget, the other component
// receives the unused space.
func GeneratedGroupIdentifier(machineType, gpuType string) string {
if machineType == "" || gpuType == "" {
return ""
}
raw := machineType + "-" + gpuType
return truncateWithHash(raw, MaxGeneratedIdentifierLength)

machine := normalizeIdentifier(machineType)
gpu := normalizeIdentifier(gpuType)
if machine == "" {
return truncateIdentifierWithHash(gpu)
}
if gpu == "" {
return truncateIdentifierWithHash(machine)
}

raw := machine + "-" + gpu
if len(raw) <= MaxGeneratedIdentifierLength {
return raw
}

suffix := hashSuffix(raw, generatedIdentifierHashLength)
readableBudget := MaxGeneratedIdentifierLength - len(suffix) - 1
machineBudget := readableBudget / 2
gpuBudget := readableBudget - machineBudget
if len(machine) < machineBudget {
machineBudget = len(machine)
gpuBudget = readableBudget - machineBudget
} else if len(gpu) < gpuBudget {
gpuBudget = len(gpu)
machineBudget = readableBudget - gpuBudget
}

return machine[:machineBudget] + "-" + gpu[:gpuBudget] + suffix
}

// MachineLabelValue returns the generated source-group identifier used as the
// MachineLabelKey node label. It is kept as a named helper for callers that
// construct selector values outside discovery.
func MachineLabelValue(machineType, gpuType string) string {
return GeneratedGroupIdentifier(machineType, gpuType)
}

// GPULabelValue returns the gpu label value, applying the same
Expand All @@ -142,36 +175,60 @@ func GPULabelValue(gpuType string) string {
if gpuType == "" {
return ""
}
return truncateWithHash(gpuType, MaxLabelValueLength)
return truncateWithHash(gpuType, MaxLabelValueLength, labelValueHashLength)
}

// truncateWithHash returns s unchanged when it fits maxLen bytes. Longer
// values retain a readable prefix and an 8-hex FNV-32a hash of the full input.
func truncateWithHash(s string, maxLen int) string {
// values retain a readable prefix and the requested number of FNV-32a hash
// digits from the full input.
func truncateWithHash(s string, maxLen, hashLength int) string {
if len(s) <= maxLen {
return s
}
h := fnv.New32a()
_, _ = h.Write([]byte(s))
suffix := fmt.Sprintf("-%08x", h.Sum32())
suffix := hashSuffix(s, hashLength)
prefixBudget := maxLen - len(suffix)
prefix := strings.TrimRight(s[:prefixBudget], "-_.")
return prefix + suffix
}

func hashSuffix(s string, hashLength int) string {
h := fnv.New32a()
_, _ = h.Write([]byte(s))
fullHash := fmt.Sprintf("%08x", h.Sum32())
return "-" + fullHash[:hashLength]
}

// SanitizeIdentifier converts a product-type or label-value string into a
// valid K8s name component: lowercases the input and replaces spaces with
// hyphens, then deterministically bounds the result to 40 bytes. The shorter
// limit leaves room for prefixes wherever the identifier is appended to a
// generated resource name or label value. Used by discovery to derive
// `ClusterConfig.Identifier` from the machine label, and by the renderer when
// an identifier needs to land in a resource name. Both call sites must agree
// on the rule so a config produced by discovery renders the same names
// downstream — single function here guarantees that.
// valid K8s name component: lowercases the input, replaces spaces with
// hyphens, removes complete "nvidia" segments, and applies common segment
// shortenings before deterministically bounding the result to 30 bytes with a
// 6-hex hash when needed. Used for single-component generated identities such
// as auto-merged GPU groups.
func SanitizeIdentifier(s string) string {
return truncateIdentifierWithHash(normalizeIdentifier(s))
}

func normalizeIdentifier(s string) string {
s = strings.ToLower(s)
s = strings.ReplaceAll(s, " ", "-")
return truncateWithHash(s, MaxGeneratedIdentifierLength)
segments := strings.Split(s, "-")
filtered := segments[:0]
for _, segment := range segments {
switch segment {
case "nvidia":
continue
case "thinksystem":
segment = "ts"
case "poweredge":
segment = "pe"
}
filtered = append(filtered, segment)
}
return strings.Join(filtered, "-")
}

func truncateIdentifierWithHash(s string) string {
return truncateWithHash(s, MaxGeneratedIdentifierLength, generatedIdentifierHashLength)
}

// LaunchKitConfig represents the l8k-config.yaml structure
Expand Down Expand Up @@ -687,11 +744,10 @@ type ClusterConfig struct {
// same kubelet resource. In Mode A this equals Identifier; in Mode B's
// per-source render units it differs.
MergedIdentifier string `yaml:"-"`
// SourceMachineLabels lists the machine-label values
// (`<machineType>-<gpuType>`) of every source group represented by the
// merged bucket. Populated only when this is a merged render group and
// the filtered set is a strict subset of its (gpuType, railCount)
// bucket — used by Scope-Aggregate templates to emit a
// SourceMachineLabels lists the generated identifier/machine-label values
// of every source group represented by the merged bucket. Populated only
// when this is a merged render group and the filtered set is a strict subset
// of its (gpuType, railCount) bucket; Scope-Aggregate templates use it to emit a
// `matchExpressions In: [...]` selector. Empty in Mode A and for
// per-source render units.
SourceMachineLabels []string `yaml:"-"`
Expand Down
Loading