Fix Spectrum-X deployment reconciliation and rail names

README.md

@@ -350,7 +350,13 @@ phase. It auto-prefers
 to `<dir>` itself. `--dry-run` does a server-side dry run. `--deploy-timeout`
 caps the whole apply+reconcile phase end-to-end (e.g. `--deploy-timeout 90m`);
 without it, deploy polls indefinitely — right for SR-IOV on large clusters
-where reconciliation can take an hour.
+where reconciliation can take an hour. `NicInterfaceNameTemplate` is the
+bounded exception: once an `InterfaceNameMismatch` is observed, Launch Kit
+retries it for up to five minutes while the NIC configuration daemon retries
+the udev rename. This gates verification of later manifests without aborting
+on the first transient mismatch. If the names still do not match after five
+minutes, deployment fails with the per-device mismatch details. A shorter
+`--deploy-timeout` still wins. Initial discovery remains unbounded.
 
 When Launch Kit installs or upgrades the Network Operator chart, its Helm
 post-renderer adds the same version annotation to chart-rendered resources.

docs/advanced/deployment.md

@@ -81,9 +81,11 @@ Kind-specific checks include per-component Network Operator state, SR-IOV per-no
 
 For `NicConfigurationTemplate` and `NicFirmwareTemplate`, Launch Kit first waits for the operator to publish matched device names in `status.nicDevices` and for that name set to reflect the current `nodeSelector`, NIC type, PCI-address, serial-number, and part-number selectors. It then evaluates only those `NicDevice` objects and waits for the corresponding `spec.configuration` or `spec.firmware` field to reflect the current template payload. A successful device condition is accepted only after its `observedGeneration` catches up with the `NicDevice` generation. A configuration template checks `FirmwareUpdateInProgress` only when the matched device carries `spec.firmware`; without a deployed firmware template, a stale firmware condition from an older device generation does not block configuration reconciliation. Other discovered NICs do not block on configuration or firmware state. Changed templates are also observation-gated before this status is accepted, so status left by an earlier generation cannot produce a false success.
 
+For `NicInterfaceNameTemplate`, `InterfaceNameMismatch` is retryable because the NIC configuration daemon can publish it while newly-written udev rules are still taking effect. Launch Kit starts a five-minute retry window when it first observes that mismatch. Initial device discovery and other ordinary in-progress states remain unbounded. Since phase-4 verification is ordered, the template gates later checks during this window. A persistent mismatch fails deployment after the local timeout and retains the per-node and per-port mismatch details.
+
 ## Timeout
 
-The default deploy budget is unbounded because SR-IOV and driver reconciliation can exceed a small fixed timeout on large clusters.
+The default deploy budget is unbounded because SR-IOV and driver reconciliation can exceed a small fixed timeout on large clusters. `NicInterfaceNameTemplate` is the only bounded exception: after the first `InterfaceNameMismatch`, its retry window is five minutes. A shorter deploy-wide timeout takes precedence.
 
 Bound the entire Helm, apply, and reconciliation operation to a maintenance window:
 
@@ -93,7 +95,7 @@ l8k deploy \
   --deploy-timeout 90m
 ```
 
-There is no independent per-manifest deadline.
+Other manifests have no independent per-manifest deadline.
 
 ## Server-Side Dry Run
 

docs/user/spectrum-x.md

@@ -207,6 +207,13 @@ l8k generate \
   --topology-file ./topology.json
 ```
 
+For the RA2.2 and RA2.3 v1alpha2 profiles, every
+`SpectrumXRailPoolConfig.spec.railTopology[].name` is consumer-visible. The
+Spectrum-X Operator uses it as both the NetworkAttachmentDefinition name and
+the device-plugin resource suffix. Launch Kit therefore renders `rail0` and
+`nvidia.com/rail0` for a per-rail workload, or `rail0p0` and
+`nvidia.com/rail0p0` for a per-rail-plane workload.
+
 Both `ipv4` and `ipv6` generate complete nv-ipam CIDRPools. IPv4 preserves the
 existing per-node `/31` allocation. IPv6 uses the standard Spectrum-X layout:
 

pkg/cmd/deploy.go

@@ -63,7 +63,10 @@ and applies them in four phases:
 Use --deploy-timeout to bound the entire end-to-end run (e.g. for a
 maintenance window). Without the flag the deploy waits indefinitely for
 reconciliation — appropriate for large SR-IOV clusters where a single
-policy can take an hour or more.
+policy can take an hour or more. NicInterfaceNameTemplate is the bounded
+exception: once observed, an interface-name mismatch is retried for up to
+five minutes so udev rules can settle, then fails before later verification
+proceeds. Initial device discovery remains unbounded.
 
 If <deployment-files>/network-operator/ exists (the layout 'l8k generate'
 produces), that subdirectory is used. Otherwise <deployment-files> itself

pkg/networkoperatorplugin/crstate/nicconfig.go

@@ -120,9 +120,11 @@ func nicTemplateValidator(kind templateKind) Validator {
 		}
 
 		var (
-			contributing  []unstructured.Unstructured
-			details       = make(map[string]string)
-			anyInProgress bool
+			contributing         []unstructured.Unstructured
+			details              = make(map[string]string)
+			anyInProgress        bool
+			anyRetryableError    bool
+			anyNonRetryableError bool
 		)
 		if templateUsesMatchedDeviceStatus(kind) {
 			devicesByName := make(map[string]*unstructured.Unstructured, len(devices.Items))
@@ -189,15 +191,18 @@ func nicTemplateValidator(kind templateKind) Validator {
 		}
 
 		// 4. Classify each contributing device.
-		var anyError bool
 		for i := range contributing {
 			d := &contributing[i]
 			label := deviceLabel(d)
-			state, reason := classifyDevice(d, kind)
+			state, reason, retryable := classifyDevice(d, kind)
 			details[label] = reason
 			switch state {
 			case StateError:
-				anyError = true
+				if retryable {
+					anyRetryableError = true
+				} else {
+					anyNonRetryableError = true
+				}
 			case StateInProgress:
 				anyInProgress = true
 			case StateSuccess:
@@ -209,8 +214,14 @@ func nicTemplateValidator(kind templateKind) Validator {
 		}
 
 		switch {
-		case anyError:
-			return Result{State: StateError, Reason: summarizeNodeStates(details), Details: details, Source: src}, nil
+		case anyRetryableError || anyNonRetryableError:
+			return Result{
+				State:     StateError,
+				Reason:    summarizeNodeStates(details),
+				Details:   details,
+				Source:    src,
+				Retryable: anyRetryableError && !anyNonRetryableError,
+			}, nil
 		case anyInProgress:
 			return Result{State: StateInProgress, Reason: summarizeNodeStates(details), Details: details, Source: src}, nil
 		default:
@@ -470,7 +481,7 @@ func deviceLabel(d *unstructured.Unstructured) string {
 // classifyDevice maps the relevant condition type+status+reason to one of
 // the four states. The mapping mirrors nic-configuration-operator's
 // controller (internal/controller/nicdevice_controller.go).
-func classifyDevice(d *unstructured.Unstructured, kind templateKind) (CRState, string) {
+func classifyDevice(d *unstructured.Unstructured, kind templateKind) (CRState, string, bool) {
 	conds, _, _ := unstructured.NestedSlice(d.Object, "status", "conditions")
 	byType := map[string]map[string]interface{}{}
 	for _, raw := range conds {
@@ -488,11 +499,13 @@ func classifyDevice(d *unstructured.Unstructured, kind templateKind) (CRState, s
 	case templateKindInterfaceName:
 		return classifyInterfaceName(byType)
 	case templateKindConfiguration:
-		return classifyConfiguration(d, byType)
+		state, reason := classifyConfiguration(d, byType)
+		return state, reason, false
 	case templateKindFirmware:
-		return classifyFirmware(d, byType)
+		state, reason := classifyFirmware(d, byType)
+		return state, reason, false
 	default:
-		return StateInProgress, "unknown template kind"
+		return StateInProgress, "unknown template kind", false
 	}
 }
 
@@ -528,26 +541,25 @@ func classifyFirmware(device *unstructured.Unstructured, byType map[string]map[s
 	}
 }
 
-// classifyInterfaceName inspects InterfaceNameApplied. Mismatch is the
-// silent-failure case we want to catch: udev rules didn't apply, so
-// downstream SR-IOV selectors that key on the new name will match
-// nothing.
-func classifyInterfaceName(byType map[string]map[string]interface{}) (CRState, string) {
+// classifyInterfaceName inspects InterfaceNameApplied. A mismatch remains an
+// error for one-shot validation, but is marked retryable so the deploy state
+// machine can give newly-written udev rules a bounded reconciliation window.
+func classifyInterfaceName(byType map[string]map[string]interface{}) (CRState, string, bool) {
 	cond, ok := byType[consts.InterfaceNameCondition]
 	if !ok {
-		return StateInProgress, "InterfaceNameApplied condition not yet set"
+		return StateInProgress, "InterfaceNameApplied condition not yet set", false
 	}
 	reason, _, _ := unstructured.NestedString(cond, "reason")
 	status, _, _ := unstructured.NestedString(cond, "status")
 	message, _, _ := unstructured.NestedString(cond, "message")
 
 	switch {
 	case reason == consts.InterfaceNameAppliedReason && status == "True":
-		return StateSuccess, "InterfaceNameApplied"
+		return StateSuccess, "InterfaceNameApplied", false
 	case reason == consts.InterfaceNameMismatchReason:
-		return StateError, fallbackMessage(message, "interface name mismatch — udev rules did not apply")
+		return StateError, fallbackMessage(message, "interface name mismatch — udev rules did not apply"), true
 	default:
-		return StateInProgress, fallbackMessage(fmt.Sprintf("InterfaceNameApplied=%s reason=%s", status, reason), "InterfaceNameApplied unknown")
+		return StateInProgress, fallbackMessage(fmt.Sprintf("InterfaceNameApplied=%s reason=%s", status, reason), "InterfaceNameApplied unknown"), false
 	}
 }
 

pkg/networkoperatorplugin/crstate/nicconfig_test.go

@@ -216,8 +216,10 @@ func TestNicInterfaceNameTemplate_AppliedSuccessfully(t *testing.T) {
 	assert.Equal(t, StateSuccess, res.State)
 }
 
-func TestNicInterfaceNameTemplate_MismatchIsError(t *testing.T) {
-	// Silent-failure case: udev rules didn't apply, names didn't take.
+func TestNicInterfaceNameTemplate_MismatchIsRetryableError(t *testing.T) {
+	// The operator can publish a mismatch while newly-written udev rules are
+	// still taking effect. One-shot validation must retain the error state;
+	// deploy uses the retryable marker to apply its bounded wait policy.
 	manifest := nicTemplateManifest(nicopKindInterfaceNameTemplate, "tpl", "ns", map[string]string{"role": "worker"})
 	live := manifest.DeepCopy()
 	c := newClient(t,
@@ -235,6 +237,7 @@ func TestNicInterfaceNameTemplate_MismatchIsError(t *testing.T) {
 	res, err := v(context.Background(), c, manifest)
 	require.NoError(t, err)
 	assert.Equal(t, StateError, res.State)
+	assert.True(t, res.Retryable)
 	assert.Contains(t, res.Reason, "interface name mismatch")
 }
 

pkg/networkoperatorplugin/crstate/state.go

@@ -49,12 +49,15 @@ const (
 // short human-readable summary; Details carries structured per-companion
 // information (e.g. per-node syncStatus for SR-IOV) for richer reports.
 // Source identifies the object that produced the result (Kind/Name) for
-// log breadcrumbs.
+// log breadcrumbs. Retryable marks a StateError that a polling caller may
+// retry under its own bounded policy; one-shot callers still treat the state
+// as an error.
 type Result struct {
-	State   CRState
-	Reason  string
-	Details map[string]string
-	Source  string
+	State     CRState
+	Reason    string
+	Details   map[string]string
+	Source    string
+	Retryable bool
 }
 
 // Validator inspects a manifest object plus whatever companion CRs it