[Feature]: Parallelism-aware preflight checks

[kwombach12]

Prerequisites

• I searched existing issues

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Feature Summary

Enable preflight health checks to target only the subset of nodes required by a workload's parallelism configuration, rather than the full cluster.

Problem/Use Case

Preflight health checks today require full gang scheduling across the entire cluster, even when a workload's parallelism configuration only requires a small subset of nodes to communicate with each other. For example, a workload with EP=32 across 4 nodes (4x8xB300) only needs those 4 nodes to validate — holding up the full cluster gang adds unnecessary overhead and delays job startup.

Proposed Solution

Run preflight health checks against only the subset of nodes relevant to a workload's parallelism configuration, using pod group gangs to coordinate scheduling across that subset. This requires changes at the scheduler level to support sub-cluster gang coordination.

Component

Preflight

[github-actions]

Welcome to NVSentinel, @kwombach12! Thanks for opening your first issue.

A maintainer will triage this shortly. In the meantime:

• Check the Contributing Guide for project conventions
• Browse existing labels for related topics

[XRFXLP]

hey @kwombach12,

I'm trying to understand the intended communication scope here.

Today, preflight NCCL tests already run only on the nodes assigned to the workload's gang. For this proposal, are we targeting workloads where the gang size is N but the actual communication groups are smaller subsets (e.g. TP/PP/EP groups in Megatron/NeMo)?

If so, where is the expected source of truth for those communication groups? Is the scheduler expected to derive them from workload metadata, or is there an existing API/annotation/CRD that exposes the communication topology to preflight?

A concrete workload example would be helpful -- for example, a job that would fail the current gang-wide preflight but should pass when validated according to its actual communication topology.

[kwombach12]

Great question! @lalitadithya and I were in a slack thread on this — to clarify: yes, we're thinking about something hierarchical where the full job gang schedules together, but preflight also validates communication within each sub-group (e.g. each tensor-parallel group independently).

Curious what you think the best approach looks like for exposing the group boundaries to preflight — whether that's derived from the job's parallelism config, an annotation, or something else. Would love your thoughts!

[XRFXLP]

we're thinking about something hierarchical where the full job gang schedules together,

This reminds me a lot of Grove's hierarchical workload model, where the hierarchy is something like:

PodCliqueSet
├─ PodCliqueScalingGroup A
│   ├─ PodClique A1
│   └─ PodClique A2
├─ PodCliqueScalingGroup B
│   ├─ PodClique B1
│   └─ PodClique B2
└─ PodCliqueScalingGroup C
    └─ PodClique C1

For preflight, it might be:

JobGang
├─ CommunicationGroup A
├─ CommunicationGroup B
└─ CommunicationGroup C

I'm wondering whether there's a more general hierarchy hiding underneath this. Instead of preflight owning a special-purpose notion of TP groups, PP stages, etc., perhaps it consumes a generic hierarchy/skeleton and the workload supplies the grouping configuration.

For example, could something like CEL be used to express group membership and relationships, with preflight simply consuming the resulting hierarchy rather than embedding workload-specific parallelism semantics?

[kwombach12]

Love this direction — a generic hierarchy that the workload supplies feels much cleaner than preflight owning TP/PP/EP semantics directly.

One thought: could we layer convenience adapters on top? So the generic mechanism is the foundation, but for common cases like tensor parallelism NVSentinel can auto-derive the group structure from standard annotations/config — meaning users don't need to hand-craft the hierarchy themselves. Feels like that could be a best of both worlds in terms of architecture and ergonomics.

Does that feel like a reasonable extension of what you're proposing?

[XRFXLP]

I agree with the hierarchical + ergonomic adapter idea, but I think the key implementation detail is the source of truth.

Today TP/PP/EP are usually expressed as framework/launcher config, not as Kubernetes communication groups. An adapter can derive the generic hierarchy only if it knows the actual pod/node membership the workload will use, not just the group sizes.

So the main question is:

At preflight time, what authoritative mapping should the adapter consume?

Possible contracts:

1. The launcher/framework writes group metadata onto pods.
2. A controller materializes a group map from the same config/order used by the launcher.
3. Preflight and the launcher share a documented deterministic ordering contract.

Without one of these, preflight could derive groups from its own pod ordering and end up validating a topology different from the workload's real communicator layout.

[XRFXLP]

We had some conversations on integrating preflight with grove(+? kai scheduler), adding @sanjaychatterjee for viz.

[kwombach12]

Really helpful breakdown — thank you. From a user experience perspective, options 1 and 2 both seem workable, but I'd have a slight preference for option 2 — having a controller materialize the group map feels like it keeps things the most seamless for the end user. Option 3 feels fragile — too easy for the ordering to drift without anyone noticing.

Would option 2 require an upstream feature request in Grove (or kai) before this could move forward?