Package preflight (and optionally primus-cli) as an installable Python distribution

[amd-ama10002-2]

Summary

There is growing interest in being able to install Primus's preflight cluster diagnostic from this branch — and ideally the primus-cli entry point — as a standalone Python package (wheel / sdist on an internal index or PyPI), rather than always having to git clone the full Primus repository.

Today the repo appears to have no Python packaging at all.

We might want to start by packaging the code for pre-flight checks without a container from this branch: dev/preflight-direct-test (we plan to merge this branch to main later)

Motivation

• Easier deployment on cluster nodes. pip install primus-preflight
  (from an internal index or PyPI) is much cleaner than asking ops to clone
  a multi-GB repo with submodules (third_party/Megatron-LM,
  torchtitan, maxtext, ...).
• Versioning and reproducibility. Sites that gate node acceptance on a
  specific preflight version could pin to a wheel rather than a git SHA.

Proposed scope (for discussion)

A few options, roughly in order of effort:

1. Minimal: primus-preflight wheel only.

   • New top-level package (e.g. primus_preflight/) or a pyproject.toml
     that builds a subset of primus/tools/preflight/ (+ the small required
     helpers from primus/tools/utils.py).
   • Optional dependency extras: [pdf] (markdown2, weasyprint),
     [plot] (matplotlib).
   • console_scripts entry point: primus-preflight = ...:main.
   • Keep the existing in-tree primus-cli preflight working unchanged.

2. Also package primus-cli itself as a Python entry point.

   • Replace (or complement) the bash primus-cli wrapper with a
     console_scripts entry point so pip install primus-cli (or
     primus) gives users primus-cli on $PATH without a clone.
   • This is more involved because the current launcher is bash and shells
     out to mode-specific scripts (runner/primus-cli-{slurm,container,direct}.sh),
     which would need to be either shipped inside the wheel or rewritten.

3. Full repo packaging.

   • One pyproject.toml for the whole primus/ package, with optional
     extras for backends (megatron / torchtitan / jax). Probably out of
     scope for a first pass given the third-party submodules.

Open questions

• Distribution target: internal AMD index, public PyPI, or both?
• How do we keep the in-tree path (primus/tools/preflight/) and the
  packaged path in sync? Single source of truth via a pyproject.toml at
  repo root that just packages the existing tree is probably easiest.
• Wheel build & publish CI: add a job to .github/workflows/ that runs on
  tags?

Acceptance criteria (suggested)

• pyproject.toml (PEP 517/518) lands that can build at least a
  primus-preflight wheel.
• pip install <wheel> followed by primus-preflight --help works in a
  clean venv with no Primus checkout on disk.
• Documented minimal install path in docs/preflight-direct.md updated
  to include the pip install route alongside the current clone-and-run
  path.
• CI builds the wheel on PR; (optional) publishes on tag.

[snakefood3232]

Packaging Primus's preflight tool as a standalone Python package will indeed streamline its deployment and improve versioning. I'd handle this by creating a pyproject.toml to build the necessary subset from primus/tools/preflight/ and setting up optional extras for dependencies like markdown2 and matplotlib. I can have a pull request ready within 3 days. I've previously built complex Python distributions with entry points, simplifying package installation and usage.