Documentation for Primus, a large-scale foundation model training framework for AMD GPUs.
| I am a... | Start here |
|---|---|
| New user | Getting started |
| User running training jobs | User guide |
| User writing YAML configurations | Configuration reference |
| Engineer tuning performance | Technical guides |
| Operator deploying to production | Operations |
| Contributor to the codebase | Developer guide |
Start here if you are new to Primus.
- Project overview: what Primus does, who it is for, key capabilities
- Installation guide: prerequisites, Docker/bare-metal/Slurm setup
- Quickstart: first training run in 5 minutes
- Glossary: terms, acronyms, and domain concepts
Core workflows and day-to-day usage.
- CLI reference:
primus-climodes, flags, and subcommands - Configuration system: YAML configuration model, presets, overrides, inheritance
- Pretraining: pretraining concepts: backends, YAML structure, parallelism, configuration inventory
- Backend training recipes: pretraining commands: copy-paste, GPU-arch-specific run commands
- Post-training: SFT and LoRA fine-tuning via Megatron Bridge
- Benchmarking: GEMM, RCCL, and dense-GEMM benchmark suites
- Preflight: cluster diagnostics and environment validation
- Projection: memory and performance projection tools
- Tuning agent: LLM-driven search for an optimal training configuration (uses projection as an oracle)
- Primus tools: catalog of all Primus tools and ecosystem projects with how-to starting points
Parameter references for Primus presets, backend-facing keys, and commonly used environment variables.
- Megatron parameters: Megatron-LM backend YAML parameters and Primus overrides
- TorchTitan parameters: Primus TorchTitan preset keys and common JobConfig fields
- MaxText parameters: Primus MaxText overlay defaults and common fields
- Megatron Bridge parameters: Megatron Bridge recipe, SFT, and pretraining fields surfaced through Primus
- Environment variables: practical reference for commonly encountered environment variables
Deep technical topics for advanced users.
- Parallelism strategies: DP, TP, PP, SP, CP, EP, FSDP explained
- Parallelism configuration: per-backend parallelism setup and batch size relationships
- Collective operations: NCCL/RCCL operations and their role in each parallelism strategy
- Performance tuning: HipBLASLt, Primus-Turbo, FP8, MoE optimization
- MoE training deep-dive: bottlenecks and Primus-Turbo optimizations for Mixture-of-Experts models
- Data preparation: tokenization, data formats, mock data
- Checkpoint management: formats, save/load, distributed checkpointing
- Multi-node networking: InfiniBand, RoCE, AINIC configuration
- Profiling and observability: Torch profiler, TraceLens, memory snapshots, projection, pp_vis
- Logging and experiment tracking: TensorBoard, WandB, MLflow setup per backend
- Fault tolerance and elastic training: graceful exit, auto-resume, in-process restart, torchft
- Determinism and reproducibility: deterministic mode, seeds, trade-offs
- Diffusion models: Flux diffusion architecture, data pipeline, and FP8 / MXFP4 training
- Native SFT and LoRA: Megatron-native SFT/LoRA runbook (BF16 / FP8 / FP4), no Megatron-Bridge dependency
Production deployment and operational guidance.
- Deployment: container, Slurm, and Kubernetes deployment
- Monitoring and logging: WandB, TensorBoard, MLflow, Primus logging
- Troubleshooting: common failures, diagnostics, and fixes
- Security: secrets handling, container security, dependencies
For contributors and maintainers.
- Architecture: system design, runtime, backends, patch system
- Contributing: development setup, code style, PR process
- Testing: test types, running tests, CI pipeline
- Extending backends: adding new training backends
- Adding models: adding model configurations per backend
- Model support matrix: supported models per backend and GPU
- CLI architecture: CLI internals: subcommand discovery, dispatch, and launch wrappers
- Backend patch notes: Primus-specific backend arguments and the files they patch
- Tooling: auxiliary analysis, benchmarking, visualization, and diagnostics tools
| Goal | Document |
|---|---|
| Understand what Primus is | Overview |
| Browse all Primus tools | Primus tools |
| Install Primus | Installation |
| Run my first training | Quickstart |
| Get an exact run command for my model/GPU | Backend training recipes |
| Write a training YAML configuration | Configuration system |
| Look up a Megatron parameter | Megatron parameters |
| Look up a TorchTitan parameter | TorchTitan parameters |
| Look up an environment variable | Environment variables |
| Understand parallelism strategies | Parallelism strategies |
| Configure parallelism for my model | Parallelism configuration |
| Tune training performance | Performance tuning |
| Train a Mixture-of-Experts model | MoE training deep-dive |
| Train a diffusion (Flux) model | Diffusion models |
| Fine-tune with native SFT / LoRA | Native SFT and LoRA |
| Auto-tune my training configuration | Tuning agent |
| Profile a training run | Profiling and observability |
| Track experiments (WandB/MLflow/TensorBoard) | Logging and experiment tracking |
| Survive node failures on long runs | Fault tolerance and elastic training |
| Reproduce results bit-for-bit | Determinism and reproducibility |
| Prepare training data | Data preparation |
| Deploy to a Slurm cluster | Deployment |
| Debug a training failure | Troubleshooting |
| Contribute to Primus | Contributing |
| Understand the code architecture | Architecture |
| Add a new training backend | Extending backends |
- Primus-Turbo: high-performance operators and kernels
- Primus-SaFE: external stability/platform layer; this repository does not include a production integration guide
- AMD ROCm documentation
- Megatron-LM
- TorchTitan
- MaxText
Need help? Open an issue on GitHub.