Skip to content

testing.md

Latest commit

 

History

History
175 lines (129 loc) · 8.73 KB

testing.md

File metadata and controls

175 lines (129 loc) · 8.73 KB
Raw

Testing guide

This guide describes where tests live, how to run them locally, and how they map to CI. For coding standards and PR workflow, see Contributing Guide. The canonical CI definition is .github/workflows/ci.yaml.

1. Test organization

Layout (simplified from the repository root):

Primus/
├── runner/
│   └── lib/
│       └── common.sh              # Shared logging/helpers sourced by the shell test runner
├── tests/
│   ├── runner/                    # Shell integration tests
│   │   ├── run_all_tests.sh       # Master shell test runner
│   │   ├── lib/                   # test_common.sh, test_config.sh, test_validation.sh
│   │   ├── helpers/               # Hook and env tests
│   │   ├── test_primus_cli.sh
│   │   ├── test_primus_cli_direct.sh
│   │   ├── test_primus_cli_container.sh
│   │   └── test_primus_cli_slurm.sh
│   ├── unit_tests/                # Python unit tests (pytest)
│   │   ├── agents/                # Tuning-agent tests
│   │   ├── backends/
│   │   ├── ci/
│   │   ├── cli/
│   │   ├── core/                  # config, backend, launcher, patches, projection, pipeline_parallel, runtime, trainer, utils
│   │   ├── megatron/              # Megatron-specific unit tests
│   │   ├── modules/
│   │   └── tools/
│   ├── trainer/                   # Integration tests (typically need GPU + data)
│   │   ├── test_megatron_trainer.py
│   │   ├── test_torchtitan_trainer.py
│   │   └── test_maxtext_trainer.py
│   ├── scripts/                   # CI unit/integration launch scripts and UT patches
│   ├── utils.py                   # Shared test utilities
│   └── run_unit_tests.py          # Optional orchestrator (walks tests/, see below)

tests/runner/run_all_tests.sh sources shared helpers from runner/lib/common.sh at the repository root (not under tests/runner/).

2. Running tests

Shell integration tests (CLI behavior, config loading, hooks, environment):

bash ./tests/runner/run_all_tests.sh

Python unit tests:

pytest tests/unit_tests/ --maxfail=1 -s

Trainer integration tests (GPU and data; might require Hugging Face access):

# Megatron
DATA_PATH=<path> pytest tests/trainer/test_megatron_trainer.py -s

# TorchTitan
DATA_PATH=<path> pytest tests/trainer/test_torchtitan_trainer.py -s

# MaxText (JAX) — often run via the orchestrator in CI
python ./tests/run_unit_tests.py --jax

tests/run_unit_tests.py walks tests/ and runs every test_*.py it finds, except for tests/trainer/test_maxtext_trainer.py in the default mode (that file is only selected when --jax is set). That means the default orchestrator run includes tests/unit_tests/ and tests/trainer/ (and any other matching tests), which is broader than pytest tests/unit_tests/ alone.

Orchestrator (default—all discovered tests except MaxText trainer):

python ./tests/run_unit_tests.py

Orchestrator (JAX / MaxText trainer only):

python ./tests/run_unit_tests.py --jax

3. Test types

  • Shell tests: Exercise runner scripts, CLI wiring, configuration loading, hook execution, and environment setup. Implemented as bash scripts under tests/runner/ and orchestrated by run_all_tests.sh.
  • Unit tests: Cover configuration parsing, preset loading, CLI behavior, patch registration, adapters, and other library logic under tests/unit_tests/.
  • Trainer tests: End-to-end training against real backends; require AMD GPUs and appropriate data paths (and sometimes tokens). See .github/workflows/ci.yaml for CI values such as DATA_PATH, MASTER_PORT, and HSA_NO_SCRATCH_RECLAIM.

Test tiers (slim on PRs, full on weekends)

Each trainer E2E case is a full training launch, so the suites hold one model per architecture and per feature path. A model that only scales the dims of an existing test is deleted, not kept in a slower tier — its recipe stays schema-checked by the example-config smoke test below. What remains is split by the @pytest.mark.weekly marker (registered in tests/conftest.py):

Tier Trigger Model E2E Unit tests
slim pull request, push to main/tags -m "not weekly" default (slow gates skipped)
full schedule cron (Sat 18:00 UTC), or workflow_dispatch with full_tests=true no filter --run-slow

The workflow derives every test step's arguments from a single PRIMUS_CI_FULL workflow-level env var, so there is one switch to flip.

Two kinds of case belong in the weekly tier: extended coverage that is not worth a PR's wall clock — secondary architectures, extra precisions, variants of a feature whose primary path stays per-PR — and new E2E cases during burn-in, promoted to the per-PR tier by deleting the marker once a weekend run has shown them green. Each marked case carries a comment saying what still covers its path on PRs.

Reproduce either tier locally:

pytest tests/trainer/test_megatron_trainer.py -m "not weekly" -s   # what PR CI runs
pytest tests/trainer/test_megatron_trainer.py -s                   # full matrix

Two things are deliberately outside this mechanism and stay hidden in both tiers: the cases --deselected in ci.yaml because they are broken on the current toolchain, and the MaxText models hidden by JAX_SKIP_UT=1.

Do not delete the last remaining test of an architecture or of a feature path. Two cases with identical extra_args are not necessarily isomorphic: the flag that makes one unique often lives in its recipe yaml, so compare those too.

Example-config smoke test

tests/unit_tests/configs/test_example_configs.py loads every yaml under examples/**/configs/ through the real config stack — env interpolation, extends: merge, module/model preset merge, experiment overrides — and asserts each resolves to a well-formed experiment whose declared framework matches its backend directory. It is CPU-only (load_primus_config does not import megatron/torchtitan/jax) and runs in seconds.

This is what makes deleting an isomorphic E2E safe: the recipe stops being trained, but a renamed model preset, a broken extends: chain or a ${VAR} without a default still fails CI, naming the exact yaml.

4. Writing new tests

  • Pytest: Add files named test_*.py under tests/unit_tests/, following existing patterns and reusing fixtures from conftest.py where present.
  • Shell: Add scripts under tests/runner/ or extend tests/runner/run_all_tests.sh to invoke new suites, consistent with existing test_primus_cli*.sh scripts.
  • Backends: Prefer tests/unit_tests/backends/<backend>/ for adapter-focused tests.

5. CI pipeline details

From .github/workflows/ci.yaml:

  • code-lint: Python 3.12 on GitHub-hosted runners. Runs pre-commit run --all-files --show-diff-on-failure, so checks follow .pre-commit-config.yaml.
  • dependency-review: Runs actions/dependency-review-action on pull requests to flag dependency changes.
  • run-unittest-torch: Self-hosted GPU runner. Installs requirements.txt, runs bash ./tests/runner/run_all_tests.sh, then pytest tests/unit_tests/ under coverage (--cov=primus --cov-report=term-missing) with specific tests --deselected (currently some megatron/cco TP-overlap and megatron/transformer/moe dispatcher cases—see the workflow file). Trainer steps set MASTER_PORT, DATA_PATH, HSA_NO_SCRATCH_RECLAIM=1, and HF_TOKEN for Megatron and TorchTitan trainer tests. A follow-up coverage step combines unit and E2E coverage.
  • run-unittest-jax: JAX runner. Installs requirements-jax.txt, runs the same shell test script, then python ./tests/run_unit_tests.py --jax with CI environment variables (for example JAX_SKIP_UT=1 and DATA_PATH as defined in the workflow).

The torch job picks its tier from PRIMUS_CI_FULL and prints the resolved tier in the job summary. See Test tiers.

The build-docker job builds images after lint passes; unit test jobs depend on code-lint, not on build-docker.

6. Pre-commit hooks

Install once per clone:

pip install pre-commit
pre-commit install

Run manually on the whole tree:

pre-commit run --all-files

Hooks include: trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files, check-merge-conflict, isort, autoflake, black, and shellcheck (as configured in .pre-commit-config.yaml). These align with the code-lint job in CI; see Contributing Guide for manual equivalents.