Add portable Retriever nightly launcher

[jioffe502]

Summary

• add one portable launcher for the four library benchmarks and all eight ViDoRe v3 domains
• make the bare command run the current checkout exactly as it exists, including local changes, without fetching or switching branches
• make --ref the explicit reproducible path: local refs run from clean detached worktrees, while remote refs such as upstream/main are fetched first and fail closed
• record working_tree_dirty in session summaries; dirty runs also write source_worktree_status.txt and label Slack reports [LOCAL CHANGES]
• isolate real benchmark children so Ray and materialized dataframe memory are released between runs
• add authenticated ViDoRe/CAS access preflight, full-SHA attribution, a standard /datasets/nv-ingest map, and a safe DeepGEMM warmup default
• let already-exported supported settings override an optional private nightly.env, which supplies only missing values
• post once when SLACK_WEBHOOK_URL is present; omit it to disable Slack or pass --no-slack for a functional test

Lead test: run the checked-out code

On a host with /datasets/nv-ingest, export the supported secrets:

export HF_TOKEN=...
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

Then run the current branch, including any local harness changes:

./ops/retriever-nightly/run-nightly.sh

That is the complete development entry point. It writes artifacts under /raid/$USER/retriever-nightly-artifacts when /raid/$USER is writable, continues through child failures, writes one terminal session_summary.json, and posts one terminal Slack report. It requires no sudo, systemd service, timer, or configuration file.

To deliberately ignore local changes and run the newest clean upstream commit:

./ops/retriever-nightly/run-nightly.sh --ref upstream/main

The same --ref interface accepts another remote branch, local branch, tag, or full SHA.

Optional functional checks

./ops/retriever-nightly/run-nightly.sh --check-vidore-access
./ops/retriever-nightly/run-nightly.sh --dry-run
./ops/retriever-nightly/run-nightly.sh \
  --no-slack \
  nemo_retriever/harness/runfiles/jp20_beir.json

The launcher performs the ViDoRe access preflight before every real invocation, including a JP20-only canary, so HF_TOKEN is required for each real launcher run.

Configuration contract

Exported variables work without a configuration file. If <nightly-root>/.config/nemo-retriever/nightly/nightly.env exists, the launcher loads it after verifying that the invoking user owns it and its mode is 600. Already-exported supported settings win; the file supplies only missing values. The launcher never requires that file and never discovers a repository .env.

Setting	Required	Purpose
HF_TOKEN	every real launcher run	Read-only access for the automatic ViDoRe preflight.
SLACK_WEBHOOK_URL	no	Enables one terminal Slack post.
RETRIEVER_DATASET_PATHS	nonstandard dataset hosts only	Path to a YAML file that replaces the checked-in /datasets/nv-ingest map.

Nonstandard dataset host

--dataset-paths and RETRIEVER_DATASET_PATHS take the path to a YAML file, not a dataset directory. A JP20-only file may contain only:

schema_version: 1
datasets:
  jp20:
    path: /raid/data/jp20
    query_file: /raid/data/jp20_query_gt.csv

Run it with:

./ops/retriever-nightly/run-nightly.sh \
  --dataset-paths /raid/data/retriever-dataset-paths.yaml \
  --no-slack \
  nemo_retriever/harness/runfiles/jp20_beir.json

Batch mode starts local models and needs no model-provider API keys. The launcher defaults VLLM_DEEP_GEMM_WARMUP=skip unless explicitly overridden.

Daily latest-main workflow

The launcher remains a one-shot developer tool. For an operator-owned daily loop, run this command inside the documented tmux loop:

./ops/retriever-nightly/run-nightly.sh --ref upstream/main

Each invocation fetches upstream/main, selects its exact commit in a clean detached worktree, runs access preflight, and then executes the suite. It never pulls, merges, or moves the invoking checkout.

Validation

• all 125 test_harness*.py tests passed after merging current upstream/main
• 33 focused run-files and launcher tests passed after the interface simplification
• Black 25.9 (120-column repository style), Flake8 7.3 for the changed Python files, Python compilation, launcher shell syntax, CLI help, and git diff --check passed
• real sequential Finance EN → Finance FR validation passed on an NVIDIA H100 NVL through retriever harness run-files with no isolation or timeout flags:
  • terminal session: success: true, exit_code: 0, dry_run: false, isolate_runs: true, two runs
  • Finance EN: 6 files, 2,942 pages, 1,854 queries, exit 0
  • Finance FR: 5 files, 2,384 pages, 1,920 queries, exit 0
  • the first spawned child and Ray session exited before the second child started; no harness/Ray process or GPU allocation remained after the session

The real run was performed immediately before commit 38e1a36b with the exact isolation simplification diff; its artifact provenance correctly records the pre-commit checkout as dirty. The subsequent upstream/main merge at b3bbec8d did not change the isolation implementation, and the merged tree passed the 125-test harness suite.

[greptile-apps]

Greptile Summary

This PR introduces a portable nightly benchmark launcher (run-nightly.sh) for running the four library benchmarks and all eight ViDoRe v3 domains, plus supporting Python infrastructure for process isolation, dirty-worktree provenance recording, and authenticated ViDoRe access preflight checks.

• Shell launcher (run-nightly.sh): Self-calling two-phase design — the first phase resolves the checkout (current tree or a clean detached worktree from --ref), the second runs retriever harness run-files with isolation enabled. Flock-based concurrency guards, atomic Slack-post deduplication, and config-file security (user-owned mode 600) are all present.
• Process isolation (runsets.py): run_runfiles now unconditionally spawns each benchmark in a fresh spawn-context subprocess with a 6-hour child timeout and bounded 30-second post-result cleanup, so Ray and GPU allocations are released between runs.
• New modules (vidore_access.py, artifacts.py): working_tree_dirty() captures dirty-tree state before runs; check_vidore_access probes one byte per HuggingFace partition to verify authenticated access without a full download.

Confidence Score: 5/5

Safe to merge; the new code is well-structured and the one notable finding is a theoretical credential-forwarding edge case in the ViDoRe access check.

The process-isolation implementation is carefully bounded (poll with 6-hour timeout, 30-second post-result join, terminate/kill escalation, message-first ordering on exit-code inspection). The shell launcher follows bash safety best practices (set -uo pipefail, proper quoting, flock guards, atomic Slack-post deduplication). The one finding is the requests.get call in vidore_access.py that forwards the Authorization header on 307 redirects — currently theoretical since HuggingFace typically uses 302 for public endpoints, and the CDN would silently ignore an unexpected header in practice.

nemo_retriever/src/nemo_retriever/harness/vidore_access.py — the allow_redirects=True + explicit Authorization header combination warrants a second look if HuggingFace's redirect behaviour ever changes.

Important Files Changed

Filename	Overview
ops/retriever-nightly/run-nightly.sh	New 577-line bash launcher; well-structured two-phase design with proper quoting, flock guards, config-file ownership validation, and bounded subprocess execution.
nemo_retriever/src/nemo_retriever/harness/runsets.py	Adds _run_prepared_benchmark_isolated with spawn-context subprocess, poll-based receive with 6-hour timeout, bounded 30-second post-result join, and correct message-first ordering when exit code is non-zero; run_runfiles unconditionally enables isolation for non-dry-run executions.
nemo_retriever/src/nemo_retriever/harness/vidore_access.py	New module probing one byte per HuggingFace partition with streaming range requests; Authorization header forwarded via raw requests.get with allow_redirects=True may leak the Bearer token to CDN hosts on 307 redirects.
nemo_retriever/src/nemo_retriever/harness/artifacts.py	Adds working_tree_dirty(), expands commit SHA regex to 64 chars for SHA-256 compat, and switches last_commit to return the full SHA rather than the 7-char short form.
nemo_retriever/src/nemo_retriever/harness/cli.py	Adds check-vidore-access CLI command delegating to check_vidore_access; exits with EXIT_MISSING_INPUT on VidoreAccessError.
nemo_retriever/tests/test_harness_nightly_launcher.py	New launcher integration tests using a fake uv binary and real git init fixtures; covers dirty-checkout labels, config-file security checks, Slack dedup, and access-preflight skip on --dry-run.
nemo_retriever/tests/test_harness_nightly_git_selection.py	New tests for the select_checkout_and_run Git selection logic; covers local refs, remote fetch, detached-worktree creation, dirty-worktree rejection, and lock-contention behaviour.
nemo_retriever/tests/test_harness_vidore_access.py	New unit tests for check_vidore_access using injected fake api and get callables; covers missing-token, bad-status, empty-response, and missing-partition failure paths.
nemo_retriever/tests/test_harness_runfiles.py	Extended with isolation-path tests: verifies that isolated child failures don't terminate the session, that the session summary records isolate_runs, and that post-result process-hang is handled gracefully.

Sequence Diagram

sequenceDiagram
    participant User
    participant Launcher as run-nightly.sh (phase 1)
    participant Checkout as Selected Checkout
    participant Launcher2 as run-nightly.sh (phase 2)
    participant Harness as retriever harness run-files
    participant Child as Isolated Child Process
    participant Slack as Slack Webhook

    User->>Launcher: ./run-nightly.sh [--ref REF] [runfiles...]
    Launcher->>Launcher: load_config_defaults (nightly.env, mode 600)
    alt --ref provided
        Launcher->>Checkout: git fetch + worktree add (detached)
    else bare command
        Launcher->>Checkout: use current checkout (allow dirty)
    end
    Launcher->>Launcher2: "RETRIEVER_SELECTED_CHECKOUT=... run-nightly.sh --check-vidore-access"
    Launcher2->>Harness: retriever harness check-vidore-access
    Harness-->>Launcher2: preflight result
    Launcher2-->>Launcher: exit 0 / non-zero
    alt preflight passed
        Launcher->>Launcher2: "RETRIEVER_SELECTED_CHECKOUT=... run-nightly.sh [args]"
        Launcher2->>Harness: retriever harness run-files --session-name ... [runfiles]
        loop "each runfile (isolate_runs=True)"
            Harness->>Child: spawn process (run_prepared_benchmark)
            Child-->>Harness: pipe send (outcome, RunOutcome)
            Child->>Child: cleanup (Ray, GPU) bounded 30s
        end
        Harness-->>Launcher2: session_summary.json + exit code
        Launcher2->>Slack: retriever harness post-slack (once, atomic marker)
        Slack-->>Launcher2: 200 OK
    else preflight failed
        Launcher->>User: log error, skip GPU work
    end

Loading

_{Reviews (11): Last reviewed commit: "fix(harness): preserve received isolated..." | Re-trigger Greptile}

[jperez999]

Just take a look at that greptile comment I think it might be valid. Wouldn't want one hang to screw the entire nightly run.