Add portable Retriever nightly launcher

nemo_retriever/developer_docs/harness_retriever_ingest_query_prd.md

@@ -1,15 +1,17 @@
 # Retriever Harness PRD: End-to-End Ingest/Query Benchmarks
 
-Last updated: 2026-07-01
+Last updated: 2026-07-21
 
 ## Implementation Status
 
-The current implementation includes the core runner described here plus two
+The current implementation includes the core runner described here plus three
 orchestration-neutral extensions: `run-files` applies a machine-local dataset
-path map to one or more checked-in runfiles, and `post-slack` renders or posts
-completed artifacts. It does not include recurring scheduling, deployment,
-locking, retry policy, or secret distribution. The harness README is the
-normative user and agent guide; this PRD records the design rationale.
+path map to one or more checked-in runfiles and gives each real child a fresh
+process; `check-vidore-access` validates remote ViDoRe evaluation data; and
+`post-slack` renders or posts completed artifacts. It does not include
+recurring scheduling, deployment, locking, retry policy, or secret
+distribution. The harness README is the normative user and agent guide; this
+PRD records the design rationale.
 
 ## Summary
 
@@ -736,7 +738,11 @@ validation through the CLI and artifact contract.
 - `retriever harness run-set <name>` expands a code-owned ablation and writes
   `expanded_runs.json`.
 - `retriever harness run-files` runs one or more checked-in requests with an
-  optional machine-local dataset path map.
+  optional machine-local dataset path map. Real children execute sequentially
+  in fresh processes while retaining one terminal session summary; dry-runs
+  stay in the parent process.
+- `retriever harness check-vidore-access` validates authenticated access to the
+  ViDoRe queries, qrels, and corpus partitions without downloading them.
 - `retriever harness post-slack --preview` reads artifacts without requiring a
   webhook or contacting Slack.
 - Every run writes `status.json`, `events.jsonl`, and `results.json`.

nemo_retriever/harness/README.md

@@ -76,10 +76,17 @@ uses the product service APIs for ingest and query while preserving the same
   overrides.
 - `run-set`: expand a code-owned benchmark group using registry paths.
 - `run-files`: execute one or more runfiles with an optional machine-local
-  dataset path map. This is the portable entry point for the checked-in suite.
+  dataset path map. Real children run sequentially in fresh processes. This is
+  the portable entry point for the checked-in suite.
+- `check-vidore-access`: validate authenticated access to the queries, qrels,
+  and corpus objects for all eight ViDoRe v3 datasets without downloading them.
 - `post-slack`: preview or post existing artifacts; it never executes a run.
 - `diff`: compare two run artifact directories by `results.json` summary metrics.
 
+For the opinionated twelve-benchmark workstation workflow, including Git
+selection, dataset defaults, Slack, and daily recurrence, use the
+[Retriever nightly launcher](../../ops/retriever-nightly/README.md).
+
 Legacy sweep, recurring-job, runner, reporting-UI, and portal
 commands are not part of this CLI surface. Scheduling and deployment belong to
 separate infrastructure, not the benchmark harness.
@@ -154,9 +161,18 @@ uv run --project nemo_retriever retriever harness run-files \
   nemo_retriever/harness/runfiles/financebench_beir.json
 ```
 
-The ViDoRe v3 library follows the same runfile-first contract. Each benchmark
-uses the VL embed model with `text_image` page embeddings and page-level BEIR
-scoring. Run one domain while validating a machine or configuration:
+The ViDoRe v3 library follows the same runfile-first contract. Before GPU work,
+export a Hugging Face read token and validate all remote evaluation partitions:
+
+```bash
+export HF_TOKEN=...
+uv run --project nemo_retriever retriever harness check-vidore-access
+```
+
+The check streams one byte from each queries, qrels, and corpus partition; it
+does not download the complete parquet objects. Each benchmark uses the VL
+embed model with `text_image` page embeddings and page-level BEIR scoring. Run
+one domain while validating a machine or configuration:
 
 ```bash
 uv run --project nemo_retriever retriever harness run-files \
@@ -191,6 +207,13 @@ default dataset paths are mounted. Prefer the checked-in runfiles for nightly
 or other orchestrated sessions because they carry per-dataset integrity gates
 and accept a machine-local path map.
 
+Real `run-files` sessions execute children sequentially, each in a fresh spawned
+process. The process boundary releases Ray and materialized dataframe memory
+before the next benchmark while preserving one parent-owned
+`session_summary.json`. A code-owned six-hour child deadline prevents one hung
+benchmark from blocking the session forever. Dry-runs stay in the parent
+process because they do not materialize batch data.
+
 `run-files` owns the session layout and execution mode. Runfiles passed to this
 command cannot set their own `output_dir`, `run_id`, or `dry_run`; use the
 session-level `--dry-run` flag instead. The session uses the following paths and
@@ -417,8 +440,12 @@ results:
 {
   "session_type": "runfiles",
   "session_name": "library_beir",
+  "run_commit": "0123456789abcdef0123456789abcdef01234567",
+  "working_tree_dirty": false,
   "success": true,
   "exit_code": 0,
+  "dry_run": false,
+  "isolate_runs": true,
   "runs": [
     {
       "benchmark": "jp20_beir",

nemo_retriever/harness/dataset_paths.example.yaml

@@ -1,4 +1,6 @@
 schema_version: 1
+# Copy this file outside the checkout, replace the paths for the host, and pass
+# the YAML file itself with --dataset-paths. Do not pass a dataset directory.
 datasets:
   jp20:
     path: /path/to/datasets/nv-ingest/jp20

nemo_retriever/src/nemo_retriever/harness/artifacts.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2024-25, NVIDIA CORPORATION & AFFILIATES.
+# SPDX-FileCopyrightText: Copyright (c) 2024-26, NVIDIA CORPORATION & AFFILIATES.
 # All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
@@ -14,7 +14,7 @@
 
 NEMO_RETRIEVER_ROOT = Path(__file__).resolve().parents[3]
 DEFAULT_ARTIFACTS_ROOT = NEMO_RETRIEVER_ROOT / "artifacts"
-_COMMIT_RE = re.compile(r"^[0-9a-fA-F]{7,40}$")
+_COMMIT_RE = re.compile(r"^[0-9a-fA-F]{7,64}$")
 
 
 def now_timestr() -> str:
@@ -25,7 +25,7 @@ def _normalize_commit(value: str | None) -> str | None:
     text = (value or "").strip()
     if not _COMMIT_RE.match(text):
         return None
-    return text[:7]
+    return text.lower()
 
 
 def _resolve_git_dir(repo_root: Path) -> Path | None:
@@ -98,7 +98,7 @@ def last_commit() -> str:
     repo_root = NEMO_RETRIEVER_ROOT.parent
     try:
         result = subprocess.run(
-            ["git", "rev-parse", "--short", "HEAD"],
+            ["git", "rev-parse", "HEAD"],
             cwd=str(repo_root),
             check=False,
             capture_output=True,
@@ -118,6 +118,25 @@ def last_commit() -> str:
     return "unknown"
 
 
+def working_tree_dirty() -> bool | None:
+    """Return whether the source checkout is modified, or ``None`` outside Git."""
+
+    repo_root = NEMO_RETRIEVER_ROOT.parent
+    try:
+        result = subprocess.run(
+            ["git", "status", "--porcelain", "--untracked-files=normal", "--ignore-submodules"],
+            cwd=str(repo_root),
+            check=False,
+            capture_output=True,
+            text=True,
+        )
+    except Exception:
+        return None
+    if result.returncode != 0:
+        return None
+    return bool(result.stdout.strip())
+
+
 def get_artifacts_root(base_dir: str | None = None) -> Path:
     if base_dir:
         return Path(base_dir).expanduser().resolve()

nemo_retriever/src/nemo_retriever/harness/cli.py

@@ -15,7 +15,7 @@
     list_benchmarks,
     list_runsets,
 )
-from nemo_retriever.harness.contracts import EXIT_INVALID, FailurePayload
+from nemo_retriever.harness.contracts import EXIT_INVALID, EXIT_MISSING_INPUT, FailurePayload
 from nemo_retriever.harness.revamp_runner import (
     HarnessRunError,
     run_benchmark,
@@ -32,6 +32,7 @@
     post_slack_payload,
     resolve_slack_webhook_url,
 )
+from nemo_retriever.harness.vidore_access import VidoreAccessError, check_vidore_access
 
 app = typer.Typer(
     help=(
@@ -45,6 +46,42 @@ def _echo_json(payload: object) -> None:
     typer.echo(json.dumps(payload, indent=2, sort_keys=False))
 
 
+@app.command("check-vidore-access")
+def check_vidore_access_command(
+    dataset_names: Annotated[
+        list[str] | None,
+        typer.Option("--dataset", help="Check one ViDoRe dataset name. Repeat to check multiple datasets."),
+    ] = None,
+    timeout_seconds: Annotated[
+        float,
+        typer.Option("--timeout-seconds", min=1.0, help="HTTP timeout for each streamed range request."),
+    ] = 20.0,
+    require_token: Annotated[
+        bool,
+        typer.Option("--require-token/--allow-anonymous", help="Require HF_TOKEN before checking public data."),
+    ] = True,
+    json_output: Annotated[bool, typer.Option("--json", help="Emit machine-readable JSON.")] = False,
+) -> None:
+    """Verify ViDoRe queries, qrels, and corpus objects without downloading them."""
+
+    try:
+        results = check_vidore_access(
+            dataset_names=dataset_names,
+            timeout_seconds=timeout_seconds,
+            require_token=require_token,
+        )
+    except VidoreAccessError as exc:
+        typer.echo(str(exc), err=True)
+        raise typer.Exit(code=EXIT_MISSING_INPUT) from exc
+
+    payload = {"success": True, "datasets": [result.to_dict() for result in results]}
+    if json_output:
+        _echo_json(payload)
+        return
+    for result in results:
+        typer.echo(f"{result.dataset}: access ok ({', '.join(result.checked_partitions)})")
+
+
 @app.command("list")
 def list_command(
     json_output: Annotated[bool, typer.Option("--json", help="Emit machine-readable JSON.")] = False,