Remove retired retriever pipeline CLI

docs/docs/extraction/releasenotes.md

@@ -19,14 +19,14 @@ Highlights for the 26.05 release include:
 
 ### Pipeline and ingestion
 
-- Legacy `nv-ingest` code paths removed; `graph_pipeline` and the graph stage registry are the canonical ingestion path  
+- Legacy `nv-ingest` and compatibility pipeline CLI code paths removed; `retriever ingest` and the graph stage registry are the canonical ingestion paths
 - Manifest-based ingest routing replaces input-type routing; `retriever ingest` is input-aware for PDF, image, audio, video, text, HTML, DOCX/PPTX, SVG, and related types  
 - `allow_no_gpu` option to skip GPU requirement during ingest for CPU-only experimentation  
 
 ### CLI
 
 - Root CLI adds first-class `retriever ingest` and `retriever query` commands with NIM URL flags, batch tuning, and LanceDB overwrite/append controls
-- For product ingest and retrieval, prefer `retriever ingest` and `retriever query`; `retriever pipeline run` remains available for compatibility and development workflows. Other top-level subcommands—including `pdf`, `html`, `eval`, `benchmark`, `harness`, `online`, `compare`, `image`, and `skill-eval`—are development and experimental
+- `retriever ingest` and `retriever query` replace the retired compatibility pipeline command. Other top-level subcommands—including `eval`, `benchmark`, `harness`, and `skill-eval`—are development and experimental
 
 ### Retriever Service and deployment
 

evaluation/bo767_recall.md

@@ -0,0 +1,70 @@
+# Evaluate BO767 Retrieval with the Retriever Harness
+
+Use the Retriever harness for the canonical BO767 end-to-end ingest, query,
+and BEIR evaluation. The checked-in runfile selects batch execution and carries
+the benchmark's worker tuning and required file, page, and query counts.
+
+Run these commands from the repository root.
+
+## Configure the Dataset Paths
+
+Copy the example path map to an untracked location:
+
+```bash
+cp nemo_retriever/harness/dataset_paths.example.yaml /tmp/retriever-dataset-paths.yaml
+```
+
+Edit the `bo767` entry so `path` points to the directory containing the 767
+documents and `query_file` points to the BO767 query/qrels CSV available on the
+machine running the benchmark.
+
+## Validate the Resolved Run
+
+Resolve the run without launching ingestion or evaluation:
+
+```bash
+uv run --project nemo_retriever retriever harness run-files \
+  --session-name bo767_beir_check \
+  --output-dir /tmp/retriever-harness-bo767-check \
+  --dataset-paths /tmp/retriever-dataset-paths.yaml \
+  --dry-run \
+  --json \
+  nemo_retriever/harness/runfiles/bo767_beir.json
+```
+
+Inspect `session_summary.json`, `expanded_runs.json`, and the child run's
+`resolved_benchmark.json` before starting the full run.
+
+## Run the Evaluation
+
+Remove `--dry-run` and choose a durable artifact directory:
+
+```bash
+uv run --project nemo_retriever retriever harness run-files \
+  --session-name bo767_beir \
+  --output-dir /local/path/to/retriever-artifacts/bo767-beir \
+  --dataset-paths /tmp/retriever-dataset-paths.yaml \
+  --json \
+  nemo_retriever/harness/runfiles/bo767_beir.json
+```
+
+BO767 is a large batch benchmark. Keep the terminal process alive until the
+harness reaches a terminal state; model startup and ingestion can be quiet for
+extended periods.
+
+## Read the Results
+
+The harness artifacts are the evaluation contract:
+
+- `status.json` reports the current phase and concise failure state.
+- `results.json` is the authoritative terminal result and summary metrics.
+- `session_summary.json` is the terminal result for the runfile session.
+- `beir_metrics.json` contains the complete BEIR metric family.
+- `query_results.jsonl` contains per-query latency and ranked hits.
+- `environment.json` records the commit and runtime context.
+
+Refer to
+[`nemo_retriever/harness/EXPECTED_RESULTS.md`](../nemo_retriever/harness/EXPECTED_RESULTS.md#bo767)
+for the expected BO767 counts and current reference metrics. For the complete
+harness contract and troubleshooting guidance, refer to
+[`nemo_retriever/harness/README.md`](../nemo_retriever/harness/README.md).

examples/README.md

@@ -14,10 +14,10 @@ Start with these guides and notebooks:
 - [Workflow: Ingest documents](https://docs.nvidia.com/nemo/retriever/latest/extraction/workflow-document-ingestion/)
 - [Adding Custom Metadata for Filtered Search/Retrieval](nemo_retriever_retriever_query_metadata_filter.ipynb) — also summarized on [Vector databases — Metadata and filtering](https://docs.nvidia.com/nemo/retriever/latest/extraction/vdbs/#metadata-and-filtering)
 
-For advanced scenarios, use these notebooks:
+For advanced scenarios, use these guides and notebooks:
 
 - [Build a Custom Vector Database Operator](building_vdb_operator.ipynb)
 - [Try Enterprise RAG Blueprint](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag)
-- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb)
+- [Evaluate BO767 retrieval with the Retriever harness](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.md)
 - [Multimodal RAG with LangChain](langchain_multimodal_rag.ipynb)
 - [Multimodal RAG with LlamaIndex](llama_index_multimodal_rag.ipynb)

nemo_retriever/README.md

@@ -97,7 +97,6 @@ The [test PDF](../data/multimodal_test.pdf) contains text, tables, charts, and i
 
 > **Note:** `retriever ingest` defaults to local, in-process execution. Use `retriever ingest batch ...` for Ray Data scale-out on larger workloads.
 > File formats and internal extraction stages are not separate root commands; configure supported behavior through `retriever ingest`.
-> `retriever pipeline run` remains callable for compatibility while existing callers migrate, but it is hidden from root help.
 
 The examples below use default local GPU inference (no `invoke_url` specified) and require the `[local]` extra and the CUDA 13 torch override from the setup steps above. For remote NIM inference without a local GPU, refer to [Run with remote inference](#run-with-remote-inference-no-local-gpu-required).
 
@@ -147,22 +146,22 @@ chunks = ingestor.ingest()  # pandas.DataFrame (batch and inprocess)
 
 ### Ingest a test corpus (CLI)
 
-`graph_pipeline` is the canonical ingestion script for building a multi-document
-LanceDB corpus. Point it at a **directory** of PDFs to produce a ready-to-query table.
+Point `retriever ingest` at a **directory** of PDFs to produce a ready-to-query
+LanceDB table.
 
 > **Corpus size matters.** LanceDB's default IVF index needs at least 16
 > chunks to train its 16 k-means partitions. Single-PDF ingestion will fail
-> at the indexing step; point `graph_pipeline` at a directory with enough
+> at the indexing step; point `retriever ingest` at a directory with enough
 > documents to clear that threshold. Replace `/your-example-dir` below with
 > the path to your own corpus.
 
 ```bash
-python -m nemo_retriever.examples.graph_pipeline \
-  /your-example-dir \
-  --vdb-kwargs-json '{"uri":"lancedb","table_name":"nemo-retriever"}'
+retriever ingest /your-example-dir \
+  --lancedb-uri lancedb \
+  --table-name nemo-retriever
 ```
 
-Chunks land at `./lancedb/nemo-retriever`, which matches the `vdb_kwargs`
+Chunks land at `./lancedb/nemo-retriever`, which matches the storage settings
 used in [Run a recall query](#run-a-recall-query) below. With the
 `[local]` extra installed (see setup), defaults point at local-GPU extraction
 and embedding. Use enough documents in the directory to clear the LanceDB IVF
@@ -174,9 +173,9 @@ through [build.nvidia.com](https://build.nvidia.com/) NIMs instead:
 ```bash
 export NVIDIA_API_KEY=nvapi-...
 
-python -m nemo_retriever.examples.graph_pipeline \
-  /your-example-dir \
-  --vdb-kwargs-json '{"uri":"lancedb","table_name":"nemo-retriever"}' \
+retriever ingest /your-example-dir \
+  --lancedb-uri lancedb \
+  --table-name nemo-retriever \
   --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
   --ocr-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v2 \
   --table-structure-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1 \
@@ -228,7 +227,7 @@ Since the ingestion job automatically populated a lancedb table with all these c
 from nemo_retriever.graph.retriever import Retriever
 
 retriever = Retriever(
-  # values used by the graph_pipeline example above
+  # values used by the retriever ingest example above
   vdb_kwargs={"uri": "lancedb", "table_name": "nemo-retriever"},
   top_k=5,
   rerank=False
@@ -454,8 +453,8 @@ print(len(result.chunks), "chunks from", {m.get("source") for m in result.metada
 print(f"{result.latency_s:.2f}s on {result.model}")
 ```
 
-Local-GPU shortcut: if you ingested with default `graph_pipeline` flags
-(`--embed` omitted, `[local]` extra installed), drop `embed_kwargs` to reuse
+Local-GPU shortcut: if you ingested with default `retriever ingest` flags
+(`[local]` extra installed), drop `embed_kwargs` to reuse
 the bundled `VL_EMBED_MODEL`.
 
 Live RAG with scoring and an LLM judge (requires a ground-truth `reference`):

nemo_retriever/developer_docs/README.md

@@ -12,4 +12,3 @@ architecture, subsystems, and developer-facing tools.
 | [Retriever Harness README](../harness/README.md) | Operator and agent instructions for the artifact-first Retriever benchmark harness. |
 | [Retriever Harness PRD](harness_retriever_ingest_query_prd.md) | Product requirements for the artifact-first Retriever ingest/query benchmark harness revamp. |
 | [Root Ingest CLI Design](root_ingest_cli_design.md) | Reviewer guide for the `retriever ingest` local, batch, and service CLI ownership split. |
-| [VDB Retrieval Refactor Scope](vdb_retrieval_refactor.md) | Motivation, in-scope paths, and ownership boundaries for graph-pipeline VDB-agnostic retrieval. |

nemo_retriever/developer_docs/harness_retriever_ingest_query_prd.md

@@ -23,10 +23,10 @@ engineers. The harness should run the new library direction end to end:
 4. Emit stable `summary_metrics` and machine-readable artifacts for humans,
    agents, and downstream reporting.
 
-This is a total revamp, not a compatibility wrapper around the current harness.
-Assume `retriever pipeline run` is deleted in the next release. The current
-`sweep`, `compare`, and legacy graph-pipeline command builder can be rewritten
-or removed if they get in the way.
+This is a total revamp, not a compatibility wrapper around the old harness. The
+retired pipeline CLI is not part of the design. The old `sweep`, `compare`, and
+graph-pipeline command builder can be rewritten or removed if they get in the
+way.
 
 The most important design choice: use a small typed benchmark registry in code,
 not a sprawling YAML configuration system. YAML/runfiles can exist as an escape
@@ -745,8 +745,7 @@ validation through the CLI and artifact contract.
 - The harness has no user-facing `--engine` flag.
 - The harness does not duplicate Typer options from `retriever ingest` or
   `retriever query`.
-- No phase-one code path invokes `retriever pipeline run` or
-  `nemo_retriever.examples.graph_pipeline`.
+- No phase-one code path invokes a retired CLI adapter.
 - CLI text formatting is explicitly non-contractual; machine consumers use
   artifact files or `--json` read-only commands.
 - Validation combines focused contract tests with functional, artifact-driven

nemo_retriever/developer_docs/root_ingest_cli_design.md

@@ -215,20 +215,9 @@ image-store URI, dry-run, and quiet output.
 Service-backed query support belongs in the query CLI/service boundary, not in
 the ingest CLI.
 
-## Pipeline Compatibility
-
-`retriever pipeline run` is not the future public ingest interface. It remains
-callable, but hidden from root help, for compatibility and development behavior
-such as:
-
-- intermediate Parquet artifacts
-- pipeline reports and runtime metrics
-- eval, recall, harness, BEIR/QA workflows
-- legacy callers not yet migrated to root ingest/query
-
-For graph ingest paths, pipeline compatibility should continue to reuse the
-canonical ingest plan/execution layer instead of shelling out to root CLI
-commands.
+The supported command-line seam is `retriever ingest` for ingestion and
+`retriever query` for retrieval. Graph ingest paths reuse the canonical ingest
+plan and execution modules rather than duplicating CLI option handling.
 
 ## Adding Or Changing A Flag