From 8e4db6695b1866a45feb35d8ae04a99b41a4b28b Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 14:27:13 -0700 Subject: [PATCH 1/5] docs(26.05): reconcile release docs from main Sync scoped 26.5.0 documentation from main into 26.05 so the release branch matches the Jun/Jul doc fixes before docs.nvidia.com publishes from 26.05. Includes extraction pages, mkdocs.yml, and nemo_retriever README/CLI/Helm docs only. --- .../extraction/agentic-retrieval-concept.md | 2 +- docs/docs/extraction/audio-video.md | 6 +- docs/docs/extraction/concepts.md | 30 +- docs/docs/extraction/custom-metadata.md | 125 ---- docs/docs/extraction/customize-extend.md | 66 ++ docs/docs/extraction/deployment-options.md | 28 +- docs/docs/extraction/embedding.md | 8 +- docs/docs/extraction/faq.md | 13 +- docs/docs/extraction/getting-started-about.md | 6 +- ...egrations-langchain-llamaindex-haystack.md | 22 - docs/docs/extraction/multimodal-extraction.md | 21 +- .../nemo-retriever-api-reference.md | 8 +- docs/docs/extraction/nimclient.md | 588 ---------------- docs/docs/extraction/overview.md | 5 +- .../prerequisites-support-matrix.md | 62 +- docs/docs/extraction/releasenotes.md | 5 +- docs/docs/extraction/starter-kits.md | 24 + docs/docs/extraction/troubleshoot.md | 40 +- docs/docs/extraction/vdbs.md | 26 +- .../extraction/workflow-agentic-retrieval.md | 23 +- .../extraction/workflow-document-ingestion.md | 22 +- .../extraction/workflow-e2e-blueprints.md | 2 +- docs/mkdocs.yml | 40 +- nemo_retriever/README.md | 71 +- nemo_retriever/docs/cli/README.md | 633 +++++++----------- nemo_retriever/docs/cli/benchmarking.md | 93 ++- nemo_retriever/helm/README.md | 374 ++++++----- 27 files changed, 833 insertions(+), 1510 deletions(-) delete mode 100644 docs/docs/extraction/custom-metadata.md create mode 100644 docs/docs/extraction/customize-extend.md delete mode 100644 docs/docs/extraction/integrations-langchain-llamaindex-haystack.md delete mode 100644 docs/docs/extraction/nimclient.md create mode 100644 docs/docs/extraction/starter-kits.md diff --git a/docs/docs/extraction/agentic-retrieval-concept.md b/docs/docs/extraction/agentic-retrieval-concept.md index a06431a359..b2bfd09cfd 100644 --- a/docs/docs/extraction/agentic-retrieval-concept.md +++ b/docs/docs/extraction/agentic-retrieval-concept.md @@ -7,4 +7,4 @@ NeMo Retriever Library focuses on document ingestion, embeddings, vector stores, **Related** - [Semantic retrieval](vdbs.md#semantic-retrieval) -- Framework examples: [LangChain, LlamaIndex, Haystack](integrations-langchain-llamaindex-haystack.md) +- Framework examples: [Starter kits](starter-kits.md) diff --git a/docs/docs/extraction/audio-video.md b/docs/docs/extraction/audio-video.md index c9031ce413..3f2df2cf9c 100644 --- a/docs/docs/extraction/audio-video.md +++ b/docs/docs/extraction/audio-video.md @@ -73,7 +73,7 @@ Use the following procedure to run the NIM on your own infrastructure. Self-host ```python from nemo_retriever import create_ingestor - from nemo_retriever.params.models import ASRParams + from nemo_retriever.common.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") @@ -100,9 +100,11 @@ Instead of running the pipeline locally, you can call Parakeet through [build.nv 2. Run inference from Python with the hosted gRPC endpoint and credentials from that page (the example below uses the default hosted gRPC hostname; confirm values in the **Get API Key** flow for your deployment). Pass hosted endpoint, function ID, and API key through `ASRParams` (`audio_endpoints`, `function_id`, `auth_token`). + For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor - from nemo_retriever.params.models import ASRParams + from nemo_retriever.common.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") diff --git a/docs/docs/extraction/concepts.md b/docs/docs/extraction/concepts.md index 4418682052..bb6ce80998 100644 --- a/docs/docs/extraction/concepts.md +++ b/docs/docs/extraction/concepts.md @@ -2,40 +2,40 @@ These terms appear throughout NeMo Retriever Library documentation. -## Job +## Job { #job } -An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations ([NeMo Retriever graph](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph)). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). +An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations. For UDFs and other extension paths, refer to [Customize & extend](customize-extend.md). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). -## Pipeline and tasks +## Pipeline and tasks { #pipeline-and-tasks } -NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. Related topics: [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). +NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. For UDFs, custom graph stages, and other extension paths, refer to [Customize & extend](customize-extend.md). -## Extraction metadata +## Extraction metadata { #extraction-metadata } Output is a **Ray Dataset** (Ray Data) or **pandas** `DataFrame` listing extracted objects (text regions, tables, images, and so on), processing notes, and timing or trace data. Field-level detail is in the [metadata reference](content-metadata.md). -## Embeddings and retrieval +## Embeddings and retrieval { #embeddings-and-retrieval } -Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, see [Vector databases](vdbs.md). For multimodal (VLM) embedding options, see [Multimodal embeddings (VLM)](embedding.md). +Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, refer to [Vector databases](vdbs.md). For multimodal (VLM) embedding options, refer to [Multimodal embeddings (VLM)](embedding.md). ## Chunking { #chunking } Chunking is built into the `.extract()` task and depends on **content type**: - **PDF, DOCX, and PPTX** — Text is grouped using built-in **page** boundaries (one chunk per page where the format has pages). -- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. See [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. -- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; see [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). +- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. Refer to [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. +- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; refer to [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). -For PDF parallelism before Ray processing (large files), see [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). +For PDF parallelism before Ray processing (large files), refer to [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). ### Token-based splitting { #token-based-splitting } -Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. Details appear in the [Python API guide](nemo-retriever-api-reference.md). +Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). -## Deployment modes +## Deployment modes { #deployment-modes } -- **Library mode** — Run without the full container stack where appropriate; see [Deployment options](deployment-options.md). -- **Kubernetes / Helm (self-hosted)** — See [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. -- **Notebooks** — [Jupyter examples](notebooks/index.md) for experimentation and RAG demos. +- **Library mode** — Run without the full container stack where appropriate; refer to [Deployment options](deployment-options.md). +- **Kubernetes / Helm (self-hosted)** — Refer to [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. +- **Notebooks** — [Jupyter examples](starter-kits.md) for experimentation and RAG demos. For a concise comparison, refer to [Deployment options](deployment-options.md). diff --git a/docs/docs/extraction/custom-metadata.md b/docs/docs/extraction/custom-metadata.md deleted file mode 100644 index 45ffa34d29..0000000000 --- a/docs/docs/extraction/custom-metadata.md +++ /dev/null @@ -1,125 +0,0 @@ -# Custom metadata and filtering - -Use this documentation to attach per-document metadata during ingestion and to narrow [LanceDB](vdbs.md) search results in [NeMo Retriever Library](overview.md). Implementation details live in the package [Vector DB operators and LanceDB](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering) README. - -## On this page { #on-this-page } - -- [Attach metadata at ingestion](#attach-metadata-at-ingestion) -- [Best practices](#best-practices) -- [Filter results during retrieval](#filter-results-during-retrieval) -- [How metadata is stored](#how-metadata-is-stored) - -## Attach metadata at ingestion { #attach-metadata-at-ingestion } - -Pass a **sidecar metadata table** on `vdb_upload` so selected columns are merged into each chunk's `content_metadata` before LanceDB upload. All three parameters must be set together: - -| Parameter | Purpose | -|-----------|---------| -| `meta_dataframe` | Path to CSV, JSON, or Parquet, or an in-memory `pandas.DataFrame` | -| `meta_source_field` | Column that identifies each document (must match ingest paths or basenames per `meta_join_key`) | -| `meta_fields` | Non-empty list of column names to copy into `content_metadata` | - -Optional `meta_join_key` controls how rows are matched to documents: `auto` (try full path then basename), `source_id` (full path), or `source_name` (basename only). - -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - -```python -import pandas as pd -from nemo_retriever import create_ingestor - -meta_df = pd.DataFrame( - { - "source": ["data/woods_frost.pdf", "data/multimodal_test.pdf"], - "meta_a": ["alpha", "bravo"], - "meta_b": [10, 20], - } -) - -hostname = "localhost" -table_name = "nemo_retriever_collection" -lancedb_uri = "s3://your-bucket/lancedb" - -ingestor = ( - create_ingestor(run_mode="service", base_url=f"http://{hostname}:7670") - .files(["data/woods_frost.pdf", "data/multimodal_test.pdf"]) - .extract( - extract_text=True, - extract_tables=True, - extract_charts=True, - extract_images=True, - text_depth="page" - ) - .embed() - .vdb_upload( - vdb_op="lancedb", - vdb_kwargs={"lancedb_uri": lancedb_uri, "table_name": table_name}, - meta_dataframe=meta_df, - meta_source_field="source", - meta_fields=["meta_a", "meta_b"], - ) -) -results = ingestor.ingest_async().result() -``` - -Set `hostname`, `table_name`, and a **remote** `lancedb_uri` (for example `s3://bucket/path`) to match your deployment—the retriever service rejects local filesystem paths. The client uploads in-memory sidecar metadata to the service before ingest; do not pass a raw local file path as `meta_dataframe` on the REST spec. For local LanceDB directories, use `run_mode="batch"` instead (refer to [Vector databases](vdbs.md)). For a step-by-step walkthrough with additional fields such as category, department, and timestamp, refer to [Vector DB operators and LanceDB — Metadata filtering](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering). - -## Best practices { #best-practices } - -- Plan metadata structure before ingestion. -- Test filter expressions with small datasets first. -- Consider performance implications of complex filters. -- Validate metadata during ingestion. -- Handle missing metadata fields gracefully. -- Log invalid filter expressions. - -## Filter results during retrieval { #filter-results-during-retrieval } - -You can use custom metadata to filter documents during retrieval operations. For **predicate pushdown**, pass a `where` SQL predicate through [`Retriever.query`](nemo-retriever-api-reference.md) (refer to [Vector databases](vdbs.md)) or chain `.where(...)` on a native LanceDB `table.search(...)` query. Application-side filtering on returned hits does not change what the database evaluates—raise `top_k` if matches might sit outside the first neighbors. - -### Example filter ideas - -Typical keys to filter on include `category`, `department`, `priority`, and `timestamp` (use comparable ISO-8601 strings for time ranges). Encode predicates in LanceDB SQL against your table columns (often the serialized `metadata` string), or inspect parsed hit metadata after search as in the example below. - -### Example: Use a Filter Expression in Search - -After ingestion is complete and documents are uploaded to LanceDB with metadata, you can narrow results in the database with a **`where`** clause, or in Python on the returned hits. - -**Native LanceDB (SQL pushdown):** connect, embed the query yourself (same model as ingestion), then chain `.where("")` on `table.search(...)` so filtering happens before the `limit`. Exact SQL depends on how `metadata` is stored; refer to [LanceDB metadata filtering](https://docs.lancedb.com/search/filtering#filtering-with-sql). - -```python -import lancedb - -# pseudocode — replace YOUR_VECTOR and YOUR_PREDICATE with real values. -db = lancedb.connect("./lancedb_data") -table = db.open_table("nemo_retriever_collection") -# table.search(YOUR_VECTOR, vector_column_name="vector").where(YOUR_PREDICATE).limit(10).to_list() -``` - -**`Retriever.query` + `where`:** LanceDB applies the predicate before ranking. For post-filter logic in Python, use a wider `top_k` first. - -```python -from nemo_retriever.retriever import Retriever - -retriever = Retriever( - vdb_kwargs={"uri": "./lancedb_data", "table_name": "nemo_retriever_collection"}, - embed_kwargs={ - "model_name": "nvidia/llama-nemotron-embed-1b-v2", - "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2", - }, -) - -hits = retriever.query( - "this is expensive", - top_k=16, - vdb_kwargs={"where": "metadata LIKE '%\"department\":\"Engineering\"%'"}, -) -``` - -For a runnable end-to-end flow (ingest, `Retriever.query`, and both filter modes), refer to [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb). - -When you ingest through the **retriever service**, upload the sidecar with [`POST /v1/ingest/sidecar`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/routers/ingest.py#L1040-L1129) (multipart file; response [`SidecarUploadResponse`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/responses.py#L60-L68)), then pass the returned `sidecar_id` as `meta_dataframe_id` with `meta_source_field` and `meta_fields` in `pipeline.vdb_upload_params` on [`POST /v1/ingest`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/requests.py#L15-L32) ([`PipelineSpec`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/pipeline_spec.py#L55-L78)). Request and response shapes, form fields, and auth headers are in the service OpenAPI UI at `/docs` (or `/openapi.json`) on your retriever base URL (for example `http://localhost:7670/docs` after `retriever service start`). Do not send a raw local path as `meta_dataframe` on the service spec. - -## How metadata is stored { #how-metadata-is-stored } - -- [Vector databases](vdbs.md) — canonical LanceDB upload and retrieval guide -- [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — runnable notebook for sidecar metadata at ingest and filtered `Retriever.query` diff --git a/docs/docs/extraction/customize-extend.md b/docs/docs/extraction/customize-extend.md new file mode 100644 index 0000000000..31fb87e7bc --- /dev/null +++ b/docs/docs/extraction/customize-extend.md @@ -0,0 +1,66 @@ +# Customize & extend + +NeMo Retriever Library ships with defaults tuned for strong recall on common document types. When those defaults are not enough, you can extend the library at several levels—from task keyword arguments on the fluent ingestor API through custom graph operators and vector-database adapters. + +Use this page to choose an extension path and find the detailed guides in the repository. + +The following table maps common needs to the right section: + +| If you need to… | Start here | +|-----------------|------------| +| Tune extraction, chunking, embedding, or upload without new code | [Start with task configuration](#start-with-task-configuration) | +| Add a small Python transformation between pipeline stages | [User-defined functions (UDFs)](#user-defined-functions-udfs) | +| Build or reuse operators stage-by-stage | [Custom graph pipelines](#custom-graph-pipelines) | +| Store vectors in a backend other than LanceDB | [Custom vector databases](#custom-vector-databases) | + +## On this page { #on-this-page } + +- [Start with task configuration](#start-with-task-configuration) +- [User-defined functions (UDFs)](#user-defined-functions-udfs) +- [Custom graph pipelines](#custom-graph-pipelines) +- [Custom vector databases](#custom-vector-databases) +- [Related Topics](#related-topics) + +## Start with task configuration { #start-with-task-configuration } + +Most customization does not require new code. Chain tasks on `create_ingestor(...)` and pass keyword arguments to control extraction, chunking, embedding, and storage—for example `extract_method`, chunking and splitting options on `.extract()`, `embed_modality` on `.embed()`, and `vdb_op` / `vdb_kwargs` on `.vdb_upload()`. + +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). For chunking behavior and pipeline concepts, refer to [Concepts](concepts.md). + +## User-defined functions (UDFs) { #user-defined-functions-udfs } + +A **user-defined function (UDF)** wraps your Python logic as a first-class pipeline stage. In the graph model, `UDFOperator` turns a plain callable into an operator you can chain with built-in stages—for example to normalize HTML, apply a custom split, or call an external service between extract and embed steps. + +Use UDFs when you need a small, self-contained transformation that is not covered by task keyword arguments. + +### Repository guides + +- [NeMo Retriever graph README — `UDFOperator`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#using-udfoperator) — API, lifecycle, and when to use `UDFOperator` versus a custom operator class +- [NimClient and custom NIM endpoints](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints) — call custom or self-hosted NIM microservices from UDF stages + +## Custom graph pipelines { #custom-graph-pipelines } + +When you need to compose pipelines stage-by-stage, reuse operators across workflows, or run the same graph in-process or with Ray Data, use the **graph execution model** instead of (or alongside) the fluent `GraphIngestor` API. + +The graph package provides `AbstractOperator`, executors (`InprocessExecutor`, `RayDataExecutor`), and operator chaining with `>>`. Built-in ingestion operators live under `nemo_retriever.operators`; you can add your own operators or UDF stages anywhere in the chain. + +For the full guide—including custom operator classes, executors, and graph shape constraints—refer to the [NeMo Retriever graph README](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). + +## Custom vector databases { #custom-vector-databases } + +The supported user path for vector storage is **[LanceDB](vdbs.md)** (`vdb_op="lancedb"`). That page covers upload, semantic retrieval, metadata filtering, and LanceDB deployment characteristics. + +To integrate a different vector store, implement the [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) interface and wire it through graph [`IngestVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py) / [`RetrieveVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py). NVIDIA validates the first-party LanceDB operator; you are responsible for testing and maintaining other backends. + +### Repository guides + +- [Vector DB package (source)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/common/vdb) — `VDB` abstract base and LanceDB reference implementation + +Partner and blueprint integrations (Elasticsearch, Pinecone, Teradata, and others) are summarized on [Vector databases — Vector database partners](vdbs.md#vector-database-partners). + +## Related Topics { #related-topics } + +- [Concepts — Pipeline and tasks](concepts.md#pipeline-and-tasks) +- [Vector databases](vdbs.md) +- [Multimodal embeddings (VLM)](embedding.md) +- [Python API guide](nemo-retriever-api-reference.md) diff --git a/docs/docs/extraction/deployment-options.md b/docs/docs/extraction/deployment-options.md index 71f0b75109..b57404685a 100644 --- a/docs/docs/extraction/deployment-options.md +++ b/docs/docs/extraction/deployment-options.md @@ -19,24 +19,16 @@ Build and run the NeMo Retriever service image with the [Docker service image gu 1. [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) 2. **NeMo Retriever Helm chart (supported):** [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) — sources in [`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) on GitHub -3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in the [NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) — use alongside the NeMo Retriever chart README for your release +3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in [About getting started](getting-started-about.md) — use alongside the NeMo Retriever chart README for your release 4. [Environment variables](environment-config.md) and [Troubleshoot](troubleshoot.md) as needed -**Core NIMs for the default extraction pipeline** (26.05): `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). See [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). +**Core NIMs for the default extraction pipeline:** `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Refer to [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). - -For audio and video extraction in Kubernetes, set `service.installFfmpeg=true` -so the service container installs `ffmpeg` and `ffprobe` at startup. This -runtime install requires package-repository network egress, a writable root -filesystem, and security policy that allows the image's scoped sudo use. If -your cluster blocks startup package installation (for example air-gapped -environments), use a custom service image that already contains `ffmpeg` and -`ffprobe`, then set `service.image.repository` and `service.image.tag`. +For audio and video extraction in Kubernetes, set `service.installFfmpeg=true` so the service container installs `ffmpeg` and `ffprobe` at startup. This runtime install requires package-repository network egress, a writable root filesystem, and security policy that allows the image's scoped sudo use. If your cluster blocks startup package installation, use a custom service image that already contains `ffmpeg` and `ffprobe`, then set `service.image.repository` and `service.image.tag`. For Parakeet ASR chart values, OpenShift-specific Helm configuration, and air-gapped alternatives, refer to [Audio and video (Parakeet ASR)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#audio-video-parakeet) and [OpenShift deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md) in the Helm chart directory. ### I want examples and notebooks -1. [Jupyter Notebooks](notebooks/index.md) -2. [Integrate with LangChain, LlamaIndex, Haystack](integrations-langchain-llamaindex-haystack.md) +1. [Jupyter Notebooks](starter-kits.md) ### I need API details and keys @@ -49,7 +41,7 @@ environments), use a custom service image that already contains `ffmpeg` and 2. [Throughput is dataset-dependent](multimodal-extraction.md#extraction-limitations-and-quality) 3. [Evaluate on your data](evaluate-on-your-data.md) -## When to use NVIDIA-hosted NIMs +## When to use NVIDIA-hosted NIMs { #when-to-use-nvidia-hosted-nims } [NVIDIA-hosted NIMs](https://build.nvidia.com/) run inference on NVIDIA-managed infrastructure. You call models with API keys (refer to [Get your API key](api-keys.md)) without operating GPU nodes yourself. @@ -61,7 +53,7 @@ Consider hosted NIMs when: **Also refer to:** [NVIDIA NIM catalog](https://build.nvidia.com/) -## When to self-host NIMs +## When to self-host NIMs { #when-to-self-host-nims } Self-hosted NIMs run on your GPUs or air-gapped hardware, typically with Kubernetes and the [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html). @@ -77,17 +69,17 @@ Consider self-hosting when: The **default document extraction pipeline** (page elements, table structure, OCR, and VL embed) runs disconnected when you mirror images and models into a private registry and configure the [NIM Operator for air-gapped environments](https://docs.nvidia.com/nim-operator/latest/air-gap.html). -On a staging host with internet access, pull from NGC, retag to your private registry, stage chart archives, then install in the enclave with registry overrides. Procedures, the 26.05 image inventory, and Helm value patterns are in [Helm — Air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#air-gapped-deployment). +On a staging host with internet access, pull from NGC, retag to your private registry, stage chart archives, then install in the enclave with registry overrides. Procedures, the chart image inventory, and Helm value patterns are in [Helm — Air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#air-gapped-deployment). !!! warning "Audio and video extraction" [Audio and video](audio-video.md) need **`ffmpeg` and `ffprobe` on `PATH`**. The bundled image omits them. Do **not** use `service.installFfmpeg=true` in an air gap (startup install needs package-repo egress). Build a custom service image on a connected staging host, mirror it, and set `service.image.repository` / `service.image.tag`. Skip this step if you do not use audio/video. -For offline image captioning, deploy the in-cluster [Nemotron 3 Nano Omni](prerequisites-support-matrix.md#image-captioning-2605) NIM and point your pipeline caption endpoint at the in-cluster HTTP URL instead of `integrate.api.nvidia.com` or other hosted APIs. +For offline image captioning, deploy the in-cluster [Nemotron 3 Nano Omni](prerequisites-support-matrix.md#image-captioning) NIM and point your pipeline caption endpoint at the in-cluster HTTP URL instead of `integrate.api.nvidia.com` or other hosted APIs. **Related** -- [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md) ([`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/26.05/nemo_retriever/helm) on GitHub) — [air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#air-gapped-deployment) -- [NeMo Retriever Library — prerequisites / deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) (supported **Helm** handoff) +- [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) ([`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) on GitHub) — [air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#air-gapped-deployment) +- [About getting started](getting-started-about.md) (prerequisites through first deployment) - [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) - [Audio and video](audio-video.md) diff --git a/docs/docs/extraction/embedding.md b/docs/docs/extraction/embedding.md index e42574a519..b6a139fac3 100644 --- a/docs/docs/extraction/embedding.md +++ b/docs/docs/extraction/embedding.md @@ -12,8 +12,6 @@ The model can embed documents in the form of an image, text, or a combination of Documents can then be retrieved given a user query in text form. The model supports images that contain text, tables, charts, and infographics. -Parameter details for `.extract()` and `.embed()` appear in the [Python API guide](nemo-retriever-api-reference.md). - ## Example with Default Text-Based Embedding When you use the multimodal model, by default, all extracted content (text, tables, charts) is treated as plain text. @@ -21,6 +19,8 @@ The following example provides a strong baseline for retrieval. - The `embed` method is called with no arguments. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor @@ -42,6 +42,8 @@ The following example enables the multimodal model to capture the spatial and st - The `embed` method is configured with `embed_modality="text_image"` to embed the extracted tables and charts as images. - This configuration is more accurate than text only, with a performance cost. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor @@ -65,6 +67,8 @@ The following example extracts and embeds each page as an image. - The `embed` method processes the page images. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor diff --git a/docs/docs/extraction/faq.md b/docs/docs/extraction/faq.md index 6b3010db52..ba299355f4 100644 --- a/docs/docs/extraction/faq.md +++ b/docs/docs/extraction/faq.md @@ -21,16 +21,17 @@ For more information, refer to [Vector databases](vdbs.md). For images that `nemoretriever-page-elements-v3` does not classify as tables, charts, or infographics, you can use our VLM caption task to create a dense caption of the detected image. That caption is then embedded along with the rest of your content. -For chart-labeled PDF regions and other caption scope limits, see [Are PDF chart or figure regions captioned when Omni is enabled?](#are-pdf-chart-or-figure-regions-captioned-when-omni-is-enabled). For more information, refer to [Extract Captions from Images](nemo-retriever-api-reference.md). +For chart-labeled PDF regions and other caption scope limits, refer to [Are PDF chart or figure regions captioned when Omni is enabled?](#are-pdf-chart-or-figure-regions-captioned-when-omni-is-enabled). For more information, refer to [Extract Captions from Images](nemo-retriever-api-reference.md). ## Are PDF chart or figure regions captioned when Omni is enabled? -No. Chart-labeled PDF regions are not routed through Omni captioning. See [Image captioning](prerequisites-support-matrix.md#image-captioning-2605) for scope, validation, and what the caption stage covers. +No. Chart-labeled PDF regions are not routed through Omni captioning. Refer to [Charts and infographics](multimodal-extraction.md#charts-and-infographics) and [Image captioning](multimodal-extraction.md#image-captioning) for caption scope and validation. ## When should I consider advanced visual parsing? -For scanned documents, or documents with complex layouts, -you can use [nemotron-parse](https://build.nvidia.com/nvidia/nemotron-parse) as an alternate PDF extraction method by setting `extract_method="nemotron_parse"`. +For scanned documents, or documents with complex layouts, +you can use [nemotron-parse](https://build.nvidia.com/nvidia/nemotron-parse) as an alternate PDF extraction method by setting `extract_method="nemotron_parse"`. +Nemotron Parse does not produce chart modality rows. For chart detection and chart-filtered retrieval, use the default **pdfium** layout path instead (refer to [Charts and infographics](multimodal-extraction.md#charts-and-infographics)). For more information, refer to [Nemotron Parse](https://build.nvidia.com/nvidia/nemotron-parse). ## Why are the environment variables different between library mode and self-hosted mode? @@ -40,11 +41,9 @@ For more information, refer to [Nemotron Parse](https://build.nvidia.com/nvidia/ For [self-hosted deployments](deployment-options.md#when-to-self-host-nims), you should set the environment variables `NGC_API_KEY` and `NIM_NGC_API_KEY`. For more information, refer to [Authentication and API keys](api-keys.md). -For advanced scenarios, you might want to set environment variables for NIM container paths, tags, and batch sizes on the ingestion runtime. Configure them in your Helm values, Kubernetes `Secret`/`ConfigMap`, or follow [Environment variables](environment-config.md). - ### Library Mode -For production environments, you should use the provided Helm charts. When you run the NeMo Retriever Library from Python (without those charts), you should set the environment variable `NVIDIA_API_KEY`. This is because the NeMo Retriever containers and the NeMo Retriever services running inside them do not have access to arbitrary variables on your laptop or jump host unless you inject them into the workload (for example via Helm, `Secret`, or the client environment as documented on [Deployment options](deployment-options.md) and [Authentication and API keys](api-keys.md)). +For production environments, you should use the provided Helm charts. When you run the NeMo Retriever Library from Python without those charts, set `NVIDIA_API_KEY` only when you call [build.nvidia.com](https://build.nvidia.com/) hosted inference—it is not required for locally deployed Hugging Face models or self-hosted NIM endpoints. For more information, refer to [Deployment options](deployment-options.md) and [Authentication and API keys](api-keys.md). For advanced scenarios, you might want to use library mode with self-hosted NIM instances. You can set custom endpoints for each NIM. diff --git a/docs/docs/extraction/getting-started-about.md b/docs/docs/extraction/getting-started-about.md index 6b21eac7a0..8f3a8ee69f 100644 --- a/docs/docs/extraction/getting-started-about.md +++ b/docs/docs/extraction/getting-started-about.md @@ -7,8 +7,8 @@ Typical order: 1. [Get your API key](api-keys.md) (NGC / API access as required by your workflow). 2. Confirm the [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) for your OS, GPU, and software stack. 3. Deploy using one of: - - [Deployment options](deployment-options.md) for how to run NeMo Retriever Library - - **Supported:** [Helm chart](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) for Kubernetes, plus [NeMo Retriever Library install docs](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) for the published charts -4. Explore [Jupyter Notebooks](notebooks/index.md) for end-to-end examples. + - [Deployment options](deployment-options.md) for library, hosted NIMs, and Kubernetes paths + - **Supported:** [Helm chart](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) for Kubernetes cluster install and upgrade +4. Explore [Jupyter Notebooks](starter-kits.md) for end-to-end examples. If you are new to the product, read [What is NeMo Retriever Library?](overview.md) and [Concepts](concepts.md) under **Introduction** first. diff --git a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md deleted file mode 100644 index 7ee0dda650..0000000000 --- a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md +++ /dev/null @@ -1,22 +0,0 @@ -# Integrate with LangChain, LlamaIndex, and Haystack - -NeMo Retriever Library is commonly used **behind** retrieval-augmented generation (RAG) apps built with popular orchestration frameworks. - -## Jupyter examples (LangChain and LlamaIndex) - -The repository includes notebooks that demonstrate multimodal RAG patterns: - -- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) -- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) - -These are also linked from [Jupyter Notebooks](notebooks/index.md) and the [FAQ](faq.md). - -## Haystack - -Haystack-related extraction modes may appear in API tables as **deprecated** in favor of current pipeline options. For up-to-date integration patterns, prefer the Python API and CLI docs, and check [Release notes](releasenotes.md) for migration notes. - -## Related - -- [Use the Python API](nemo-retriever-api-reference.md) -- [Use the CLI](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli) -- [Chunking](concepts.md#chunking), [Upload data](vdbs.md), [Filter search](custom-metadata.md) diff --git a/docs/docs/extraction/multimodal-extraction.md b/docs/docs/extraction/multimodal-extraction.md index 5e6a5a4fb5..f44aee36cf 100644 --- a/docs/docs/extraction/multimodal-extraction.md +++ b/docs/docs/extraction/multimodal-extraction.md @@ -27,7 +27,7 @@ NeMo Retriever Library accepts multiple document and media types. A current list For PDFs, NeMo Retriever Library typically uses **pdfium**-based extraction with configurable depth and paths. Scanned or mixed pages may use hybrid, OCR-oriented, or Nemotron Parse methods. For `extract_method` options such as `pdfium`, `pdfium_hybrid`, `ocr`, and `nemotron_parse`, refer to the [Python API reference](nemo-retriever-api-reference.md). !!! note - `extract_method="nemotron_parse"` requires the Nemotron Parse NIM client dependencies. Install them with the `nemotron-parse` extra, for example `pip install "nemo-retriever[nemotron-parse]"`, before running PDF extraction through Nemotron Parse. + `extract_method="nemotron_parse"` requires the Nemotron Parse NIM client dependencies. Install them with the `nemotron-parse` extra, for example `pip install "nemo-retriever[nemotron-parse]"`, before running PDF extraction through Nemotron Parse. This path does not produce chart modality rows; for chart detection, refer to [Charts and infographics](#charts-and-infographics). **Related** @@ -49,7 +49,18 @@ NeMo Retriever Library detects tables as structured page elements, processes the Charts and infographic regions are classified with other page layout elements (tables, text blocks, titles) and processed through layout detection and OCR. `extract_charts` and `extract_infographics` are enabled by default. Outputs use the same metadata schema as other extracted objects. -Chart-labeled PDF regions are **not** routed through the Omni caption stage; they remain on the layout-and-OCR path. For scope and validation guidance, see [Image captioning](prerequisites-support-matrix.md#image-captioning-2605). +!!! important "Chart modality requires the default layout path" + [Nemotron Parse v1.2](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2) semantic classes do not include `Chart` or `Infographic`. The model labels regions as `Text`, `Table`, `Picture`, `Caption`, `List-item`, `Section-header`, and similar types instead. + + When you set `extract_method="nemotron_parse"`: + + - The pipeline does not produce `chart` or `infographic` modality rows, even when `extract_charts=True` or `extract_infographics=True`. + - Chart- and infographic-filtered retrieval (for example, queries scoped to figure or chart content) returns no hits. + - Chart-heavy and infographic-heavy pages are typically emitted as `Picture` or other non-chart modalities. + + For chart and infographic detection and modality-specific retrieval, use the default **pdfium** layout path (page-elements detection and OCR), not `extract_method="nemotron_parse"`. + +For how chart-labeled PDF regions interact with captioning, refer to [Image captioning](#image-captioning). For natural-language infographic descriptions, optionally enable [image captioning](#image-captioning) and set `caption_infographics=True` when you need VLM captions on infographic regions. @@ -63,7 +74,7 @@ For natural-language infographic descriptions, optionally enable [image captioni Scanned PDFs and image-only pages rely on OCR and hybrid paths that combine native text extraction with OCR when needed. For extract methods such as `ocr` and `pdfium_hybrid`, refer to the [Python API reference](nemo-retriever-api-reference.md). -OCR artifacts depend on how you deploy. **Helm / NIM:** the production chart uses **Nemotron OCR v1** (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). **Local Hugging Face inference:** the default engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes defaults and the Helm-vs-local split, see [OCR artifacts (Helm vs local Hugging Face)](prerequisites-support-matrix.md#nemotron-ocr-v2-language-mode) in the support matrix. +When you run extraction locally with Hugging Face weights, the default OCR engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes deployment, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. **Related** @@ -77,11 +88,13 @@ Image captioning generates natural-language descriptions for unstructured image **Captioning is optional** — enable it in your ingest configuration (for example, the `caption` API or pipeline flag) when you need natural-language descriptions of image content. Reasoning traces are disabled by default for captioning. +Chart-classified PDF regions stay on the layout/OCR path; only non-chart image regions and optional infographics (`caption_infographics=True`) receive Omni captions. + **Related** - [Multimodal embeddings (VLM)](embedding.md) - [Metadata reference](content-metadata.md) -- [Image captioning](prerequisites-support-matrix.md#image-captioning-2605) +- [Image captioning](prerequisites-support-matrix.md#image-captioning) ## Metadata and content schema { #metadata-and-content-schema } diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index da21b30a40..2841b9799f 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -1,10 +1,10 @@ # NeMo Retriever API Reference -## PDF pre-splitting for parallel ingest +## PDF pre-splitting for parallel ingest { #pdf-pre-splitting-for-parallel-ingest } Large PDFs are split into page batches before Ray processing so extraction can run in parallel. This happens on the default ingest path; you do not need extra configuration for typical workloads. -To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). See [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. +To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). Refer to [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. **Python client (`pdf_split_config`):** Only `create_ingestor(run_mode="service")` implements `.pdf_split_config(pages_per_chunk=...)`, which records page-chunking settings in the request pipeline spec for the remote gateway. Local graph ingest (`run_mode="inprocess"` or `"batch"`) raises `NotImplementedError` if you call this method; PDFs are split automatically on the default ingest path without client-side configuration. @@ -13,6 +13,6 @@ To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray acto filters: - "!^pdf_split_config$" -::: nemo_retriever.retriever +::: nemo_retriever.graph.retriever -::: nemo_retriever.params +::: nemo_retriever.common.params diff --git a/docs/docs/extraction/nimclient.md b/docs/docs/extraction/nimclient.md deleted file mode 100644 index 755db2b366..0000000000 --- a/docs/docs/extraction/nimclient.md +++ /dev/null @@ -1,588 +0,0 @@ -# NimClient Usage Guide for NeMo Retriever Library - -The `NimClient` class provides a unified interface for connecting to and interacting with NVIDIA NIM Microservices. -This documentation demonstrates how to create custom NIM integrations for use in [NeMo Retriever Library](overview.md) pipelines and User Defined Functions (UDFs). - -The NimClient architecture consists of two main components: - -1. **NimClient**: The client class that handles communication with NIM endpoints via gRPC or HTTP protocols -2. **ModelInterface**: An abstract base class that defines how to format input data, parse output responses, and process inference results for specific models - -For advanced usage patterns, refer to the existing model interfaces in `nemo_retriever/src/nemo_retriever/api/internal/primitives/nim/model_interface/`. - - -## Quick Start - -For ingest and pipeline APIs used with NimClient in UDFs, refer to the [Python API guide](nemo-retriever-api-reference.md). - -### Basic NimClient Creation - -```python -from nemo_retriever.api.util.nim import create_inference_client -from nemo_retriever.api.internal.primitives.nim import ModelInterface - -# Create a custom model interface (refer to examples below) -model_interface = MyCustomModelInterface() - -# Define endpoints (gRPC, HTTP) -endpoints = ("grpc://my-nim-service:8001", "http://my-nim-service:8000") - -# Create the client -client = create_inference_client( - endpoints=endpoints, - model_interface=model_interface, - auth_token="your-ngc-api-key", # Optional - infer_protocol="grpc", # Optional: "grpc" or "http" - timeout=120.0, # Optional: request timeout - max_retries=5 # Optional: retry attempts -) - -# Perform inference -data = {"input": "your input data"} -results = client.infer(data, model_name="your-model-name") -``` - -### Using Environment Variables - -```python -import os -from nemo_retriever.api.util.nim import create_inference_client - -# Use environment variables for configuration -auth_token = os.getenv("NGC_API_KEY") -grpc_endpoint = os.getenv("NIM_GRPC_ENDPOINT", "grpc://localhost:8001") -http_endpoint = os.getenv("NIM_HTTP_ENDPOINT", "http://localhost:8000") - -client = create_inference_client( - endpoints=(grpc_endpoint, http_endpoint), - model_interface=model_interface, - auth_token=auth_token -) -``` - -## Creating Custom Model Interfaces - -To integrate a new NIM, you need to create a custom `ModelInterface` subclass that implements the required methods. - -### Basic Model Interface Template - -```python -from typing import Dict, Any, List, Tuple, Optional -import numpy as np -from nemo_retriever.api.internal.primitives.nim import ModelInterface - -class MyCustomModelInterface(ModelInterface): - """ - Custom model interface for My Custom NIM. - """ - - def __init__(self, model_name: str = "my-custom-model"): - """Initialize the model interface.""" - self.model_name = model_name - - def name(self) -> str: - """Return the name of this model interface.""" - return "MyCustomModel" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - """ - Prepare and validate input data before formatting. - - Parameters - ---------- - data : dict - Raw input data - - Returns - ------- - dict - Validated and prepared data - """ - # Validate required fields - if "input_text" not in data: - raise KeyError("Input data must include 'input_text'") - - # Ensure input is in the expected format - if not isinstance(data["input_text"], str): - raise ValueError("input_text must be a string") - - return data - - def format_input( - self, - data: Dict[str, Any], - protocol: str, - max_batch_size: int, - **kwargs - ) -> Tuple[List[Any], List[Dict[str, Any]]]: - """ - Format input data for the specified protocol. - - Parameters - ---------- - data : dict - Prepared input data - protocol : str - Communication protocol ("grpc" or "http") - max_batch_size : int - Maximum batch size for processing - **kwargs : dict - Additional parameters - - Returns - ------- - tuple - (formatted_batches, batch_data_list) - """ - if protocol == "http": - return self._format_http_input(data, max_batch_size, **kwargs) - elif protocol == "grpc": - return self._format_grpc_input(data, max_batch_size, **kwargs) - else: - raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") - - def _format_http_input( - self, - data: Dict[str, Any], - max_batch_size: int, - **kwargs - ) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]: - """Format input for HTTP protocol.""" - input_text = data["input_text"] - - # Create HTTP payload - payload = { - "model": kwargs.get("model_name", self.model_name), - "input": input_text, - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.7), - } - - # Return as single batch - return [payload], [{"original_input": input_text}] - - def _format_grpc_input( - self, - data: Dict[str, Any], - max_batch_size: int, - **kwargs - ) -> Tuple[List[np.ndarray], List[Dict[str, Any]]]: - """Format input for gRPC protocol.""" - input_text = data["input_text"] - - # Convert to numpy array for gRPC - text_array = np.array([[input_text.encode("utf-8")]], dtype=np.object_) - - return [text_array], [{"original_input": input_text}] - - def parse_output( - self, - response: Any, - protocol: str, - data: Optional[Dict[str, Any]] = None, - **kwargs - ) -> Any: - """ - Parse the raw model response. - - Parameters - ---------- - response : Any - Raw response from the model - protocol : str - Communication protocol used - data : dict, optional - Original batch data - **kwargs : dict - Additional parameters - - Returns - ------- - Any - Parsed response data - """ - if protocol == "http": - return self._parse_http_response(response) - elif protocol == "grpc": - return self._parse_grpc_response(response) - else: - raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") - - def _parse_http_response(self, response: Dict[str, Any]) -> str: - """Parse HTTP response.""" - if isinstance(response, dict): - # Extract the generated text from response - if "choices" in response: - return response["choices"][0].get("text", "") - elif "output" in response: - return response["output"] - else: - raise RuntimeError("Unexpected response format") - return str(response) - - def _parse_grpc_response(self, response: np.ndarray) -> str: - """Parse gRPC response.""" - if isinstance(response, np.ndarray): - # Decode bytes response - return response.flatten()[0].decode("utf-8") - return str(response) - - def process_inference_results( - self, - output: Any, - protocol: str, - **kwargs - ) -> Any: - """ - Post-process the parsed inference results. - - Parameters - ---------- - output : Any - Parsed output from parse_output - protocol : str - Communication protocol used - **kwargs : dict - Additional parameters - - Returns - ------- - Any - Final processed results - """ - # Apply any final processing (e.g., filtering, formatting) - if isinstance(output, str): - return output.strip() - return output -``` - -## Real-World Examples - -### Text Generation Model Interface - -```python -class TextGenerationModelInterface(ModelInterface): - """Interface for text generation NIMs (e.g., LLaMA, GPT-style models).""" - - def name(self) -> str: - return "TextGeneration" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - if "prompt" not in data: - raise KeyError("Input data must include 'prompt'") - return data - - def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): - prompt = data["prompt"] - - if protocol == "http": - payload = { - "model": kwargs.get("model_name", "llama-2-7b-chat"), - "messages": [{"role": "user", "content": prompt}], - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.7), - "top_p": kwargs.get("top_p", 0.9), - "stream": False - } - return [payload], [{"prompt": prompt}] - else: - raise ValueError("Only HTTP protocol supported for this model") - - def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): - if protocol == "http" and isinstance(response, dict): - choices = response.get("choices", []) - if choices: - return choices[0].get("message", {}).get("content", "") - return str(response) - - def process_inference_results(self, output: Any, protocol: str, **kwargs): - return output.strip() if isinstance(output, str) else output -``` - -### Image Analysis Model Interface - -```python -import base64 -from nemo_retriever.api.util.image_processing.transforms import numpy_to_base64 - -class ImageAnalysisModelInterface(ModelInterface): - """Interface for image analysis NIMs (e.g., vision models).""" - - def name(self) -> str: - return "ImageAnalysis" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - if "images" not in data: - raise KeyError("Input data must include 'images'") - - # Ensure images is a list - if not isinstance(data["images"], list): - data["images"] = [data["images"]] - - return data - - def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): - images = data["images"] - prompt = data.get("prompt", "Describe this image.") - - # Convert images to base64 if needed - base64_images = [] - for img in images: - if isinstance(img, np.ndarray): - base64_images.append(numpy_to_base64(img)) - elif isinstance(img, str) and img.startswith("data:image"): - # Already base64 encoded - base64_images.append(img.split(",")[1]) - else: - base64_images.append(str(img)) - - # Batch images - batches = [base64_images[i:i + max_batch_size] - for i in range(0, len(base64_images), max_batch_size)] - - payloads = [] - batch_data_list = [] - - for batch in batches: - if protocol == "http": - messages = [] - for img_b64 in batch: - messages.append({ - "role": "user", - "content": f'{prompt} ' - }) - - payload = { - "model": kwargs.get("model_name", "llava-1.5-7b-hf"), - "messages": messages, - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.1) - } - payloads.append(payload) - batch_data_list.append({"images": batch, "prompt": prompt}) - - return payloads, batch_data_list - - def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): - if protocol == "http" and isinstance(response, dict): - choices = response.get("choices", []) - return [choice.get("message", {}).get("content", "") for choice in choices] - return [str(response)] - - def process_inference_results(self, output: Any, protocol: str, **kwargs): - if isinstance(output, list): - return [result.strip() for result in output] - return output -``` - -## Using NimClient in UDFs - -### Basic UDF with NimClient - -```python -from nemo_retriever.api.internal.primitives.ingest_control_message import IngestControlMessage -from nemo_retriever.api.util.nim import create_inference_client -import os - -def analyze_document_with_nim(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF that uses a custom NIM to analyze document content.""" - - # Create NIM client - model_interface = TextGenerationModelInterface() - client = create_inference_client( - endpoints=( - os.getenv("ANALYSIS_NIM_GRPC", "grpc://analysis-nim:8001"), - os.getenv("ANALYSIS_NIM_HTTP", "http://analysis-nim:8000") - ), - model_interface=model_interface, - auth_token=os.getenv("NGC_API_KEY"), - infer_protocol="http" - ) - - # Get the document DataFrame - df = control_message.get_payload() - - # Process each document - for idx, row in df.iterrows(): - if row.get("content"): - # Prepare analysis prompt - prompt = f"Analyze the following document content and provide a summary: {row['content'][:1000]}" - - # Perform inference - try: - results = client.infer( - data={"prompt": prompt}, - model_name="llama-2-7b-chat", - max_tokens=256, - temperature=0.3 - ) - - # Add analysis to metadata - if results: - analysis = results[0] if isinstance(results, list) else results - df.at[idx, "custom_analysis"] = analysis - - except Exception as e: - print(f"NIM inference failed: {e}") - df.at[idx, "custom_analysis"] = "Analysis failed" - - # Update the control message with processed data - control_message.payload(df) - return control_message -``` - -### Advanced UDF with Batching - -```python -def batch_image_analysis_udf(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF that performs batched image analysis using NIM.""" - - # Create image analysis client - model_interface = ImageAnalysisModelInterface() - client = create_inference_client( - endpoints=( - os.getenv("VISION_NIM_GRPC", "grpc://vision-nim:8001"), - os.getenv("VISION_NIM_HTTP", "http://vision-nim:8000") - ), - model_interface=model_interface, - auth_token=os.getenv("NGC_API_KEY") - ) - - df = control_message.get_payload() - - # Collect all images for batch processing - image_rows = [] - images = [] - - for idx, row in df.iterrows(): - if "image_data" in row and row["image_data"]: - image_rows.append(idx) - images.append(row["image_data"]) - - if images: - try: - # Batch process all images - results = client.infer( - data={ - "images": images, - "prompt": "Describe the content and key elements in this image." - }, - model_name="llava-1.5-7b-hf", - max_tokens=200 - ) - - # Apply results back to DataFrame - for idx, result in zip(image_rows, results): - df.at[idx, "image_description"] = result - - except Exception as e: - print(f"Batch image analysis failed: {e}") - for idx in image_rows: - df.at[idx, "image_description"] = "Analysis failed" - - control_message.payload(df) - return control_message -``` - -## Configuration and Best Practices - -### Environment Variables - -Set these environment variables for your NIM endpoints: - -```bash -# NIM endpoints -export MY_NIM_GRPC_ENDPOINT="grpc://my-nim-service:8001" -export MY_NIM_HTTP_ENDPOINT="http://my-nim-service:8000" - -# Authentication -export NGC_API_KEY="your-ngc-api-key" - -# Optional: timeouts and retries -export NIM_TIMEOUT=120 -export NIM_MAX_RETRIES=5 -``` - -### Performance Optimization - -1. **Use gRPC when possible**: Generally faster than HTTP for high-throughput scenarios -2. **Batch processing**: Process multiple items together to reduce overhead -3. **Connection reuse**: Create NimClient instances once and reuse them -4. **Appropriate timeouts**: Set reasonable timeouts based on your model's response time -5. **Error handling**: Always handle inference failures gracefully - -### Error Handling - -```python -def robust_nim_udf(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF with comprehensive error handling.""" - - try: - client = create_inference_client( - endpoints=(grpc_endpoint, http_endpoint), - model_interface=model_interface, - auth_token=auth_token, - timeout=60.0, - max_retries=3 - ) - except Exception as e: - print(f"Failed to create NIM client: {e}") - return control_message - - df = control_message.get_payload() - - for idx, row in df.iterrows(): - try: - results = client.infer(data=input_data, model_name="my-model") - df.at[idx, "nim_result"] = results - except TimeoutError: - print(f"NIM request timed out for row {idx}") - df.at[idx, "nim_result"] = "timeout" - except Exception as e: - print(f"NIM inference failed for row {idx}: {e}") - df.at[idx, "nim_result"] = "error" - - control_message.payload(df) - return control_message -``` - -## Troubleshooting - -### Common Issues - -* **Connection Errors** – Verify NIM service is running and endpoints are correct -* **Authentication Failures** – Check NGC_API_KEY is valid and properly set -* **Timeout Errors** – Increase timeout values or check NIM service performance -* **Format Errors** – Ensure your ModelInterface formats data correctly for your NIM -* **Memory Issues** – Use appropriate batch sizes to avoid memory exhaustion - -### NIM Triton Limit Memory - -If you encounter memory issues, try increasing the `NIM_TRITON_CUDA_MEMORY_POOL_MB` parameter. This adjustment typically does not affect performance. - -If memory issues persist, you can reduce the `NIM_TRITON_RATE_LIMIT` value — even down to 1. However, lowering this parameter affects performance. - -### Debugging Tips - -```python -import logging - -# Enable debug logging -logging.getLogger("nemo_retriever.api.internal.primitives.nim").setLevel(logging.DEBUG) - -# Test your model interface separately -model_interface = MyCustomModelInterface() -test_data = {"input": "test"} - -# Test data preparation -prepared = model_interface.prepare_data_for_inference(test_data) -print(f"Prepared data: {prepared}") - -# Test input formatting -formatted, batch_data = model_interface.format_input(prepared, "http", 1) -print(f"Formatted input: {formatted}") -``` - -## Related Topics - -- [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph) diff --git a/docs/docs/extraction/overview.md b/docs/docs/extraction/overview.md index f6266aa0d2..f4837ae63b 100644 --- a/docs/docs/extraction/overview.md +++ b/docs/docs/extraction/overview.md @@ -48,6 +48,5 @@ NeMo Retriever Library supports the following file types: - [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) - [Deployment options](deployment-options.md) — library, Helm, hosted vs self-hosted NIMs in one place - [Deploy on Kubernetes with Helm](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) -- [NeMo Retriever Library — prerequisites / deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) (supported Helm charts) -- [Notebooks](notebooks/index.md) -- [NVIDIA AI Blueprints catalog](https://build.nvidia.com/explore/discover) — solution cards, enterprise RAG blueprints, and end-to-end patterns (including [Enterprise RAG — multimodal PDF data extraction](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag)); for integration pathways, refer to [Integrations](integrations-langchain-llamaindex-haystack.md). +- [Notebooks](starter-kits.md) +- [NVIDIA AI Blueprints catalog](https://build.nvidia.com/explore/discover) — solution cards, enterprise RAG blueprints, and end-to-end patterns (including [Enterprise RAG — multimodal PDF data extraction](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag)); for integration pathways, refer to [Starter kits](starter-kits.md). diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index b8910f2ea8..fb4315daab 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -2,7 +2,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your software stack, deployment hardware, and—if you use them—advanced features (audio and video, Nemotron Parse, VLM image captioning, reranking) against the guidance in this page. -## Software Requirements +## Software Requirements { #software-requirements } - Linux operating systems (Ubuntu 22.04 or later recommended) - [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) (NVIDIA Driver >= `580`, CUDA >= `13.0`) @@ -11,8 +11,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw - For audio and video, `ffmpeg` and `ffprobe` must be on `PATH` (for example `sudo apt-get install -y --no-install-recommends ffmpeg` on Debian/Ubuntu). `ffmpeg-python` and `nemo-retriever[multimedia]` do not install these binaries. - On Helm with package-repo access, set `service.installFfmpeg=true`. For - air-gapped clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). + For container and Kubernetes guidance, refer to [Audio and video](audio-video.md). - For PDF extraction with `extract_method="nemotron_parse"`, install the Nemotron Parse client dependencies with `pip install "nemo-retriever[nemotron-parse]"` (pulls `open-clip-torch`, which provides the `open_clip` module required by the Nemotron Parse @@ -23,7 +22,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw When you use UV, create the environment with Python 3.12 — for example, `uv venv --python 3.12`. This matches the `requires-python` metadata in the library packages. -## Hardware Requirements +## Hardware Requirements { #hardware-requirements } The full ingestion pipeline is designed to consume significant CPU and memory resources to achieve maximal parallelism. Resource usage scales up to the limits of your deployed system. @@ -60,32 +59,32 @@ For production deployments processing large volumes of documents, consider: Ensure your deployment environment meets these specifications before running the full pipeline. Resource-constrained environments may experience performance degradation. -## Core and Advanced Pipeline Features +## Core and Advanced Pipeline Features { #core-and-advanced-pipeline-features } The NeMo Retriever Library extraction core pipeline features run on a single A10G or better GPU. -### Default Helm NIMs +### Default Helm NIMs { #default-helm-nims } -The production Helm chart enables these NIM microservices **by default** (for example via `nimOperator.*.enabled=true`): +The production Helm chart enables these NIM microservices **by default** (for example through `nimOperator.*.enabled=true`): | Helm flag | NIM | Role | |-----------|-----|------| | `page_elements` | [nemotron-page-elements-v3](https://huggingface.co/nvidia/nemotron-page-elements-v3) | Page layout and element detection | | `table_structure` | [nemotron-table-structure-v1](https://huggingface.co/nvidia/nemotron-table-structure-v1) | Table structure extraction | -| `ocr` | [nemotron-ocr-v1](https://huggingface.co/nvidia/nemotron-ocr-v1) | Image OCR | +| `ocr` | [nemotron-ocr-v2](https://huggingface.co/nvidia/nemotron-ocr-v2) | Image OCR | | `vlm_embed` | [llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | Multimodal (VL) embedding | ### OCR artifacts (Helm vs local Hugging Face) { #nemotron-ocr-v2-language-mode } !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, see [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v2** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. - **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. + **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: -- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` +- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` Default VL embedder container and model for release deployments: @@ -94,54 +93,43 @@ Default VL embedder container and model for release deployments: ### Optional Helm NIMs (not auto-wired) { #optional-helm-nims-not-auto-wired-by-default } -These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). For **26.05 production**, disable keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#nim-operator-sub-stack). +These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). In production, disable keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#nim-operator-sub-stack). | Helm flag | NIM | Role | |-----------|-----|------| | `rerankqa` | [llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2) | Reranking for improved retrieval accuracy | | `nemotron_parse` | [nemotron-parse](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2) | Optional PDF `extract_method="nemotron_parse"` (default PDF extraction uses **pdfium**) | -| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning for 26.05 when you enable the caption stage | +| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning when you enable the caption stage | | `audio` | [parakeet-1-1b-ctc-en-us](https://huggingface.co/nvidia/parakeet-ctc-1.1b) | [Audio and video](audio-video.md) transcription | -### Image captioning (26.05) { #image-captioning-2605 } +### Image captioning { #image-captioning } -For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. - -!!! important "PDF chart regions are not captioned by Omni" - - When **nemotron-page-elements-v3** classifies a PDF region as **chart**, that region is processed through layout detection and OCR—not the Omni caption stage. Enabling the caption NIM and the `caption` pipeline stage does **not** send chart-labeled figures to `/v1/chat/completions`. - - The caption stage covers: - - - Unstructured content in the `images` column (standalone image files and page-element regions **not** classified as table, chart, or infographic) - - Optional infographic regions when you set `caption_infographics=True` on `CaptionParams` (the VLM caption is stored in `caption`, separate from OCR `text`) - - To validate caption traffic during ingest, inspect metadata such as `page_elements_v3_counts_by_label`. If the figure is labeled `chart`, expect no Omni chat-completions requests for that region even when captioning is enabled. +Use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. Optional features listed in the table above require additional GPU support, disk space, and feature-specific system dependencies beyond the four default NIMs. For published NIM model IDs and deployment-specific constraints, use the product support matrices linked under [Related Topics](#related-topics) below. -## Model Hardware Requirements +## Model Hardware Requirements { #model-hardware-requirements } NeMo Retriever Library supports the following GPU hardware given system constraints in the table. - **HF model weights** — approximate Hugging Face checkpoint footprint (files such as `model*.safetensors`, `weights.pth`, or other published weight bundles in the model repository). Values are rounded from the current public file listing and can change when the repository is updated. -- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, see the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). +- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, refer to the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). Model repositories and NIM references are linked in [Core and Advanced Pipeline Features](#core-and-advanced-pipeline-features) above. -**B200 and audio/video extraction (26.05):** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR via `nimOperator.audio`) is **not supported on B200** or other Blackwell GPUs. Core PDF and multimodal extraction on B200 is unchanged. See footnote ⁴ below. +**B200, H200 NVL, and audio/video extraction:** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR through `nimOperator.audio`) is **not supported on B200**, other Blackwell GPUs, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Refer to footnote ⁴ below. | Feature | HF Model Weights | GPU Option | [RTX Pro 6000](https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/) | [B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) | [H200 NVL](https://www.nvidia.com/en-us/data-center/h200/) | [H100](https://www.nvidia.com/en-us/data-center/h100/) | [A100 80GB](https://www.nvidia.com/en-us/data-center/a100/) | A100 40GB | [A10G](https://aws.amazon.com/ec2/instance-types/g5/) | L40S | [RTX PRO 4500 Blackwell](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4500/) | |---------|------------------|------------|--------|--------|--------|--------|--------|--------|--------|--------|------------------------| | GPU | — | Memory | 96GB | 180GB | 141GB | 80GB | 80GB | 40GB | 24GB | 48GB | 32GB GDDR7 (GB203) | | Core Features | ~4.8 GiB combined: embed VL 1b ~3.1 GiB; page-elements ~0.41 GiB; table-structure ~0.81 GiB; OCR ~0.51 GiB | Total GPUs | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | Core Features | — | Total Disk Space | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | -| Audio/video extraction (parakeet-1-1b-ctc-en-us) | ~4.0 GiB (`model.safetensors`; the repo also ships `parakeet-ctc-1.1b.nemo` of similar size—use one format to avoid roughly doubling disk use) | Additional Dedicated GPUs | Not supported⁴ | Not supported⁴ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | Not supported⁴ | -| | — | Additional Disk Space | Not supported⁴ | Not supported⁴ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | Not supported⁴ | -| nemotron-parse | ~3.5 GiB | Additional Dedicated GPUs | Not supported | 1 | Not supported | 1 | 1 | 1 | 1 | 1 | Not supported² | -| nemotron-parse | — | Additional Disk Space | Not supported | ~16GB | Not supported | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | Not supported² | +| Audio/video extraction (parakeet-1-1b-ctc-en-us) | ~4.0 GiB (`model.safetensors`; the repo also ships `parakeet-ctc-1.1b.nemo` of similar size—use one format to avoid roughly doubling disk use) | Additional Dedicated GPUs | Not supported⁴ | Not supported⁴ | Not supported⁴ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | Not supported⁴ | +| | — | Additional Disk Space | Not supported⁴ | Not supported⁴ | Not supported⁴ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | Not supported⁴ | +| nemotron-parse | ~3.5 GiB | Additional Dedicated GPUs | Not supported | 1 | Not supported | 1 | 1 | 1 | 1 | 1 | 1 | +| nemotron-parse | — | Additional Disk Space | Not supported | ~16GB | Not supported | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | ~62 GiB (BF16); ~33 GiB (FP8); ~21 GiB (NVFP4) | Additional Dedicated GPUs | 1 | 1 | 1 | 1 | 1 | Not supported | Not supported | 2 | Not supported³ | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | — | Additional Disk Space (HF) | ~21–62GB | ~21–62GB | ~21–62GB | ~21–62GB | ~21–62GB | Not supported | Not supported | ~21–62GB | Not supported³ | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | — | Additional Disk Space (NIM) | ~80GB | ~80GB | ~80GB | ~80GB | ~80GB | Not supported | Not supported | ~80GB | Not supported³ | @@ -150,17 +138,15 @@ Model repositories and NIM references are linked in [Core and Advanced Pipeline ¹ On other supported GPUs, Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`) may require a runtime TensorRT engine build (no prebuilt profile in the chart image). -⁴ On **B200** and other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, self-hosted [audio/video extraction](audio-video.md) via Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported**. Core PDF and multimodal extraction on Blackwell is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a non-Blackwell dedicated GPU, [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. - -² Nemotron Parse fails to start on 32GB. +⁴ Self-hosted [audio/video extraction](audio-video.md) through Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported** on **B200**, other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a supported dedicated GPU (for example H100 or A100), [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. -³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; see the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. +³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; refer to the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. \* GPUs with less than 80GB VRAM cannot run the reranker concurrently with the core pipeline. To perform recall testing with the reranker on these GPUs, shut down the core pipeline NIM microservices and run only the embedder, reranker, and your vector database. -## Related Topics +## Related Topics { #related-topics } - [Troubleshooting](troubleshoot.md) - [Release Notes](releasenotes.md) diff --git a/docs/docs/extraction/releasenotes.md b/docs/docs/extraction/releasenotes.md index e8c5b34b86..dd1a2b483a 100644 --- a/docs/docs/extraction/releasenotes.md +++ b/docs/docs/extraction/releasenotes.md @@ -25,8 +25,8 @@ Highlights for the 26.05 release include: ### CLI -- Root CLI adds `retriever ingest` and `retriever query` with NIM URL flags, batch tuning, and LanceDB overwrite/append controls, plus `retriever pipeline` for graph execution -- For product use, only `retriever ingest`, `retriever query`, and `retriever pipeline` (for example `retriever pipeline run`) are supported; other top-level subcommands—including `pdf`, `html`, `eval`, `benchmark`, `harness`, `online`, `compare`, `image`, and `skill-eval`—are development and experimental +- Root CLI adds first-class `retriever ingest` and `retriever query` commands with NIM URL flags, batch tuning, and LanceDB overwrite/append controls +- For product ingest and retrieval, prefer `retriever ingest` and `retriever query`; `retriever pipeline run` remains available for compatibility and development workflows. Other top-level subcommands—including `pdf`, `html`, `eval`, `benchmark`, `harness`, `online`, `compare`, `image`, and `skill-eval`—are development and experimental ### Retriever Service and deployment @@ -64,7 +64,6 @@ Highlights for the 26.05 release include: ### Packaging and platform -- GA PyPI install: `uv pip install nemo-retriever==26.5.0` (refer to the library [quickstart](https://github.com/NVIDIA/NeMo-Retriever/tree/26.05/nemo_retriever#setup-your-environment)) - Optional install extras (`[local]`, `[multimedia]`, `[llm]`, `[tabular]`, `[nemotron-parse]`, `[service]`, and others), including slim remote/NIM-only installs on Mac and Windows ### Helm chart diff --git a/docs/docs/extraction/starter-kits.md b/docs/docs/extraction/starter-kits.md new file mode 100644 index 0000000000..08b366a6eb --- /dev/null +++ b/docs/docs/extraction/starter-kits.md @@ -0,0 +1,24 @@ +# Starter Kits for NeMo Retriever Library + +To get started using [NeMo Retriever Library](overview.md), you can try one of the ready-made notebooks that are available. + +## Dataset Downloads for Benchmarking + +If you plan to run benchmarking or evaluation tests, you must download the [Benchmark Datasets (Bo20, Bo767, Bo10k)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/digital_corpora_download.ipynb) from Digital Corpora. This is a prerequisite for all benchmarking operations. + +## Getting Started + +To get started with the basics, try one of the following guides or notebooks: + +- [Quickstart: retriever CLI](../reference/retriever-cli-quickstart.md) +- [Workflow: Ingest documents](workflow-document-ingestion.md) +- [Adding Custom Metadata for Filtered Search/Retrieval](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — also summarized on [Vector databases — Metadata and filtering](vdbs.md#metadata-and-filtering) + + +For more advanced scenarios, try one of the following notebooks: + +- [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb) +- [Try Enterprise RAG Blueprint](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) +- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb) +- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) +- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) diff --git a/docs/docs/extraction/troubleshoot.md b/docs/docs/extraction/troubleshoot.md index 4d0bc9c8d9..127fee07a6 100644 --- a/docs/docs/extraction/troubleshoot.md +++ b/docs/docs/extraction/troubleshoot.md @@ -4,8 +4,8 @@ Use this documentation to troubleshoot issues that arise when you use [NeMo Retr ## Can't process long, non-language text strings -NeMo Retriever Library is designed to process language and language-length strings. -If you submit a document that contains extremely long, or non-language text strings, +NeMo Retriever Library is designed to process language and language-length strings. +If you submit a document that contains extremely long, or non-language text strings, such as a DNA sequence, errors or unexpected results occur. ## Can't process malformed input files @@ -17,7 +17,7 @@ When you run a job you might see errors similar to the following: - File may be malformed - Failed to format paragraph -These errors can occur when your input file is malformed. +These errors can occur when your input file is malformed. Verify or fix the format of your input file, and try resubmitting your job. ## Audio or video extraction reports missing media dependencies { #audio-or-video-extraction-reports-missing-media-dependencies } @@ -33,7 +33,7 @@ VideoFrameActor requires media dependencies; missing: ffprobe. The `ffmpeg-python` wrapper and `nemo-retriever[multimedia]` do not install the `ffmpeg` or `ffprobe` binaries the pipeline executes. -For air-gapped or locked-down clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). +For air-gapped or locked-down clusters, refer to [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). **Connected environments:** @@ -60,7 +60,7 @@ This path fails with `allowPrivilegeEscalation: false` or `readOnlyRootFilesyste ## Can't start new thread error -In rare cases, when you run a job you might an see an error similar to `can't start new thread`. +In rare cases, when you run a job you might an see an error similar to `can't start new thread`. This error occurs when the maximum number of processes available to a single user is too low. To resolve the issue, set or raise the maximum number of processes (`-u`) by using the [ulimit](https://ss64.com/bash/ulimit.html) command. Before you change the `-u` setting, consider the following: @@ -76,26 +76,30 @@ ulimit -u 10000 ## Out-of-Memory (OOM) Error when Processing Large Datasets -When you process a very large dataset with thousands of documents, you might encounter an Out-of-Memory (OOM) error. -This happens because, by default, NeMo Retriever Library stores the results from every document in system memory (RAM). +When you process a very large dataset with thousands of documents, you might encounter an Out-of-Memory (OOM) error. +This happens because NeMo Retriever Library materializes extraction results in system memory (RAM) while the job runs. If the total size of the results exceeds the available memory, the process fails. -To resolve this issue, use the `save_to_disk` method. -For details, refer to [Working with Large Datasets: Saving to Disk](nemo-retriever-api-reference.md). +To reduce memory pressure, try one or more of the following: + +- Process documents in smaller batches instead of submitting the entire corpus in one job. +- Route outputs to a sink (for example, `.vdb_upload(...)`, `.webhook(...)`, or `.store(...)`) so results are written out instead of held in memory until the job finishes. +- In `run_mode="service"`, pass `return_results=False` to `.ingest(...)` when you do not need the full result payload returned to the client. For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). +- Increase available host or pod memory for the ingest workload. ## Embedding service fails to start with an unsupported batch size error -On certain hardware, for example RTX 6000, +On certain hardware, for example RTX 6000, the embedding service might fail to start and you might see an error similar to the following. ```bash ValueError: Configured max_batch_size (30) is larger than the model''s supported max_batch_size (3). ``` -If you are using hardware where the embedding NIM uses the ONNX model profile, -you must set `EMBEDDER_BATCH_SIZE=3` in your environment. +If you are using hardware where the embedding NIM uses the ONNX model profile, +you must set `EMBEDDER_BATCH_SIZE=3` in your environment. You can set the variable in your .env file or directly in your environment. @@ -122,18 +126,18 @@ For local GPU inference with Nemotron Parse, combine extras: pip install "nemo-retriever[local,nemotron-parse]" ``` -See also [What is NeMo Retriever Library?](overview.md) and [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md#software-requirements). +Also refer to [What is NeMo Retriever Library?](overview.md) and [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md#software-requirements). ## Extract method nemotron-parse doesn't support image files -Currently, extraction with Nemotron parse doesn't support image files, only scanned PDFs. +Currently, extraction with Nemotron parse doesn't support image files, only scanned PDFs. To work around this issue, convert image files to PDFs before you use `extract_method="nemotron_parse"`. ## Too many open files error -In rare cases, when you run a job you might an see an error similar to `too many open files` or `max open file descriptor`. +In rare cases, when you run a job you might an see an error similar to `too many open files` or `max open file descriptor`. This error occurs when the open file descriptor limit for your service user account is too low. To resolve the issue, set or raise the maximum number of open file descriptors (`-n`) by using the [ulimit](https://ss64.com/bash/ulimit.html) command. Before you change the `-n` setting, consider the following: @@ -149,8 +153,8 @@ ulimit -n 10000 ## Triton server INFO messages incorrectly logged as errors -Sometimes messages are incorrectly logged as errors, when they are information. -When this happens, you can ignore the errors, and treat the messages as information. +Sometimes messages are incorrectly logged as errors, when they are information. +When this happens, you can ignore the errors, and treat the messages as information. For example, you might see log messages that look similar to the following. ```bash @@ -185,4 +189,4 @@ ERROR 2025-04-24 22:49:44.434 nimutils.py:68] } - [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) - [Deployment options](deployment-options.md) - [Deploy with Helm](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) -- [NeMo Retriever Library — prerequisites / deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) (supported **Helm** charts) +- [About getting started](getting-started-about.md) (prerequisites and deployment) diff --git a/docs/docs/extraction/vdbs.md b/docs/docs/extraction/vdbs.md index 67f3b64a3f..d1b6e4de1e 100644 --- a/docs/docs/extraction/vdbs.md +++ b/docs/docs/extraction/vdbs.md @@ -60,7 +60,7 @@ Pass `vdb_op="lancedb"` to `vdb_upload`, or construct a `LanceDB` instance and p For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). ```python -from nemo_retriever.vdb.lancedb import LanceDB +from nemo_retriever.common.vdb.lancedb import LanceDB vdb = LanceDB( uri="./lancedb_data", # Path to LanceDB database directory @@ -83,7 +83,7 @@ When using the `Ingestor` with `vdb_upload`, pass `vdb_op="lancedb"` or a `Lance Semantic retrieval uses dense embeddings to find content that is similar in meaning to a query. In NeMo Retriever Library, the default vector path is LanceDB. Use these resources together with the sections on this page: -- [Metadata and filtering](#metadata-and-filtering) for sidecar metadata at ingest and query-time filters +- [Metadata and filtering](#metadata-and-filtering) for custom metadata at ingest and filtered retrieval - [Concepts](concepts.md) for broader pipeline and search patterns - [Use the NeMo Retriever Library Python API](nemo-retriever-api-reference.md) for `Retriever.query` and `LanceDB.retrieval` parameters @@ -91,10 +91,7 @@ Semantic retrieval uses dense embeddings to find content that is similar in mean ## Metadata and filtering { #metadata-and-filtering } -This page covers LanceDB upload and retrieval. **Metadata is not duplicated here.** - -- **Published guide** — [Custom metadata and filtering](custom-metadata.md) (sidecar `meta_*` on `vdb_upload`, compact JSON in LanceDB, server-side `where` on `Retriever.query`, and client-side `filter_hits_by_content_metadata`). -- **Canonical reference** — [Vector DB operators and LanceDB — Metadata filtering](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering) in `nemo_retriever/src/nemo_retriever/vdb/README.md` (operator behavior and examples). +Refer to the [metadata filtering notebook](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) for an end-to-end example of adding custom metadata fields to your documents and filtering retrieval results with that metadata. ## LanceDB deployment characteristics { #lancedb-deployment-characteristics } @@ -120,19 +117,19 @@ NeMo Retriever Library integrates with vector databases used for RAG collections ### Backends with `VDB` implementations (retriever adapters) { #vdb-backends-implementations } -NeMo Retriever graph operators [`IngestVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/operators.py) and [`RetrieveVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/operators.py) wrap concrete classes that implement the [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/adt_vdb.py) interface (`run` for ingest, `retrieval` for search). The library ships one first-party backend: +NeMo Retriever graph operators [`IngestVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py) and [`RetrieveVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py) wrap concrete classes that implement the [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) interface (`run` for ingest, `retrieval` for search). The library ships one first-party backend: | Backend | Project | Implementation | |---------|---------|----------------| -| **LanceDB** | [LanceDB](https://lancedb.com/) · [documentation](https://lancedb.github.io/lancedb/) | [`lancedb.py`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/lancedb.py) — pass `vdb_op="lancedb"` (recommended). | +| **LanceDB** | [LanceDB](https://lancedb.com/) · [documentation](https://lancedb.github.io/lancedb/) | [`lancedb.py`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py) — pass `vdb_op="lancedb"` (recommended). | On `GraphIngestor.vdb_upload`, omitting `vdb_op` does not select LanceDB; refer to [Upload to LanceDB](#upload-to-lancedb). -Pass `vdb_op="lancedb"` or a `LanceDB` instance. To integrate another vector database, subclass [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/adt_vdb.py) and pass your operator instance as `vdb` (refer to [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb)). +Pass `vdb_op="lancedb"` or a `LanceDB` instance. To integrate another vector database, subclass [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) and pass your operator instance as `vdb` (refer to [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb)). ### RAG Blueprint and partner vector stores { #rag-blueprint-and-partner-vector-stores } -Some deployments use a different vector store than the default LanceDB path on this page—for example the [NVIDIA RAG Blueprint](https://docs.nvidia.com/rag/latest/index.html) (Docker Compose or Helm) or a partner package that subclasses the same [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/vdb/adt_vdb.py) interface. Use the following public references when you wire those stacks to ingestion and retrieval: +Some deployments use a different vector store than the default LanceDB path on this page—for example the [NVIDIA RAG Blueprint](https://docs.nvidia.com/rag/latest/index.html) (Docker Compose or Helm) or a partner package that subclasses the same [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) interface. Use the following public references when you wire those stacks to ingestion and retrieval: | Vector store | Where to configure or implement | |--------------|--------------------------------| @@ -144,7 +141,7 @@ Testing and release cadence for these integrations follow the owning project (RA ### More information (embeddings & custom `VDB`) { #vector-database-partners-more-info } -- [Custom metadata and filtering](custom-metadata.md) and the package [VDB README (metadata filtering)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering) +- [Metadata filtering notebook](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) and the package [VDB README (metadata filtering)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/common/vdb#metadata-filtering) - [Multimodal embeddings (VLM)](embedding.md) - [NeMo Retriever Text Embedding NIM](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/overview.html) - [NVIDIA NIM catalog](https://build.nvidia.com/) for embedding and retrieval-related NIMs @@ -153,12 +150,13 @@ Testing and release cadence for these integrations follow the owning project (RA NVIDIA documents and validates the first-party LanceDB operator for this library. If you integrate a different vector store, you are responsible for testing and maintaining that integration. -To implement a custom operator, follow the `VDB` abstract interface described in [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb). +To implement a custom operator, follow the `VDB` abstract interface described in [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb). For an overview of all customization paths (UDFs, graph pipelines, and embeddings), refer to [Customize & extend](customize-extend.md). ## Related Topics { #related-topics } -- [Custom metadata and filtering](custom-metadata.md) -- [Vector DB operators and LanceDB (source)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb) +- [Metadata and filtering](#metadata-and-filtering) +- [Customize & extend](customize-extend.md) +- [Vector DB operators and LanceDB (source)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/common/vdb) - [Use the NeMo Retriever Library Python API](nemo-retriever-api-reference.md) - [Store Extracted Images](nemo-retriever-api-reference.md) - [Environment Variables](environment-config.md) diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index 78b48cdc47..d29b36331b 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,11 +4,30 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. +## MCP access for agents + +`retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API. + +For local stdio-based agents, run the MCP server as a shim that points at an existing retriever service: + +```bash +retriever service mcp-stdio \ + --service-url http://localhost:7670 \ + --api-token "$NEMO_RETRIEVER_API_TOKEN" +``` + +For remote agents, expose the retriever service URL and configure the agent to connect to: + +```text +https:///mcp +``` + +The `ingest_documents` MCP tool accepts either paths visible to the MCP server process or inline `content_base64` document bytes. Use inline base64 for remote agents whose local files are not present on the service host. + **Where to go next** Use these pages together with your orchestration layer: -- [Semantic retrieval](vdbs.md#semantic-retrieval), [Custom metadata and filtering](custom-metadata.md), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality and reranking notes +- [Semantic retrieval](vdbs.md#semantic-retrieval), [Metadata and filtering](vdbs.md#metadata-and-filtering), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality, reranking, and evaluation guidance - [Agentic retrieval (concept)](agentic-retrieval-concept.md) -- [Evaluate on your data](evaluate-on-your-data.md), which includes retrieval evaluation guidance - [Release notes](releasenotes.md), which may mention agentic retrieval updates diff --git a/docs/docs/extraction/workflow-document-ingestion.md b/docs/docs/extraction/workflow-document-ingestion.md index 853de9e1a0..cba0164d63 100644 --- a/docs/docs/extraction/workflow-document-ingestion.md +++ b/docs/docs/extraction/workflow-document-ingestion.md @@ -2,7 +2,7 @@ This page covers extracting content from documents and turning that content into a searchable vector collection in one place so you can scroll and search (for example with Ctrl+F) instead of jumping across multiple short workflow stubs. -## Ingest and extract +## Ingest and extract { #ingest-and-extract } Document ingestion is the step where NeMo Retriever Library reads your files (PDFs, Office documents, images, and other [supported formats](multimodal-extraction.md#supported-file-types-and-formats)), runs extraction and optional enrichment, and returns structured content you can embed and index. @@ -14,9 +14,9 @@ Follow these steps: Pipeline concepts and stage overview appear in [Key concepts](concepts.md). Default chunking behavior is summarized under [Chunking](concepts.md#chunking). -`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). For directory-scale corpus ingest, the `graph_pipeline` CLI below is the canonical path used in evaluation workflows. +`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). -## Choose how you call the library +## Choose how you call the library { #choose-how-you-call-the-library } The following examples match the [NeMo Retriever Library README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md). They assume a checkout of the [NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever) repository and the `batch` run mode with local GPU inference unless you configure remote NIMs. @@ -47,20 +47,4 @@ result = ingestor.ingest() # ``pandas.DataFrame`` (``batch`` and ``inprocess``) Run the above with your working directory at the repository root (so `data/multimodal_test.pdf` resolves), or adjust `documents` to the absolute path of the test PDF. -### Ingest a test corpus (CLI) - -`graph_pipeline` is the canonical ingestion script used throughout the [QA evaluation guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/evaluation/README.md#step-1-ingest-and-embed-pdfs-nemo-retriever). Point it at a **directory** of PDFs to produce a ready-to-query LanceDB table. - -!!! note "Corpus size and LanceDB indexing" - - LanceDB's default IVF index needs enough chunks to train its partitions (often on the order of tens of chunks). A single small PDF can be insufficient; use a directory with enough documents for your index settings. Replace `/your-example-dir` with your corpus path. - -```bash -python -m nemo_retriever.examples.graph_pipeline \ - /your-example-dir \ - --vdb-kwargs-json '{"uri":"lancedb","table_name":"nemo-retriever"}' -``` - -For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](api-keys.md#nvidia-api-key) and pass the `--*-invoke-url` / `--embed-invoke-url` options shown in the [README remote inference section](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md#ingest-a-test-corpus-cli). - **Next:** [Semantic retrieval](vdbs.md#semantic-retrieval) when serving queries (also refer to [Evaluate on your data](evaluate-on-your-data.md) for reranking and quality checks). diff --git a/docs/docs/extraction/workflow-e2e-blueprints.md b/docs/docs/extraction/workflow-e2e-blueprints.md index 16aa4bb3d6..93203a6374 100644 --- a/docs/docs/extraction/workflow-e2e-blueprints.md +++ b/docs/docs/extraction/workflow-e2e-blueprints.md @@ -5,4 +5,4 @@ Use these external resources for end-to-end RAG implementations with NeMo Retrie - [Enterprise RAG - multimodal PDF data extraction](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) - [NVIDIA AI Blueprints catalog](https://build.nvidia.com/explore/discover) -For framework-specific integration patterns, see [Framework integrations](integrations-langchain-llamaindex-haystack.md). +For framework-specific integration patterns, refer to [Starter kits](starter-kits.md). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index a944ad6c8b..57a3ea6d78 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -84,8 +84,9 @@ nav: - "Authentication and API keys": extraction/api-keys.md - "3. Deployment options": - "Compare deployment options": extraction/deployment-options.md + - "OpenShift deployment (Helm)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md - "Helm chart (Kubernetes)": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm - - "Docker Compose (unsupported, developer)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md + - "Docker service image": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md - "4. Core workflows": - "Workflow: Ingest documents into a searchable VDB collection": extraction/workflow-document-ingestion.md - "Workflow: Audio & video ingestion": extraction/audio-video.md @@ -95,26 +96,22 @@ nav: # Single vector-DB page (vdbs.md). Deep links: in-page "On this page" TOC and redirects # (for example extraction/vector-db-partners.md → vdbs.md#vector-database-partners). - "Vector databases": extraction/vdbs.md - - "7. Retrieval & ranking": - - "Custom metadata and filtering": extraction/custom-metadata.md - - "8. Deployment & operations": + - "7. Deployment & operations": - "Ray and distributed ingest": extraction/ray-logging.md - - "9. Customize & extend": - - Extending/Customizing NeMo Retriever Library with custom code: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph - - "NimClient and custom NIM endpoints": extraction/nimclient.md - - "10. Integrations & ecosystem": - - "Framework integrations": extraction/integrations-langchain-llamaindex-haystack.md - - "Starter kits": extraction/notebooks/index.md - - "11. Evaluation & benchmarks": + - "8. Customize & extend": + - "Customize & extend": extraction/customize-extend.md + - "9. Integrations & ecosystem": + - "Starter kits": extraction/starter-kits.md + - "10. Evaluation & benchmarks": - "Evaluate on your own documents": extraction/evaluate-on-your-data.md - - "12. Reference": + - "11. Reference": - "API guide": extraction/nemo-retriever-api-reference.md # TODO: after nv-ingest code removal, update this link when CLI docs are relocated. - "CLI reference": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli - "Quickstart: retriever CLI": reference/retriever-cli-quickstart.md - Environment variables: extraction/environment-config.md - "Metadata reference": extraction/content-metadata.md - - "13. Support & community": + - "12. Support & community": - Troubleshooting: extraction/troubleshoot.md - FAQ: extraction/faq.md - Contributing: extraction/contributing.md @@ -159,8 +156,11 @@ plugins: extraction/hosted-nims-when-to-use.md: extraction/deployment-options.md extraction/releasenotes-nv-ingest.md: extraction/releasenotes.md extraction/ngc-api-key.md: extraction/api-keys.md - extraction/notebooks.md: extraction/notebooks/index.md + extraction/notebooks/index.md: extraction/starter-kits.md + extraction/notebooks.md: extraction/starter-kits.md extraction/data-store.md: extraction/vdbs.md + extraction/custom-metadata.md: extraction/vdbs.md#metadata-and-filtering + extraction/integrations-langchain-llamaindex-haystack.md: extraction/starter-kits.md extraction/nemoretriever-parse.md: extraction/multimodal-extraction.md#text-and-layout-extraction extraction/supported-file-types.md: extraction/multimodal-extraction.md#supported-file-types-and-formats extraction/text-layout-extraction.md: extraction/multimodal-extraction.md#text-and-layout-extraction @@ -185,7 +185,11 @@ plugins: extraction/chunking.md: extraction/concepts.md#chunking extraction/quickstart-library-mode.md: extraction/deployment-options.md extraction/workflow-video-ocr.md: extraction/audio-video.md - extraction/user-defined-stages.md: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph + extraction/user-defined-stages.md: extraction/customize-extend.md + extraction/user-defined-functions/index.md: extraction/customize-extend.md#user-defined-functions-udfs + extraction/user-defined-functions.md: extraction/customize-extend.md#user-defined-functions-udfs + extraction/customize-and-extend.md: extraction/customize-extend.md + extraction/nimclient.md: https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints - site-urls markdown_extensions: @@ -207,11 +211,11 @@ markdown_extensions: - admonition - footnotes -# MkDocs 1.6+: exclude legacy duplicate pages (still in repo for parity). +# MkDocs 1.6+: exclude suite landing and legacy duplicate pages (still in repo for parity). # extraction/chunking.md — removed from nav; content is under concepts.md (redirect_maps keeps old URLs). -# Root index.md is not in nav; redirect_maps sends index.md → extraction/overview.md so /latest/ (Docs Hub tile) resolves. -# Use /index.md in exclude_docs only (bare index.md would exclude every index.md, e.g. extraction/notebooks/index.md). +# Use /index.md (docs root only); bare index.md would exclude every index.md (e.g. under subfolders). exclude_docs: | + /index.md extraction/chunking.md extraction/helm.md extraction/choose-your-path.md diff --git a/nemo_retriever/README.md b/nemo_retriever/README.md index 7c2747cd42..24c24bd942 100644 --- a/nemo_retriever/README.md +++ b/nemo_retriever/README.md @@ -21,7 +21,7 @@ Before starting, make sure your system meets the following requirements: - The host is running CUDA 13.x so that `libcudart.so.13` is available. - Your GPUs are visible to the system and compatible with CUDA 13.x. ​ -If optical character recognition (OCR) fails with a `libcudart.so.13` error, install the CUDA 13 runtime for your platform and update `LD_LIBRARY_PATH` to include the CUDA lib64 directory, then rerun the pipeline. +If optical character recognition (OCR) fails with a `libcudart.so.13` error, install the CUDA 13 runtime for your platform and update `LD_LIBRARY_PATH` to include the CUDA lib64 directory, then rerun the pipeline. For example, the following command can be used to update the `LD_LIBRARY_PATH` value. @@ -44,7 +44,15 @@ For **local GPU inference** (Nemotron models running on your GPU), install with ```bash uv venv retriever --python 3.12 source retriever/bin/activate -uv pip install "nemo-retriever[local]==26.5.0" +uv pip install "nemo-retriever[local]" +``` + +The `[local]` extra resolves stable Nemotron extraction packages by default. To +try prerelease/nightly Nemotron packages from PyPI within the same supported +major-version windows, opt in with `--pre`: + +```bash +uv pip install --pre "nemo-retriever[local]==26.05-RC1" ``` Install matching **ingestion client** and **ingestion runtime** wheels at the same version when your workflow expects them (refer to the [NeMo Retriever Library prerequisites](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) for the exact PyPI coordinates for your release). @@ -55,7 +63,7 @@ For **remote NIM inference only** (no local GPU required), the base package is s uv python install 3.12 uv venv retriever --python 3.12 source retriever/bin/activate -uv pip install nemo-retriever==26.5.0 +uv pip install nemo-retriever ``` Install matching **ingestion client** and **ingestion runtime** wheels at the same version when your workflow expects them (refer to the [NeMo Retriever Library prerequisites](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) for the exact PyPI coordinates for your release). @@ -65,7 +73,7 @@ This creates a dedicated Python environment and installs the `nemo-retriever` Py If your PDF pipeline uses `extract_method="nemotron_parse"`, install the Nemotron Parse client dependencies with the `nemotron-parse` extra: ```bash -uv pip install "nemo-retriever[nemotron-parse]==26.5.0" +uv pip install "nemo-retriever[nemotron-parse]" ``` For local GPU inference with Nemotron Parse, combine the extras as `nemo-retriever[local,nemotron-parse]`. @@ -87,14 +95,15 @@ Skip this step if you are using remote NIM inference only. The [test PDF](../data/multimodal_test.pdf) contains text, tables, charts, and images. Additional test data resides [here](../data/). -> **Note:** `batch` is the primary intended run_mode of operation for this library. Other modes are experimental and subject to change or removal. +> **Note:** `retriever ingest` defaults to local, in-process execution. Use `retriever ingest batch ...` for Ray Data scale-out on larger workloads. +> `retriever pipeline run` keeps its legacy `--run-mode` flag for compatibility and development workflows. -The examples below use default local GPU inference (no `invoke_url` specified) and require the `[local]` extra and the CUDA 13 torch override from the setup steps above. For remote NIM inference without a local GPU, see [Run with remote inference](#run-with-remote-inference-no-local-gpu-required). +The examples below use default local GPU inference (no `invoke_url` specified) and require the `[local]` extra and the CUDA 13 torch override from the setup steps above. For remote NIM inference without a local GPU, refer to [Run with remote inference](#run-with-remote-inference-no-local-gpu-required). ### Ingest a test pdf ```python from nemo_retriever import create_ingestor -from nemo_retriever.io import to_markdown, to_markdown_by_page +from nemo_retriever.common.io import to_markdown, to_markdown_by_page from pathlib import Path documents = [str(Path("../data/multimodal_test.pdf"))] @@ -159,7 +168,7 @@ used in [Run a recall query](#run-a-recall-query) below. With the and embedding. For a realistic retrieval corpus, see [QA evaluation -- Step 1](./src/nemo_retriever/evaluation/README.md#step-1-ingest-and-embed-pdfs-nemo-retriever). -**No local GPU?** Set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) (see [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)) and route extraction and embedding +**No local GPU?** Set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)) and route extraction and embedding through [build.nvidia.com](https://build.nvidia.com/) NIMs instead: ```bash @@ -216,7 +225,7 @@ Since the ingestion job automatically populated a lancedb table with all these c ### Run a recall query ```python -from nemo_retriever.retriever import Retriever +from nemo_retriever.graph.retriever import Retriever retriever = Retriever( # values used by the graph_pipeline example above @@ -321,7 +330,7 @@ embedding model in `embed_kwargs` must match the one used during ingestion so query vectors land in the same embedding space as the stored chunks. ```python -from nemo_retriever.retriever import Retriever +from nemo_retriever.graph.retriever import Retriever from nemo_retriever.llm import LiteLLMClient retriever = Retriever( @@ -426,7 +435,7 @@ ingestor = ( ### Render results as markdown If you want a readable markdown view of extracted results, pass a single document's extraction -records to `nemo_retriever.io.to_markdown`. The helper returns one markdown string (or `None` +records to `nemo_retriever.common.io.to_markdown`. The helper returns one markdown string (or `None` if there is no content), with per-page sections joined under a single document heading. For multi-document runs, pass one document at a time—for example, `to_markdown(results[0])`. @@ -508,7 +517,7 @@ ingestor = ( .embed( model_name="nvidia/llama-nemotron-embed-vl-1b-v2", #works with plain "text"s, "image"s, and "text_image" pairs - embed_modality="text_image" + embed_modality="text_image" ) ) ``` @@ -520,7 +529,7 @@ ingestor = ingestor.files(documents).extract(method="nemotron_parse") ## Run with remote inference, no local GPU required: -For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) as an environment variable (see [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)). +For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/#nvidia-api-key) as an environment variable (refer to [Authentication and API keys](https://nvidia.github.io/NeMo-Retriever/extraction/api-keys/)). ```python ingestor = ( @@ -625,38 +634,22 @@ sudo apt install python3.12-dev After installing the headers, restart the pipeline. -## ViDoRe Harness Sweep - -The harness includes BEIR-style ViDoRe dataset presets in `nemo_retriever/harness/test_configs.yaml` and a ready-made sweep definition in `nemo_retriever/harness/vidore_sweep.yaml`. - -The ViDoRe harness datasets are configured to: - -- read PDFs from `/datasets/retrieval-eval/vidore_v3_corpus_pdf/...` -- ingest with `embed_modality: text_image` -- embed at `embed_granularity: page` -- enable `extract_page_as_image: true` and `extract_infographics: true` -- evaluate with BEIR-style `ndcg` and `recall` metrics - -To run the full ViDoRe sweep: - -```bash -cd ~/NeMo-Retriever/nemo_retriever -retriever-harness sweep --runs-config harness/vidore_sweep.yaml -``` +## Retriever Harness -The same commands also work under the main CLI as `retriever harness ...` if you prefer a single top-level command namespace. +The developer harness runs code-owned benchmarks through `retriever harness`. +Use `retriever harness list --runsets` to see available benchmark names and +runsets, then run one benchmark with `retriever harness run `. -### Pipeline image storage +### Ingest image storage -Use the pipeline CLI to persist extracted image assets to local storage or any +Use root ingest to persist extracted image assets to local storage or any fsspec-compatible URI: ```bash -retriever pipeline run ./data \ +retriever ingest ./data \ --store-images-uri ./processed_docs/images ``` -The store stage writes the image payloads produced by the configured pipeline. -With `--embed-granularity page`, stored assets are page images. With -`--embed-granularity element`, stored assets are element images. Store is not -currently configured through the harness. +The store stage writes the image payloads produced by ingest. With +`--embed-granularity page`, stored assets are page images. With +`--embed-granularity element`, stored assets are element images. diff --git a/nemo_retriever/docs/cli/README.md b/nemo_retriever/docs/cli/README.md index f4b3d5ab54..a84c4cf9bb 100644 --- a/nemo_retriever/docs/cli/README.md +++ b/nemo_retriever/docs/cli/README.md @@ -1,81 +1,56 @@ -# Retriever CLI — replacement examples for the legacy ingestion-service CLI - -This folder contains `retriever` command-line examples that deliver the same -end-user outcomes as the legacy **ingestion-service** CLI examples that used to -live under `docs/`, `api/`, `client/`, and `deploy/` in older repository layouts. - -The historical CLI documentation is **not removed** from the ecosystem — these files sit -alongside it as a new-CLI counterpart you can link to or migrate to. - -## Supported vs development / experimental subcommands - -For product use and published examples, treat only these top-level subcommands as -the **supported public path**: - -- **`retriever ingest`** — ingest documents into LanceDB -- **`retriever query`** — query an existing LanceDB table - -`retriever pipeline` remains available as a **development / compatibility** -wrapper, including `retriever pipeline run`, while ingestion behavior migrates -onto the same implementation used by `retriever ingest`. Prefer `retriever -ingest` and `retriever query` for user-facing workflows. - -Any other top-level `retriever` subcommand — including but not limited to -`pipeline`, `pdf`, `html`, `txt`, `audio`, `chart`, `benchmark`, `harness`, -`eval`, `recall`, `service`, `local`, `compare`, `image`, and `skill-eval` — -is **development and experimental**. These commands may change without public -compatibility guarantees. - -## Key shape difference - -The legacy **ingestion-service** CLI was a **single command that talks to a running REST service on -`localhost:7670`** and composes work via repeated `--task extract|split|caption|embed|dedup|filter|udf`. - -`retriever` is a **multi-subcommand Typer app**. Public ingest/query examples -should map to `retriever ingest INPUT_PATH` followed by `retriever query ...`. -`retriever pipeline run INPUT_PATH` is still present for development workflows -that need pipeline-only evaluation, runtime summaries, Parquet export, or -service-mode compatibility. - -| Old intent | New subcommand | -|------------|----------------| -| Extract + embed + store a batch of documents | `retriever ingest` | -| Run an ad-hoc PDF extraction stage | `retriever pdf stage` | -| Run an HTML / text / audio / chart stage | `retriever html run`, `retriever txt run`, `retriever audio extract`, `retriever chart run` | -| Upload stage output to LanceDB | `retriever ingest` | -| Query LanceDB + compute recall@k | `retriever recall vdb-recall` | -| Run a QA evaluation sweep | `retriever eval run` | -| Serve / submit to the online REST API | `retriever online serve` / `retriever online stream-pdf` | -| Benchmark stage throughput | `retriever benchmark {split,extract,audio-extract,page-elements,ocr,all}` | -| Benchmark orchestration | `retriever harness {run,sweep,nightly,summary,compare}` | - -Rows that use subcommands other than `ingest` or `query` are -[development and experimental](#supported-vs-development--experimental-subcommands). - -## Contents - -| Topic | Location | Replaces example(s) in | -|-------|----------|------------------------| -| Quick start | [below](#quick-start) | Legacy service quickstart; **Helm** + [NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) | -| CLI reference | [below](#cli-reference) | Prior `cli-reference` pages under `docs/docs/extraction/` | -| Client usage walk-through | [below](#client-usage-walk-through) | `client/client_examples/examples/cli_client_usage.ipynb` | -| PDF pre-splitting | [API guide](../../../docs/docs/extraction/nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest); [Large PDF page batches](#large-pdf-page-batches) below | Prior extraction docs | -| Benchmarking | [`benchmarking.md`](benchmarking.md) | `docs/docs/extraction/benchmarking.md` and `nemo_retriever/harness/HANDOFF.md` | +# Retriever CLI + +This page describes the public `retriever` command-line workflow for document +ingest and retrieval. + +For product-facing examples, prefer these commands: + +- `retriever ingest` - ingest supported documents and media into a Retriever index. +- `retriever query` - query a local LanceDB table written by local or batch ingest. +- `retriever query service` - query a Retriever service deployment. + +`retriever pipeline run` remains available as a development and compatibility +command for legacy pipeline workflows, evaluation, intermediate artifacts, and +pipeline-specific debugging. It is not the preferred public ingest interface. + +## Public ingest shape + +`retriever ingest` defaults to local, in-process ingest: + +```bash +retriever ingest DOCUMENTS... +``` + +Explicit modes are also available: + +```bash +retriever ingest local DOCUMENTS... +retriever ingest batch DOCUMENTS... +retriever ingest service DOCUMENTS... +``` + +The root ingest CLI uses subcommands instead of a `--run-mode` flag. Choose +the command that matches where ingest runs and where results are stored. + +| Command | What It Does | Writes To | Use When | +|---|---|---|---| +| `retriever ingest ...` | Local in-process ingest | local LanceDB | Default local ingest and CI/small corpus runs. | +| `retriever ingest local ...` | Local in-process ingest | local LanceDB | Same as the default, but explicit. | +| `retriever ingest batch ...` | Ray-backed batch ingest | local LanceDB | Larger or batch-tuned runs. | +| `retriever ingest service ...` | Sends documents to a Retriever service | service-configured storage | Remote service ingest. | + +This separation keeps invalid flag combinations out of the parser. For example, +service ingest does not expose LanceDB target flags, Ray tuning, local endpoint +configuration, local embed backend selection, or local media controls. > Use `retriever ingest` and `retriever query` for product-facing workflows. -> `retriever pipeline` is development / compatibility only; see -> [Supported vs development / experimental subcommands](#supported-vs-development--experimental-subcommands). +> `retriever pipeline run` is development / compatibility only. ## Quick start -For deployment of NeMo Retriever / **NIM** containers, use -[nemo_retriever/helm](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) -and the [NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) -Helm install guides. - -### Ingest a PDF +### Ingest a PDF locally ```bash retriever ingest ./data/multimodal_test.pdf \ @@ -85,19 +60,49 @@ retriever ingest ./data/multimodal_test.pdf \ --embed-model-name nvidia/llama-nemotron-embed-1b-v2 ``` -Then query the LanceDB table: +Then query the default LanceDB table: ```bash retriever query "What is in this document?" \ --embed-model-name nvidia/llama-nemotron-embed-1b-v2 ``` -Development-only pipeline features such as `--save-intermediate`, runtime -summaries, and post-ingest evaluation remain on `retriever pipeline run` while -the public path is restricted to ingest/query. +By default, local ingest writes to `lancedb/nemo-retriever` and `retriever query` +reads from the same table. + +The plain `retriever query` examples below apply to local and batch ingest output +written to LanceDB. Use `retriever query service` to query a Retriever service. + +### Ingest a larger corpus with batch mode + +```bash +retriever ingest batch ./data/pdf_corpus \ + --profile fast-text \ + --pdf-extract-workers 4 \ + --embed-workers 2 +``` + +Batch mode exposes Ray runtime and batch tuning flags such as `--ray-address`, +`--pdf-extract-workers`, `--ocr-workers`, and `--embed-workers`. -Route stages to self-hosted or hosted NIM endpoints by passing only the URLs you -want to override: +### Ingest through a Retriever service + +```bash +retriever ingest service ./data/pdf_corpus \ + --service-url http://localhost:7670 \ + --service-concurrency 8 +``` + +Use `--service-api-token` or `NEMO_RETRIEVER_API_TOKEN` when the service requires +a bearer token. Service ingest does not expose `--lancedb-uri`; the service +configures its vector database. Query the service with: + +```bash +retriever query service "What is in this corpus?" \ + --service-url http://localhost:7670 +``` + +### Route ingest to hosted or self-hosted NIM endpoints ```bash export NVIDIA_API_KEY=nvapi-... @@ -108,365 +113,247 @@ retriever ingest ./data/multimodal_test.pdf \ --table-structure-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1 \ --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \ --embed-model-name nvidia/llama-nemotron-embed-1b-v2 +``` + +`NVIDIA_API_KEY` is required only when those URLs point at hosted +build.nvidia.com endpoints. `NGC_API_KEY` is used separately when pulling or +running self-hosted NIM containers. + +For NVIDIA inference hub rerank models that expose the Cohere-style rerank +route, pass the full `/v1/rerank` URL and the model name shown in the hub +snippet: + +```bash +export NGC_INFERENCE_API_KEY=... retriever query "What is in this document?" \ --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \ --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \ - --reranker-invoke-url https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-nemotron-rerank-vl-1b-v2/reranking + --reranker-invoke-url https://inference-api.nvidia.com/v1/rerank \ + --reranker-model-name nvidia/nvidia/llama-3.2-nv-rerankqa-1b-v2 \ + --reranker-api-key-env NGC_INFERENCE_API_KEY ``` ### Query result controls -`retriever query` returns compact JSON hits with `source`, `page_number`, and `text`. -By default it retrieves and returns `--top-k` rows. Use these controls when you -need a wider candidate pool or a narrower result shape: +Both `retriever query` and `retriever query service` return compact JSON hits +with `source`, `page_number`, and `text`. Use `--candidate-k`, `--page-dedup`, +and `--content-types` to control how results are selected after vector +retrieval: ```bash -# Retrieve 30 candidates, then return the best 10. -retriever query "where is the warranty limitation discussed?" \ - --candidate-k 30 - -# Keep only the first hit from each document page. -retriever query "which pages discuss operating costs?" \ - --top-k 5 \ - --candidate-k 30 \ - --page-dedup - -# Search a wider pool, then keep only table rows. retriever query "annual revenue by region" \ --top-k 5 \ --candidate-k 40 \ --content-types table ``` -`--top-k` is the final number of hits returned. `--candidate-k` is the wider -candidate pool retrieved before page deduplication, content-type filtering, and -final truncation. It must be greater than or equal to `--top-k`, and should -usually be larger when page deduplication or content-type filtering might -otherwise remove too many of the top retrieved rows. Page deduplication and -content-type filtering are applied after vector retrieval, preserving the -retriever's ranking order and truncating the final output to `--top-k`. -When querying a table ingested with an explicit embedding model, pass the same -`--embed-model-name` to `retriever query`. -`--content-types` accepts comma-separated content types such as `text`, `table`, -`chart`, `image`, and `infographic`. `images` is accepted as an alias for -captioned image rows emitted by ingest. Hits with missing or unknown content -types are excluded while `--content-types` is active. +`--top-k` is the final number of results to return after filtering and +deduplication. `--candidate-k` is the number of raw results to retrieve from +LanceDB or the Retriever service before filtering, page deduplication, and +final truncation. If omitted, the candidate pool is the same size as +`--top-k`. Set `--candidate-k` larger than `--top-k` when page deduplication +or content-type filtering might remove too many of the nearest retrieved rows. +It must always be greater than or equal to `--top-k`. -`NVIDIA_API_KEY` is required only when those URLs point at hosted -build.nvidia.com endpoints. `NGC_API_KEY` is used separately when pulling or -running self-hosted NIM containers. +Page deduplication and content-type filtering are applied after vector +retrieval, preserving retriever ranking order and truncating the final output to +`--top-k`. When querying a local table ingested with an explicit embedding +model, pass the same `--embed-model-name` to `retriever query`. -### What you get +`--content-types` accepts comma-separated content types such as `text`, `table`, +`chart`, `image`, and `infographic`. `images` is accepted as an alias for +captioned image rows emitted by ingest. This option filters by content-type +metadata only; it does not filter by source, page, or other metadata +predicates. Hits with missing or unknown content-type metadata are excluded +while `--content-types` is active. In service mode, results must include +content-type metadata to match this filter. Default display values in the JSON +output are not used for content-type matching. -- Extracted text, tables, and charts as rows in LanceDB at `./lancedb` (default - table name `nemo-retriever`). -- Compact JSON retrieval hits from `retriever query`, including source, page, - and text fields. -- Extracted image assets when `retriever ingest` is run with - `--store-images-uri`. -- Pipeline-only development artifacts such as extraction Parquet, runtime - summaries, and evaluation reports remain available through - `retriever pipeline run`. -- Progress and stage logs on stderr. +### Agentic retrieval -### Inspect the results +`--agentic` swaps the single dense pass for an LLM-driven ReAct loop: the agent +issues several retrieval sub-queries, fuses the candidates, and selects a final +ranking. It searches the same LanceDB table built by `retriever ingest`, so it is +a drop-in alternative to standard retrieval — add `--agentic` and name the chat +model the agent drives with `--agentic-llm-model` (required): ```bash -ls ./lancedb -``` - -```python -import lancedb - -db = lancedb.connect("./lancedb") -tbl = db.open_table("nemo-retriever") -print(tbl.to_pandas().head()) -``` - -Or query via the Retriever Python client (`nemo_retriever/README.md`): - -```python -from nemo_retriever.retriever import Retriever - -retriever = Retriever( - vdb_kwargs={"uri": "lancedb", "table_name": "nemo-retriever"}, - embed_kwargs={ - "model_name": "nvidia/llama-nemotron-embed-1b-v2", - "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2", - }, - top_k=5, -) -hits = retriever.query( - "Given their activities, which animal is responsible for the typos?" -) +retriever query "how does the ingestion pipeline handle tables?" \ + --agentic \ + --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 + +# remote agent + embedding endpoints, fewer reasoning rounds +retriever query "summarize the deployment options" \ + --agentic \ + --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \ + --agentic-invoke-url http://localhost:9000/v1/chat/completions \ + --embed-invoke-url http://localhost:8000/v1 \ + --agentic-react-max-steps 5 ``` -### Larger datasets - -- Batch ingest: `retriever ingest ./data/pdf_corpus --run-mode batch`. -- Tune throughput with `--pdf-extract-workers`, `--pdf-extract-batch-size`, - `--page-elements-workers`, `--page-elements-batch-size`, `--ocr-workers`, - `--ocr-batch-size`, `--embed-workers`, and `--embed-batch-size`. -- For CI or debugging: `--run-mode inprocess` skips Ray startup. +Unlike the dense path (which returns text-enriched hits), agentic mode returns +the agent's ranked document IDs as JSON, each annotated with the source that +produced it (`final_results`, `rrf`, or `selection_agent`). It reuses the same +`--top-k`, `--lancedb-uri`, `--table-name`, `--embed-invoke-url`, and +`--embed-model-name` options as standard retrieval. + +**How it works.** Each agentic query runs `Query → ReActAgentOperator → (RRF +fusion) → SelectionAgentOperator → ranked results`: + +- `ReActAgentOperator` runs the per-query ReAct loop; every `retrieve` tool call + delegates to the standard `Retriever`, so the agent searches the same vector + DB and embedding config as dense retrieval. +- `RRFAggregatorOperator` fuses candidates from the loop's multiple searches with + reciprocal rank fusion. +- `SelectionAgentOperator` runs a final LLM selection pass over the fused set and + emits the ranked document IDs. + +Agentic-only knobs (apply only with `--agentic`): + +- `--agentic-invoke-url` — OpenAI-compatible chat-completions endpoint for the + agent LLM; defaults to the operators' built-in endpoint when omitted. +- `--agentic-reasoning-effort` (default `high`) — `reasoning_effort` forwarded on + agentic LLM calls. +- `--agentic-backend-top-k` (default `20`) — candidates pulled from the vector DB + per retrieval call. +- `--agentic-react-max-steps` (default `50`) — maximum ReAct loop iterations. +- `--agentic-text-truncation` (default `0`) — max characters of each candidate + shown to the agent; `0` disables truncation. +- `--agentic-temperature` (default `0.0`) — sampling temperature for agentic LLM + calls (`0.0` = greedy). -## CLI reference - -`retriever` is the Typer app installed with the `nemo-retriever` package. Subcommand -support policy: [Supported vs development / experimental subcommands](#supported-vs-development--experimental-subcommands). - -Document ingestion for users is `retriever ingest INPUT_PATH`, followed by -`retriever query` for retrieval. `retriever pipeline run INPUT_PATH` is retained -as a development / compatibility wrapper for pipeline-only behavior. +## Common ingest options + +### Local and batch ingest + +These options apply to `retriever ingest`, `retriever ingest local`, and +`retriever ingest batch` unless otherwise noted. + +| Option | Default | Notes | +|---|---|---| +| `DOCUMENTS...` | required | Files, directories, or shell globs. Supported file families are detected automatically. | +| `--profile` | `auto` | `auto` is normal manifest-routed ingest. `fast-text` is a PDF/document text-only profile for faster fallback ingest. | +| `--lancedb-uri` | `lancedb` | LanceDB database URI. | +| `--table-name` | `nemo-retriever` | LanceDB table name. Must match query-time storage flags. | +| `--overwrite/--append` | overwrite | Overwrite the table by default; use `--append` to add rows. | +| `--method` | planner default | PDF extraction method such as `pdfium` or `nemotron_parse`. | +| `--extract-text`, `--extract-tables`, `--extract-charts` | planner default | Enable or disable extraction families. | +| `--ocr-version` | planner default | OCR engine version for local extraction. | +| `--ocr-lang` | planner default | OCR v2 language selector for local extraction. | +| `--caption` | off | Add a captioning stage. | +| `--dedup` | off | Add image deduplication before captioning and embedding. | +| `--text-chunk` | off | Enable token chunking during extraction. | +| `--store-images-uri` | unset | Store extracted images at a local path or fsspec-compatible URI. | +| `--dry-run` | off | Print the resolved ingest plan without creating an ingestor. | +| `--quiet/--no-quiet` | quiet | Suppress verbose progress output by default. | + +Batch-only options include `--ray-address`, `--ray-log-to-driver`, +`--pdf-split-batch-size`, `--pdf-extract-workers`, `--ocr-workers`, +`--table-structure-workers`, `--nemotron-parse-workers`, `--embed-workers`, and +related batch-size / CPU / GPU tuning flags. + +### Service ingest + +`retriever ingest service` exposes only service-supported request controls. +It does not expose LanceDB target flags, Ray tuning, local endpoint URLs/API +keys, local embed backend selection, `--ocr-lang`, or local audio/video media +controls. + +| Option | Default | Notes | +|---|---|---| +| `DOCUMENTS...` | required | Files, directories, or shell globs sent to the service client. | +| `--service-url` | `http://localhost:7670` | Retriever service base URL. | +| `--service-concurrency` | `8` | Maximum concurrent document uploads. | +| `--service-api-token` | env fallback | Bearer token; also reads `NEMO_RETRIEVER_API_TOKEN`. | +| `--profile` | `auto` | Same profile names as local and batch ingest where supported. | +| `--caption`, `--dedup`, `--text-chunk` | off | Service-supported ingest controls. | +| `--store-images-uri` | unset | Service-accessible image storage URI. | +| `--dry-run` | off | Print the resolved service ingest request. Tokens are redacted. | + +## Examples + +### Custom LanceDB location ```bash -retriever --version -retriever --help -retriever ingest --help -retriever query --help +retriever ingest ./data/multimodal_test.pdf \ + --lancedb-uri ./my-lancedb \ + --table-name my-corpus ``` -### Extract a PDF with defaults - ```bash -retriever ingest ./data/test.pdf \ - --run-mode inprocess +retriever query "What is in this document?" \ + --lancedb-uri ./my-lancedb \ + --table-name my-corpus ``` -Results go to LanceDB (`./lancedb`, table `nemo-retriever` by default). Use -`retriever pipeline run --save-intermediate` only when you need development -Parquet artifacts. - -### Text chunking and PDF page batches - -Splitting is intrinsic to the pipeline. Control text chunks with `--text-chunk`. For -PDF pre-splitting and `--pdf-split-batch-size`, see -[PDF pre-splitting](../../../docs/docs/extraction/nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest) -and [Large PDF page batches](#large-pdf-page-batches): +### Fast text-only PDF fallback ```bash -retriever pipeline run ./data/test.pdf \ - --input-type pdf \ - --no-extract-tables --no-extract-charts \ - --text-chunk --text-chunk-max-tokens 512 --text-chunk-overlap-tokens 64 \ - --save-intermediate ./processed_docs +retriever ingest ./data/pdf_corpus \ + --profile fast-text \ + --embed-model-name nvidia/llama-nemotron-embed-1b-v2 ``` -There is no split-only mode without extraction; narrow flags to text extraction if you -only need chunk boundaries. - -### Nemotron OCR v2 language mode { #nemotron-ocr-v2-language-mode } - -The default OCR engine for **local** extraction (Hugging Face weights, no remote -`--ocr-invoke-url`) is **Nemotron OCR v2**, which runs in **multilingual** mode -by default (`multi`). - -| Flag | Values | Notes | -|------|--------|-------| -| `--ocr-lang` | `multi` (default), `english` | v2 only — English-only selector | -| `--ocr-version` | `v2` (default), `v1` | `v1` is the legacy English-only engine | +### OCR language mode ```bash -retriever pipeline run ./data/scanned.pdf \ - --input-type pdf \ - --method pdfium_hybrid \ +retriever ingest ./data/scanned.pdf \ + --ocr-version v2 \ --ocr-lang english - -retriever ingest ./data/scanned.pdf --ocr-version v1 ``` -Set the equivalent `ocr_lang` and `ocr_version` fields on `ExtractParams` (or the -ingest API) in Python. - -Remote OCR NIM endpoints choose their own model and language behavior. Local -`--ocr-lang` and `--ocr-version` are not sent on remote requests. For hosted -examples until OCR v2 is published on build.nvidia.com, keep -`--ocr-invoke-url` pointed at `nemotron-ocr-v1` (see [Quick start](#quick-start)). +For mixed-script documents, use `--ocr-lang multi` where supported by the local +OCR engine. -### PDF and Office documents - -Run once per input type (`--input-type doc` matches `*.docx` and `*.pptx`): +### Text chunking ```bash -retriever pipeline run ./data/test.pdf \ - --input-type pdf \ - --method pdfium \ - --text-chunk --text-chunk-max-tokens 512 \ - --save-intermediate ./processed_docs - -retriever pipeline run ./data/test.docx \ - --input-type doc \ - --text-chunk --text-chunk-max-tokens 512 \ - --save-intermediate ./processed_docs -``` - -Mixed PDF and docx in one invocation is not supported. - -### Large PDF page batches - -```bash -retriever pipeline run ./data/test.pdf \ - --input-type pdf \ - --method pdfium \ - --extract-text --no-extract-tables --no-extract-charts \ - --pdf-split-batch-size 64 \ - --save-intermediate ./processed_docs +retriever ingest ./data/test.pdf \ + --text-chunk \ + --text-chunk-max-tokens 512 \ + --text-chunk-overlap-tokens 64 ``` -### Caption images +### Captioning and image storage ```bash -retriever pipeline run ./data/test.pdf \ - --input-type pdf \ - --method pdfium \ +retriever ingest ./data/test.pdf \ --caption \ - --caption-model-name nvidia/nemotron-3-nano-omni-30b-a3b-reasoning \ --caption-invoke-url https://integrate.api.nvidia.com/v1/chat/completions \ --api-key "${NVIDIA_API_KEY}" \ - --store-images-uri ./processed_docs/images \ - --save-intermediate ./processed_docs + --store-images-uri ./processed_docs/images ``` -Custom caption prompts and `reasoning` flags are not exposed on the CLI — use -`nemo_retriever.ingestor.Ingestor.caption(...)` in Python. +## Results and diagnostics -### Directory of documents +Local and batch ingest report the number of input files and LanceDB rows written: -```bash -retriever pipeline run ./data/pdf_corpus \ - --input-type pdf \ - --method pdfium \ - --save-intermediate ./processed_docs +```text +Ingested 20 file(s) -> 1884 row(s) in LanceDB lancedb/nemo-retriever. ``` -There is no `dataset.json` loader; pass a directory or glob of files. +Service ingest reports the row count returned by the service result when +available: -### Store images to object storage - -```bash -retriever pipeline run ./data/test.pdf \ - --input-type pdf \ - --method pdfium \ - --store-images-uri s3://my-bucket/images \ - --save-intermediate ./processed_docs +```text +Ingested 20 file(s) -> 1940 row(s) through retriever service http://localhost:7670. ``` -Image URIs are written to row metadata. Use `--store-actors` to tune object-storage -write concurrency. +Use `--dry-run` on any ingest mode to inspect the resolved request without +creating an ingestor or contacting the service. -### Where results live +## Development / compatibility command -- **LanceDB** — `--lancedb-uri lancedb` (default). Default table name depends on the - subcommand: `retriever ingest` and `retriever query` use `nemo-retriever`; `retriever - pipeline run` still uses `nv-ingest`. Query via `retriever recall vdb-recall …` or - `nemo_retriever.retriever.Retriever`. -- **Parquet** — `--save-intermediate ` writes `/extraction.parquet`. -- **Images** — `--store-images-uri ` (local path or fsspec URI). Storage follows - `--embed-granularity` (page vs element images). +Use `retriever pipeline run` only when you need pipeline-specific behavior that +is intentionally not part of the first-class ingest/query workflow, such as: -### Errors and exit codes - -`retriever pipeline run` exits **0** on success and **non-zero** on validation or -pipeline failures. Use `--debug` or `--log-file ` for diagnostics. - -## Client usage walk-through - -Counterpart to `client/client_examples/examples/cli_client_usage.ipynb`. Covers help, a -single-PDF run, a batch directory run, and inspecting results. Drop these cells into a -notebook (e.g. `retriever_client_usage.ipynb`) if you prefer. - -### Help - -```bash -retriever --help -retriever pipeline run --help -``` - -Top-level `--help` lists the subcommand tree; `pipeline run --help` shows the -ingest-specific flags used below. - -### Run a single PDF - -```bash -retriever pipeline run "${SAMPLE_PDF0}" \ - --input-type pdf \ - --method pdfium \ - --extract-text --extract-tables --extract-charts \ - --dedup --dedup-iou-threshold 0.45 \ - --store-images-uri "${OUTPUT_DIRECTORY_SINGLE}/images" \ - --save-intermediate "${OUTPUT_DIRECTORY_SINGLE}" -``` - -- Table/structure detectors are chosen automatically; there is no CLI flag to pick a - specific table-extraction backend. -- `--dedup` with `--dedup-iou-threshold` removes duplicate image elements. -- There is no image scale/aspect-ratio filter in the `retriever` CLI today. -- `--store-images-uri` persists image assets at the configured embed granularity. - -### Run a batch of PDFs - -```bash -# $PDF_DIR is a directory of PDFs. -retriever pipeline run "${PDF_DIR}" \ - --input-type pdf \ - --method pdfium \ - --extract-text --extract-tables --extract-charts \ - --dedup --dedup-iou-threshold 0.45 \ - --store-images-uri "${OUTPUT_DIRECTORY_BATCH}/images" \ - --save-intermediate "${OUTPUT_DIRECTORY_BATCH}" -``` - -- Pass a directory or glob; there is no built-in `dataset.json` loader. -- Tune throughput with `--pdf-split-batch-size`, `--pdf-extract-batch-size`, etc. - -### Inspect results - -The batch walk-through above uses `retriever pipeline run`, which writes LanceDB table -`nv-ingest` by default. After `retriever ingest`, use `nemo-retriever` instead (see -[Inspect the results](#inspect-the-results)). - -```python -import pyarrow.parquet as pq -import lancedb - -df = pq.read_table(f"{OUTPUT_DIRECTORY_BATCH}/extraction.parquet").to_pandas() -print(df[["source_id", "text", "content_type"]].head()) - -db = lancedb.connect("./lancedb") -tbl = db.open_table("nv-ingest") -print(tbl.to_pandas().head()) -``` +- `--save-intermediate` Parquet artifacts. +- runtime metrics and pipeline reports. +- eval, recall, harness, or BEIR/QA workflows. +- legacy compatibility while callers migrate to `retriever ingest` and + `retriever query`. -## Gaps with no retriever-CLI equivalent (kept out of this folder) - -The following legacy **ingestion-service** CLI examples are **not** migrated here because the -new CLI does not yet expose an equivalent — continue to use the **ingestion-service** CLI -for these cases: - -- `--task 'udf:{…}'` — user-defined functions ([NeMo Retriever Graph](../../src/nemo_retriever/graph/README.md#nemo-retriever-graph)). `retriever` does not expose UDFs. -- `--task 'filter:{content_type:"image", min_size:…, min_aspect_ratio:…, max_aspect_ratio:…}'`. - The image scale/aspect-ratio filter stage is not reproduced in the new CLI. -- Bare service submission (legacy CLI `--doc foo.pdf` with no extract tasks - and full content-type metadata returned by the service). `retriever online submit` - is currently a stub — only `retriever online stream-pdf` is implemented. -- `gen_dataset.py` dataset creation with enumeration and sampling. -- `--collect_profiling_traces --zipkin_host --zipkin_port`. Use - `--runtime-metrics-dir` / `--runtime-metrics-prefix` instead for a different - metrics flavor. - -## Conventions used in the examples - -- Input paths assume you invoke `retriever` from the `nemo_retriever/` - directory (or point at absolute paths). -- `--save-intermediate ` writes the extraction DataFrame as - `/extraction.parquet` for inspection. LanceDB output goes to `--lancedb-uri` - (defaults to `./lancedb`). -- `--store-images-uri ` stores extracted image assets to a local path or - an fsspec URI (e.g. `s3://bucket/prefix`). Page granularity stores page - images; element granularity stores element images. -- `--run-mode inprocess` skips Ray and is ideal for single-file demos and CI; - `--run-mode batch` (the default) uses Ray Data for throughput. - -Run `retriever pipeline run --help` for the authoritative flag list. +Run `retriever pipeline run --help` for the compatibility command flag list. diff --git a/nemo_retriever/docs/cli/benchmarking.md b/nemo_retriever/docs/cli/benchmarking.md index e2200dae36..61331aa447 100644 --- a/nemo_retriever/docs/cli/benchmarking.md +++ b/nemo_retriever/docs/cli/benchmarking.md @@ -13,46 +13,86 @@ per-stage micro-benchmarks. ## Harness (development / experimental) -Run from the repository root (or any directory; pass `--config` if needed). Uses -`--dataset` and `--preset` against `nemo_retriever/harness/test_configs.yaml`. +Run from the repository root or any directory. The harness uses code-owned +benchmark names from `nemo_retriever.harness.benchmark_registry`; use +`retriever harness list` to discover the available benchmarks and runsets. ```bash -# Named dataset from nemo_retriever/harness/test_configs.yaml -retriever harness run --dataset bo767 --preset PE_GE_OCR_TE_DENSE +# List benchmark registry entries, optionally including runsets +retriever harness list +retriever harness list --runsets -# Default active profile (jp20 + single_gpu in test_configs.yaml) -retriever harness run --dataset jp20 +# Inspect one concrete benchmark spec +retriever harness show jp20_beir -# Custom directory on disk -retriever harness run --dataset /path/to/your/data +# Run one benchmark and write stable artifacts +retriever harness run jp20_beir -# Override a single config key -retriever harness run --dataset bo767 --override run_mode=inprocess +# Run one benchmark in batch mode +retriever harness run bo767_beir --mode batch + +# Override a resolved config key for this run +retriever harness run bo767_beir --set query.top_k=5 + +# Expand and run a code-owned benchmark runset +retriever harness run-set jp20_core ``` Related commands: ```bash -retriever harness --help # run, sweep, nightly, summary, compare, portal +retriever harness --help +retriever harness list --help +retriever harness show --help retriever harness run --help -retriever harness sweep --help -retriever harness nightly --help -retriever harness summary --help -retriever harness compare --help +retriever harness run-set --help +retriever harness diff --help ``` -Sweep and nightly examples: +### Agentic BEIR evaluation + +Harness runs use the standard dense retrieval path unless agentic retrieval is +enabled in the resolved benchmark query config. Set `query.agentic: true` in a +code-owned benchmark or runfile, or use repeatable `--set` overrides on the CLI. +The agentic harness path runs the same ReAct retrieval graph used by root query, +but only after ingest and only for BEIR evaluation (`evaluation.mode: beir`). +`retriever pipeline run` does not expose agentic evaluation flags. + +Minimal BEIR override example: ```bash -retriever harness sweep --runs-config nemo_retriever/harness/nightly_config.yaml -retriever harness nightly --runs-config nemo_retriever/harness/nightly_config.yaml --dry-run +retriever harness run jp20_beir \ + --set query.agentic=true \ + --set query.agentic_llm_model=nvidia/llama-3.3-nemotron-super-49b-v1.5 ``` +Useful agentic query overrides: + +- `query.agentic_llm_model` — chat model used by the ReAct and selection agents; + required when `query.agentic=true`. +- `query.agentic_invoke_url` — OpenAI-compatible chat-completions endpoint. Omit + to use the built-in NVIDIA endpoint. +- `query.agentic_backend_top_k` — backend candidate pool per ReAct retrieval + call. Must be at least the final requested metric depth (`max(evaluation.ks)`). +- `query.agentic_react_max_steps` — maximum ReAct loop iterations per query + (defaults to `50`). +- `query.agentic_text_truncation` — max characters of each candidate shown to + the agent; `0` disables truncation. +- `query.agentic_num_concurrent` — number of queries the agent batch runs + concurrently (defaults to `1`). +- `query.agentic_temperature` — defaults to `0.0`; hosted/default NVIDIA + endpoints are validated as `0.0..1.0`, while other OpenAI-compatible endpoints + allow `0.0..2.0`. +- `query.agentic_reasoning_effort` — optional provider-specific field forwarded + only when configured. + ### Image storage -Image persistence is configured on `retriever pipeline run`, not on the harness. -Use `--store-images-uri ` (local path or fsspec URI). Stored assets follow -`--embed-granularity` (page vs element images). +For normal ingest, configure image persistence on `retriever ingest` with +`--store-images-uri ` (local path or fsspec URI). The harness does not +configure store directly; `retriever pipeline run --store-images-uri ` +remains available for pipeline-specific compatibility workflows. Stored assets +follow `--embed-granularity` (page vs element images). ## Per-stage micro-benchmarks @@ -80,10 +120,11 @@ Each benchmark reports rows/sec (or chunk rows/sec for audio) for its actor. ## Notes -- **Configuration:** `retriever harness` uses `--dataset` / `--preset` / - `--override KEY=VALUE` against - `nemo_retriever/harness/test_configs.yaml`. -- **Launcher:** for internal benchmarking, `retriever harness run …` is the - benchmark orchestration entry point (development / experimental; no guarantees). +- **Configuration:** `retriever harness` uses code-owned benchmarks/runsets from + `nemo_retriever.harness.benchmark_registry`; use `--set KEY=VALUE` for small + per-run config overrides. +- **Launcher:** for internal benchmarking, `retriever harness run BENCHMARK` and + `retriever harness run-set RUNSET` are the benchmark orchestration entry points + (development / experimental; no guarantees). - **Stage benchmarks:** `retriever benchmark …` is specific to the retriever CLI and covers per-stage throughput rather than full harness orchestration. diff --git a/nemo_retriever/helm/README.md b/nemo_retriever/helm/README.md index 121a41fdd5..5ba9b9291c 100644 --- a/nemo_retriever/helm/README.md +++ b/nemo_retriever/helm/README.md @@ -50,6 +50,7 @@ nemo_retriever/helm/ ├── Chart.yaml ├── values.yaml ├── README.md <-- this file +├── openshift.md <-- OpenShift restricted-v2 install guide ├── .helmignore └── templates/ ├── _helpers.tpl @@ -66,7 +67,7 @@ nemo_retriever/helm/ └── nims/ ├── nemotron-page-elements-v3.yaml # NIMCache + NIMService ├── nemotron-table-structure-v1.yaml # NIMCache + NIMService - ├── nemotron-ocr-v1.yaml # NIMCache + NIMService (OCR) + ├── nemotron-ocr-v2.yaml # NIMCache + NIMService (OCR) ├── llama-nemotron-embed-vl-1b-v2.yaml # NIMCache + NIMService (VLM embed) ├── llama-nemotron-rerank-vl-1b-v2.yaml # NIMCache + NIMService (optional; not auto-wired) ├── nemotron-parse.yaml # NIMCache + NIMService (optional; not auto-wired) @@ -78,15 +79,15 @@ nemo_retriever/helm/ ## Quick start -### 1. Service image +### 1. Service image { #1-service-image } -The chart defaults to the staging image published to NGC: +The chart defaults to the GA image published to NGC: ``` -nvcr.io/nvstaging/nim/nemo-retriever-service:043020205-001 +nvcr.io/nvidia/nemo-microservices/nrl-service:26.5.0 ``` -Pulling from `nvcr.io/nvstaging` requires an NGC pull secret — either set +Pulling from `nvcr.io` requires an NGC pull secret — either set `ngcImagePullSecret.create=true` (see below) or pre-create one in the namespace named `ngc-secret`. @@ -188,7 +189,7 @@ NIM (the VL reranker `rerankqa`, Nemotron Parse, Omni 30B, and the Parakeet `audio` ASR NIM) is **disabled by default** to honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md); -see [Recommended minimal install (26.05)](#recommended-minimal-install-2605) +refer to [Recommended minimal install](#recommended-minimal-install-2605) for the opt-in `--set` flags that turn any of them on. ```bash @@ -211,14 +212,15 @@ helm install retriever ./nemo_retriever/helm \ --set ngcApiSecret.password=$NGC_API_KEY ``` -> The VL reranker (`rerankqa`), Nemotron Parse, the Nemotron 3 Nano Omni 30B caption NIM, and the Parakeet `audio` ASR NIM are **all off by default** in 26.05 — they only reconcile when you explicitly opt in. Opt-in flags: +> The VL reranker (`rerankqa`), Nemotron Parse, the Nemotron 3 Nano Omni 30B caption NIM, the generic answer-generation LLM (`answer_llm`, Super-49B defaults), and the Parakeet `audio` ASR NIM are **all off by default** — they only reconcile when you explicitly opt in. Opt-in flags: > > * VL reranker — `--set nimOperator.rerankqa.enabled=true` > * Nemotron Parse — `--set nimOperator.nemotron_parse.enabled=true` > * Omni 30B captioner — `--set nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` +> * Answer generation LLM — `--set nimOperator.answer_llm.enabled=true` > * Parakeet ASR — `--set nimOperator.audio.enabled=true` (also set `serviceConfig.nimEndpoints.audioGrpcEndpoint=audio:50051` to wire ASR into the service, plus `service.installFfmpeg=true` if your image does not bundle ffmpeg) > -> This matches the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and avoids silently pulling ≈ 62 GiB of Omni weights or claiming a second dedicated GPU on a "default" install. See the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table for per-NIM GPU and disk costs. +> This matches the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and avoids silently pulling ≈ 62 GiB of Omni weights, loading a large two-GPU LLM, or claiming extra dedicated GPUs on a "default" install. Refer to the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table for per-NIM GPU and disk costs. The chart auto-wires the operator-managed in-cluster URLs of the four "core" NIMs into the service's `nim_endpoints` block: @@ -227,7 +229,7 @@ The chart auto-wires the operator-managed in-cluster URLs of the four | --- | ------------------------ | ----------- | | `nimOperator.page_elements` | `nemotron-page-elements-v3` | `/v1/infer` | | `nimOperator.table_structure` | `nemotron-table-structure-v1` | `/v1/infer` | -| `nimOperator.ocr` | `nemotron-ocr-v1` | `/v1/infer` | +| `nimOperator.ocr` | `nemotron-ocr-v2` | `/v1/infer` | | `nimOperator.vlm_embed` | `llama-nemotron-embed-vl-1b-v2` | `/v1/embeddings` | Track operator reconciliation with: @@ -286,8 +288,8 @@ short list of knobs you'll touch first. | Path | Default | Notes | |-------------------------------|------------------------------------|-------| -| `service.image.repository` | `localhost:32000/nemo-retriever-service` | Override to a published image. | -| `service.image.tag` | `latest` | | +| `service.image.repository` | `nvcr.io/nvidia/nemo-microservices/nrl-service` | GA NGC image; override to pin a different build or use a local registry. | +| `service.image.tag` | `26.5.0` | | | `service.replicas` | `1` | Hard cap = 1 while SQLite is the backend. | | `service.installFfmpeg` | `false` | Install `ffmpeg`/`ffprobe` at container startup by setting `INSTALL_FFMPEG=true`. Requires network egress, writable root filesystem, and sudo/setuid allowed. Not for air-gapped clusters — use a custom image instead. | | `service.resources.requests` | `16 / 16Gi` | Tune in tandem with `serviceConfig.pipeline.*Workers`. | @@ -295,19 +297,20 @@ short list of knobs you'll touch first. | `service.gpu.enabled` | `false` | The service does **not** need a GPU. | For audio and video extraction, set `service.installFfmpeg=true` when your -cluster allows runtime package installation. For air-gapped clusters, see -[Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment). +cluster allows runtime package installation. **OpenShift restricted-v2** blocks +that path — use a prebuilt service image instead; refer to [Audio and video on restricted OpenShift](./openshift.md#audio-and-video-ffmpeg-on-restricted-openshift). +For air-gapped clusters, refer to [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment). ### Audio and video (Parakeet ASR) { #audio-video-parakeet } To run self-hosted Parakeet for [audio and video extraction](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/audio-video.md): -1. Set `nimOperator.audio.enabled=true` (it is on by default; disable other optional NIMs you do not need per [Recommended minimal install (26.05)](#recommended-minimal-install-2605)). -2. Pin the ASR `NIMService` to a **dedicated GPU** with `nimOperator.audio.resources`, `nodeSelector`, or `tolerations` (see [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html)). +1. Set `nimOperator.audio.enabled=true` (it is on by default; disable other optional NIMs you do not need per [Recommended minimal install](#recommended-minimal-install-2605)). +2. Pin the ASR `NIMService` to a **dedicated GPU** with `nimOperator.audio.resources`, `nodeSelector`, or `tolerations` (refer to [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html)). 3. Confirm the GPU SKU in [Model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) (footnote ⁴ lists Blackwell limitations). -4. Set `service.installFfmpeg=true` when the retriever service will process audio or video (see `service.installFfmpeg` above). +4. Set `service.installFfmpeg=true` when the retriever service will process audio or video on clusters that allow runtime package install (refer to `service.installFfmpeg` above). On **OpenShift restricted-v2**, use a [prebuilt service image](./openshift.md#audio-and-video-ffmpeg-on-restricted-openshift) instead. -The retriever service picks up the in-cluster ASR endpoint when `nimOperator.audio` is enabled; see [NIM Operator sub-stack](#nim-operator-sub-stack). +The retriever service picks up the in-cluster ASR endpoint when `nimOperator.audio` is enabled; refer to [NIM Operator sub-stack](#nim-operator-sub-stack). ### Service configuration (rendered into `retriever-service.yaml`) @@ -315,12 +318,20 @@ The retriever service picks up the in-cluster ASR endpoint when `nimOperator.aud |---------------------------------------------------|---------|-------| | `serviceConfig.server.port` | `7670` | Container + Service port. | | `serviceConfig.pipeline.realtimeWorkers` | `24` | Per-pod realtime worker count. | -| `serviceConfig.pipeline.batchWorkers` | `48` | Per-pod batch worker count. See [Timeouts and alleviating ingest failures](#timeouts-and-alleviating-ingest-failures) if embed or pool errors appear under load. | -| `serviceConfig.nimEndpoints.*InvokeUrl` | `""` | Override the auto-resolved NIM Operator URL. Available knobs: `pageElementsInvokeUrl`, `tableStructureInvokeUrl`, `ocrInvokeUrl`, `embedInvokeUrl`, and `captionInvokeUrl` (see [Image captioning (Omni 30B)](#image-captioning-omni-30b)). | +| `serviceConfig.pipeline.batchWorkers` | `48` | Per-pod batch worker count. Refer to [Timeouts and alleviating ingest failures](#timeouts-and-alleviating-ingest-failures) if embed or pool errors appear under load. | +| `serviceConfig.nimEndpoints.*InvokeUrl` | `""` | Override the auto-resolved NIM Operator URL. Available knobs: `pageElementsInvokeUrl`, `tableStructureInvokeUrl`, `ocrInvokeUrl`, `embedInvokeUrl`, and `captionInvokeUrl` (refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b)). | | `serviceConfig.nimEndpoints.captionModelName` | `""` | Model id sent to the remote VLM. Auto-set to `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` whenever a caption URL is resolved. | -| `serviceConfig.vectordb.enabled` | `true` | Deploy the LanceDB vectordb Pod. When `true` the chart **requires** a resolvable embed endpoint (see [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint)); `helm install` / `helm upgrade` fails fast otherwise. | +| `serviceConfig.llm.enabled` | `false` | Enables `POST /v1/answer`. Auto-flips to true when `nimOperator.answer_llm` is enabled and the operator URL resolves. | +| `serviceConfig.llm.apiBase` | `""` | OpenAI-compatible LLM base URL. Explicit value wins; otherwise `answer_llm` opt-in resolves to `http://answer-llm:8000/v1` by default. | +| `serviceConfig.llm.apiKeySecret.name` | `""` | Optional Secret name for external LLM credentials. Explicit values win; otherwise operator-managed `answer_llm` mounts its `authSecret` as `NEMO_RETRIEVER_LLM_API_KEY` so LiteLLM/OpenAI has a credential value without writing it to the ConfigMap. | +| `serviceConfig.llm.apiKeySecret.key` | `api_key` | Secret key for external LLM credentials. Operator-managed `answer_llm` uses `NGC_API_KEY` from `nimOperator.answer_llm.authSecret` when no explicit LLM Secret is set. | +| `serviceConfig.llm.model` | `""` | Optional explicit LiteLLM model id. Leave empty to inherit `nimOperator.answer_llm.model` when using the operator-managed answer LLM; set it for external endpoints. | +| `serviceConfig.llm.ragSystemPromptPrefix` | `""` | Optional explicit RAG prompt prefix. Leave empty unless an endpoint needs model-specific prompt directives. | +| `serviceConfig.llm.reasoningEnabled` | `true` | Request-level reasoning toggle for `/v1/answer`. Defaults to true for external OpenAI-compatible providers; set false for Nemotron endpoints that should receive portable no-reasoning controls. | +| `serviceConfig.vectordb.enabled` | `true` | Deploy the LanceDB vectordb Pod. When `true` the chart **requires** a resolvable embed endpoint (refer to [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint)); `helm install` / `helm upgrade` fails fast otherwise. | | `serviceConfig.vectordb.lancedbUri` | `/data/vectordb` | LanceDB on the vectordb Pod's PVC. | | `serviceConfig.vectordb.embedModel` | `nvidia/llama-nemotron-embed-vl-1b-v2` | Passed to vectordb + worker `embed_model_name`. | +| `serviceConfig.vectordb.embedModelProviderPrefix` | `""` | Optional LiteLLM provider prefix prepended to the remote embed model name. | #### VectorDB and the embed endpoint { #vectordb-and-the-embed-endpoint } @@ -345,7 +356,7 @@ resolved. Pick one of: 3. --set serviceConfig.vectordb.enabled=false ``` -Resolution order matches the rest of the chart (see [Mix and match NIM +Resolution order matches the rest of the chart (refer to [Mix and match NIM sources](#3-install-with-the-nim-operator-in-cluster-nims)): 1. Explicit `serviceConfig.nimEndpoints.embedInvokeUrl` always wins. @@ -355,6 +366,91 @@ sources](#3-install-with-the-nim-operator-in-cluster-nims)): `apps.nvidia.com/v1alpha1` CRDs are installed in the cluster. 3. Otherwise the chart fails the install. +#### Answer generation (operator-managed LLM) { #answer-generation-llm } + +Enable the generic `answer_llm` NIM slot to add service-mode answer +generation on top of the VectorDB query path. The slot defaults to the +Super-49B NIM, but the image, model id, service name, resources, +profile filter, and environment can be overridden for another +OpenAI-compatible LLM NIM. + +```bash +helm upgrade --install retriever ./nemo_retriever/helm \ + --set nimOperator.answer_llm.enabled=true +``` + +When the NIM Operator CRDs are present, the chart renders an `answer-llm` +NIMCache/NIMService by default and writes this block into +`retriever-service.yaml`: + +```yaml +llm: + enabled: true + model: "openai/nvidia/llama-3.3-nemotron-super-49b-v1.5" + api_base: "http://answer-llm:8000/v1" + rag_system_prompt_prefix: null + reasoning_enabled: true +``` + +The retriever service then exposes `POST /v1/answer`, which calls the +VectorDB pod's `/v1/query` endpoint for context and sends those chunks to +the configured LLM endpoint. The `answer_llm` NIM deployment leaves +reasoning defaults model-neutral; `/v1/answer` controls reasoning per +request. By default, `serviceConfig.llm.reasoningEnabled=true`, so requests +leave reasoning behavior to the LLM endpoint defaults and avoid sending +provider-specific `chat_template_kwargs` to external OpenAI-compatible +endpoints. Set `serviceConfig.llm.reasoningEnabled=false` for Nemotron +endpoints that should skip reasoning; the service then adds both `/no_think` +and `chat_template_kwargs.enable_thinking=false`. The default Super-49B NIMService +resources request two GPUs (`nvidia.com/gpu: 2`) to match the bundled +tensor-parallel NIM profile. Override `resources`, `modelProfile`, or +`env` for deployments that use a different profile or hardware topology. +When `answer_llm` is enabled and no explicit `serviceConfig.llm.apiKeySecret` +is set, the service also mounts `nimOperator.answer_llm.authSecret` as +`NEMO_RETRIEVER_LLM_API_KEY`; OpenAI-compatible clients require a +credential value even for in-cluster NIM endpoints, and the key is never +rendered into the ConfigMap. + +For example, to try Nemotron 3 Nano as the answer LLM on an A100 80GB +node, override the operator-managed slot instead of adding a second +hard-coded LLM service: + +```bash +helm upgrade --install retriever ./nemo_retriever/helm \ + --set nimOperator.answer_llm.enabled=true \ + --set nimOperator.answer_llm.nimServiceName=nemotron-3-nano \ + --set nimOperator.answer_llm.image.repository=nvcr.io/nim/nvidia/nemotron-3-nano \ + --set nimOperator.answer_llm.image.tag=1.7.0-variant \ + --set nimOperator.answer_llm.model=openai/nvidia/nemotron-3-nano-30b-a3b \ + --set-json nimOperator.answer_llm.modelProfile='{"profiles":["5f89f01a0af587fd8bae50c611b1f358f92effdb9fb29362e1af0a986e5561c3"]}' \ + --set-json nimOperator.answer_llm.resources='{"limits":{"nvidia.com/gpu":1},"requests":{"nvidia.com/gpu":1}}' \ + --set nimOperator.answer_llm.env[0].name=NIM_HTTP_API_PORT \ + --set-string nimOperator.answer_llm.env[0].value=8000 \ + --set nimOperator.answer_llm.env[1].name=NIM_SERVED_MODEL_NAME \ + --set-string nimOperator.answer_llm.env[1].value=nvidia/nemotron-3-nano-30b-a3b \ + --set nimOperator.answer_llm.env[2].name=NIM_TENSOR_PARALLEL_SIZE \ + --set-string nimOperator.answer_llm.env[2].value=1 +``` + +Use the repository and tag available in your NGC environment; staging +registries can use the same override shape with `nvstaging` image names +or tags. `nimOperator.answer_llm.model` is the LiteLLM model id used by +the retriever service; for an OpenAI-compatible in-cluster NIM, keep the +`openai/` prefix there and set `NIM_SERVED_MODEL_NAME` to the raw model +name advertised by the NIM. Replace the default Super-49B `modelProfile`, +`resources`, and `env` when the target model requires a different +GPU/profile setup. Leaving `modelProfile` empty preserves NIM +Operator auto-discovery, but for Nano it can cache every advertised +profile on first reconciliation; pin a known-compatible profile when you +know the target GPU topology. + +`serviceConfig.llm.apiBase` and `serviceConfig.llm.model` can be set +explicitly to point `/v1/answer` at an external OpenAI-compatible LLM +instead of deploying an answer LLM in-cluster. For external credentials, +create a Kubernetes Secret and set `serviceConfig.llm.apiKeySecret.name` +plus `serviceConfig.llm.apiKeySecret.key`; Helm mounts the Secret as an +environment variable instead of writing the key into the ConfigMap. + ### NIM Operator sub-stack Each NIM block under `nimOperator.` renders a `NIMCache` + `NIMService` @@ -370,23 +466,26 @@ pair gated on three conditions ALL holding: | `nimOperator.page_elements.enabled` | `true` | Page-elements detector NIM. | | `nimOperator.table_structure.enabled` | `true` | Table-structure detector NIM. | | `nimOperator.ocr.enabled` | `true` | OCR NIM. | -| `nimOperator.ocr.image` | `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` | Default OCR NIM image. | +| `nimOperator.ocr.image` | `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` | Default OCR NIM image. | | `nimOperator.vlm_embed.enabled` | `true` | Multimodal embedding NIM (also used by the vectordb Pod). | | `nimOperator.vlm_embed.nimServiceName` | `llama-nemotron-embed-vl-1b-v2` | NIMService / in-cluster DNS name. | | `nimOperator.vlm_embed.image` | `nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0` | Default VLM embed NIM image. | -| `nimOperator.rerankqa.enabled` | `false` | VL reranker NIM (optional; not auto-wired). Set `true` to opt in. Default `false` so 26.05 installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and do not silently provision an extra ≈ 3.1 GiB GPU NIM. The image points at the **VL** SKU (`llama-nemotron-rerank-vl-1b-v2`) per [prerequisites-support-matrix.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#default-helm-nims) — the text-only `llama-nemotron-rerank-1b-v2` silently degrades multimodal reranking and is not the documented POR. | -| `nimOperator.nemotron_parse.enabled` | `false` | Structured-parse NIM (optional). Set `true` when using `extract_method="nemotron_parse"`. Default `false` so 26.05 installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md). Image tag follows the [image tag conventions](#image-tag-conventions). | -| `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled` | `false` | Omni 30B caption NIM (optional). Set `true` to enable image captioning — see [Image captioning (Omni 30B)](#image-captioning-omni-30b). Default `false` so 26.05 installs do not silently pull ≈ 62 GiB of BF16 weights or claim a second dedicated GPU. Image tag follows the [image tag conventions](#image-tag-conventions). | +| `nimOperator.rerankqa.enabled` | `false` | VL reranker NIM (optional; not auto-wired). Set `true` to opt in. Default `false` so chart installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and do not silently provision an extra ≈ 3.1 GiB GPU NIM. The image points at the **VL** SKU (`llama-nemotron-rerank-vl-1b-v2`) per [prerequisites-support-matrix.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#default-helm-nims) — the text-only `llama-nemotron-rerank-1b-v2` silently degrades multimodal reranking and is not the documented POR. | +| `nimOperator.nemotron_parse.enabled` | `false` | Structured-parse NIM (optional). Set `true` when using `extract_method="nemotron_parse"`. Default `false` so chart installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md). Image tag follows the [image tag conventions](#image-tag-conventions). | +| `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled` | `false` | Omni 30B caption NIM (optional). Set `true` to enable image captioning — refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b). Default `false` so chart installs do not silently pull ≈ 62 GiB of BF16 weights or claim a second dedicated GPU. Image tag follows the [image tag conventions](#image-tag-conventions). | +| `nimOperator.answer_llm.enabled` | `false` | Generic answer-generation LLM NIM (optional; Super-49B defaults). Set `true` to enable `/v1/answer` — refer to [Answer generation (operator-managed LLM)](#answer-generation-llm). Default `false` so installs do not silently claim answer-generation GPUs. | +| `nimOperator.answer_llm.model` | `openai/nvidia/llama-3.3-nemotron-super-49b-v1.5` | LiteLLM/OpenAI model id inherited by `serviceConfig.llm.model` when the operator-managed answer LLM is enabled and no explicit service model is set. | +| `nimOperator.answer_llm.ragSystemPromptPrefix` | `""` | Optional prompt prefix inherited by `serviceConfig.llm.ragSystemPromptPrefix` only when explicitly set. Leave empty to keep the operator-managed LLM model-neutral and use `serviceConfig.llm.reasoningEnabled` for request-level reasoning control. | | `nimOperator.audio.enabled` | `false` | Parakeet ASR NIM (optional). Set `true` for audio/video transcription; pair with `serviceConfig.nimEndpoints.audioGrpcEndpoint=audio:50051` so the retriever-service can reach it. | | `nimOperator..image.repository` | `nvcr.io/nim/nvidia/...` | Per-NIM image. | | `nimOperator..image.pullSecrets` | `[ngc-secret]` | Referenced by the NIMService CR. | | `nimOperator..authSecret` | `ngc-api` | NIM auth Secret name. | | `nimOperator..storage.pvc.size` | `25Gi` (50Gi for vlm_embed/rerankqa, 100Gi parse, 300Gi VL) | NIMCache PVC size. | | `nimOperator..replicas` | `1` | Per-NIMService replica count. | -| `nimOperator.nimServiceGpuLimit` | `1` | Default `nvidia.com/gpu` limit on every NIMService when per-NIM `resources` is `{}`. Set to `null` for operator-only reconciliation (not reliable on all NIM Operator versions — see [GPU limits and `helm upgrade`](#gpu-limits-and-helm-upgrade)). | +| `nimOperator.nimServiceGpuLimit` | `1` | Default `nvidia.com/gpu` limit on every NIMService when per-NIM `resources` is `{}`. Set to `null` for operator-only reconciliation (not reliable on all NIM Operator versions — refer to [GPU limits and `helm upgrade`](#gpu-limits-and-helm-upgrade)). | | `nimOperator..resources` | `{}` | Per-NIM override of the whole `resources` block. Empty uses `nimServiceGpuLimit`; non-empty replaces the chart default (may require `--force-conflicts` on later `helm upgrade`). | -| `nimOperator.modelProfile` | `{}` | Chart-wide NIMCache GPU/profile filter. Applied to every NIMCache that does not have its own override. See [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | -| `nimOperator..modelProfile` | `{}` | Per-NIM NIMCache GPU/profile filter. Non-empty values REPLACE the chart-wide default (no merge). See [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | +| `nimOperator.modelProfile` | `{}` | Chart-wide NIMCache GPU/profile filter. Applied to every NIMCache that does not have its own override. Refer to [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | +| `nimOperator..modelProfile` | `{}` | Per-NIM NIMCache GPU/profile filter. Non-empty values REPLACE the chart-wide default (no merge). Refer to [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | | `nimOperator..expose.service.port` | `8000` (9000 for audio) | HTTP port. | | `nimOperator..expose.service.grpcPort` | `8001` (50051 for audio) | gRPC port. | @@ -394,7 +493,7 @@ pair gated on three conditions ALL holding: > are auto-wired into the retriever-service config. Optional NIMs may reconcile > when `nimOperator..enabled` is `true` in `values.yaml`, but the > retriever-service won't call them unless you wire your pipeline to use them. -> For 26.05, prefer the [minimal install](#recommended-minimal-install-2605) overrides. +> For minimal installs, prefer the [minimal install](#recommended-minimal-install-2605) overrides. #### Filtering cached GPU profiles { #filtering-cached-gpu-profiles } @@ -465,7 +564,7 @@ Every NIM in this chart pins an exact NGC image tag in `values.yaml` | Family | Example | Meaning | | ------ | ------- | ------- | | Plain semver | `nemotron-page-elements-v3:1.8.0` | A standard NIM release, identical bytes on every pull. Used by the four core NIMs and the reranker / ASR NIMs. | -| `-variant` | `nemotron-parse-v1.2:1.7.0-variant`, `nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | The Nemotron Parse and Nemotron 3 Nano Omni 30B builds that ship per-GPU TensorRT engine variants the NIM Operator selects from at reconciliation time (see the Omni and Parse rows in the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table). The `-variant` suffix is the NGC tag that ships alongside the 26.05 chart and matches footnote ³ of the support matrix. | +| `-variant` | `nemotron-parse-v1.2:1.7.0-variant`, `nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | The Nemotron Parse and Nemotron 3 Nano Omni 30B builds that ship per-GPU TensorRT engine variants the NIM Operator selects from at reconciliation time (refer to the Omni and Parse rows in the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table). The `-variant` suffix is the NGC tag that ships alongside this chart and matches footnote ³ of the support matrix. | For air-gapped mirror pipelines: mirror the *exact* tag — both the plain semver and the `-variant` form — and do not substitute `:latest`. @@ -483,18 +582,18 @@ helm upgrade --install retriever ./nemo_retriever/helm \ and validate against the same release of the retriever service before production rollout. -**Charts and captioning (26.05).** Charts and infographics use **page_elements** +**Charts and captioning.** Charts and infographics use **page_elements** and **ocr** (no `graphic_elements` operator NIM in this chart). For image -captioning, set `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` — see +captioning, set `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` — refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b) for the chart-side wiring and -[Image captioning (26.05)](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning-2605) +[Image captioning](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning) for the product matrix. #### Image captioning (Omni 30B) { #image-captioning-omni-30b } The Nemotron 3 Nano Omni VLM is the canonical image-caption NIM for -26.05. When you enable it, +this chart. When you enable it, ```bash helm upgrade --install retriever ./nemo_retriever/helm \ @@ -531,7 +630,7 @@ Resolution order mirrors every other NIM endpoint (see the `serviceConfig.nimEndpoints.captionModelName` follows the same order — it defaults to the canonical Omni remote model id (`nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`, matching -`nemo_retriever.caption.model_profiles.OMNI_REMOTE_MODEL_ID`) whenever +`nemo_retriever.common.modality.caption.model_profiles.OMNI_REMOTE_MODEL_ID`) whenever the chart resolves any caption URL. Override only when pointing at a different VLM SKU. @@ -583,7 +682,7 @@ block). Confirm `image.repository` and `image.tag` before you upgrade. |------|------| | `nimOperator.nimCache.keepOnUninstall` | When `true`, NIMCache CRs survive `helm uninstall` (`helm.sh/resource-policy: keep`). NIMService CRs are always removed. Set `false` for dev clusters that should fully tear down on uninstall. | | `nimOperator.ocr.enabled` | Reconcile the OCR `NIMService` | -| `nimOperator.ocr.image.repository` | NIM image (default `nvcr.io/nim/nvidia/nemotron-ocr-v1`) | +| `nimOperator.ocr.image.repository` | NIM image (default `nvcr.io/nim/nvidia/nemotron-ocr-v2`) | | `nimOperator.ocr.image.tag` | Pin the image tag for reproducible upgrades | Override the auto-wired in-cluster URL with `serviceConfig.nimEndpoints.ocrInvokeUrl` @@ -937,163 +1036,106 @@ sanity check before opening Grafana. --- -## OpenShift deployment { #openshift-deployment } - -The chart defaults target generic Kubernetes clusters that allow fixed numeric -UIDs (`runAsUser` / `runAsGroup` / `fsGroup` **1000**). **OpenShift 4.x** -namespaces under the default **restricted-v2** Security Context Constraint (SCC) -and **Pod Security Admission (PSA) `restricted`** profile assign a per-namespace -UID/GID range instead. A stock `helm install` without overrides therefore fails -SCC validation, emits PSA warnings, or crashes on log paths the random UID cannot -write. - -We do **not** change chart defaults for OpenShift-only behavior (that would affect -other platforms). Use the overrides below on OpenShift, or save the YAML block -into a local values file and pass `-f `. - -### Cluster posture (typical QA / hardened namespaces) - -| Control | Typical default on a new OpenShift project | -| --- | --- | -| SCC | **restricted-v2** (first match in priority order) | -| PSA | `pod-security.kubernetes.io/warn=restricted` (and often `audit=restricted`; `enforce` may be unset on dev clusters) | -| UID assignment | SCC injects `runAsUser` / `fsGroup` from the namespace range (for example `1000750000–1000759999`) | - -On clusters with **PSA `enforce=restricted`**, missing container `securityContext` -fields become hard rejections, not warnings. - -### Override reference (maps to chart limitations) - -| Symptom on stock install | Cause | Helm override | -| --- | --- | --- | -| `FailedCreate`: UID/GID **1000** not in namespace range | Hardcoded `service.podSecurityContext` UID/GID/fsGroup | Omit `runAsUser`, `runAsGroup`, and `fsGroup`; keep only `runAsNonRoot: true` | -| PSA warning: `allowPrivilegeEscalation`, capabilities, `seccompProfile` | Empty `service.securityContext` | Set restricted baseline on `service.securityContext` (see sample below) | -| `PermissionError` on `/var/lib/nemo-retriever/retriever-service.log` when `persistence.enabled=false` | Default log path is image-owned; random UID cannot write without a PVC | Point `serviceConfig.logging.file` at `/tmp/...` (chart mounts `emptyDir` at `/tmp`) | -| `CreateContainerConfigError`: non-numeric image `USER nemo` on **vectordb** | Vectordb container has no `securityContext` block for SCC to annotate | Disable vectordb for smoke tests, or patch the vectordb Deployment after install (below) | -| PSA warnings on **otel-collector** | Otel Deployment has no `securityContext` in the chart | `topology.otel.enabled=false` unless you patch that Deployment | +## Tracing and Zipkin -### Recommended value overrides +Helm installs the chart-owned OpenTelemetry Collector and Zipkin backend on by +default. This is intentional: the legacy 26.1.2 Helm chart shipped with a +managed Zipkin deployment enabled, so the new chart keeps a default trace +backend available for functional parity. Pod trace export is also enabled by +default for retriever service pods and chart-managed NIMs: ```yaml -# OpenShift overrides for nemo-retriever Helm chart (restricted-v2 / PSA restricted). -# Save locally, then: helm install retriever ./nemo_retriever/helm -f .yaml ... +topology: + otel: + enabled: true + zipkin: + enabled: true service: - podSecurityContext: - runAsNonRoot: true - # Do NOT set runAsUser, runAsGroup, or fsGroup — OpenShift SCC assigns them. - securityContext: - allowPrivilegeEscalation: false - capabilities: - drop: ["ALL"] - seccompProfile: - type: RuntimeDefault - -serviceConfig: - logging: - # Writable without persistence PVC (chart always mounts emptyDir at /tmp). - file: /tmp/retriever-service.log - vectordb: - # Set false for minimal service-only validation; see vectordb patch below if enabled. - enabled: false + otel: + enabled: true -topology: +nimOperator: otel: - enabled: false + enabled: true ``` -When **`persistence.enabled=true`**, you can keep the default log path under -`persistence.mountPath` (`/var/lib/nemo-retriever`) because the PVC is mounted and -SCC-assigned `fsGroup` applies. When persistence is off, always relocate logs to -`/tmp` (or another path backed by `service.extraVolumes`). - -### Example install on OpenShift 4.20 (service-only smoke test) +Because Zipkin is chart-owned by default, an upgrade with default values can +create a Zipkin Deployment and Service. Set `topology.zipkin.enabled=false` +before upgrading if your deployment uses an external backend or should not run +chart-owned Zipkin. -Matches QA validation with external NIMs disabled, no persistence, and no results -PVC: +With default values, retriever service pods and chart-managed NIMs emit OTLP to +the chart's OpenTelemetry Collector, which exports traces to the chart-owned +Zipkin service. Set `service.otel.enabled=false` or +`nimOperator.otel.enabled=false` to opt out by surface. Open a job and read the +Zipkin lookup key from either the JSON body or the `x-trace-id` response header: ```bash -oc new-project nemo-retriever - -oc create secret docker-registry ngc-secret -n nemo-retriever \ - --docker-server=nvcr.io --docker-username='$oauthtoken' \ - --docker-password="$NGC_API_KEY" +kubectl port-forward svc/tracing-smoke-nemo-retriever 7670:80 -oc create secret generic ngc-api -n nemo-retriever \ - --from-literal=NGC_API_KEY="$NGC_API_KEY" \ - --from-literal=NGC_CLI_API_KEY="$NGC_API_KEY" +curl -s -D headers.txt -o job.json \ + -X POST http://localhost:7670/v1/ingest/job \ + -H 'content-type: application/json' \ + -d '{"expected_documents":1}' -helm install retriever ./nemo_retriever/helm -n nemo-retriever \ - -f .yaml \ - --set ngcImagePullSecret.create=false \ - --set ngcApiSecret.create=false \ - --set nims.enabled=false \ - --set persistence.enabled=false \ - --set retrieverResults.enabled=false +TRACE_ID=$(jq -r .trace_id job.json) +grep -i x-trace-id headers.txt ``` -Verify pods: +Port-forward Zipkin and query the trace directly: ```bash -oc get pods -n nemo-retriever -oc describe pod -l app.kubernetes.io/name=nemo-retriever -n nemo-retriever +kubectl port-forward svc/tracing-smoke-nemo-retriever-zipkin 9411:9411 +curl "http://localhost:9411/api/v2/trace/${TRACE_ID}" ``` -You should see SCC-assigned numeric `runAsUser` on containers that declare a -`securityContext` block, and no PSA warnings once overrides are applied. +Common opt-out and override knobs: -### Enabling the vectordb Deployment on OpenShift +```yaml +topology: + zipkin: + enabled: false # do not deploy chart-owned Zipkin + exporter: + enabled: false # keep Zipkin deployed, but do not export traces to it + endpoint: http://external-zipkin:9411/api/v2/spans -`serviceConfig.vectordb.enabled=true` renders a **vectordb** container from the -same image (`USER nemo`, non-numeric). The chart does not yet expose a -`securityContext` value for that container. After `helm install`, patch the -Deployment so OpenShift can inject a numeric UID into the container spec: +service: + otel: + enabled: false # do not inject service pod instrumentation env -```bash -RELEASE=retriever -NS=nemo-retriever -VDB_DEPLOY="${RELEASE}-nemo-retriever-vectordb" - -oc patch deployment "$VDB_DEPLOY" -n "$NS" --type=json -p='[ - {"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": { - "allowPrivilegeEscalation": false, - "capabilities": {"drop": ["ALL"]}, - "runAsNonRoot": true, - "seccompProfile": {"type": "RuntimeDefault"} - }} -]' +nimOperator: + otel: + enabled: false # do not inject inherited NIM OTLP env + page_elements: + otel: + enabled: false # per-NIM opt-out + ocr: + otel: + env: + TRITON_OTEL_RATE: "10" # per-NIM Triton OTel override ``` -Re-apply the patch after `helm upgrade` if the Deployment is recreated. A future -chart release may add first-class `topology.vectordb.securityContext` values. +Set `topology.zipkin.exporter.endpoint` when you run your own Zipkin-compatible +collector. Set `topology.otel.enabled=false` to disable the chart-owned collector +and all chart-rendered collector wiring. -### Enabling the OpenTelemetry collector on OpenShift - -The chart’s otel-collector Deployment likewise lacks `securityContext` fields. -Prefer `topology.otel.enabled=false` (as in the sample values) unless you operate -your own collector or patch `*-otel` the same way as vectordb. - -### What we intentionally do not require on OpenShift - -Do **not** bind the namespace to **anyuid** SCC or set PSA `enforce=privileged` -unless your security team explicitly approves it. The overrides above are intended -to keep **restricted-v2** / PSA **restricted** posture. +--- -### Related documentation +## OpenShift deployment { #openshift-deployment } -- [Pre-Requisites & Support Matrix](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md) -- [Deployment options](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) +OpenShift install procedures, **restricted-v2** / PSA **restricted** value overrides, prebuilt `ffmpeg` images, internal registry pull secrets, optional NIM `LD_LIBRARY_PATH` tuning, and install examples are in **[OpenShift deployment](./openshift.md)**. Pass `-f openshift-restricted.yaml` from that guide when you install on OpenShift. --- ## Air-gapped deployment { #air-gapped-deployment } -See [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment) for overview and workflow. Chart-specific reference for mirroring: +Refer to [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment) for overview and workflow. Chart-specific reference for mirroring: -### Container images to mirror (26.05 chart defaults) +### Container images to mirror (chart defaults) -Verify tags on the Git branch or tag you ship (for example `26.05` or -`26.5.0`). Defaults below match +Verify tags on the Git branch or tag you ship (for example `main` or +your release tag). Defaults below match [`values.yaml`](./values.yaml) on the current chart. | Role | `nimOperator` key | Default image (`repository:tag`) | @@ -1101,11 +1143,12 @@ Verify tags on the Git branch or tag you ship (for example `26.05` or | Retriever service | — | `service.image.repository`:`service.image.tag` (override for production) | | Page elements | `page_elements` | `nvcr.io/nim/nvidia/nemotron-page-elements-v3:1.8.0` | | Table structure | `table_structure` | `nvcr.io/nim/nvidia/nemotron-table-structure-v1:1.8.0` | -| OCR | `ocr` | `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` | +| OCR | `ocr` | `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` | | VL embed | `vlm_embed` | `nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0` | | VL reranker (optional) | `rerankqa` | `nvcr.io/nim/nvidia/llama-nemotron-rerank-vl-1b-v2:1.10.0` | | Nemotron Parse (optional) | `nemotron_parse` | `nvcr.io/nim/nvidia/nemotron-parse-v1.2:1.7.0-variant` | | Omni caption (optional) | `nemotron_3_nano_omni_30b_a3b_reasoning` | `nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | +| Answer LLM (optional, Super-49B default) | `answer_llm` | `nvcr.io/nim/nvidia/llama-3.3-nemotron-super-49b-v1.5:2.0.5` | | Parakeet ASR (optional) | `audio` | `nvcr.io/nim/nvidia/parakeet-1-1b-ctc-en-us:1.5.0` | GPU SKU support for `audio` is in [Model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements). @@ -1135,7 +1178,8 @@ imagePullSecrets: - name: my-private-registry ngcImagePullSecret: - create: false # use secrets that authenticate to YOUR mirror + create: false + name: "" # Explicitly empty — clears the default "ngc-secret" nimOperator: page_elements: @@ -1146,15 +1190,15 @@ nimOperator: # Repeat for table_structure, ocr, vlm_embed, and any optional keys you enable. ``` -- Set `nimOperator..image.pullSecrets` to the Secret name your - `NIMService` resources should use (defaults to `ngc-secret`). +- Set `nimOperator..image.pullSecrets` to your mirror pull secret + (for example `my-private-registry`; chart default is `ngc-secret`). - Leave `serviceConfig.nimEndpoints.*` empty when operator-managed NIMs are in-cluster; set explicit URLs only for external or mirrored services outside the chart. - For **offline captioning**, enable `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning` and point the pipeline - caption endpoint at the in-cluster NIM URL (see - [Image captioning (26.05)](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning-2605)). + caption endpoint at the in-cluster NIM URL (refer to + [Image captioning](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning)). ### Mirroring pattern @@ -1209,7 +1253,7 @@ helm template r nemo_retriever/helm \ ``` Both renders should succeed cleanly and parse as valid Kubernetes manifests -(`kubectl apply --dry-run=client -f /tmp/r.yaml`). See [VectorDB and the +(`kubectl apply --dry-run=client -f /tmp/r.yaml`). Refer to [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint) for why `helm template r nemo_retriever/helm` without flags is rejected as a misconfiguration. From 830c0bdc781a8c4d7b803c6a2f4f1614aed946d1 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 14:43:01 -0700 Subject: [PATCH 2/5] docs(26.05): fix ASRParams import path in audio-video examples Import ASRParams from nemo_retriever.params.models instead of the non-existent nemo_retriever.common.params.models, which would raise ModuleNotFoundError when users copy the Parakeet ASR workflow. --- docs/docs/extraction/audio-video.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/extraction/audio-video.md b/docs/docs/extraction/audio-video.md index 3f2df2cf9c..6a6cfa21a6 100644 --- a/docs/docs/extraction/audio-video.md +++ b/docs/docs/extraction/audio-video.md @@ -73,7 +73,7 @@ Use the following procedure to run the NIM on your own infrastructure. Self-host ```python from nemo_retriever import create_ingestor - from nemo_retriever.common.params.models import ASRParams + from nemo_retriever.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") @@ -104,7 +104,7 @@ Instead of running the pipeline locally, you can call Parakeet through [build.nv ```python from nemo_retriever import create_ingestor - from nemo_retriever.common.params.models import ASRParams + from nemo_retriever.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") From 8f34b60fad0a1d2a18de795f281ff565036da4da Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 15:19:08 -0700 Subject: [PATCH 3/5] docs(26.05): fix API reference module paths and pin minimal-install link Point mkdocstrings at nemo_retriever.retriever and nemo_retriever.params (the modules that exist in this branch) so the API reference renders, and pin the deployment-options minimal-install Helm README link to the 26.05 branch to avoid chart drift from main. --- docs/docs/extraction/deployment-options.md | 2 +- docs/docs/extraction/nemo-retriever-api-reference.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/extraction/deployment-options.md b/docs/docs/extraction/deployment-options.md index b57404685a..7d350925c5 100644 --- a/docs/docs/extraction/deployment-options.md +++ b/docs/docs/extraction/deployment-options.md @@ -22,7 +22,7 @@ Build and run the NeMo Retriever service image with the [Docker service image gu 3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in [About getting started](getting-started-about.md) — use alongside the NeMo Retriever chart README for your release 4. [Environment variables](environment-config.md) and [Troubleshoot](troubleshoot.md) as needed -**Core NIMs for the default extraction pipeline:** `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Refer to [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). +**Core NIMs for the default extraction pipeline:** `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Refer to [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). For audio and video extraction in Kubernetes, set `service.installFfmpeg=true` so the service container installs `ffmpeg` and `ffprobe` at startup. This runtime install requires package-repository network egress, a writable root filesystem, and security policy that allows the image's scoped sudo use. If your cluster blocks startup package installation, use a custom service image that already contains `ffmpeg` and `ffprobe`, then set `service.image.repository` and `service.image.tag`. For Parakeet ASR chart values, OpenShift-specific Helm configuration, and air-gapped alternatives, refer to [Audio and video (Parakeet ASR)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#audio-video-parakeet) and [OpenShift deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md) in the Helm chart directory. diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index 2841b9799f..1b057431d3 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -13,6 +13,6 @@ To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray acto filters: - "!^pdf_split_config$" -::: nemo_retriever.graph.retriever +::: nemo_retriever.retriever -::: nemo_retriever.common.params +::: nemo_retriever.params From b5b9ed6fc9d2bae8501df21fc7c7023b02abce3a Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 15:29:05 -0700 Subject: [PATCH 4/5] docs(26.05): align README/helm examples and links with 26.05 chart Fix import paths (nemo_retriever.io, nemo_retriever.retriever), correct OCR NIM to nemotron-ocr-v1:1.3.0 and VL reranker tag to 1.11.0 to match the 26.05 chart values, and pin deployment-options air-gap/Helm handoff links to the 26.05 branch. Addresses Greptile review comments on PR 2311. --- docs/docs/extraction/deployment-options.md | 4 ++-- nemo_retriever/README.md | 6 +++--- nemo_retriever/helm/README.md | 10 +++++----- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/docs/extraction/deployment-options.md b/docs/docs/extraction/deployment-options.md index 7d350925c5..063632f6f0 100644 --- a/docs/docs/extraction/deployment-options.md +++ b/docs/docs/extraction/deployment-options.md @@ -69,7 +69,7 @@ Consider self-hosting when: The **default document extraction pipeline** (page elements, table structure, OCR, and VL embed) runs disconnected when you mirror images and models into a private registry and configure the [NIM Operator for air-gapped environments](https://docs.nvidia.com/nim-operator/latest/air-gap.html). -On a staging host with internet access, pull from NGC, retag to your private registry, stage chart archives, then install in the enclave with registry overrides. Procedures, the chart image inventory, and Helm value patterns are in [Helm — Air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#air-gapped-deployment). +On a staging host with internet access, pull from NGC, retag to your private registry, stage chart archives, then install in the enclave with registry overrides. Procedures, the chart image inventory, and Helm value patterns are in [Helm — Air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#air-gapped-deployment). !!! warning "Audio and video extraction" @@ -79,7 +79,7 @@ For offline image captioning, deploy the in-cluster [Nemotron 3 Nano Omni](prere **Related** -- [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) ([`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm) on GitHub) — [air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#air-gapped-deployment) +- [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md) ([`nemo_retriever/helm`](https://github.com/NVIDIA/NeMo-Retriever/tree/26.05/nemo_retriever/helm) on GitHub) — [air-gapped deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#air-gapped-deployment) - [About getting started](getting-started-about.md) (prerequisites through first deployment) - [Pre-Requisites & Support Matrix](prerequisites-support-matrix.md) - [Audio and video](audio-video.md) diff --git a/nemo_retriever/README.md b/nemo_retriever/README.md index 24c24bd942..3dcc626696 100644 --- a/nemo_retriever/README.md +++ b/nemo_retriever/README.md @@ -103,7 +103,7 @@ The examples below use default local GPU inference (no `invoke_url` specified) a ### Ingest a test pdf ```python from nemo_retriever import create_ingestor -from nemo_retriever.common.io import to_markdown, to_markdown_by_page +from nemo_retriever.io import to_markdown, to_markdown_by_page from pathlib import Path documents = [str(Path("../data/multimodal_test.pdf"))] @@ -225,7 +225,7 @@ Since the ingestion job automatically populated a lancedb table with all these c ### Run a recall query ```python -from nemo_retriever.graph.retriever import Retriever +from nemo_retriever.retriever import Retriever retriever = Retriever( # values used by the graph_pipeline example above @@ -330,7 +330,7 @@ embedding model in `embed_kwargs` must match the one used during ingestion so query vectors land in the same embedding space as the stored chunks. ```python -from nemo_retriever.graph.retriever import Retriever +from nemo_retriever.retriever import Retriever from nemo_retriever.llm import LiteLLMClient retriever = Retriever( diff --git a/nemo_retriever/helm/README.md b/nemo_retriever/helm/README.md index 5ba9b9291c..0590777a9f 100644 --- a/nemo_retriever/helm/README.md +++ b/nemo_retriever/helm/README.md @@ -67,7 +67,7 @@ nemo_retriever/helm/ └── nims/ ├── nemotron-page-elements-v3.yaml # NIMCache + NIMService ├── nemotron-table-structure-v1.yaml # NIMCache + NIMService - ├── nemotron-ocr-v2.yaml # NIMCache + NIMService (OCR) + ├── nemotron-ocr-v1.yaml # NIMCache + NIMService (OCR) ├── llama-nemotron-embed-vl-1b-v2.yaml # NIMCache + NIMService (VLM embed) ├── llama-nemotron-rerank-vl-1b-v2.yaml # NIMCache + NIMService (optional; not auto-wired) ├── nemotron-parse.yaml # NIMCache + NIMService (optional; not auto-wired) @@ -229,7 +229,7 @@ The chart auto-wires the operator-managed in-cluster URLs of the four | --- | ------------------------ | ----------- | | `nimOperator.page_elements` | `nemotron-page-elements-v3` | `/v1/infer` | | `nimOperator.table_structure` | `nemotron-table-structure-v1` | `/v1/infer` | -| `nimOperator.ocr` | `nemotron-ocr-v2` | `/v1/infer` | +| `nimOperator.ocr` | `nemotron-ocr-v1` | `/v1/infer` | | `nimOperator.vlm_embed` | `llama-nemotron-embed-vl-1b-v2` | `/v1/embeddings` | Track operator reconciliation with: @@ -466,7 +466,7 @@ pair gated on three conditions ALL holding: | `nimOperator.page_elements.enabled` | `true` | Page-elements detector NIM. | | `nimOperator.table_structure.enabled` | `true` | Table-structure detector NIM. | | `nimOperator.ocr.enabled` | `true` | OCR NIM. | -| `nimOperator.ocr.image` | `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` | Default OCR NIM image. | +| `nimOperator.ocr.image` | `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` | Default OCR NIM image. | | `nimOperator.vlm_embed.enabled` | `true` | Multimodal embedding NIM (also used by the vectordb Pod). | | `nimOperator.vlm_embed.nimServiceName` | `llama-nemotron-embed-vl-1b-v2` | NIMService / in-cluster DNS name. | | `nimOperator.vlm_embed.image` | `nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0` | Default VLM embed NIM image. | @@ -1143,9 +1143,9 @@ your release tag). Defaults below match | Retriever service | — | `service.image.repository`:`service.image.tag` (override for production) | | Page elements | `page_elements` | `nvcr.io/nim/nvidia/nemotron-page-elements-v3:1.8.0` | | Table structure | `table_structure` | `nvcr.io/nim/nvidia/nemotron-table-structure-v1:1.8.0` | -| OCR | `ocr` | `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` | +| OCR | `ocr` | `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` | | VL embed | `vlm_embed` | `nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0` | -| VL reranker (optional) | `rerankqa` | `nvcr.io/nim/nvidia/llama-nemotron-rerank-vl-1b-v2:1.10.0` | +| VL reranker (optional) | `rerankqa` | `nvcr.io/nim/nvidia/llama-nemotron-rerank-vl-1b-v2:1.11.0` | | Nemotron Parse (optional) | `nemotron_parse` | `nvcr.io/nim/nvidia/nemotron-parse-v1.2:1.7.0-variant` | | Omni caption (optional) | `nemotron_3_nano_omni_30b_a3b_reasoning` | `nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | | Answer LLM (optional, Super-49B default) | `answer_llm` | `nvcr.io/nim/nvidia/llama-3.3-nemotron-super-49b-v1.5:2.0.5` | From a1259052aa6f61b518d1f6ae0f5c39cc009d9e35 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 15:33:21 -0700 Subject: [PATCH 5/5] docs(26.05): flag answer_llm as not in the 26.05 chart The 26.05 chart ships no answer_llm values block, NIM template, or serviceConfig.llm keys, so the advertised answer-generation flag has no effect on this chart. Add a note softening the operator-managed answer LLM claims until the chart matches. Addresses Greptile review comment on PR 2311. --- nemo_retriever/helm/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/nemo_retriever/helm/README.md b/nemo_retriever/helm/README.md index 0590777a9f..d1fadf7543 100644 --- a/nemo_retriever/helm/README.md +++ b/nemo_retriever/helm/README.md @@ -217,7 +217,7 @@ helm install retriever ./nemo_retriever/helm \ > * VL reranker — `--set nimOperator.rerankqa.enabled=true` > * Nemotron Parse — `--set nimOperator.nemotron_parse.enabled=true` > * Omni 30B captioner — `--set nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` -> * Answer generation LLM — `--set nimOperator.answer_llm.enabled=true` +> * Answer generation LLM — `--set nimOperator.answer_llm.enabled=true` (**not in the 26.05 chart**; refer to [Answer generation (operator-managed LLM)](#answer-generation-llm)) > * Parakeet ASR — `--set nimOperator.audio.enabled=true` (also set `serviceConfig.nimEndpoints.audioGrpcEndpoint=audio:50051` to wire ASR into the service, plus `service.installFfmpeg=true` if your image does not bundle ffmpeg) > > This matches the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and avoids silently pulling ≈ 62 GiB of Omni weights, loading a large two-GPU LLM, or claiming extra dedicated GPUs on a "default" install. Refer to the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table for per-NIM GPU and disk costs. @@ -368,6 +368,8 @@ sources](#3-install-with-the-nim-operator-in-cluster-nims)): #### Answer generation (operator-managed LLM) { #answer-generation-llm } +> **Note:** Operator-managed answer generation is not included in the 26.05 chart. This branch ships no `answer_llm` values block or NIM template and no `serviceConfig.llm` keys, so `--set nimOperator.answer_llm.enabled=true` has no effect on this chart. The configuration described in this section applies to a later chart release; do not rely on it for a 26.05 deployment. + Enable the generic `answer_llm` NIM slot to add service-mode answer generation on top of the VectorDB query path. The slot defaults to the Super-49B NIM, but the image, model id, service name, resources,