Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/docs/extraction/custom-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ You can upload custom metadata for documents during ingestion.
By uploading custom metadata you can attach additional information to documents,
and use it for filtering results during retrieval operations.
For example, you can add author metadata to your documents, and filter by author when you retrieve results.
To create filters at query time, use predicates supported by [LanceDB SQL](https://lancedb.github.io/lancedb/sql/) against your table schema (custom fields are serialized into the `metadata` column with your ingested chunks). For a worked example, see the repository notebook linked at the end of this page.
To create filters at query time, use predicates supported by [LanceDB SQL](https://docs.lancedb.com/search/sql) against your table schema (custom fields are serialized into the `metadata` column with your ingested chunks). For a worked example, see the repository notebook linked at the end of this page.

Use this documentation to use custom metadata to filter search results when you work with [NeMo Retriever Library](overview.md).

Expand Down Expand Up @@ -177,4 +177,4 @@ print(f"{q_results}")
## Related Content

- For a notebook that uses the CLI to add custom metadata and filter query results, see [metadata_and_filtered_search.ipynb
](https://github.com/NVIDIA/nv-ingest/blob/main/examples/metadata_and_filtered_search.ipynb).
](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/metadata_and_filtered_search.ipynb).
6 changes: 3 additions & 3 deletions docs/docs/extraction/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ This documentation contains the Frequently Asked Questions (FAQ) for [NeMo Retri

You can use the CLI or Python APIs to perform extraction only, and then consume the results.
Using the Python API, `results` is a list object with one entry.
For code examples, see the Jupyter notebooks [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/nv-ingest/blob/main/examples/llama_index_multimodal_rag.ipynb)
and [Multimodal RAG with LangChain](https://github.com/NVIDIA/nv-ingest/blob/main/examples/langchain_multimodal_rag.ipynb).
For code examples, see the Jupyter notebooks [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/llama_index_multimodal_rag.ipynb)
and [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/langchain_multimodal_rag.ipynb).



Expand Down Expand Up @@ -61,7 +61,7 @@ For production environments, you should use the provided Helm charts. For [libra

For advanced scenarios, you might want to use library mode with self-hosted NIM instances.
You can set custom endpoints for each NIM.
For examples of `*_ENDPOINT` variables, refer to [nv-ingest/docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml).
For examples of `*_ENDPOINT` variables, refer to [nv-ingest/docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/docker-compose.yaml).



Expand Down
16 changes: 8 additions & 8 deletions docs/docs/extraction/notebooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,25 @@ To get started using [NeMo Retriever Library](overview.md), you can try one of t

## Dataset Downloads for Benchmarking

If you plan to run benchmarking or evaluation tests, you must download the [Benchmark Datasets (Bo20, Bo767, Bo10k)](https://github.com/NVIDIA/nv-ingest/blob/main/evaluation/digital_corpora_download.ipynb) from Digital Corpora. This is a prerequisite for all benchmarking operations.
If you plan to run benchmarking or evaluation tests, you must download the [Benchmark Datasets (Bo20, Bo767, Bo10k)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/evaluation/digital_corpora_download.ipynb) from Digital Corpora. This is a prerequisite for all benchmarking operations.

## Getting Started

To get started with the basics, try one of the following guides or notebooks:

- [CLI Client Quick Start Guide](quickstart-guide.md#ingest_cli_example) — walkthrough for `nv-ingest-cli` / `retriever` with the self-hosted stack
- [Python Client Quick Start Guide](quickstart-guide.md#ingest_python_example) — walkthrough for the Python client with the self-hosted stack
- [How to add metadata to your documents and filter searches](https://github.com/NVIDIA/nv-ingest/blob/main/examples/metadata_and_filtered_search.ipynb)
- [How to reindex a collection](https://github.com/NVIDIA/nv-ingest/blob/main/examples/reindex_example.ipynb)
- [How to add metadata to your documents and filter searches](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/metadata_and_filtered_search.ipynb)
- [How to reindex a collection](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/reindex_example.ipynb)


For more advanced scenarios, try one of the following notebooks:

- [Build a Custom Vector Database Operator](https://github.com/NVIDIA/nv-ingest/blob/main/examples/building_vdb_operator.ipynb)
- [Try Enterprise RAG Blueprint](https://github.com/NVIDIA/nv-ingest/blob/main/deploy/pdf-blueprint.ipynb)
- [Evaluate bo767 retrieval recall accuracy with NV-Ingest and Milvus](https://github.com/NVIDIA/nv-ingest/blob/main/evaluation/bo767_recall.ipynb)
- [Multimodal RAG with LangChain](https://github.com/NVIDIA/nv-ingest/blob/main/examples/langchain_multimodal_rag.ipynb)
- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/nv-ingest/blob/main/examples/llama_index_multimodal_rag.ipynb)
- [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/building_vdb_operator.ipynb)
- [Try Enterprise RAG Blueprint](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/deploy/pdf-blueprint.ipynb)
- [Evaluate bo767 retrieval recall accuracy with NV-Ingest and Milvus](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/evaluation/bo767_recall.ipynb)
- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/langchain_multimodal_rag.ipynb)
- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/llama_index_multimodal_rag.ipynb)



Expand Down
12 changes: 6 additions & 6 deletions docs/docs/extraction/quickstart-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This guide helps you get started using [NeMo Retriever Library](overview.md) in

## Step 1: Start Containers

Use the provided [docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml) to start all needed services with a few commands.
Use the provided [docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/docker-compose.yaml) to start all needed services with a few commands.

!!! warning

Expand All @@ -16,11 +16,11 @@ If you prefer, you can run on Kubernetes by using [our Helm chart](https://githu

a. Git clone the repo:

`git clone https://github.com/nvidia/nv-ingest`
`git clone -b 26.03 https://github.com/NVIDIA/NeMo-Retriever`

b. Change the directory to the cloned repo by running the following code.

`cd nv-ingest`.
`cd NeMo-Retriever`.

c. [Generate API keys](ngc-api-key.md) and authenticate with NGC with the `docker login` command.

Expand Down Expand Up @@ -58,7 +58,7 @@ f. Start core services. By default, the pipeline uses **LanceDB** as the vector

!!! tip

By default, we have [configured log levels to be verbose](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml). It's possible to observe service startup proceeding. You will notice a lot of log messages. Disable verbose logging by configuring `NIM_TRITON_LOG_VERBOSE=0` for each NIM in [docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml).
By default, we have [configured log levels to be verbose](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/docker-compose.yaml). It's possible to observe service startup proceeding. You will notice a lot of log messages. Disable verbose logging by configuring `NIM_TRITON_LOG_VERBOSE=0` for each NIM in [docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/docker-compose.yaml).

!!! tip

Expand Down Expand Up @@ -431,7 +431,7 @@ python src/util/image_viewer.py --file_path ./processed_docs/image/multimodal_te

!!! tip

Beyond inspecting the results, you can read them into things like [llama-index](https://github.com/NVIDIA/nv-ingest/blob/main/examples/llama_index_multimodal_rag.ipynb) or [langchain](https://github.com/NVIDIA/nv-ingest/blob/main/examples/langchain_multimodal_rag.ipynb) retrieval pipelines. Also, checkout our [Enterprise RAG Blueprint on build.nvidia.com](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) to query over document content pre-extracted with the retriever pipeline.
Beyond inspecting the results, you can read them into things like [llama-index](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/llama_index_multimodal_rag.ipynb) or [langchain](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/examples/langchain_multimodal_rag.ipynb) retrieval pipelines. Also, checkout our [Enterprise RAG Blueprint on build.nvidia.com](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) to query over document content pre-extracted with the retriever pipeline.



Expand Down Expand Up @@ -474,7 +474,7 @@ Ensure the same image tags and `docker-compose.yaml` version are used in both en

## Docker Compose override files

The default [docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml) might exceed VRAM on a single GPU for some hardware. Override files reduce per-service memory, batch sizes, or concurrency so the full pipeline can run on the available GPU. To use an override, pass a second `-f` file after the base compose file; Docker Compose merges them and the override takes precedence.
The default [docker-compose.yaml](https://github.com/NVIDIA/NeMo-Retriever/blob/26.03/docker-compose.yaml) might exceed VRAM on a single GPU for some hardware. Override files reduce per-service memory, batch sizes, or concurrency so the full pipeline can run on the available GPU. To use an override, pass a second `-f` file after the base compose file; Docker Compose merges them and the override takes precedence.

| Override file | GPU target |
|---------------|------------|
Expand Down
Loading