Support local in-process vLLM LLMs for agentic retrieval

docs/docs/extraction/prerequisites-support-matrix.md

@@ -106,7 +106,7 @@ When you call [NVIDIA-hosted NIMs](deployment-options.md#when-to-use-nvidia-host
 | llama-nemotron-rerank-vl-1b-v2 | `https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-nemotron-rerank-vl-1b-v2/reranking` | Optional VL reranker |
 | nemotron-parse | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/nemotron-parse` | Optional `extract_method="nemotron_parse"` |
 | nemotron-3-nano-omni-30b-a3b-reasoning | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | Optional image captioning |
-| llama-3.3-nemotron-super-49b-v1.5 | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Optional `/v1/answer` (Helm `answer_llm`) and agentic RAG; not part of the default extraction pipeline. Helm auto-wires to the in-cluster NIM when `nimOperator.answer_llm` is enabled |
+| llama-3.3-nemotron-super-49b-v1.5 | `https://integrate.api.nvidia.com/v1/chat/completions` with model ID `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Optional `/v1/answer` (Helm `answer_llm`) and OpenAI-compatible agentic RAG endpoint mode; not part of the default extraction pipeline. Agentic query/harness runs default to local in-process vLLM instead. Helm auto-wires to the in-cluster NIM when `nimOperator.answer_llm` is enabled |
 | parakeet-1-1b-ctc-en-us | `grpc.nvcf.nvidia.com:443` (function ID from [build.nvidia.com](https://build.nvidia.com/)) | Optional ASR; refer to [Parakeet hosted inference](audio-video.md#parakeet-hosted-inference-build-nvidia) |
 
 For local Hugging Face OCR language mode (`multi` vs `english`), Helm OCR image overrides, and local model install, refer to [OCR and scanned documents](multimodal-extraction.md#ocr-and-scanned-documents), [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration), and [CLI — OCR language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#ocr-language-mode).

docs/docs/extraction/workflow-agentic-retrieval.md

@@ -4,6 +4,30 @@
 
 NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks.
 
+
+## Local agentic retrieval
+
+The `retriever query --agentic` and harness BEIR agentic paths default to an
+in-process local vLLM agent LLM. If no agent model is provided, the library loads
+`nemotron-8b` (`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`) on the local CUDA host.
+The larger `super-49b` profile is also supported. Other custom in-process LLMs
+are not supported yet because the agent loop depends on OpenAI-style tool-call
+messages; use an OpenAI-compatible endpoint for custom models.
+
+```bash
+retriever query "find documents about parser behavior" --agentic
+```
+
+For custom or already deployed chat models, opt into the endpoint path:
+
+```bash
+retriever query "find documents about parser behavior" \
+  --agentic \
+  --agentic-llm-backend openai_compatible \
+  --agentic-llm-model custom-remote-model \
+  --agentic-invoke-url http://localhost:9000/v1/chat/completions
+```
+
 ## MCP access for agents
 
 `retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API.

nemo_retriever/README.md

@@ -314,31 +314,29 @@ Agentic retrieval runs an LLM-driven ReAct loop over an existing LanceDB index.
 It does not ingest documents. Build the index with one of the ingestion flows
 above, then query the same `lancedb_uri`, `table_name`, and embedding model.
 
-The agent LLM supports hosted NVIDIA inference or a local OpenAI-compatible
-chat-completions endpoint. Pass `--agentic-invoke-url` for a local vLLM or
-self-hosted NIM server; when omitted, the agent uses the built-in NVIDIA hosted
-endpoint. Embedding follows the same local/remote split as dense retrieval:
-CPU-only hosts default to hosted endpoints; GPU-capable hosts use local GPU
-embedding unless you pass `--embed-invoke-url`, for example
-`--embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings`.
-
-**Hosted inference.** For [build.nvidia.com](https://build.nvidia.com/) hosted
-inference, set `NVIDIA_API_KEY`. On CPU-only machines, the CPU embedding actor
-and agent LLM use the hosted NVIDIA endpoints by default:
+By default, the agent LLM runs in process with local vLLM and `nemotron-8b`
+(`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`). This requires a CUDA GPU host and the
+`[local]` extra. GPU placement follows process-level vLLM behavior, so set
+`CUDA_VISIBLE_DEVICES` before starting the command. Embedding follows the same
+local/remote split as dense retrieval; pass `--embed-invoke-url` to use a remote
+embedding endpoint.
 
-```bash
-export NVIDIA_API_KEY=nvapi-...
+**Local in-process vLLM agent LLM.** Omit `--agentic-invoke-url` to load the
+supported local agent LLM directly in the Python process. `nemotron-8b` is the
+default; `super-49b` is also supported when the process has enough visible GPUs.
 
-retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
+```bash
+CUDA_VISIBLE_DEVICES=0 retriever query "Given their activities, which animal is responsible for the typos in my documents?" \
   --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
   --lancedb-uri lancedb \
   --table-name nemo-retriever \
   --embed-model-name nvidia/llama-nemotron-embed-1b-v2
 ```
 
-**Local agent LLM.** Point `--agentic-invoke-url` at your OpenAI-compatible
-chat-completions server (for example vLLM or a self-hosted NIM):
+**OpenAI-compatible agent endpoint.** Pass `--agentic-invoke-url` when you want a
+custom model or a separately hosted chat-completions server, such as vLLM server
+mode or a self-hosted NIM. When an invoke URL is provided, `--agentic-llm-model`
+is required and is sent as the remote model ID.
 
 ```bash
 retriever query "What is RAG?" \
@@ -350,18 +348,14 @@ retriever query "What is RAG?" \
   --embed-model-name nvidia/llama-nemotron-embed-1b-v2
 ```
 
-Native local Hugging Face LLM inference for the agent is in active development;
-this README will be updated when that path ships.
-
 Unlike dense retrieval, agentic mode returns ranked document IDs as JSON, not
 text-enriched hits.
 
 For a quick smoke test, reduce agent work:
 
 ```bash
-retriever query "What is RAG?" \
+CUDA_VISIBLE_DEVICES=0 retriever query "What is RAG?" \
   --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
   --lancedb-uri lancedb \
   --table-name nemo-retriever \
   --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
@@ -370,9 +364,9 @@ retriever query "What is RAG?" \
   --agentic-backend-top-k 1
 ```
 
-You can run the same flow from Python. Set `NVIDIA_API_KEY` for hosted inference,
-or pass `invoke_url` on `QueryAgenticOptions` for a local chat-completions
-endpoint.
+You can run the same flow from Python. Omit `invoke_url` for the default local
+in-process vLLM backend, or pass `invoke_url` on `QueryAgenticOptions` for a
+separate OpenAI-compatible chat-completions endpoint.
 
 ```python
 from nemo_retriever.cli.query_workflow import agentic_query_documents
@@ -384,8 +378,8 @@ from nemo_retriever.query.options import (
     QueryStorageOptions,
 )
 
-# Hosted: set NVIDIA_API_KEY=nvapi-... in the environment.
-# Local agent LLM: pass invoke_url="http://localhost:9000/v1/chat/completions".
+# Local in-process vLLM: set CUDA_VISIBLE_DEVICES before starting Python.
+# Remote agent LLM: pass invoke_url="http://localhost:9000/v1/chat/completions".
 results = agentic_query_documents(
     QueryRequest(
         query="What is RAG?",
@@ -399,7 +393,10 @@ results = agentic_query_documents(
         ),
         agentic=QueryAgenticOptions(
             enabled=True,
-            llm_model="nvidia/llama-3.3-nemotron-super-49b-v1.5",
+            llm_model="nemotron-8b",
+            local_llm_backend="vllm",
+            local_gpu_memory_utilization=0.8,
+            local_tensor_parallel_size=1,
         ),
     )
 )

nemo_retriever/docs/cli/README.md

@@ -174,18 +174,22 @@ output are not used for content-type matching.
 `--agentic` swaps the single dense pass for an LLM-driven ReAct loop: the agent
 issues several retrieval sub-queries, fuses the candidates, and selects a final
 ranking. It searches the same LanceDB table built by `retriever ingest`, so it is
-a drop-in alternative to standard retrieval — add `--agentic` and name the chat
-model the agent drives with `--agentic-llm-model` (required):
+a drop-in alternative to standard retrieval.
+
+By default, agentic retrieval runs the agent LLM in process with local vLLM and
+`nemotron-8b` (`nvidia/Llama-3.1-Nemotron-Nano-8B-v1`). This requires a CUDA GPU
+host and the local extras installed. Provide `--agentic-invoke-url` when you want
+a custom model or a separately hosted OpenAI-compatible endpoint.
 
 ```bash
+# default local vLLM agent LLM: nemotron-8b
 retriever query "how does the ingestion pipeline handle tables?" \
-  --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5
+  --agentic
 
-# remote agent + embedding endpoints, fewer reasoning rounds
+# custom/self-hosted model through an OpenAI-compatible endpoint
 retriever query "summarize the deployment options" \
   --agentic \
-  --agentic-llm-model nvidia/llama-3.3-nemotron-super-49b-v1.5 \
+  --agentic-llm-model custom-remote-model \
   --agentic-invoke-url http://localhost:9000/v1/chat/completions \
   --embed-invoke-url http://localhost:8000/v1 \
   --agentic-react-max-steps 5
@@ -197,8 +201,8 @@ produced it (`final_results`, `rrf`, or `selection_agent`). It reuses the same
 `--top-k`, `--lancedb-uri`, `--table-name`, `--embed-invoke-url`, and
 `--embed-model-name` options as standard retrieval.
 
-**How it works.** Each agentic query runs `Query → ReActAgentOperator → (RRF
-fusion) → SelectionAgentOperator → ranked results`:
+**How it works.** Each agentic query runs `Query -> ReActAgentOperator -> (RRF
+fusion) -> SelectionAgentOperator -> ranked results`:
 
 - `ReActAgentOperator` runs the per-query ReAct loop; every `retrieve` tool call
   delegates to the standard `Retriever`, so the agent searches the same vector
@@ -210,17 +214,21 @@ fusion) → SelectionAgentOperator → ranked results`:
 
 Agentic-only knobs (apply only with `--agentic`):
 
+- `--agentic-llm-model` — local profile alias/model ID when no invoke URL is
+  provided (`nemotron-8b` by default; `super-49b` also supported), or the remote
+  model ID when `--agentic-invoke-url` is provided.
 - `--agentic-invoke-url` — OpenAI-compatible chat-completions endpoint for the
-  agent LLM; defaults to the operators' built-in endpoint when omitted.
+  agent LLM. Providing it routes agent LLM calls to that remote endpoint.
 - `--agentic-reasoning-effort` (default `high`) — `reasoning_effort` forwarded on
-  agentic LLM calls.
+  OpenAI-compatible agentic LLM calls; ignored by the local adapter.
+- `--agentic-temperature` (default `0.0`) — sampling temperature for agent LLM
+  calls. Local and non-NVIDIA OpenAI-compatible endpoints allow up to `2.0`;
+  NVIDIA-hosted endpoints allow up to `1.0`.
 - `--agentic-backend-top-k` (default `20`) — candidates pulled from the vector DB
   per retrieval call.
 - `--agentic-react-max-steps` (default `50`) — maximum ReAct loop iterations.
 - `--agentic-text-truncation` (default `0`) — max characters of each candidate
   shown to the agent; `0` disables truncation.
-- `--agentic-temperature` (default `0.0`) — sampling temperature for agentic LLM
-  calls (`0.0` = greedy).
 
 <!-- --8<-- [end:quickstart] -->
 

nemo_retriever/docs/cli/benchmarking.md

@@ -57,20 +57,48 @@ code-owned benchmark or runfile, or use repeatable `--set` overrides on the CLI.
 The agentic harness path runs the same ReAct retrieval graph used by root query,
 but only after ingest and only for BEIR evaluation (`evaluation.mode: beir`).
 
+By default, agentic harness evaluation uses the in-process local vLLM backend
+with `nemotron-8b`. Custom LLMs are not supported in process yet; run them behind
+an OpenAI-compatible chat-completions endpoint and set `query.agentic_invoke_url`.
+
 Minimal BEIR override example:
 
+```bash
+retriever harness run jp20_beir \
+  --set query.agentic=true
+```
+
+Larger supported local profile:
+
+```bash
+retriever harness run jp20_beir \
+  --set query.agentic=true \
+  --set query.agentic_llm_model=super-49b \
+  --set query.agentic_local_tensor_parallel_size=2
+```
+
+Custom/self-hosted OpenAI-compatible endpoint:
+
 ```bash
 retriever harness run jp20_beir \
   --set query.agentic=true \
-  --set query.agentic_llm_model=nvidia/llama-3.3-nemotron-super-49b-v1.5
+  --set query.agentic_llm_model=custom-remote-model \
+  --set query.agentic_invoke_url=http://localhost:9000/v1/chat/completions
 ```
 
 Useful agentic query overrides:
 
-- `query.agentic_llm_model` — chat model used by the ReAct and selection agents;
-  required when `query.agentic=true`.
-- `query.agentic_invoke_url` — OpenAI-compatible chat-completions endpoint. Omit
-  to use the built-in NVIDIA endpoint.
+- `query.agentic_llm_model` — local profile alias/model ID when no invoke URL is
+  provided (`nemotron-8b` by default; `super-49b` also supported), or the remote
+  model ID when `query.agentic_invoke_url` is provided.
+- `query.agentic_invoke_url` — OpenAI-compatible chat-completions endpoint.
+  Providing it routes agent LLM calls to that remote endpoint.
+- `query.agentic_local_gpu_memory_utilization`,
+  `query.agentic_local_tensor_parallel_size`, `query.agentic_local_max_model_len`,
+  and `query.agentic_local_max_num_seqs` — harness-only local vLLM resource and
+  scheduling controls for benchmark runs. Use environment variables such as
+  `CUDA_VISIBLE_DEVICES` and the standard Hugging Face cache environment for
+  placement and model cache control.
 - `query.agentic_backend_top_k` — backend candidate pool per ReAct retrieval
   call. Must be at least the final requested metric depth (`max(evaluation.ks)`).
 - `query.agentic_react_max_steps` — maximum ReAct loop iterations per query
@@ -79,9 +107,9 @@ Useful agentic query overrides:
   the agent; `0` disables truncation.
 - `query.agentic_num_concurrent` — number of queries the agent batch runs
   concurrently (defaults to `1`).
-- `query.agentic_temperature` — defaults to `0.0`; hosted/default NVIDIA
-  endpoints are validated as `0.0..1.0`, while other OpenAI-compatible endpoints
-  allow `0.0..2.0`.
+- `query.agentic_temperature` — defaults to `0.0`; local and non-NVIDIA
+  OpenAI-compatible endpoints allow `0.0..2.0`, while hosted/default NVIDIA
+  endpoints are validated as `0.0..1.0`.
 - `query.agentic_reasoning_effort` — optional provider-specific field forwarded
   only when configured.
 

nemo_retriever/src/nemo_retriever/cli/query/app.py

@@ -16,10 +16,7 @@
 from nemo_retriever.cli.query import options as opts
 from nemo_retriever.cli.query_workflow import agentic_query_documents as query_agentic_documents
 from nemo_retriever.cli.query_workflow import query_documents_with_metadata as query_local_documents_with_metadata
-from nemo_retriever.query.agentic_options import (
-    agentic_backend_top_k_error,
-    agentic_temperature_error,
-)
+from nemo_retriever.query.agentic_options import agentic_backend_top_k_error, agentic_temperature_error
 from nemo_retriever.cli.shared import (
     ROOT_CLI_ERRORS,
     quiet_capture,
@@ -201,16 +198,22 @@ def _local_command(
         embed_invoke_url = os.environ.get("EMBED_INVOKE_URL") or None
     rerank = rerank or bool(reranker_invoke_url) or bool(reranker_model_name) or bool(reranker_backend)
     silence_noisy_libraries()
-    if agentic and not agentic_llm_model:
-        typer.echo("Error: --agentic requires --agentic-llm-model.", err=True)
-        raise typer.Exit(1)
-
     if agentic:
+        if agentic_invoke_url and not agentic_llm_model:
+            typer.echo(
+                "Error: --agentic-invoke-url requires --agentic-llm-model.",
+                err=True,
+            )
+            raise typer.Exit(1)
+        if not agentic_invoke_url and not agentic_llm_model:
+            agentic_llm_model = "nemotron-8b"
+
         backend_error = agentic_backend_top_k_error(agentic_backend_top_k, target_top_k=top_k)
         if backend_error:
             typer.echo(f"Error: {backend_error}", err=True)
             raise typer.Exit(1)
-        temperature_error = agentic_temperature_error(agentic_temperature, invoke_url=agentic_invoke_url)
+        temperature_invoke_url = agentic_invoke_url or "local://in-process"
+        temperature_error = agentic_temperature_error(agentic_temperature, invoke_url=temperature_invoke_url)
         if temperature_error:
             typer.echo(f"Error: {temperature_error}", err=True)
             raise typer.Exit(1)

nemo_retriever/src/nemo_retriever/cli/query/options.py

@@ -159,7 +159,10 @@
     str | None,
     typer.Option(
         "--agentic-llm-model",
-        help="Chat model the agent drives. Required when --agentic is set.",
+        help=(
+            "Chat model the agent drives. Defaults to nemotron-8b for local in-process runs; "
+            "required when --agentic-invoke-url is provided."
+        ),
     ),
 ]
 AgenticInvokeUrlOption = Annotated[

nemo_retriever/src/nemo_retriever/harness/resolution.py

@@ -129,6 +129,10 @@ def _override_child_keys(prefix: str, paths: set[str]) -> set[str]:
     "query.agentic",
     "query.agentic_llm_model",
     "query.agentic_invoke_url",
+    "query.agentic_local_gpu_memory_utilization",
+    "query.agentic_local_tensor_parallel_size",
+    "query.agentic_local_max_model_len",
+    "query.agentic_local_max_num_seqs",
     "query.agentic_reasoning_effort",
     "query.agentic_backend_top_k",
     "query.agentic_react_max_steps",
@@ -438,6 +442,10 @@ def build_query_request(resolved: dict[str, Any], query_text: str) -> QueryReque
             enabled=bool(query.get("agentic", False)),
             llm_model=query.get("agentic_llm_model"),
             invoke_url=query.get("agentic_invoke_url"),
+            local_gpu_memory_utilization=float(query.get("agentic_local_gpu_memory_utilization") or 0.8),
+            local_tensor_parallel_size=int(query.get("agentic_local_tensor_parallel_size") or 1),
+            local_max_model_len=query.get("agentic_local_max_model_len"),
+            local_max_num_seqs=query.get("agentic_local_max_num_seqs"),
             reasoning_effort=query.get("agentic_reasoning_effort"),
             backend_top_k=int(query.get("agentic_backend_top_k") or 20),
             react_max_steps=int(query.get("agentic_react_max_steps") or 50),