Remove retired retriever pipeline CLI

[jioffe502]

Summary

Remove the hidden retriever pipeline compatibility command now that supported CLI workflows are owned by retriever ingest and retriever query.

This PR:

• unregisters and deletes the legacy pipeline CLI and its examples.graph_pipeline compatibility shim;
• removes helpers, examples, and documentation that existed only for that adapter;
• removes tests that exclusively exercised the retired command;
• preserves the active ServiceIngestor coverage from the mixed pipeline-helper test file in test_ingest_service.py;
• replaces the retired BO767 notebook with a harness-based evaluation guide;
• consolidates the surviving LanceDB row-count and recall-summary helpers with their remaining domain code;
• updates ingestion and evaluation documentation to compose the supported ingest, query, harness, and standalone evaluation commands.

Scope boundary

This PR does not change workflow YAML, Dockerfiles or Docker scripts, Hugging Face token handling, tokenizer pre-download behavior, documentation build CI, Ray configuration snapshots, or any of the other cleanup candidates from the earlier version of this PR. Those can be evaluated independently in later PRs.

Compatibility boundary

This is a deliberate removal of the unsupported compatibility adapter, not the underlying Retriever implementation.

• retriever ingest and retriever query remain the supported CLI workflows.
• GraphIngestor, ServiceIngestor, graph execution, LanceDB upload/query, BEIR, recall, QA evaluation, and the benchmark harness remain.
• Pipeline-only convenience flags such as combined ingest-and-evaluate and intermediate-Parquet output are not carried forward as a compatibility promise. Standalone evaluation commands remain available.

Tests

• test_graph_pipeline_cli.py is removed because it exclusively tested the deleted CLI.
• Pipeline-adapter tests are removed from test_pipeline_helpers.py and test_beir_evaluation.py.
• Three active ServiceIngestor tests are retained in test_ingest_service.py.
• Root CLI tests now assert that pipeline is not callable, while ingest and query remain available.

Validation

• Full Retriever suite: 2,481 passed, 110 skipped, 10 deselected; 26 subtests passed.
• Focused recall, BEIR, and documentation tests: 93 passed, 2 skipped.
• uvx pre-commit run --all-files: all hooks passed.
• Direct CLI boundary: retriever ingest --help and retriever query --help succeed; retriever pipeline --help is rejected as an unknown command.

Reviewer decision requested

Please confirm that the hidden retriever pipeline compatibility command and its adapter-only support code are not compatibility surfaces we intend to maintain.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Summary

This PR removes the hidden retriever pipeline compatibility CLI and all adapter-only support code, consolidating surviving helpers into their canonical homes and migrating active ServiceIngestor tests to a new test_ingest_service.py.

• Deletes ~4,200 lines of pipeline CLI, examples, and pipeline-only helpers (cli/pipeline/__main__.py, detection_summary.py, lancedb_read.py, parquet_to_lancedb.py, examples/graph_pipeline.py); no surviving code references the deleted modules.
• Moves print_run_summary/_fmt_time/_evaluation_metric_sort_key into tools/recall/core.py and lancedb_row_count into common/vdb/lancedb.py; all callers (beir_eval.py, recall_eval.py, portal/app.py) are updated to use the new import paths.
• Adds pipeline to the parametrized "must-be-rejected" CLI test list and removes the now-stale "pipeline is callable" assertion in test_root_cli_workflow.py.

Confidence Score: 5/5

Safe to merge — this is a deletion-and-consolidation PR with no functional logic changes to surviving code paths.

All imports of the deleted modules have been fully cleaned up with no remaining references. The consolidated helpers are byte-for-byte copies of the originals, and all callers are updated. The only finding is a missing SPDX license header in the new test_recall_summary.py.

No files require special attention; the only nit is the missing SPDX header in nemo_retriever/tests/test_recall_summary.py.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/cli/main.py	Removes the pipeline entry from the lazy subapp registry; one-line clean deletion.
nemo_retriever/src/nemo_retriever/tools/recall/core.py	Consolidates print_run_summary, _fmt_time, and _evaluation_metric_sort_key from the deleted detection_summary.py into this module; logic is identical to the original.
nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py	Adds lancedb_row_count (moved from the deleted lancedb_read.py) and updates a docstring; all call sites are updated accordingly.
nemo_retriever/tests/test_ingest_service.py	New test file preserving the three ServiceIngestor tests migrated from the deleted test_pipeline_helpers.py; SPDX header present and tests are well-structured.
nemo_retriever/tests/test_recall_summary.py	New test file (renamed from test_detection_summary.py) covering metric sort order in print_run_summary; missing required SPDX license header.
nemo_retriever/tests/test_root_cli_workflow.py	Removes the pipeline --help callable assertion; adds pipeline to the parametrized list of commands that must be rejected as unknown.
nemo_retriever/src/nemo_retriever/harness/portal/app.py	Updates lancedb_row_count import path from deleted lancedb_read to the new location in lancedb; no logic change.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    CLI["retriever CLI\n(cli/main.py)"] --> INGEST["retriever ingest"]
    CLI --> QUERY["retriever query"]
    CLI --> RECALL["retriever recall"]
    CLI --> EVAL["retriever eval"]
    CLI -. removed .-> PIPELINE["retriever pipeline\n❌ DELETED"]

    INGEST --> SI["ServiceIngestor\n(ingest/service.py)"]
    INGEST --> GI["GraphIngestor\n(ingestor/graph_ingestor.py)"]

    SI --> VDB["IngestVdbOperator\n(operators/vdb.py)"]
    GI --> VDB

    VDB --> RECORDS["to_client_vdb_records\n(common/vdb/records.py)"]

    RECALL --> CORE["print_run_summary\n(tools/recall/core.py)\n← moved from detection_summary.py"]
    EVAL --> CORE

    PORTAL["harness/portal/app.py"] --> LANCEDB_FN["lancedb_row_count\n(common/vdb/lancedb.py)\n← moved from lancedb_read.py"]

    style PIPELINE fill:#ffcccc,stroke:#cc0000
    style CORE fill:#ccffcc,stroke:#009900
    style LANCEDB_FN fill:#ccffcc,stroke:#009900

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    CLI["retriever CLI\n(cli/main.py)"] --> INGEST["retriever ingest"]
    CLI --> QUERY["retriever query"]
    CLI --> RECALL["retriever recall"]
    CLI --> EVAL["retriever eval"]
    CLI -. removed .-> PIPELINE["retriever pipeline\n❌ DELETED"]

    INGEST --> SI["ServiceIngestor\n(ingest/service.py)"]
    INGEST --> GI["GraphIngestor\n(ingestor/graph_ingestor.py)"]

    SI --> VDB["IngestVdbOperator\n(operators/vdb.py)"]
    GI --> VDB

    VDB --> RECORDS["to_client_vdb_records\n(common/vdb/records.py)"]

    RECALL --> CORE["print_run_summary\n(tools/recall/core.py)\n← moved from detection_summary.py"]
    EVAL --> CORE

    PORTAL["harness/portal/app.py"] --> LANCEDB_FN["lancedb_row_count\n(common/vdb/lancedb.py)\n← moved from lancedb_read.py"]

    style PIPELINE fill:#ffcccc,stroke:#cc0000
    style CORE fill:#ccffcc,stroke:#009900
    style LANCEDB_FN fill:#ccffcc,stroke:#009900

Loading

_{Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'upstream/m..." | Re-trigger Greptile}

[greptile-apps]

Direct IoU unit tests removed without replacement

The TestCalculateIou class (testing identical boxes → 1.0, no overlap → 0.0, zero-area boxes → 0.0, and partial overlap ratio) was deleted alongside deduplicate.py. The inlined _calculate_iou in dedup.py is now exercised only through the higher-level dedup_images call in test_drops_partial_overlap_above_threshold. Boundary conditions like identical boxes, non-overlapping boxes, and zero-area degenerate cases are no longer explicitly covered — a regression in the IoU formula would not be caught at the unit level.

Prompt To Fix With AI

This is a comment left during a code review.
Path: nemo_retriever/tests/test_dedup.py
Line: 20

Comment:
**Direct IoU unit tests removed without replacement**

The `TestCalculateIou` class (testing identical boxes → 1.0, no overlap → 0.0, zero-area boxes → 0.0, and partial overlap ratio) was deleted alongside `deduplicate.py`. The inlined `_calculate_iou` in `dedup.py` is now exercised only through the higher-level `dedup_images` call in `test_drops_partial_overlap_above_threshold`. Boundary conditions like identical boxes, non-overlapping boxes, and zero-area degenerate cases are no longer explicitly covered — a regression in the IoU formula would not be caught at the unit level.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[jperez999]

This change still supports the case when no metadata information about the table type or the model exists correct? It still works as expected in that case?

[jperez999]

If we are deleting this file, we need to add a new file that explains how to run this evaluation with the harness.

[jperez999]

am I reading this right? this file only has this method? if so, is it possible to move this to one of the other files in this common/vdb/ sub folder that still exist?

[jperez999]

Same as above, there are other metric logic functions, we should group them altogether, or is there a reason these should stay here?