Add semantic deduplication for ranked crawl chunks

[benmaster82]

What

Adds an optional embedding-based semantic deduplication stage to research
chunk selection, layered after the existing lexical Jaccard filter. It removes
syndicated, paraphrased, or reworded chunks that share the same meaning but have
low literal token overlap, which the Jaccard filter cannot catch.

How

• chunk_pool_selection_service: cosine-similarity dedup against already-accepted
  chunks in ranked order, so the highest-ranked member of a duplicate group is
  kept. Cooperates with per-source quotas and the fill pass, and records one
  rejection diagnostic per dropped chunk (chunk_id, reason, similarity,
  matched_chunk_id).
• hybrid_embed_search_service: attaches the dense document embedding already
  computed during ranking to each ranked chunk, so semantic dedup reuses it with
  no extra embedding calls. The embedding is dropped again after selection so it
  never bloats trace files or public payloads.
• config: adds chunk_semantic_dedupe_enabled (default true) and
  chunk_semantic_dedupe_threshold (default 0.92, conservative) with coercion and
  annotated config comments. SearchResult and existing shapes are unchanged.
• research pipeline: wires the new settings through selection.

Design notes

Lexical Jaccard stays as a cheap first pass; semantic dedup ru
already-bounded oversampled pool, comparing each candidate to the running
selected output. A conservative default threshold avoids mergi
that happen to be phrased similarly, and the stage can be disabled entirely.

Acceptance criteria

• Identifies semantically similar chunks despite low Jaccard o
• Retains the highest-ranked duplicate: done
• Works across varied source URLs including syndicated/paraphr
• Cooperates with per-source quotas and fill: done
• Reuses existing embeddings, no redundant computation: done
• Config in JSON settings: done
• Targeted tests for paraphrases, related-but-distinct chunks,
  variation, disabled state, embedding reuse: done
• README settings and design tradeoffs: done

Testing

python -m unittest discover tests -> 293 passed (1 skipped).
Also drove the real research pipeline end-to-end with the ONNX
a paraphrase pair measured cosine 0.57 with Jaccard token overlap 0.18; with
semantic dedup enabled the lower-ranked paraphrase was dropped
both were kept.

[MarcellM01]

Thank you so much for the implementation, sorry for the delay, I will merge this in today 😄 !

[MarcellM01]

Reviewed against issue #12 and current main. The implementation reuses ranked chunk embeddings, preserves ranked-order and quota/fill behavior, adds focused coverage and documentation, merges cleanly, and passes the full local suite (293 tests, 1 skipped). Thank you, Ben!