data: add RAM-bounded sequential reference dedup pipeline

[AuraIis]

Why

A measured 1M-document combined-index sample projected 44.1 GB LSH RSS for all 30.44M reference documents, before the additional exact-hash set, on a host with only 49 GB available. The combined index is therefore unsafe.

Pipeline

dedup_de_sequential.py runs the existing hardened dedup implementation once per reference:

fresh → ref1 → survivors → ref2 → ... → final

• peak RAM is bounded by the largest individual reference
• full ordered SHA-256 manifest remains mandatory
• each stage retains its own report and reference attribution
• final report aggregates exact, near, invalid and retained counts
• reference order is recorded
• exact/near classification is explicitly first-match attribution
• intermediate JSONL files are deleted after successful completion unless requested
• in-place input/output is rejected

The retained set is the union-of-reference rejection result; only exact/near attribution can depend on reference order.

Tests cover two-reference aggregation, stage counts, final retention, intermediate cleanup and unsafe in-place output.