Part of #553 (group 1 — independent; no relayhistory work required). This is a pure refactor with zero behaviour change; it exists so that the group-2 relayhistory backend is a drop-in.
Current structure
crates/relayburn-sdk/src/ingest/ingest.rs hard-wires three harness passes:
ingest_all (:303-411): stale-stamp cleanup → source_fingerprint(&roots) stat-only fast path (:218-268) → load cursors → ingest_claude_into → ingest_codex_into → ingest_opencode_into → gap warnings → save_cursors_if_changed → write_source_fingerprint (only if !had_skips).
- Each
ingest_*_into walks a root (ingest/walk.rs), detects rotation via (inode, mtime, size) (:626-645, :762-778, :907-926), calls a crate-private parser (parse_claude_session_incremental, parse_codex_session_incremental, parse_opencode_session_incremental), appends turns via Ledger::append_turns, then apply_parsed_extras (:1257-1287) over the DerivedRecords trait (:1153-1173: content(), events(), relationships(), tool_result_events(), user_turns(), request_id_lookup(), turns()).
- Claude additionally accumulates
ClaudeRelationshipEvidence per file and runs reconcile_claude_session_relationships once at the end (:703-726).
IngestRoots (:112-119) is the only injection point for harness roots; default_session_roots() feeds the watch loop (ingest/watch_loop.rs, ingest/fs_events.rs).
ingest_claude_transcript_path (:502-582) is the --hook claude fast path; ingest_claude_session (:480-492) encodes cwd into the project dir name.
lib.rs:45-48 keeps reader, ingest, ledger, analyze private, and the parse entry points are not re-exported — the boundary is already private, which is what makes this refactor safe.
Scope
- Define in
ingest/backend.rs:
pub(crate) trait SessionSourceBackend {
/// Stat-only change probe; `None` means "cannot cheaply tell, run a pass".
fn fingerprint(&self) -> Option<String>;
/// One pass over everything new since `state`; returns parsed batches plus the new state.
fn pass(&mut self, state: &BackendState, opts: &PassOptions, sink: &mut dyn BatchSink) -> Result<PassReport, IngestError>;
/// Single-session fast path used by `--hook claude` and `ingest_claude_session`.
fn hydrate_one(&mut self, locator: &SessionLocator, sink: &mut dyn BatchSink) -> Result<PassReport, IngestError>;
/// Paths the watch loop should subscribe to (empty → polling only).
fn watch_roots(&self) -> Vec<PathBuf>;
}
BatchSink receives (SourceKind, session_id, &dyn DerivedRecords) plus the pending-stamp candidate (PendingStampSessionCandidate, ingest/pending_stamps.rs:88-95) and the gap-warning inputs (ingest/gap.rs record_session_gap args). BackendState wraps today's archive_state.upstream_cursors_json as an opaque string so the built-in backend keeps its FileCursor map unchanged (ingest/cursors.rs:74-82, unknown variants must still round-trip).
- Move the three
ingest_*_into bodies, walk.rs, cursors.rs, source_fingerprint, and the Claude cross-file reconcile into ingest/backend/builtin.rs implementing the trait. ingest_all becomes: cleanup stamps → backend.fingerprint() compare → backend.pass(...) → save state → write fingerprint, with the same had_skips semantics and the same ordering guarantee (cursors before fingerprint, :386-404).
LedgerOpenOptions / RawIngestOptions gain an internal backend: BackendChoice defaulting to Builtin(IngestRoots). Do not add a public API yet; the group-2 issue adds the relayhistory choice.
- Watch loop (
watch_loop.rs:383-417) asks backend.watch_roots() instead of default_session_roots().
ingest_claude_transcript_path / ingest_claude_session route through hydrate_one with a SessionLocator::ClaudeTranscript{path} / SessionLocator::ClaudeSession{cwd, session_id}; keep the post-parse re-stat + EOF cursor write (:563-567) inside the builtin backend.
Acceptance
cargo test --workspace green with no snapshot or golden changes (crates/relayburn-cli/tests/golden.rs, ingest/orchestration_tests.rs, gap_warning_tests.rs, watch_loop_tests.rs, pending_stamps_compat_tests.rs).
- A new test
ingest/backend_tests.rs runs ingest_all with a RecordingBackend mock that yields one hand-built DerivedRecords per harness and asserts the ledger rows, pending-stamp resolution, and gap warning are identical to the builtin path on tests/fixtures/claude/simple-turn.jsonl.
packages/sdk-node/src/index.d.ts unchanged.
AGENTS.md "Adding ingest support" updated to describe the backend seam (one paragraph; the full rewrite happens at cutover).
Out of scope
Depending on relayhistory; changing cursor formats; deleting readers.
Part of #553 (group 1 — independent; no relayhistory work required). This is a pure refactor with zero behaviour change; it exists so that the group-2 relayhistory backend is a drop-in.
Current structure
crates/relayburn-sdk/src/ingest/ingest.rshard-wires three harness passes:ingest_all(:303-411): stale-stamp cleanup →source_fingerprint(&roots)stat-only fast path (:218-268) → load cursors →ingest_claude_into→ingest_codex_into→ingest_opencode_into→ gap warnings →save_cursors_if_changed→write_source_fingerprint(only if!had_skips).ingest_*_intowalks a root (ingest/walk.rs), detects rotation via(inode, mtime, size)(:626-645,:762-778,:907-926), calls a crate-private parser (parse_claude_session_incremental,parse_codex_session_incremental,parse_opencode_session_incremental), appends turns viaLedger::append_turns, thenapply_parsed_extras(:1257-1287) over theDerivedRecordstrait (:1153-1173:content(),events(),relationships(),tool_result_events(),user_turns(),request_id_lookup(),turns()).ClaudeRelationshipEvidenceper file and runsreconcile_claude_session_relationshipsonce at the end (:703-726).IngestRoots(:112-119) is the only injection point for harness roots;default_session_roots()feeds the watch loop (ingest/watch_loop.rs,ingest/fs_events.rs).ingest_claude_transcript_path(:502-582) is the--hook claudefast path;ingest_claude_session(:480-492) encodescwdinto the project dir name.lib.rs:45-48keepsreader,ingest,ledger,analyzeprivate, and the parse entry points are not re-exported — the boundary is already private, which is what makes this refactor safe.Scope
ingest/backend.rs:BatchSinkreceives(SourceKind, session_id, &dyn DerivedRecords)plus the pending-stamp candidate (PendingStampSessionCandidate,ingest/pending_stamps.rs:88-95) and the gap-warning inputs (ingest/gap.rsrecord_session_gapargs).BackendStatewraps today'sarchive_state.upstream_cursors_jsonas an opaque string so the built-in backend keeps itsFileCursormap unchanged (ingest/cursors.rs:74-82, unknown variants must still round-trip).ingest_*_intobodies,walk.rs,cursors.rs,source_fingerprint, and the Claude cross-file reconcile intoingest/backend/builtin.rsimplementing the trait.ingest_allbecomes: cleanup stamps →backend.fingerprint()compare →backend.pass(...)→ save state → write fingerprint, with the samehad_skipssemantics and the same ordering guarantee (cursors before fingerprint,:386-404).LedgerOpenOptions/RawIngestOptionsgain an internalbackend: BackendChoicedefaulting toBuiltin(IngestRoots). Do not add a public API yet; the group-2 issue adds the relayhistory choice.watch_loop.rs:383-417) asksbackend.watch_roots()instead ofdefault_session_roots().ingest_claude_transcript_path/ingest_claude_sessionroute throughhydrate_onewith aSessionLocator::ClaudeTranscript{path}/SessionLocator::ClaudeSession{cwd, session_id}; keep the post-parse re-stat + EOF cursor write (:563-567) inside the builtin backend.Acceptance
cargo test --workspacegreen with no snapshot or golden changes (crates/relayburn-cli/tests/golden.rs,ingest/orchestration_tests.rs,gap_warning_tests.rs,watch_loop_tests.rs,pending_stamps_compat_tests.rs).ingest/backend_tests.rsrunsingest_allwith aRecordingBackendmock that yields one hand-builtDerivedRecordsper harness and asserts the ledger rows, pending-stamp resolution, and gap warning are identical to the builtin path ontests/fixtures/claude/simple-turn.jsonl.packages/sdk-node/src/index.d.tsunchanged.AGENTS.md"Adding ingest support" updated to describe the backend seam (one paragraph; the full rewrite happens at cutover).Out of scope
Depending on relayhistory; changing cursor formats; deleting readers.