Skip to content

[G1] Introduce a SessionSourceBackend seam in ingest so the built-in readers can be swapped for relayhistory #554

Description

@willwashburn

Part of #553 (group 1 — independent; no relayhistory work required). This is a pure refactor with zero behaviour change; it exists so that the group-2 relayhistory backend is a drop-in.

Current structure

crates/relayburn-sdk/src/ingest/ingest.rs hard-wires three harness passes:

  • ingest_all (:303-411): stale-stamp cleanup → source_fingerprint(&roots) stat-only fast path (:218-268) → load cursors → ingest_claude_intoingest_codex_intoingest_opencode_into → gap warnings → save_cursors_if_changedwrite_source_fingerprint (only if !had_skips).
  • Each ingest_*_into walks a root (ingest/walk.rs), detects rotation via (inode, mtime, size) (:626-645, :762-778, :907-926), calls a crate-private parser (parse_claude_session_incremental, parse_codex_session_incremental, parse_opencode_session_incremental), appends turns via Ledger::append_turns, then apply_parsed_extras (:1257-1287) over the DerivedRecords trait (:1153-1173: content(), events(), relationships(), tool_result_events(), user_turns(), request_id_lookup(), turns()).
  • Claude additionally accumulates ClaudeRelationshipEvidence per file and runs reconcile_claude_session_relationships once at the end (:703-726).
  • IngestRoots (:112-119) is the only injection point for harness roots; default_session_roots() feeds the watch loop (ingest/watch_loop.rs, ingest/fs_events.rs).
  • ingest_claude_transcript_path (:502-582) is the --hook claude fast path; ingest_claude_session (:480-492) encodes cwd into the project dir name.

lib.rs:45-48 keeps reader, ingest, ledger, analyze private, and the parse entry points are not re-exported — the boundary is already private, which is what makes this refactor safe.

Scope

  1. Define in ingest/backend.rs:
    pub(crate) trait SessionSourceBackend {
        /// Stat-only change probe; `None` means "cannot cheaply tell, run a pass".
        fn fingerprint(&self) -> Option<String>;
        /// One pass over everything new since `state`; returns parsed batches plus the new state.
        fn pass(&mut self, state: &BackendState, opts: &PassOptions, sink: &mut dyn BatchSink) -> Result<PassReport, IngestError>;
        /// Single-session fast path used by `--hook claude` and `ingest_claude_session`.
        fn hydrate_one(&mut self, locator: &SessionLocator, sink: &mut dyn BatchSink) -> Result<PassReport, IngestError>;
        /// Paths the watch loop should subscribe to (empty → polling only).
        fn watch_roots(&self) -> Vec<PathBuf>;
    }
    BatchSink receives (SourceKind, session_id, &dyn DerivedRecords) plus the pending-stamp candidate (PendingStampSessionCandidate, ingest/pending_stamps.rs:88-95) and the gap-warning inputs (ingest/gap.rs record_session_gap args). BackendState wraps today's archive_state.upstream_cursors_json as an opaque string so the built-in backend keeps its FileCursor map unchanged (ingest/cursors.rs:74-82, unknown variants must still round-trip).
  2. Move the three ingest_*_into bodies, walk.rs, cursors.rs, source_fingerprint, and the Claude cross-file reconcile into ingest/backend/builtin.rs implementing the trait. ingest_all becomes: cleanup stamps → backend.fingerprint() compare → backend.pass(...) → save state → write fingerprint, with the same had_skips semantics and the same ordering guarantee (cursors before fingerprint, :386-404).
  3. LedgerOpenOptions / RawIngestOptions gain an internal backend: BackendChoice defaulting to Builtin(IngestRoots). Do not add a public API yet; the group-2 issue adds the relayhistory choice.
  4. Watch loop (watch_loop.rs:383-417) asks backend.watch_roots() instead of default_session_roots().
  5. ingest_claude_transcript_path / ingest_claude_session route through hydrate_one with a SessionLocator::ClaudeTranscript{path} / SessionLocator::ClaudeSession{cwd, session_id}; keep the post-parse re-stat + EOF cursor write (:563-567) inside the builtin backend.

Acceptance

  • cargo test --workspace green with no snapshot or golden changes (crates/relayburn-cli/tests/golden.rs, ingest/orchestration_tests.rs, gap_warning_tests.rs, watch_loop_tests.rs, pending_stamps_compat_tests.rs).
  • A new test ingest/backend_tests.rs runs ingest_all with a RecordingBackend mock that yields one hand-built DerivedRecords per harness and asserts the ledger rows, pending-stamp resolution, and gap warning are identical to the builtin path on tests/fixtures/claude/simple-turn.jsonl.
  • packages/sdk-node/src/index.d.ts unchanged.
  • AGENTS.md "Adding ingest support" updated to describe the backend seam (one paragraph; the full rewrite happens at cutover).

Out of scope

Depending on relayhistory; changing cursor formats; deleting readers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions