Generate research archive manifests for segment batches

[erskingardner]

Problem

A directory of segment files is not enough metadata for a reproducible research dataset. Researchers need a manifest describing what files are present, their hashes, sizes, event counts, time ranges, and the software version used to produce them.

Why this matters

Manifests make archive releases auditable, resumable, and citable. They also let researchers verify downloads before processing terabytes of data.

Suggested implementation

Add a manifest generator, possibly as a mode of segment-inspect or a separate segment-manifest CLI.

The manifest should be machine-readable, probably JSONL with one row per segment.

Candidate fields:

• path or object key
• segment_number
• sha256
• compressed_bytes
• decompressed_bytes
• record_count
• min_created_at
• max_created_at
• first_event_id
• last_event_id
• kind_counts
• pensieve_git_rev
• notepack_git_rev
• generated_at
• format_version

Acceptance criteria

• Manifest generation works for a segment directory.
• Each manifest row includes SHA-256 and record count.
• Each manifest row includes min/max created_at.
• Manifest output is deterministic for unchanged inputs.
• Manifest generation fails clearly on malformed segments unless an explicit skip flag is used.
• Tests cover deterministic output and hash/count correctness.
• just precommit passes before merging.