Allow incremental ingestion without reprocessing entire history #35
Description
Overview
Enhance the ingestion workflow to support incremental data processing so that running ingestion does not reprocess the entire ledger history every time.
Acceptance Criteria
- Tracks last processed ledger: System persists the last successfully processed ledger. Upon subsequent ingestion runs, only new ledgers are processed.
- Idempotent execution: Running the ingestion multiple times without new data does not cause duplicate processing, errors, or side effects.
- CLI command available: A command-line interface is available to trigger incremental ingestion, e.g.,
python -m astroml.ingestion.incremental(or similar, per project conventions).
Notes
- Ensure the marker for the last processed ledger is robust and persists across restarts/crashes.
- Handle initialization (if no previous state is found) by starting from the configured start ledger or from the earliest available in the database.
- Update README with usage instructions for the new CLI.