Skip to content

Make reporting handle large datasets (payload scale, pagination, streaming) #1537

Description

@vybe

Summary

Make the reporting pipeline hold up when a report carries a large dataset. Today the payload is a single JSON blob capped at 256 KB (413 over cap) stored in one TEXT column — fine for KPIs, but it fails for a report of thousands of rows.

Context

Part of the Agent Reports epic. As agents report real tabular results (lead lists, scan results, transaction histories), the 256 KB cap and single-blob storage/transfer become the ceiling. This issue covers scaling the create, store, list, and render paths — without breaking the small-report common case.

Acceptance Criteria

  • Reports well beyond today's 256 KB cap are supported (raise/stream the cap, or chunk/paginate the payload) with a defined new ceiling
  • List endpoints stay metadata-only (no payload) so the Reports tab is unaffected by large payloads (preserve the ReportSummary vs full-payload split)
  • Detail/render path paginates or lazy-loads large tabular payloads instead of shipping the whole blob at once
  • Create path validates/streams large bodies without pinning a backend coroutine or blowing memory
  • Retention sweep still handles large rows efficiently (chunked delete already exists)
  • Backward compatible with existing small reports

Technical Notes

db/reports.py storage model (consider off-row / paginated storage for tabular payloads), services/report_service.py, routers/reports.py (REPORT_PAYLOAD_MAX_BYTES). Dual-track migration if the schema changes (SQLite db/migrations.py + Alembic). Coordinate with the export issue (streaming). OSS-core.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions