feat: implement data export scheduling system (#267)

[Georgechisom]

Overview

Implemented a comprehensive data export scheduling system that enables users to schedule recurring data exports with automated email delivery and history tracking.

Changes Made

Core Features Implemented

1. Export Scheduler Service (src/lib/export-scheduler/)

• Scheduler Service: Main orchestrator for scheduled exports with periodic checking
• Cron Parser: Parse and validate cron expressions, calculate next run times
• Storage Layer: IndexedDB-based persistence for templates, schedules, and history
• Exporter: Multi-format data export engine (CSV, JSON, XLSX, PDF)
• Notification Service: Email notification system for export completion/failure

2. API Endpoints (src/app/api/exports/)

• GET/POST /api/exports/templates - Template management
• GET/PATCH/DELETE /api/exports/templates/:id - Individual template operations
• GET/POST /api/exports/schedules - Schedule management
• GET/PATCH/DELETE /api/exports/schedules/:id - Individual schedule operations
• GET /api/exports/history - Export history retrieval
• POST /api/exports/execute - Immediate export execution

3. UI Pages (src/pages/exports/)

• Dashboard (/exports) - Main page with tabs for templates, schedules, and history
• Create Template (/exports/templates/new) - Form to create export templates
• Create Schedule (/exports/schedules/new) - Form to create export schedules

4. Testing (src/lib/export-scheduler/__tests__/)

• Cron parser tests
• Exporter tests
• Scheduler service tests

Technical Implementation

Export Formats Supported

• CSV: Comma-separated values with proper escaping
• JSON: Structured JSON output
• XLSX: Excel-compatible XML format
• PDF: HTML-based PDF generation

Scheduling Features

• Cron-based scheduling with validation
• Predefined frequencies: daily, weekly, monthly
• Custom cron expressions support
• Next run time calculation
• Enable/disable schedules

Queue System Integration

• Leverages existing task queue (src/lib/queue)
• Background job processing
• Exponential backoff retry logic
• Dead letter queue for failed jobs

Email Notifications

• Success notifications with download links
• Failure notifications with error details
• HTML and plain text email templates
• Multiple recipient support

Data Persistence

• IndexedDB for client-side storage
• Templates, schedules, and history tracking
• Indexed queries for efficient retrieval
• Automatic cleanup of old history

File Structure

src/
├── lib/export-scheduler/
│   ├── __tests__/
│   │   ├── cron-parser.test.ts
│   │   ├── exporter.test.ts
│   │   └── scheduler-service.test.ts
│   ├── cron-parser.ts
│   ├── exporter.ts
│   ├── index.ts
│   ├── notification-service.ts
│   ├── scheduler-service.ts
│   ├── storage.ts
│   ├── types.ts
│   └── README.md
├── app/api/exports/
│   ├── execute/route.ts
│   ├── history/route.ts
│   ├── schedules/
│   │   ├── [id]/route.ts
│   │   └── route.ts
│   └── templates/
│       ├── [id]/route.ts
│       └── route.ts
└── pages/exports/
    ├── index.tsx
    ├── schedules/new.tsx
    └── templates/new.tsx

Usage Examples

Starting the Scheduler

import { schedulerService } from '@/lib/export-scheduler';

// Start checking for due schedules every minute
schedulerService.start(60000);

Creating an Export Template

const template = await createTemplate(
  {
    name: 'Monthly Course Report',
    format: 'csv',
    dataSource: 'courses',
    columns: ['id', 'name', 'enrollments', 'created_at'],
  },
  userId,
);

Creating a Schedule

const schedule = await createSchedule(
  {
    templateId: template.id,
    name: 'Monthly Export',
    frequency: 'monthly',
    emailDelivery: true,
    emailRecipients: ['admin@example.com'],
  },
  userId,
  nextRunAt,
);

Testing

Run tests with:

npm test src/lib/export-scheduler/__tests__

Acceptance Criteria

✅ Automated recurring exports delivered reliably

• Cron-based scheduler checks for due exports
• Queue system ensures reliable execution
• Retry logic handles transient failures

✅ Scheduling for recurring exports with cron/queues

• Full cron expression support
• Predefined frequencies (daily, weekly, monthly)
• Integration with existing task queue

✅ Email delivery support

• Automatic email notifications
• Success and failure notifications
• Download links in emails

✅ Export history tracking

• Complete history of all exports
• Status tracking (pending, processing, completed, failed)
• File metadata (size, format, download URL)

Dependencies

• idb (already in package.json) - IndexedDB wrapper
• Existing task queue system
• Existing email service infrastructure
• Existing API client

Future Enhancements

• Cloud storage integration (S3, Azure Blob)
• Advanced filtering and data transformations
• Export compression (ZIP)
• Webhook notifications
• Template sharing between users
• Data encryption for sensitive exports
• Rate limiting and quotas
• Export preview functionality

Breaking Changes

None - This is a new feature with no impact on existing functionality.

Migration Notes

No migration required. The feature uses IndexedDB for client-side storage and doesn't require database schema changes.

Screenshots

Exports Dashboard

The main dashboard provides three tabs:

• Templates: Manage reusable export templates
• Schedules: View and manage scheduled exports
• History: Track all export executions

Create Template

Form to create export templates with:

• Template name and description
• Export format selection (CSV, JSON, XLSX, PDF)
• Data source selection
• Column configuration

Create Schedule

Form to create export schedules with:

• Schedule name
• Template selection
• Frequency configuration (daily, weekly, monthly, custom cron)
• Email delivery settings

Documentation

Comprehensive documentation available in:

• src/lib/export-scheduler/README.md - Full API documentation
• Inline code comments throughout the implementation

Checklist

• Code follows project style guidelines
• Self-review completed
• Code commented where necessary
• Documentation updated
• Tests added for new functionality
• All tests pass locally
• No breaking changes introduced
• Feature is backward compatible

Issue

Closes #267 - Data Export Scheduling

[drips-wave]

@Georgechisom Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits