feat: Implement Concurrent Mission Queue with BullMQ

[nik-kale]

Summary

This PR implements a production-grade concurrent mission queue using BullMQ and Redis, enabling 3 missions to execute simultaneously instead of sequential blocking. Includes job tracking, retry logic, and queue management endpoints.

Changes

• Created missionQueue.ts:
  • BullMQ queue and worker setup
  • Configurable concurrency (default: 3)
  • Exponential backoff retry (3 attempts)
  • Job lifecycle management
• Queue Features:
  • Concurrent execution (3 missions at once)
  • Automatic retries on failure
  • Job progress tracking
  • Queue position visibility
  • Graceful shutdown handling
• API Endpoints:
  • GET /api/queue/status - Queue metrics
  • GET /api/queue/job/:jobId - Job details and position
• Updated mission creation:
  • Missions added to queue instead of direct execution
  • Returns mission ID immediately
  • Execution happens asynchronously

Type of Change

• New feature
• Bug fix
• Breaking change
• Documentation update
• Performance improvement

Performance Improvements

Metric	Before	After	Improvement
Concurrent missions	1	3	300%
Queue management	None	BullMQ	✅
Retry logic	None	3 attempts	✅
Job tracking	None	Full visibility	✅
Browser instances	1	Up to 3	300%

Queue Architecture

Client → API → Queue → Worker Pool (3) → Browser Agents
                ↓
              Redis (job storage)

Queue Features

Concurrency

• 3 concurrent workers processing missions
• Configurable via maxConcurrentMissions config
• Automatic load balancing

Retry Logic

• 3 retry attempts on failure
• Exponential backoff: 2s, 4s, 8s
• Failed jobs preserved for debugging

Job Management

• Queue position: Know your place in line
• Progress tracking: 0-100% completion
• State tracking: waiting, active, completed, failed, delayed

Metrics

• Jobs waiting
• Jobs active
• Jobs completed
• Jobs failed
• Worker count

API Examples

Create Mission (queued)

POST /api/missions
{
  "prompt": "Check dashboard health"
}

Response: 201
{
  "missionId": "abc-123"
}

Check Queue Status

GET /api/queue/status

Response:
{
  "status": {
    "waiting": 5,
    "active": 3,
    "completed": 42,
    "failed": 2,
    "delayed": 0,
    "workers": 3,
    "concurrency": 3
  }
}

Check Job Position

GET /api/queue/job/abc-123

Response:
{
  "job": {
    "id": "abc-123",
    "state": "waiting",
    "progress": 0,
    "position": 3,
    "attempts": 0
  }
}

Configuration

Requires Redis:

REDIS_HOST=localhost
REDIS_PORT=6379

Testing

• ✅ No linting errors
• ✅ Queue initializes successfully
• ✅ Workers process jobs concurrently
• ✅ Retry logic functions
• ✅ Queue status endpoint works
• ✅ Job tracking accurate

Scalability

• Horizontal scaling: Add more workers
• Redis cluster: For high availability
• Job persistence: Survives restarts
• Rate limiting: 10 jobs/second max

Graceful Shutdown

• Waits for active jobs to complete
• Closes queue connections cleanly
• No job data loss

Checklist

• Code follows project style guidelines
• Self-review completed
• Queue properly configured
• No new warnings introduced
• Retry logic tested
• Metrics endpoints functional

Related Issues

Addresses Feature #6 from FEATURE_OPPORTUNITIES.md - Priority Score: 1.5

Notes

• Requires Redis running locally or configured
• BullMQ already in package.json dependencies
• Queue survives server restarts
• Future: Add queue dashboard UI
• Future: Implement priority queuing

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[nik-kale]

Closing PR - queue implementation files were removed by user. Implementation available for future reference.