bug: Scheduled execution silently skipped when previous execution times out at trigger time

[vybe]

Summary

When a scheduled execution runs for exactly as long as (or longer than) the schedule interval and times out at the moment the next execution is due, the next scheduled execution is silently skipped. APScheduler's max_instances=1 constraint causes the new job to be rejected, and no execution record is created. This leads to missing heartbeats/scheduled tasks with no audit trail.

Component

Scheduler Service

Priority

P1 - Scheduled tasks silently fail, no workaround without reducing execution time or increasing interval

Error

[WARNING] apscheduler.scheduler: Execution of job "agent:Heartbeat (trigger: cron[...], next run at: 2026-03-01 08:00:00 UTC)" skipped: maximum number of running instances reached (1)

Location

• File: src/scheduler/service.py
• Function: Job configuration where max_instances is set

Root Cause

APScheduler is configured with max_instances=1 per schedule job. When an execution times out after exactly 1 hour (e.g., 07:00 → 08:00), the timeout handling and the new job trigger happen nearly simultaneously at 08:00. APScheduler sees the previous instance as still running and skips the new one entirely.

The timeline observed:

1. 07:00:00 - Execution starts
2. 08:00:00 - Timeout fires (exactly 3600 seconds)
3. 08:00:00 - APScheduler tries to start new execution, sees "1 instance running", skips
4. 08:00:00 - Previous execution completes with failed status
5. 09:00:00 - Next execution starts (skipped 08:00 entirely)

The skipped execution:

• Is NOT recorded in schedule_executions table
• Only appears as a WARNING in scheduler logs
• Leaves no audit trail in the database

Reproduction Steps

1. Create an agent with a schedule (e.g., hourly at :00)
2. Configure timeout to equal the schedule interval (e.g., 3600 seconds for hourly)
3. Send a task that takes longer than the timeout to complete
4. Observe that when timeout fires at exactly :00, the next scheduled execution is skipped
5. Check schedule_executions - no record exists for the skipped execution
6. Check scheduler logs - see "skipped: maximum number of running instances" warning

Suggested Fix

Option A: Record skipped executions

When APScheduler fires the EVENT_JOB_MAX_INSTANCES event, create a schedule_executions record with status='skipped' and skipped_reason='max_instances_reached'. This preserves audit trail.

from apscheduler.events import EVENT_JOB_MAX_INSTANCES

def on_job_skipped(event):
    # Create execution record with status='skipped'
    db.add_schedule_execution(
        schedule_id=extract_schedule_id(event.job_id),
        status='skipped',
        message='Skipped: previous execution still running',
        triggered_by='schedule'
    )

scheduler.add_listener(on_job_skipped, EVENT_JOB_MAX_INSTANCES)

Option B: Use max_instances > 1 with explicit locking

Allow APScheduler to queue the job but use Redis distributed locking at the execution level to prevent actual concurrent execution. This way:

• Job gets queued
• Execution record gets created with status='queued'
• When lock is acquired, execution proceeds
• If lock cannot be acquired within reasonable time, mark as status='skipped'

Option C: Configure misfire_grace_time and coalesce

Adjust APScheduler's misfire handling:

job = scheduler.add_job(
    func,
    trigger=trigger,
    max_instances=1,
    misfire_grace_time=300,  # Allow 5 min grace
    coalesce=True,           # Combine missed runs
    replace_existing=True
)

This still skips but allows a grace window after the conflict resolves.

Recommended Approach

Option A is the minimum fix for audit trail. The system should never silently drop scheduled work.

Additionally, consider a warning/notification when timeout_seconds >= interval_seconds to alert users that overlapping executions will be skipped.

Environment

• Trinity version: 77c45e4
• Docker: Ubuntu-based containers
• APScheduler: (version from requirements.txt)

Related

• Scheduler service: src/scheduler/service.py
• Execution table: schedule_executions
• APScheduler events: https://apscheduler.readthedocs.io/en/stable/modules/events.html

[vybe]

Fixed in commit 4a04dea.

When APScheduler's max_instances=1 causes a scheduled job to be skipped (because the previous execution is still running at trigger time), the execution is now recorded in the database with status='skipped' instead of being silently dropped.

Changes:

• Added SKIPPED status to ExecutionStatus enum
• Added APScheduler EVENT_JOB_MAX_INSTANCES event listener
• Added create_skipped_execution() database methods
• Frontend displays skipped executions with purple styling
• WebSocket broadcasts schedule_execution_skipped event
• Added 12 tests for skipped execution functionality

Skipped executions now appear in the execution history, providing operators with an audit trail for debugging scheduling issues.