Make scheduled tasks not scale in frequency with number of backend replicas

[theosanderson]

We have scheduled tasks in backend - ideally frequency shouldn't scale with number of backend replicas. We should make it so we ensure only one replica tries to run scheduled tasks.

[theosanderson]

@theosanderson-agent what would some options be here

[theosanderson-agent]

A few approaches, roughly from lightest-touch to most architectural. Context: we have 4 Spring @Scheduled tasks (UseNewerProcessingPipelineVersionTask, CleanUpStaleSequencesInProcessingTask, CleanUpAuxTableTask, SeqSetCrossRefCitationsTask), all backed by the shared Postgres DB, running inside the loculus-backend Deployment (replicas.backend, default 1). With N replicas, each one fires every timer independently → N× the work.

Option 1 — ShedLock (recommended)

ShedLock is purpose-built for exactly this: "make sure your scheduled tasks run at most once at the same time." Add the dependency + a small LockProvider bean (it has a JDBC/Postgres provider, so no new infra — just one shedlock table via a Flyway migration), then annotate each method:

@Scheduled(fixedDelayString = "...")
@SchedulerLock(name = "cleanUpStaleSequences", lockAtMostFor = "10m", lockAtLeastFor = "30s")
fun task() { ... }

Pros: minimal, declarative, per-task tuning, battle-tested, works regardless of how we scale replicas. Cons: one new dependency + a migration.

Option 2 — Postgres advisory locks (zero new dependency)

We already have Postgres + Exposed. Wrap each task in a transaction-scoped advisory lock and bail if another replica holds it:

fun <T> runIfLockAcquired(key: Long, block: () -> T): T? = transaction {
    val acquired = exec("SELECT pg_try_advisory_xact_lock($key)") { rs -> rs.next(); rs.getBoolean(1) }!!
    if (acquired) block() else null  // xact lock auto-releases at commit
}

Give each task a distinct constant key. Pros: no new dependency or table, no infra. Cons: a little manual wiring per task; we own the helper. Functionally equivalent to ShedLock for our needs.

Option 3 — Dedicated single-replica scheduler

Disable the @Scheduled beans on the API replicas (e.g. behind a config flag / Spring profile) and run them in either:

• a separate loculus-backend-image Deployment pinned to replicas: 1 with scheduling enabled, or
• Kubernetes CronJobs that hit internal trigger endpoints.

Pros: cleanly decouples scheduling from API scaling; no locking needed. Cons: more moving parts in the chart; the CronJob variant needs endpoints; the single-replica-deployment variant has a brief gap during rollout.

Option 4 — StatefulSet ordinal / k8s leader election

Switch the backend to a StatefulSet and only run tasks on pod-0, or use a Kubernetes Lease-based leader election. Heavier than we need given a DB is right there.

---

Recommendation: Option 1 (ShedLock) if we're fine adding a dependency — it's the standard, lowest-effort-to-maintain answer. Option 2 (advisory locks) if we'd rather not add a dependency; it reuses Postgres and is only ~15 lines of helper. Both leave the Deployment/replica model untouched and just guard the tasks. Happy to put up a draft PR for whichever you prefer.

[theosanderson]

@theosanderson-agent make a demo PR implementing ShedLock

[theosanderson-agent]

Opened a demo PR implementing ShedLock: #6705.

Approach: ShedLock makes each @Scheduled task acquire a named, shared lock before running, so only one replica runs a given task per interval — execution frequency no longer scales with replica count. Locks live in a small shedlock table in the existing Postgres DB (new migration V1.31), so no extra infrastructure is needed.

What it does:

• Adds shedlock-spring + shedlock-provider-jdbc-template (7.7.0), wired to the existing DataSource with usingDbTime() (lock timing uses the DB clock, immune to inter-replica clock drift).
• @EnableSchedulerLock(defaultLockAtMostFor = "PT30M") as a crash safety net.
• Annotates all four scheduled tasks (CleanUpStaleSequencesInProcessing, UseNewerProcessingPipelineVersion, CleanUpAuxTable, SeqSetCrossRefCitations) with @SchedulerLock, each lockAtMostFor sized above its expected runtime.

It does not bump replicas.backend (still 1) — it just makes running >1 replica safe. Compile, ktlint, the context-load test, and a DB-backed endpoint test (Flyway confirms the shedlock migration applies) all pass.