fix(server): hold migration advisory lock across the whole migration (PERF-027) - #1998
Draft
serenakeyitan wants to merge 1 commit into
Draft
fix(server): hold migration advisory lock across the whole migration (PERF-027)#1998serenakeyitan wants to merge 1 commit into
serenakeyitan wants to merge 1 commit into
Conversation
The old preflight acquired pg_try_advisory_lock(hashtext('drizzle_migrations'))
and released it immediately, then ran drizzle migrate() on a separate,
unlocked connection. Under multi-replica rollout every replica could pass
preflight simultaneously and execute DDL/backfills in parallel, causing
duplicate-object errors, duplicated backfill work, or failed boot replicas.
runMigrations now opens one max:1 connection, acquires the session-level
advisory lock on it (same key, same 15s poll/timeout, same contention error
message), and keeps it held across the drizzle migrate() call and the table
count; closing the session in finally releases the lock on both success and
error paths. Non-owner replicas keep polling; once the owner releases, they
acquire the lock, see the advanced journal, and no-op.
Closes #1690
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1690 — [PERF-027][High] Startup migrations are not serialized across replicas.
Problem
runMigrations(packages/server/src/db/migrate.ts) only probed the advisory lock: it acquiredpg_try_advisory_lock(hashtext('drizzle_migrations'))and released it immediately, then ran drizzlemigrate()on a separate, unlocked connection.drizzle-orm@0.44.7takes no advisory lock of its own, so during a multi-replica rollout every replica could pass the preflight simultaneously and execute DDL/backfills in parallel — duplicate-object errors, duplicated backfill work, or failed boot replicas.Approach
Hold one session-level advisory lock on a single dedicated connection for the entire migration, per the issue's recommended direction:
runMigrationsopens onemax: 1postgres.js client (one physical session), pollspg_try_advisory_lock(unchanged 1s interval / 15s default timeout), and on success keeps the lock held.migrate()runs on that same connection; session-level advisory locks survive the transactions drizzle opens, so journal read + DDL/backfills + journal insert are all covered.client.end()infinallycloses the session, which releases the lock server-side on both success and error paths — a failed migration cannot strand the lock, and no explicit unlock can mask an in-flight error on a dead connection.migration lock contentionerror still fails the boot fast for the orchestrator to retry.Lock key, timeout defaults, error message, and the 20s
runMigrationsstage budget inbootstrap-server.tsare all unchanged, so existing monitoring/behavior contracts hold.bootstrap-server.tsgets a comment update only.Verification
pnpm check,pnpm typecheck,pnpm build: pass (the one pre-existing biome error inpackages/client/src/__tests__/assistant-text.test.tsis onmainand untouched by this PR).pnpm test: full suite green locally (server: 245 files / 2672 tests).bootstrap-migration-lock.test.tsextended from 2 to 4 cases:migration lock contentionerror after timeout;migrate()executes — this test fails against the old preflight-then-unlock code — and grabbable again afterrunMigrationsreturns;QA note
This touches the boot path. Deterministic serialization behavior is covered by the product tests above; for release-candidate validation the existing case
packages/qa/cases/cross-surface/release-boot-health.md(production image boots + self-runs migrations + health surfaces) already covers this surface, so formal QA can reuse it if requested.