Skip to content

Latest commit

 

History

History
165 lines (138 loc) · 8.78 KB

File metadata and controls

165 lines (138 loc) · 8.78 KB

Direct Execution

Table of Contents

Some ALTER statements are deterministically refused by the MySQL schema-change engine — dropping a primary key or adding a foreign key can never survive its online copy, and explicit ALGORITHM= / LOCK= clauses conflict with the assertions it prepends. By default those statements block the apply, and the plan comment says so up front.

Some refused changes are still genuinely necessary — the canonical case is a primary-key reshape on a small table. Without direct execution, the only path is running the DDL by hand against the target, outside SchemaBot, with no PR trail, no plan, and no audit record. Direct execution brings that category of change inside the system: under an explicit per-environment policy, a refused statement can run verbatim as native MySQL DDL.

A direct statement behaves nothing like a normal SchemaBot apply. It is synchronous, it blocks writes to the table for its full duration, there is no throttling or checkpointing, and it cannot be reverted. The policy exists to bound those consequences, and every uncertain input fails closed.

Lock acquisition is bounded too. Native DDL queues on the table's metadata lock behind any open transaction that has touched the table, and by default MySQL lets it queue essentially forever — with every query arriving after it queueing behind the DDL, stalling all traffic to the table. Direct statements run on a session with a short lock_wait_timeout, so a busy table fails the apply fast with a retryable "table is busy" error instead of stalling. The bound is configurable per policy via the lock_acquisition_timeout config field; the engine applies a short default when it is not set.

Routing

The policy is configured per database environment (see Configuration → Direct Execution). At plan time, each engine-refused statement is resolved against it, and the plan records a per-table execution-mode verdict:

engine refuses statement (e.g. primary-key reshape)
        │ direct_execution policy for this database/environment?
        ├─ absent or disabled ───────────────► blocked
        ├─ table row count unavailable ──────► blocked
        ├─ estimated rows > max_table_rows ──► blocked
        ├─ exact bounded count > bound ──────► blocked
        └─ within bound ─────────────────────► direct: statement runs verbatim
                                               as native MySQL DDL, with its own
                                               progress entry and outcome metric

The same predicate is re-evaluated at apply time before anything runs, so a verdict cannot go stale between plan and execution — a table that grew past the bound after planning is blocked at apply, not run. Statements the policy does not route stay blocked and keep the blocked-apply gate behavior: the apply is rejected up front with the engine's refusal reason.

The size bound

max_table_rows is the blast-radius cap. How long writes stay blocked during native DDL is roughly proportional to table size, so the bound expresses "only run this on tables small enough that the write outage is acceptable" — and the operator enabling the policy decides what that means per environment.

The gate runs in two steps. The first reads information_schema TABLE_ROWS, the InnoDB optimizer's sampled estimate, with statistics caching disabled (information_schema_stats_expiry = 0) so it sees current statistics rather than a cached value up to a day old. That estimate is trusted only to block — it can be off by a meaningful factor in either direction, so it is never allowed to approve on its own. When the estimate is within bound, a verdict for direct execution is corroborated with an exact row count whose scan is capped just past the bound, so a stale or undercounting estimate can never approve a table that actually exceeds the limit. A missing table, a NULL or negative estimate, or a failed count query never passes the gate — unknown size is blocked, not assumed small.

Engine compatibility

Direct execution is implemented by the MySQL (Spirit) engine. The contract it sits on is deliberately engine-neutral, so other engines can adopt it without reshaping any shared surface:

  • The verdict vocabularyExecutionModeDirect / ExecutionModeBlocked and the human-readable ModeReason on a planned table change — lives in pkg/engine as plain strings, alongside TableChange.ExecutionMode.
  • The policy transport is the engine interface's generic metadata channel. Each engine interprets its own policy keys; no engine is forced to read them.
  • The PR workflow keys purely off the execution mode recorded on table changes and aggregates across shards, so any engine that emits a direct verdict inherits the same disclosure and consent UX with no webhook changes.
  • Config validation is the opt-in gate. A direct_execution block is only accepted on databases whose engine implements routing; on any other engine it fails at startup rather than being silently ignored.

An engine that adopts direct execution owns three pieces: its refusal detector (which statements it deterministically cannot run), its size estimator (for MySQL, TABLE_ROWS; a PostgreSQL engine would use its catalog's row estimate), and its executor. Everything else — policy schema, verdicts, metrics, and the PR consent flow — is shared. Two contract requirements come with those pieces:

  • The estimator must map an unknown or negative estimate to blocked — a negative value is a "no real estimate" sentinel, not a count (PostgreSQL's pg_class.reltuples reports -1 for a never-analyzed table).
  • The executor must bound lock acquisition and fail fast on a busy table, the way the MySQL engine bounds lock_wait_timeout. On PostgreSQL that is lock_timeout, and it matters even more there: a DDL queued on a lock blocks new reads of the table as well as writes.

Engine notes:

  • Sharded MySQL (Strata): each shard's engine instance evaluates its own estimate, so max_table_rows is a per-shard bound, and one over-bound or unknown-size shard blocks the whole apply through the normal any-shard-blocked aggregation. A direct statement that does run executes per shard, not atomically across shards — the same property every sharded change has.
  • PlanetScale/Vitess: excluded by design. Raw DDL against vtgate would bypass Vitess online DDL — schema tracking, revert, and the deploy workflow — which is the reason that engine exists. Config validation rejects the policy on these databases at startup.

Operator consent in the PR workflow

The apply command normally proceeds to execution in one step. A plan containing direct-execution changes never does:

  • The plan comment renders a ⚙️ direct-execution section listing each table, the statement, and why the engine refused it.
  • schemabot apply stops at a locked apply comment that repeats the ⚙️ disclosure and asks for schemabot apply-confirm — nothing executes, and the comment the operator confirms against is the one that spells out the consequences.
  • schemabot apply-confirm verifies the PR head still matches the plan the operator confirmed against and re-plans against the live target before executing; a re-plan that resolves to blocked is rejected. On the automatic path, a re-plan that gains direct changes downgrades to manual confirmation instead of executing.
  • --defer-cutover is rejected on an all-direct plan — a direct statement has no cutover to defer. On a mixed plan it applies to the engine-driven statements only, and the disclosure says so. A rejection at confirm time preserves the pending confirmation, so re-running apply-confirm without the flag executes the confirmed plan.

Observability

Every routing outcome increments schemabot.direct_execution.statements_total with the database and an outcome attribute: completed, failed, or stopped for executed statements; blocked_policy_disabled, blocked_size_limit, or blocked_size_unknown for statements the policy did not route. Direct executions are rare, operator-consented events — a spike in failed means native DDL is erroring on the target (check the apply logs for the statement and MySQL error), and a spike in blocked_size_unknown means row counts are unavailable (check target connectivity and information_schema access).