sonar-belt-factory: 6-chain consolidated belt + blue→green expansion gate + zero-downtime swap

[zkSoju]

What

Consolidates the sonar belt to a 6-chain footprint and ships the blue→green promotion machinery that put it live at zero downtime.

• Green = 6-chain consolidated belt (config.yaml: ETH 1 · OP 10 · Base 8453 · Arbitrum 42161 [new] · Berachain 80094 · Zora 7777777 [new], 41 contracts). Dockerfile.belt BELT_CONFIG build-arg (blue unchanged).
• Promotion gate (scripts/promotion-gate.js, 54/54 tests) — EXPANSION-mode: shared entities green ≥ blue − floor (non-lossy, green MAY exceed); new chains have no blue baseline → verified by Part-4 raw-L1 eth_getLogs ground-truth (golden-tx identity; empty-200 = GAP per KF-012). Independent EXPECTED_CHAINS completeness (green can't self-attest). Fail-closed throughout. redactUrl credential guard.
• Swap (scripts/promote.sh, bats 6/6) — the only swap path: runs the gate as a non-skippable precondition (fail-closed), sole BELT_UPSTREAM writer, --rollback (no gate, blue stays hot), --dry-run. Caddyfile admin localhost-only (Option B §7.4).

Verified live (2026-05-22)

event	polls	5xx
swap blue→green	182	0
rollback green→blue	96	0
re-promote blue→green	161	0

Green now serves all 6 chains via the stable alias — 0 consumer config changes (G3). Gate-gated (expansion PASS), zero-downtime (G1), rollback exercised (G4/R-A). Cross-model review+audit caught and fixed 2 fail-closed blockers + 1 secrets HIGH before going live.

Deferred (post-merge, soak-gated)

• Retire blue — irreversible; only after the rollback-window soak with green clean. Blue kept hot.
• Cleanup: remove temp green-hasura public domain; rotate green DB password + belt-hasura-green admin secret; revoke the project token.
• promote.sh follow-up: stop default-publishing the Score #163 signal to the gateway (cred over-exposure) — wire per #163 (score-side).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0c6274aad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fetch schema per deployment before superset check

makeFetchSnapshot() ignores the GraphQL endpoint for schema data and always reads schema.graphql from disk, so Part 3 compares the same local schema for both blue and green. That means the gate can report PASS even if the deployed green schema removed/changed fields relative to blue, which defeats the additive-safety check and can allow a breaking swap.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require GREEN_GRAPHQL_URL for real promotion runs

Defaulting greenUrl to blueUrl creates a fail-open path where a misconfigured run validates blue against itself and can still pass, even though green was never checked. Because promote.sh treats this gate as the non-skippable precondition before flipping BELT_UPSTREAM, this can promote an unvalidated green deployment in any context where the extra expansion guards are not forcing failure.

Useful? React with 👍 / 👎.

[zkSoju]

Summary

Verdict: REQUEST_CHANGES. The PR shows strong operational intent around zero-downtime promotion, but two gate checks currently pass by construction: blue-vs-blue endpoint parity and local-vs-local schema comparison. For a belt promotion system, the decisive question is not whether the script is elegant; it is whether it can refuse a beautiful lie.

Promotion gates are production boundary objects. Netflix Hystrix taught the same lesson at service-call scale: the guardrail matters most when the happy path is persuasive. In Loa terms, this is kaironic time: knowing when to stop the swap is part of the craft.

Findings

{
  "schema_version": 1,
  "findings": [
    {
      "id": "F1",
      "title": "Promotion can pass without a green endpoint",
      "severity": "HIGH",
      "category": "safety",
      "file": "scripts/promotion-gate.js:478",
      "description": "GREEN_GRAPHQL_URL defaults to BLUE_GRAPHQL_URL, so a real promote can accidentally run blue-vs-blue self-parity and pass without validating the green deployment. scripts/promote.sh does not require GREEN_GRAPHQL_URL before flipping BELT_UPSTREAM to green.",
      "suggestion": "In expansion mode or when invoked from promote.sh, require GREEN_GRAPHQL_URL to be explicitly set and different from BLUE_GRAPHQL_URL. Keep self-parity behind an explicit SELF_PARITY=1 or test-only mode.",
      "confidence": 0.95,
      "faang_parallel": "Google SRE launch checklists require validating the actual candidate, not an alias of production.",
      "metaphor": "This is checking the same passport twice and declaring two travelers cleared.",
      "teachable_moment": "Fail-closed gates must make the dangerous default impossible. Test conveniences need explicit names so they cannot leak into live operations.",
      "connection": "This is the bridge loop at the deployment boundary: blue and green must meet as distinct witnesses before the hounfour can bless the swap."
    },
    {
      "id": "F2",
      "title": "Live schema gate compares the same local schema for blue and green",
      "severity": "HIGH",
      "category": "correctness",
      "file": "scripts/promotion-gate.js:386",
      "description": "makeFetchSnapshot reads schema.graphql from disk for both blue and green snapshots, so Part 3 passes by construction during live runs and cannot detect that the deployed green Hasura schema removed or changed fields relative to blue.",
      "suggestion": "Fetch each deployment's actual GraphQL schema via introspection or accept separate BLUE_SCHEMA_PATH and GREEN_SCHEMA_PATH inputs. Fail closed if either schema cannot be retrieved.",
      "confidence": 0.9,
      "faang_parallel": "Amazon service teams rely on live contract tests because repository schemas can diverge from deployed reality.",
      "metaphor": "It is comparing two printed menus while the kitchen may be serving something else.",
      "teachable_moment": "A compatibility gate must observe the deployed artifact, not the intended artifact. Otherwise it certifies documentation, not behavior.",
      "connection": "This is schema-is-not-the-contract in practice: the contract includes the live endpoint and its invariants."
    },
    {
      "id": "F3",
      "title": "Schema parser misses common GraphQL directives with arguments",
      "severity": "MEDIUM",
      "category": "correctness",
      "file": "scripts/promotion-gate.js:175",
      "description": "parseSchema only allows bare directives between the type name and body. A declaration such as `type Action @entity(name: \"actions\") { ... }` is not parsed, which can omit blue types from the superset comparison and hide breaking removals.",
      "suggestion": "Use a GraphQL parser package already available in the project, or update the parser to handle directives with arguments. Add tests with type and enum directives containing arguments.",
      "confidence": 0.82,
      "faang_parallel": "Facebook's GraphQL ecosystem standardized AST parsing because regex-level schema handling misses valid language constructs.",
      "metaphor": "This is a customs form that only recognizes middle names if they have no punctuation.",
      "teachable_moment": "When a domain has a grammar, use the grammar. Parsers buy correctness across edge cases future maintainers will not remember.",
      "connection": "The gate should behave like a cheval: the vessel persists even as schema syntax arrives in different valid forms."
    },
    {
      "id": "F4",
      "title": "Expansion mode does not fail on unexpected green-only chains",
      "severity": "MEDIUM",
      "category": "safety",
      "file": "scripts/promotion-gate.js:104",
      "description": "checkBlockHeights records every green-only chain as deferred in expansion mode, even when that chain is not in EXPECTED_CHAINS. This can allow a misconfigured green deployment to include an unintended extra chain without failing the promotion gate.",
      "suggestion": "When expectedChains is provided, fail if green contains any chain not present in blue and not listed in expectedChains.",
      "confidence": 0.72,
      "faang_parallel": "Kubernetes admission control commonly distinguishes allowed drift from unknown drift; unspecified resources are rejected.",
      "metaphor": "Expansion mode should open named doors, not leave the whole building unlocked.",
      "teachable_moment": "Allowlist semantics are strongest when unexpected additions fail loudly. Deferred should mean planned, not merely unrecognized.",
      "connection": "This is Loa room discipline: allowed inputs must be explicit, and forbidden context should not sneak in under exploration."
    },
    {
      "id": "F5",
      "title": "Shell command uses unescaped service name",
      "severity": "LOW",
      "category": "security",
      "file": "scripts/sync-dashboard.cjs:29",
      "description": "GREEN_PG_SERVICE is interpolated into an execSync shell command. Although this is a local dashboard script, a malformed environment value can execute additional shell syntax.",
      "suggestion": "Use execFileSync with argument arrays, for example `execFileSync(\"railway\", [\"variables\", \"--service\", PG_SERVICE, \"--json\"], ...)`.",
      "confidence": 0.78,
      "faang_parallel": "Chrome and Android build tooling avoid shell interpolation for user-controlled arguments for the same reason.",
      "metaphor": "A shell string is a shared microphone; argument arrays give each word its own channel.",
      "teachable_moment": "Even local scripts become production habits. Prefer APIs that make injection structurally impossible.",
      "connection": "Operational tooling is part of the hounfour too; every helper script should preserve the same boundary discipline."
    },
    {
      "id": "F6",
      "title": "Promotion path has focused fail-closed test coverage",
      "severity": "PRAISE",
      "category": "testing",
      "file": "test/promote.bats:38",
      "description": "The promote.sh tests explicitly verify that a failing gate produces no Railway writes and that rollback does not run the gate. These are the right invariants for the highest-risk operational path.",
      "suggestion": "Keep these tests as required checks for any future promotion script changes.",
      "confidence": 0.95,
      "faang_parallel": "Google SRE postmortem practice turns outage lessons into regression checks on the exact control plane path.",
      "metaphor": "This is testing that the emergency brake stops the train before testing the upholstery.",
      "teachable_moment": "The most valuable tests assert irreversible side effects do not happen under failure.",
      "connection": "This is kaironic time encoded as test coverage: the system knows when not to move."
    },
    {
      "id": "F7",
      "title": "Raw-L1 check treats empty logs as a hard failure",
      "severity": "PRAISE",
      "category": "correctness",
      "file": "scripts/promotion-gate.js:251",
      "description": "checkRawL1 correctly distinguishes an empty eth_getLogs response from a successful proof and fails closed, which directly addresses silent log-loss risk on new chains.",
      "suggestion": "Retain this behavior and require configured golden samples for every expansion chain.",
      "confidence": 0.95,
      "faang_parallel": "Netflix Hystrix treated absence of signal as a first-class failure mode, not a quiet success.",
      "metaphor": "No smoke from the signal fire is not proof the mountain is safe.",
      "teachable_moment": "Empty responses need domain meaning. In promotion gates, silence is usually evidence to stop.",
      "connection": "This is strong bridge-loop behavior: the new chain must speak before it is trusted."
    }
  ]
}

Callouts

[Operational] The promotion tests are pointed at the right invariants: no Railway writes after a failed gate, and rollback remaining independent from promotion certification. That is mature control-plane thinking.

[Correctness] The Raw-L1 empty-log handling is excellent because it refuses ambiguity. Keep that posture across the schema and endpoint checks: every gate should prove the candidate, not merely avoid crashing.

Reviewed with: unknown v0.0.0

[zkSoju]

Bridgebuilder findings addressed (commit f4c3063):

• ✅ F1 (HIGH) — expansion mode + promote.sh now refuse unless GREEN_GRAPHQL_URL is set AND ≠ BLUE_GRAPHQL_URL (no blue-vs-blue self-parity in a real promote). +3 bats tests.
• ✅ F3 (MED) — parseSchema/parseEnums now handle directives with args (@entity(name:"x")) so such a type can't be silently dropped from the superset check. +2 tests.
• ✅ F4 (MED) — expansion now fails a green-only chain not in EXPECTED_CHAINS (unplanned drift), instead of deferring it. +3 tests.
• 📝 F2 (HIGH) — documented limitation, deferred: both belts deploy the identical committed schema.graphql (verified — same 93-entity table set) and the counts query is the live entity-presence check (untracked entity → fail-closed). The real fix (live introspection→SDL) is zero-dep-hard; low-risk given the identical-schema reality. Tracked as a follow-up.
• 📝 F5 (LOW) — deferred: sync-dashboard.cjs execSync→execFileSync (local monitoring tool, env-trusted, not in the promotion path).

All fixes only add fail-closed conditions. Tests: 59 vitest + 9 bats green; hardened gate re-verified — live green still EXPANSION PASS. Thanks for the sharp review — F1 was a real hole. 🙏