feat(tern): one deployment correlates to exactly one remote apply

[aparajon]

Why this matters

Deployment-keyed dispatch attaches every sibling operation into the deployment's one data-plane apply, so all of a deployment's operation rows must record the same remote apply id — that shared id is what an operator greps in the data plane's storage and logs during an incident. Nothing enforced this: a dispatch result carrying a different id (an in-flight apply spanning a dispatch-key rollout, or a data plane that lost its keyed apply and minted a fresh one) would be persisted silently, splitting one deployment across two remote applies and breaking every correlation built on top.

What it does

• persistRemoteApplyID fails closed when storing a dispatch's remote apply id would correlate the operation's deployment to a second remote apply — either because the deployment's siblings already recorded a different id, or because the siblings themselves already disagree. Sibling deployments of the same apply are exempt: they own their own remote applies.
• storage.DeploymentRemoteApplyID resolves the single remote apply id shared by a deployment's operations (with the legacy engine-resume-context carrier honored), erroring on disagreement instead of picking one. The write-path guard consumes it now; the status/progress read model consumes it next.
• ApplyOperation.RemoteApplyID names the external-id-else-legacy-carrier lookup both sites used.
• New counter schemabot.remote_apply_deployment_id_conflict_total (database, environment, deployment) counts the refusals, documented in the metrics README with the operator action.

How it moves us toward the northstar

One data-plane apply per deployment; operations dispatch into it.

0  per-shard reconcile                          ✅ shipped
pre vschema-only deployment-scoped shape        ✅ shipped
pre op-lease drives settle projection-safely    ✅ shipped
1  dispatch granularity
   ├─ data-plane sibling attach                 ✅ merged
   └─ deployment-keyed idempotency + echo       ✅ merged
2  correlation persistence                      ⬅ this PR
   one shared external_id per deployment, fail-closed on divergence
3  op-scoped data-plane drive (shared apply)    up next
4  read model exposes the deployment apply id   up next
5  CLI renders it as APPLY ID in deployment mode
6  test matrix

Opened by Claude (Fable 5).

[Copilot]

Pull request overview

This PR strengthens SchemaBot’s remote (gRPC) apply correlation model by enforcing the invariant that a single deployment must map to exactly one remote data-plane apply ID, failing closed when a dispatch response would split a deployment across multiple remote applies. This improves operator triage by ensuring there’s a single remote apply identifier to correlate across control-plane and data-plane logs/storage.

Changes:

• Add a write-path guard (guardDeploymentRemoteApplyID) that rejects persisting a remote apply ID when sibling operations for the same deployment already recorded a different ID (or already disagree).
• Introduce shared storage helpers (ApplyOperation.RemoteApplyID() + storage.DeploymentRemoteApplyID) to consistently resolve the remote apply ID (including legacy engine_resume_context fallback) and fail closed on disagreement.
• Add a new metric (schemabot.remote_apply_deployment_id_conflict_total) and document the operator action in the metrics README.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/tern/grpc_client.go	Enforces deployment-level single-remote-apply-ID persistence with a fail-closed guard and emits a new conflict metric.
pkg/tern/grpc_client_deployment_id_test.go	Adds focused tests for the new deployment-shared remote apply ID invariant on the gRPC client write path.
pkg/storage/deployment_remote_apply.go	Adds helpers to resolve per-operation and per-deployment remote apply IDs (with legacy fallback) and fail closed on disagreement.
pkg/storage/deployment_remote_apply_test.go	Adds unit tests covering RemoteApplyID() and DeploymentRemoteApplyID() behavior, including disagreement cases.
pkg/metrics/README.md	Documents the new conflict counter and the expected operator response.
pkg/metrics/metrics.go	Adds RecordRemoteApplyDeploymentIDConflict for emitting the new counter with deployment dimension.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.