scripts/cutover/execute-cutover.ts and scripts/cutover/rollback.ts are the two scripts that flip shadow mode around a cutover. Both report the step succeeded without confirming the flag actually moved, and in opposite directions from each other.
The cutover path passes while shadow mode is still on
disableShadowMode has two exits and both are PASS. The second one is reached precisely when shadow mode is still engaged:
console.log(' ACTION REQUIRED: Set SHADOW_MODE=false in Railway environment variables');
console.log(' Then redeploy the discord-bot service.');
return { step, status: 'PASS', message: 'Shadow mode disable instruction issued', timestamp: new Date() };
Steps 1-3 in executeCutover each gate on status === 'FAIL' and return early. Step 4 pushes its result and falls straight through to the announcement step and log.outcome = 'SUCCESS'.
So with the flag still on, --confirm prints:
[OK] Disable shadow mode: Shadow mode disable instruction issued
[OK] Post announcement: Announcement: "..."
Cutover COMPLETED
and exits 0. The announcement goes to the community, the incumbent gets turned off, and Outpost answers nobody — because shadow mode never came off.
Also, currentValue is interpolated into the already-disabled message but dropped from the still-on one. The branch where the raw value is the whole diagnosis is the branch that omits it.
The rollback path never reads the flag at all
enableShadowMode prints three ACTION REQUIRED lines and returns status: 'DONE'. It does not call isShadowMode(), does not read process.env.SHADOW_MODE, and verifies nothing. executeRollback gates only step 1, so the outcome is SUCCESS regardless.
This is the worse half, because it is the incident path. Production is double-posting, the operator runs rollback --confirm, reads Rollback COMPLETED, and moves on to the 30-minute monitoring step while Outpost keeps posting at real reporters the whole time.
Suggested shape
Make the unresolved branch not a pass, carry the raw value, and gate the step the way the three above it are gated:
return {
step,
status: 'FAIL',
message:
`Shadow mode is still ON (SHADOW_MODE=${currentValue ?? 'unset'}). ` +
'Set it to false in Railway, redeploy, and re-run — cutover is not complete while the flag is engaged.',
timestamp: new Date(),
};
If a manual step reading as a hard failure is wrong, a third status (ACTION_REQUIRED) treated as non-SUCCESS in the aggregation works too. "PASS / COMPLETED" for a step that just verified the flag is still on is the part that needs to go.
The rollback fix mirrors it, and isShadowMode() helps there: an unrecognized value reads as ON, which is the right answer for a rollback.
Test gap that goes with it
scripts/__tests__/cutover.test.ts covers runHealthChecks, verifyTicketData, executeCutover and executeRollback, but there is no test for disableShadowMode and nothing in the file sets SHADOW_MODE. Worth adding the same fence the queue suites got in #233 — it.each(['TRUE', '1', 'yes', 'on', ' true ', '', 'flase']) asserting the step does not report the cutover done.
Use the delete-when-absent restore from packages/outpost/queue/src/__tests__/ai-response.test.ts: assigning undefined stores the string "undefined", which is an unrecognized value under the current rules and would leak shadow-mode-ON into every later test in the file.
Both behaviours predate #233 — that PR routed the comparison through isShadowMode(), which fixed which direction the message is wrong in and is what made this visible. Filed separately so that diff stays about the predicate.
scripts/cutover/execute-cutover.tsandscripts/cutover/rollback.tsare the two scripts that flip shadow mode around a cutover. Both report the step succeeded without confirming the flag actually moved, and in opposite directions from each other.The cutover path passes while shadow mode is still on
disableShadowModehas two exits and both arePASS. The second one is reached precisely when shadow mode is still engaged:Steps 1-3 in
executeCutovereach gate onstatus === 'FAIL'and return early. Step 4 pushes its result and falls straight through to the announcement step andlog.outcome = 'SUCCESS'.So with the flag still on,
--confirmprints:and exits 0. The announcement goes to the community, the incumbent gets turned off, and Outpost answers nobody — because shadow mode never came off.
Also,
currentValueis interpolated into the already-disabled message but dropped from the still-on one. The branch where the raw value is the whole diagnosis is the branch that omits it.The rollback path never reads the flag at all
enableShadowModeprints three ACTION REQUIRED lines and returnsstatus: 'DONE'. It does not callisShadowMode(), does not readprocess.env.SHADOW_MODE, and verifies nothing.executeRollbackgates only step 1, so the outcome isSUCCESSregardless.This is the worse half, because it is the incident path. Production is double-posting, the operator runs
rollback --confirm, readsRollback COMPLETED, and moves on to the 30-minute monitoring step while Outpost keeps posting at real reporters the whole time.Suggested shape
Make the unresolved branch not a pass, carry the raw value, and gate the step the way the three above it are gated:
If a manual step reading as a hard failure is wrong, a third status (
ACTION_REQUIRED) treated as non-SUCCESSin the aggregation works too. "PASS / COMPLETED" for a step that just verified the flag is still on is the part that needs to go.The rollback fix mirrors it, and
isShadowMode()helps there: an unrecognized value reads as ON, which is the right answer for a rollback.Test gap that goes with it
scripts/__tests__/cutover.test.tscoversrunHealthChecks,verifyTicketData,executeCutoverandexecuteRollback, but there is no test fordisableShadowModeand nothing in the file setsSHADOW_MODE. Worth adding the same fence the queue suites got in #233 —it.each(['TRUE', '1', 'yes', 'on', ' true ', '', 'flase'])asserting the step does not report the cutover done.Use the delete-when-absent restore from
packages/outpost/queue/src/__tests__/ai-response.test.ts: assigningundefinedstores the string"undefined", which is an unrecognized value under the current rules and would leak shadow-mode-ON into every later test in the file.Both behaviours predate #233 — that PR routed the comparison through
isShadowMode(), which fixed which direction the message is wrong in and is what made this visible. Filed separately so that diff stays about the predicate.