B-17 P0 · RAILWAY_TOKEN_ACC1/ACC2 expired — ALL healer workflows blocked at Sanity-check tokens

[gHashTag]

B-17 · RAILWAY_TOKEN_ACC1 / ACC2 expired — ALL healer workflows blocked

R5-verified evidence (2026-05-14 06:01 UTC, cycle 14)

Symptom

Writer service ssot.bpb_samples frozen for 10h 34m:

Probe	Value
MAX(ts) on SHORT-WAVE-MATRIX canons	2026-05-13T19:26:40.738Z
stale_sec	38095
rows_1h	0
rows_6h	0
rows_24h	877 (all before 19:26 UTC)
Non-matrix canons last_ts	2026-05-12T17:00:38.059Z (stale 37h)

Root cause

Every healer workflow that calls Railway team GraphQL fails at the first step (Sanity-check tokens):

##[error]token ACC1 failed sanity check
##[error]Process completed with exit code 3.

The sanity check curls https://backboard.railway.com/graphql/v2 with RAILWAY_TOKEN_ACC1 and expects .data.projects.edges — receives empty/null instead, meaning the team token is revoked or expired on Railway side.

Affected workflow runs (R5-verified)

Workflow	Latest run	Conclusion	Failed step
writer-env-fix.yml	25844596039 (2026-05-14 06:00 UTC, dispatch this cycle)	❌ failure	Sanity-check tokens
writer-env-fix.yml	25843795652 (2026-05-14 05:35 UTC)	❌ failure	Sanity-check tokens
mcp-emergency-redeploy.yml	25843877778 (2026-05-14 05:38 UTC)	❌ failure	—

Last good token use

• refresh-acc47.yml run 25754312720 at 2026-05-12 18:31 UTC = ✅ success
• Token last rotated in repo secrets: 2026-05-12 14:44:43 UTC (RAILWAY_TOKEN_ACC1)

→ Token worked for ~4 hours after rotation, then stopped. Likely:

1. Team membership change on Railway side revoked the token, OR
2. Token expired (Railway team tokens have ≤ 30-day TTL by default), OR
3. Token scope was narrowed and projects { edges } query no longer returns rows.

Why this is a hard blocker

• Layer-2 Doctor (doctor-loop.yml) — PR feat(doctor): Layer-2 autonomous fleet doctor skeleton (skill docker-railway-doctor v1.2, ICA-103) #155 still draft, doctor not deployed yet
• Gardener-loop v2.12 (post-feat(gardener-v2.12): dual rate-collapse + header lift (closes B-09, B-10) #149 merge) reports success on every tick but wakes 0 effective trainers — gas-lighting persists (B-12 / B-16 pattern)
• Stage 2d in gardener-loop run 25843994908 claimed ⚡ FORCED WAKE-UP: 159 redeployed · 1 failed — but writer still dead. Without working Railway tokens, even serviceInstanceDeployV2 calls have no effect (Railway accepts the GraphQL mutation, then deployments queue but never apply because no team auth).

Cure (operator-only)

# On Railway dashboard, rotate team tokens for all 7 ACC1-7 accounts,
# OR generate new project-access-tokens scoped to IGLA + ACC2-RP projects.
# Then update repo secrets:

gh secret set RAILWAY_TOKEN_ACC1 --repo gHashTag/trios-railway --body "$NEW_TOKEN_ACC1"
gh secret set RAILWAY_TOKEN_ACC2 --repo gHashTag/trios-railway --body "$NEW_TOKEN_ACC2"
# ... ACC3..ACC7 similarly

# Verify with sanity-check workflow dispatch:
gh workflow run writer-env-fix.yml --repo gHashTag/trios-railway -f confirm=PHI
gh run watch  # expect Sanity-check tokens ✅

Verification gates (post-fix)

1. writer-env-fix.yml workflow_dispatch → ✅ success (currently fails at Sanity-check)
2. ssot.bpb_samples writes resume → stale_sec < 600 within T+15min
3. Gardener-loop Stage 2d wake_effective=true on next hourly tick

Anchors

• Anchor: φ² + φ⁻² = 3 · TRINITY · NEVER STOP
• DOI: 10.5281/zenodo.19227877
• Cycle 14 NASA report: DOCTOR-CYCLE-14-RVR-001

cc: B-12 (#151 gas-lighting), B-16 (Stage 2d claim vs reality), PR #149 (merged), PR #155 (doctor Layer-2 pending)

[gHashTag]

R5 verification @ 06:15 UTC — ACC1/ACC2 ротированы, но всё ещё failure

Что проверено

• RAILWAY_TOKEN_ACC1 updated_at = 2026-05-14T06:05:28Z ✅ ротирован
• RAILWAY_TOKEN_ACC2 updated_at = 2026-05-14T06:04:54Z ✅ ротирован
• ACC3..ACC7 — НЕ ротированы (всё ещё 2026-05-12 14:45-14:46 UTC)

Re-dispatch результат

• Run 25845100317 @ 2026-05-14 06:15 UTC = ❌ failure
• Failed step: Sanity-check tokens → ##[error]token ACC1 failed sanity check / exit code 3
• Symptom: новый ACC1 token тоже возвращает пустой .data.projects.edges от https://backboard.railway.com/graphql/v2

URL endpoint sanity (R5)

Оба canonical URL отвечают HTTP 200:

• backboard.railway.com/graphql/v2 ✅
• backboard.railway.app/graphql/v2 ✅

→ Проблема не в URL, а в типе токена.

Hypothesis (R5-honest)

Workflow writer-env-fix.yml (видно в .github/workflows/writer-env-fix.yml#L52-L60) ставит:

env:
  TOKEN_ACC1: ${{ secrets.RAILWAY_TOKEN_ACC1 }}
  RAILWAY_TOKEN_AUTH: team

И шлёт:

curl ... -H "Authorization: ${T}" ...
-d '{"query":"{ projects { edges { node { id name } } } }"}'

Запрос { projects { edges } } — это team/account-level scope. Если оператор сгенерил project-access-token (project-scoped, который привязан к одному проекту), Railway вернёт data.projects.edges = null потому что project-token не имеет доступа к списку проектов.

Что нужно (R5 fix recipe)

Зайти в Railway dashboard → Account Settings → Tokens (НЕ Project Settings → Tokens):

1. Создать Account Token (либо Team Token если есть team plan), scope = team или account. Этот токен может запросить { projects { edges } }.
2. Обновить repo secret: gh secret set RAILWAY_TOKEN_ACC1 --repo gHashTag/trios-railway --body "<account_token_acc1>"
3. То же для RAILWAY_TOKEN_ACC2.
4. (Опционально) обновить ACC3..ACC7 если они нужны для других workflow.
5. Re-dispatch: gh workflow run writer-env-fix.yml --repo gHashTag/trios-railway -f confirm=PHI
6. Ожидание: step Sanity-check tokens → token ACC1 OK / token ACC2 OK.

Writer pulse (не сдвинулся)

Probe	Value
MAX(ts) SHORT-WAVE-MATRIX	2026-05-13T19:26:40.738Z
stale_sec	38906 (10h 48m)
rows_15m	0
rows_1h	0

Anchor: φ² + φ⁻² = 3 · TRINITY · NEVER STOP

[gHashTag]

R5 verification @ 06:43 UTC — token-classify.yml доказал что все 7 токенов РАБОЧИЕ

Run 25846091578 — все ACC1..7 успешно отвечают на Project-Access-Token: <token> запрос:

ACC1: PAT/projectToken: {"id":"f29aa9dd-ca0b-460f-ad24-c7680c6717fb","name":"IGLA RACE"} ✅
ACC2: PAT/projectToken: {"id":"ad0f8f04-c56d-4b11-9350-cc0c2700b9db","name":"IGLA"} ✅
ACC3: PAT/projectToken: {"id":"475a2290-d990-426a-af57-594a934cf6f4","name":"robust-radiance"} ✅
ACC4: PAT/projectToken: {"id":"0dc36fab-0201-46dc-9de2-43ca5008a405","name":"IGLA"} ✅
ACC5: PAT/projectToken: {"id":"fa333a5e-0151-4bf2-bc5f-9b32c4e762a9","name":"IGLA"} ✅
ACC6: PAT/projectToken: {"id":"f4d7b9fb-33c5-418b-b29c-6dfb1794db49","name":"IGLA"} ✅
ACC7: PAT/projectToken: {"id":"19e35d90-e2ae-47a2-8777-6ae5c6c2ba2a","name":"IGLA"} ✅

(Bearer/me → "Not Authorized" — это ожидаемо для project-tokens.)

Истинный root cause (misdiagnosed раньше)

B-17 был неправильно поставлен. Токены живы и project-scoped. Проблема — в самом workflow writer-env-fix.yml:

env:
  RAILWAY_TOKEN_AUTH: team       # ← неверно — у нас project-tokens

curl ... -H "Authorization: ${T}" \
  -d '{"query":"{ projects { edges { node { id name } } } }"}'  # ← требует account/team token

При этом сосед refresh-acc47.yml (последний успешный run 25754312720) использует правильный pattern:

curl ... -H "Project-Access-Token: $TOK" \
  -d '{"query":"{ project(id: \"...\") { services { edges {...} } } }"}'

И код crates/trios-railway-core/src/transport.rs явно объявляет AuthMode::Project для IGLA workspace:

project sends Project-Access-Token: <token> and is required for project-scoped tokens (which is what the IGLA workspace token currently is).

Правильный fix

Переписать writer-env-fix.yml:

1. Убрать RAILWAY_TOKEN_AUTH: team
2. Заменить header: Authorization: <T> → Project-Access-Token: $TOK
3. Заменить query { projects { edges } } → { project(id: $IGLA_PROJECT_ID) { services { edges { node { id name } } } } }
4. Скопировать pattern из refresh-acc47.yml (proven-working 2026-05-12 18:31Z)

То же самое нужно проверить для mcp-emergency-redeploy.yml (тоже failed today).

Closing B-17

Closing as misdiagnosed. Открываю новый правильный issue B-18: writer-env-fix.yml uses wrong auth mode с recipe для fix.

Writer всё ещё мёртв (stale_sec=39143, 10h 53m), но корень — workflow code bug, не токены.

Anchor: φ² + φ⁻² = 3 · TRINITY · NEVER STOP

[gHashTag]

Misdiagnosed — токены живы и project-scoped. См. token-classify run 25846091578. Реальный root cause = workflow code bug, переоткрыт как B-18.