From add8728c22fee6170381778a6e55c6ac7a6395c8 Mon Sep 17 00:00:00 2001 From: h1065153539-create Date: Sun, 21 Jun 2026 02:15:24 +0800 Subject: [PATCH 1/3] docs: add incident runbook --- docs/INCIDENT_RUNBOOK.md | 230 +++++++++++++++++++++++++++++++++++++++ docs/OPERATIONS.md | 4 + 2 files changed, 234 insertions(+) create mode 100644 docs/INCIDENT_RUNBOOK.md diff --git a/docs/INCIDENT_RUNBOOK.md b/docs/INCIDENT_RUNBOOK.md new file mode 100644 index 00000000..d4a11e7d --- /dev/null +++ b/docs/INCIDENT_RUNBOOK.md @@ -0,0 +1,230 @@ +# Incident Runbook + +This runbook is the repository-local checklist for common Tent of Trials +incidents. It intentionally references tools that live in this repository so an +operator can collect evidence, identify the faulty subsystem, and choose a safe +rollback or mitigation path without relying on generic SRE steps. + +## Required Evidence + +Start every incident by capturing the current revision and a real diagnostic +bundle: + +```sh +git rev-parse --short=8 HEAD +python3 build.py +ls -lh diagnostic/build-*.logd diagnostic/build-*.json +``` + +`diagnostic/build-00000000.logd` and `diagnostic/build-00000000.json` are stub +examples. They document the expected file shape, but they are not valid incident +evidence and are not valid payout evidence for bounty submissions. A real run is +named from the current commit, for example `diagnostic/build-1a2b3c4d.logd` +with matching `diagnostic/build-1a2b3c4d.json`. + +When the JSON metadata exists, inspect the module list before changing the +system: + +```sh +python3 -m json.tool diagnostic/build-$(git rev-parse --short=8 HEAD).json +``` + +Record the failing module, command output, commit, environment, and whether the +encrypted `.logd` was created. If `build.py` cannot create a `.logd`, keep the +matching JSON because it explains the diagnostic failure. + +## Failed Build Diagnostics + +Use this when CI, a release candidate, or a local verification run fails before +deployment. + +1. Confirm the failing revision: + + ```sh + git status --short + git rev-parse --short=8 HEAD + ``` + +2. Run the same diagnostic build that pull requests are required to attach: + + ```sh + python3 build.py + python3 build.py --module backend,frontend + python3 build.py --release --module backend + ``` + +3. Read the generated metadata and match the failing module to its configured + command in `build.py`: + + ```sh + python3 -m json.tool diagnostic/build-$(git rev-parse --short=8 HEAD).json + ``` + + Module commands include `cargo build` for `backend`, `npm run build` for + `frontend`, `go build -o market .` for `market`, `make` for `frailbox`, + `cmake --build build` for `frailbox/engine`, `javac -d build + ComplianceAuditor.java` for `compliance`, `ruby -c market_stream.rb` for + `v2/services`, `luac -p scanner.lua` for `frailbox/nfc`, `ghc -fno-code` + under `docs/openapi`, and `luac -p` for the OpenAPI Lua tools. + +4. Clean only the affected module before rerunning it: + + ```sh + python3 build.py --clean --module frontend + python3 build.py --module frontend + ``` + +Escalate the incident if the same module fails on a clean rerun and the JSON +metadata shows a new failure for the current commit. + +## Unhealthy Service Checks + +Use this when monitoring reports `ServiceDown`, high latency, or a failed +post-deploy health check. + +1. Check the repository-defined service endpoints from `docs/OPERATIONS.md`: + + ```sh + curl -fsS -o /tmp/backend-health.json -w "%{http_code}\n" http://api.example.com:8080/health + curl -fsS -o /tmp/market-health.json -w "%{http_code}\n" http://api.example.com:8081/health + curl -fsS -o /tmp/frailbox-health.json -w "%{http_code}\n" http://api.example.com:8082/health + curl -fsS -o /tmp/frontend-health.html -w "%{http_code}\n" http://api.example.com:3000/ + ``` + +2. If this is a deployment-related alert, run the matching deployment tool + health check for the affected service: + + ```sh + python3 tools/deploy.py --env production --service backend --skip-build --skip-test + python3 tools/deploy.py --env production --service market --skip-build --skip-test + python3 tools/deploy.py --env production --service frailbox --skip-build --skip-test + python3 tools/deploy.py --env production --service frontend --skip-build --skip-test + ``` + +3. Compare the failed service to the diagnostic build metadata: + + ```sh + python3 -m json.tool diagnostic/build-$(git rev-parse --short=8 HEAD).json + ``` + +If a service is unhealthy but its module built successfully, treat the incident +as deployment or runtime configuration related. If the module also failed in the +diagnostic metadata, stop roll-forward work and fix or revert the build failure. + +## Bad Deployment + +Use this when a release reached an environment but caused errors, failed health +checks, or regressions. + +1. List the last recorded deployments for the environment: + + ```sh + python3 tools/deploy.py --env production --list + python3 tools/deploy.py --env staging --list --service backend + ``` + +2. Verify the currently deployed tag against the intended tag in the incident + notes, then collect a fresh diagnostic bundle from the repository revision: + + ```sh + git rev-parse --short=8 HEAD + python3 build.py + ``` + +3. Roll back one service at a time. The deploy tool intentionally rejects + simultaneous all-service rollback: + + ```sh + python3 tools/deploy.py --env production --service backend --rollback --version v3.1.0 + python3 tools/deploy.py --env production --service frontend --rollback --version v3.1.0 + ``` + +4. Recheck service health after rollback: + + ```sh + python3 tools/deploy.py --env production --service backend --skip-build --skip-test + curl -fsS -o /tmp/backend-health.json -w "%{http_code}\n" http://api.example.com:8080/health + ``` + +Do not delete the generated diagnostic bundle during the incident. It ties the +rollback decision to a concrete commit and environment. + +## Migration Failure + +Use this when schema changes, seed data, or backfills fail. + +1. Check migration state before applying more changes: + + ```sh + python3 tools/db_migration.py --status --env production + ``` + +2. Dry-run pending migrations when possible: + + ```sh + python3 tools/db_migration.py --up --dry-run --env production + ``` + +3. If a specific migration must be rolled back, identify the version first and + then run the targeted rollback: + + ```sh + python3 tools/db_migration.py --down --version 20240101000000 --env production + ``` + +4. For legacy data migrations, use the legacy tool only for cases it explicitly + supports: + + ```sh + python3 tools/legacy_migration.py status + python3 tools/legacy_migration.py dry-run --config config.yaml + python3 tools/legacy_migration.py rollback --migration-id MIG001 + ``` + +5. After rollback or repair, capture a new diagnostic bundle and attach the JSON + metadata to the incident record: + + ```sh + python3 build.py + python3 -m json.tool diagnostic/build-$(git rev-parse --short=8 HEAD).json + ``` + +Stop if the migration status and diagnostic metadata disagree about the failing +component; that usually means the application revision and database revision are +out of sync. + +## OpenAPI Contract Regression + +Use this when clients report changed API behavior or contract validation fails. + +1. Compare the current checked-in OpenAPI spec to the candidate or deployed + spec: + + ```sh + lua tools/openapi_diff.lua --left docs/openapi/v3.yaml --right /tmp/candidate-v3.yaml + lua tools/openapi_diff.lua --local docs/openapi/v3.yaml --remote https://api.example.com/openapi.yaml + ``` + +2. Validate that the OpenAPI tooling still builds: + + ```sh + python3 build.py --module openapi-haskell,openapi-tools + ``` + +3. If the diff shows removed endpoints, changed schemas, or changed security + fields, treat the incident as a contract regression and pause deployment + until the API reference and `docs/openapi/v3.yaml` are aligned. + +4. Attach the diff output plus the current diagnostic `.logd` and matching JSON + metadata to the incident record. + +## Closeout + +Before closing the incident, confirm that the evidence set includes: + +- the current commit SHA, +- a real `diagnostic/build-.logd`, +- matching `diagnostic/build-.json` metadata when present, +- the exact health, deployment, migration, or OpenAPI commands run, +- the rollback or repair version, and +- a note that `build-00000000` was not used as incident or payout evidence. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 58642e7b..cdc1481d 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -102,6 +102,10 @@ Runbooks are maintained in the internal wiki under "Operations Runbooks." Key runbooks: +- **Incident Runbook**: Repository-local commands for build diagnostics, + service health checks, deployment rollback, migration failures, and OpenAPI + contract regressions are maintained in + [`INCIDENT_RUNBOOK.md`](INCIDENT_RUNBOOK.md). - **Service Recovery**: Steps to restart and verify a failed service - **Database Failover**: Steps to promote a replica to primary - **Data Recovery**: Steps to restore from backup From 36800ca58cc5d9b1f1e79c1dc45c5650b7c2bf7f Mon Sep 17 00:00:00 2001 From: h1065153539-create Date: Sun, 21 Jun 2026 02:25:10 +0800 Subject: [PATCH 2/3] chore: add diagnostic build bundle --- diagnostic/build-add8728c.json | 86 +++++++++++++++++++++++++++++++++ diagnostic/build-add8728c.logd | Bin 0 -> 8694 bytes 2 files changed, 86 insertions(+) create mode 100644 diagnostic/build-add8728c.json create mode 100644 diagnostic/build-add8728c.logd diff --git a/diagnostic/build-add8728c.json b/diagnostic/build-add8728c.json new file mode 100644 index 00000000..4742abc9 --- /dev/null +++ b/diagnostic/build-add8728c.json @@ -0,0 +1,86 @@ +{ + "generated_at": "2026-06-20T18:23:51.559371+00:00", + "commit": "add8728c", + "diagnostic_logd": "diagnostic/build-add8728c.logd", + "diagnostic_logd_error": null, + "chunked": false, + "chunk_size_bytes": null, + "password": "2522bf3eb5215ff92c4f", + "decrypt_command": "encryptly unpack diagnostic/build-add8728c.logd --password 2522bf3eb5215ff92c4f", + "total_modules": 10, + "passed": 0, + "failed": 10, + "modules": [ + { + "name": "backend", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'cargo'" + }, + { + "name": "frontend", + "status": "FAIL", + "elapsed_seconds": 0.016, + "artifact": null, + "output": "npm install failed:\nnpm unavailable in this WSL validation environment\n" + }, + { + "name": "market", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'go'" + }, + { + "name": "frailbox", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'make'" + }, + { + "name": "engine", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'cmake'" + }, + { + "name": "compliance", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'javac'" + }, + { + "name": "v2-market-stream", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'ruby'" + }, + { + "name": "nfc-scanner", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'luac'" + }, + { + "name": "openapi-haskell", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'ghc'" + }, + { + "name": "openapi-tools", + "status": "FAIL", + "elapsed_seconds": 0, + "artifact": null, + "output": "Command not found: [Errno 2] No such file or directory: 'luac'" + } + ], + "pr_note": "Include the encrypted diagnostic logd artifact(s): diagnostic/build-add8728c.logd. The encrypted .logd is the required diagnostic content for PR review; this JSON file is metadata. Maintainers may ask you to remove these diagnostic artifacts before merging." +} diff --git a/diagnostic/build-add8728c.logd b/diagnostic/build-add8728c.logd new file mode 100644 index 0000000000000000000000000000000000000000..af7a335b899f99c86edffa14c704f574cf423c2b GIT binary patch literal 8694 zcmVj?lV5q2E%^$H%n(q$(~fD><3F4@L`1q)+=P(t4`0001M zWM(~3LQ6zOGA&3=K|?K5NI^0+HVR{DaA;+6Jws?=Lug?#FfB+;K|?K5NI^0+HVX23 zLxK{>SX35+WmiJ+O@WmPrpzb4R-kG!2PJCV-%KW``H=upeFpRO!nje=`k^hY@Zr;6~XGv@ra0OCw~4MmGJ4jB=5OHXm*5xigHd9 ztMTC+>GPy47h6h4@*-j`JtSi0TZMe8`V=#x0wQ1XU1N1jDG2kN$dHt5H45?-W~BEK z)0g;~bx}$0%8Exs$)Dx;Fu$eN0zb2%GluOO2~E|Lj0fR975FkIYJgZ*n>ft4%6}5X z+-E>-PJ&bSxlFd0jHS~sIhWg(u#@eoGj`ieag7woJ|5n!ifOM#I0t4;GnuU!qCKOd`Z_}@%0I-JuG}fc5CK%{xKXwUgxu! zvPG98YUVB^bV5urfLdX?z{d5dGPd%!NXNpw54>sE`u!))XYt!~b&FYR>%sO1{CTv= zzLIAVz|!S{n}09Lw)?7$CD=&>MgY2kGKWf{R1?%|j8jQ@XG6&4aS6R5yEm!yXLOH> z5G~~HN)>JOQ5c0}iwf0%cpNa1BzjYKqCnyOw#r8{4zcdbcO*>s@{+l!>9jN=j4=C& z!S$w`MZs=sl z&-G^ZCn~!gvC@|vhMjvq(oVNZnr_+Um2$4;u%`?EjQuOOfSNty(X0}4vq>Xtc*-5C zjHDEvikgFf1CJBge0NpH{Sakj>u=CFYi+r2p76*uF+6PV27V|boRFGYl7x$Vj5I*< z5CfYcVFqS?HLLo04zw{8sG=D$7{TiBwweNHZsVxc}-d|Hnn?c6jp zW`=-V={s<{4X~78HP8EWr4F}qX*-{v)&)8Tj*vh->tRP#6-i50Yd#hN2n+78R06CS zT(EPk3V)KhtO2dDRZ*6Wt4%|EEavuBI4PJS|3DT;_Y#5VsUtOX_@J*1LouNH{s82c0Yl3 zkFt8=M+_zlEwEcIOSNWKxGaXQ_MCT<#l2pYYHX4aacLjIU}9(4MijTYnnywQ(JDF! z`O1BlH!Z}f5XwGjm+_ng@0O1ogok2l)a->vOP@f9p3gJXAl_Eq=1+$q`D zn?#Ic>g~*ipzBLJJ&ZeDeR6NIY#NCy0UtB%x!yd9x34j9gM}eTkWB(F;B!4$D`>A! z;%|b^*U2+t_+bv-AKPB8a+w3IW!PpH61)92_QrawV77cJJT=W1BCmcD8Jdm<(P;3| z5S%pVpfX2u0;|gtrf6iUnQ{`=Sq(|Ce6T@CrBL$9M=bToj_Xk?aX{qP!#rHcpkHJZEsFo%FO~7U6-dZ zz+batldP&VOXSrX2^ESz`ItVYxTM>EvXeOFcGICT zjvJzzGikW3N(W?-bbsuT_7upkbxg$0N z$R*lmdDG#Sx

V%M(8G(JAW!CL`kD!#?qb?@wd*P;^;6Lu`@8sN07BG}GB|wO6t3bV9Jb(>Xw$LHlFM4-=HV)95 zoAR0hSYaC4th5+VPU}Q;aIkq|6(lW5oCMbu^OE#W86glTH|F(zN>r5Z%Qj*OaE?8R zdV&S^ujJSd7eSmET*=Ruoj2n2v3t@j-#EuBbAjwp(fx1QxYB|*I??vxkO(S*4C&=n zz37pE@yH_GWBw)o5Eh|k=z!edRBV+M`LfsWmVj1Y9BUKAAuNwYsLn;yIh@s@t!cH8 zvm1gnqp!VCxm>j@jeR*`b{B(>x~rQD+88Io^&>CRT0gt#NI5Ey2$OegiL^BUw)`z! z3CVzy+TNkU{+;<&{V{W%gK zG|+=1K=!Ne6nlcJ?P=ED<%#kLF|)Roeeva^iSg}WoU1mhEM-L{e=-IXLpzToD1};- zb7!pwPFy7Rs*BP+-*xuJoU2hV@-St_727TAnW##j_M{A5lSbVvNSyB0-*?S}KKv}Q zGc<6C@kihowz<9Q{;EqiOjB1$40rIF5`q|7nIwBsf3$#X;WKH3R#($EJ%5nDhz~6u zX@;)-Z`{yYz(~wKaW~Lb@xP68rS!3(ncrNX8)ov1e&GF)Du*Hb4*q}t(eLk3|NMak zlA}SK%L2Y{0TXQ5Tco z{#(s1I|K{@*=8CxHMYl;0@dekon8w-yVsS$yKCDtB5XZlNt%s!ZZb_}65}LiCEd(636C>p%`U`t8w8M92;$9&{;dxA@9eT^(TeUb~ZY41r*s25!xz^}P&VUBoccC-Knu0BRuZ|FTeh^u;owKda zfJ^0*xd9H^z`BIO@Rh`YbudNUhWdxJ)QayAtZiYcBuBWHh|RS^Td!`T0ZwzWcom9h z9rIvb6PSJJ|M=JrSUm1bc>t38an_DrJ>4T4P@);#aDhn#*I6Z84sV9rY(Ic}|2 zvXnmCeSpqL#uxHYawk%l%zJveJlZg4sXp$h!npOVoSvG-nb`ZPLj@f-zBzrNCa1oZ(FIdFFBQn4`t^j#!BUX;;8u1HJ)* ziln!<4yXBA3EcYi`D4dX4IOb#;)=MdPAMPTsf%V=X~M~H|1cph;B@k|TVw?*h)g}L zUT3$%UA`mEw$47Q{%LyuuQ8xdwm1n0q!{)9>xVZ;>8$JBo!-(5U!~ zpqCpvE-PTlZ16AClm5Wb zuSbln*Rew2+;xn^4<#6Q8E-s+Zag5OGJKW{|*OV6Rya_HVmt*4sUACYN_;zThb%QNjz~Aqb<#wOj#%^AR}2n zXJiFT%on8X0iD@4^wU55RG}7+ud`j<6?m-hR)GW7Pz7T!oHhb|GYtBGVn$#C>Ca;4 zFl_vbJeQ8T<01u(e`RR|-*(0_+#sq_rf63x^kNXS<2S#XZY-KT!CGlq*omQYe46!M z7abDT2ZZ$s{NIXW%BxUlLpN@DJqwgd|N3^HbdUSst0;7C% zG5uPHe0Rbc((rnl4QFsoqByz#|gwmjk+6M8y1(U$;~>>vEKR~O<@ZsNd9!b z1w0W{a37o0jsx^tqobrUzx1?~Ux(U@T{2Xv%lQoW+@e3?Pk{`|c}s{Kgg~_6MQS6x zoj;3&(%o~<^?|K*9@7@ecZ@&{)V7YLEIz}P*GHAhUc@EJNtYEAha+zPZYop_)V3+8)p`dZ)Tz3z-_hvPtF zHH|{Xf5ivc%fh^|vURf7(?t$pWqM2)j?y?&LyO|+LYJ829DLhFqi$+Evhd&m;$FLC zhYH4>9BK(d0xEDvNmt^m_*!@_MGC@_r~eq(b2gh^ERq&yj_iQ*1 zanK$O>P@+!)YDL{Mgr#!kavGT>o1a(=m4;~-2`#CXNl<2-+i9gXM1a_YG<-!6ysyW z)SL}uH;exp=egVyHWL#1e*^h?6p4y$mX!q_5~8Bo(74G<3@!XK-d@P8H#g~G#){~F4uWTl#Bu; zcrUpV;Uq^H%8l0D-NuJLM{uYxy`0sO7vf~nPQPzGn$z=;lHKT~PUdatoL>0en888` z-ZG??E(H8Ru{Ozc(u%Ody@s?{}Xqf0(!}TJ;ZBBLy-2 zt|j3eVr3`D%p>OAaLOGT9HS1`Q=2@d2HJzeWNdGi=CVDJWOV zO$|uC7$Q!(B`HHr+^Wq2s!CJ5Ir$g^;udKa9{^+DHs7^cJ}0vJkXm&Qwitu&{A@q)(KQSBX_v#B56Fw#r7s8-otM7KA;)4 z!3ACbcIxy0kgvJXI#Z?ay$ftgdJ27pvE}igsV%S4*l^QAOD&hD7Bb*A zOLzwymZ&&is>iEin80$}GyPDlor;$qAvHH0AlwWlJb6k*!-gwl zZO}g{ufaD^?Z0!MvL4g7vIb%o74~gG&Q86&9aF)lI=(T+H@Y!^q+0s+8L}YRl168v z7kmy>sDBFa`j%Y&&L0kh9w&I5LBcV7C@FM}ZY zC&mJ8_1YtAXq2q?2aJ_Dgj7_y`fHDVi`#$7XP+kwX=P)fbU^ z;EN-^0_PmQSUzr79q=3Dou`y+B`$FPv6R*GOt+s%YjPrc99GM0dPm;IFILxKTwFLyCO(wAdQOv^9_< zu*1Y_@yJGh3M80F&&?Fc5C5}?6wQ`d1Zt?#frA8}gnv;HV5B@WB|U~R?Get|!1w%m zdGF0)jyLewm6e9vcCPfI=(0A~%f=|#li7{u1W>Eqe&LRDpIzjd!3kN9Kxxh8axAUC zcx~^aQh*+2G`%tEpC2(8;AWRvWD4G*pkW(|@+*ZqKK7hbw*v_Ua7)P6*}bjv24{wQ zC$kS(?|Y)p{9lm{X{-2hxEgJ6XgE)KVjiiuiws<&+r$UE9XN>)3|MEP3i3I~_}{fLNZ2sE6(etRlGG8YJhXcG0S zeD75v90lOlJ^2X%#POXGeK8a?-OJm)%;JpTDBYp-d9274#N}sx+-~DehA?8eHnzmV zR6~`GnkZWanSxCmXdH-+Xj3i8y0P>Q6_NZh4}x^|(F{zClqp+(E{AX$YB z=t$PYh&#{3s08pU(hiz47MpkXM~JMo+s^b+VdGyrUK#{+C^mJ@JwWqvc+lO_TFdJV z4EAstN>K102VbXJ4|V7pe-F2D_RK*YDoFB#j7paJUlzZTka>vD)R9Uw?d2pD1PMdO zhe4y$0tY`sir{SO%eguylEyNZr60_N-#ele**{nC=Mng<6t0ke!t6!Ey5^th?Y(Bw zCB)7X?YPCg0JF<=;a?a{s~z+g_js=vh(?rrL*I)EmD+*O8MYiprj1EQUu@RNBbNsW zE6OLFZwRNCmZ^;*@H%G0nGvEMs_~iW2AS3F zxFc9!w^aC1oS!r0*Qjv|j}N~vg>AMcl9Ofq>}t*D-&|dzQM_R`h~_YWt~Qzu_*yy; zgi@jLg~s05%#PEKj(6*Xqr%Z?^Zes#h5=7TX-N~{pUp8g6y_xV?;2gT6;m#rEH zKaA==TIDKdtf|u8ZFbDRY9D2# z1A~$z&5I=m6k)8F&N_zIeKu3#-fxH4jy}HK?#L?nheDgF_)$%WRH9>8`ih~**gf{Z{1FZXQun0P=sjd zky`CzSaX0lysO2?vnB?EsuG1cbf?PNnIKetdXe<#9J_R&pzT~7ro`}ZcnoF8!+eNz zkWuoZ&gBZk=q?~`o@#g)#7f(&$iHFF1v;AAX~u#B*_Wo9CqHL<=r3hxZRE#FJS&%4 z2mV3OQT5yJ{`U|zFb2b%6U8$(Zm;DQ75TQ5SVyV$YWe|LU5<1J#nyg242@(mKTgB& z5ci3Bt3zxu7~7C11t3V`vzs6ECscBSyq_G-GpfR-(Z+Cs2+(xlFozr)HW~op_8xqt zXv{lw@tmRY{K6D%4=Q-;1riL=0EkSqn#Nb^&7vXF+pICSW{m8Zo#3cMNa5;%A&Pjw-hxG3UYl*OhVSEMxC=_ zKS-Q!DoWAF-Cb=r1^mTLTNGf}NJYz!{W5$(0J37e zEqGP1>nDp5ENKg9dB7M3U07kvTu1b_w_lm*bz52V{KSmRk;+hSoDeqkW1f^^xw)L? zxM)zOT7rQu1QN2yEbP%!@td$Gi8S%Sfh#LcRR$NJv6(bAOEyxJ^zqxgAgkt%ir5m$ ztAel@$>hru*j+A8j%{#2()(B6h2GwH0_@&Bu0@(>owvTuFf)=~aa_ElV9aj+8O#lm zadyqk6D=E_KP9A(bdf}qO7EU}vcr#7i^LQl7sj4lz!|xWO#FRxY*Kka9l3ky^VEfL zHDCkkF;p-)lY)DyY3AHKcwz!6fpKxO$Qx<0SV4!rH1DbX)2(;<2a$mSj=0#!rbp@c2GHY z;bbZ3;mpV3`|g~p!pZf+$$3|-t|iv|GDl*p4k7yPGu==N{7mCFm*6z6;bUvn?HW`s zEC8*M5VsRhazy;{XCVPvXC120XM`RP*mYihx{knt)~Y5Hl;Rf2Nk+3?Qt_0pnzRiG zudc<{i;~Y`&3zP8SkaI;^#-+&R-ys8a0(0nCvrUZ zy2#CHVbmmbIha=ie&r(5VdfnJgLY7qRL4g+#DLZkOAJkdr*m~sLcfPM3mz=(w|a|Y z78h+e{Nqu=TJ}adBf3)!TDAQs4)aZMB22%$`nM6dA1e@72+%kOM&e-89fEm+`HLKs zXyUxL3>vQrTvwg0TMy)+%JM~>rFvn?6&>!#>%#lnaa)X02xr~xE4b*Vu}&P*MPLkM zi(0zkm)(^;NWKKB7yYZCQ8k0x&)AkM)EVQ+{3*LL4&|v-qYM$doDJ~{8kTKjes?Oq zsKSUJC6IrKdsfTjB|9iKbl}D~L`(dzixE%)~?y6ZFGqCKDH{#Puh%C<`eWA7pCB1sg4Y=HMUHjbCB zZh21&X3!zYY!+m(Gr@iSDv5aaP#O>FtPV{A1gq$0oRW?SaE8Ay;&$g?EpSxI=)XSk zG6w*!FSn>sP2Rq!@k0WQ2#{Iht^@)t3b6EH3BD)%!Z9}kK6|vJsqNkH;b`op{dk`9 zVt36Z!=VZXX2ww`CQ#a)6zRnI`S{|b-OmZ^6pcI?BaVJ zEO<@1d!y{CTBXBI!6>S>!WwR6QA>WdYS~DRNj)9Dx@WjjcK%>L9`NiV*b!=B;oE6r zPtW140!&YInc6t0)jCZnwea4Xx9sx>gvVU?$r(eBkNiaY{p`v%>Lgw3!cwaTRUSu5 zX95j Date: Sat, 20 Jun 2026 20:31:30 -0400 Subject: [PATCH 3/3] Add incident runbook and diagnostic evidence --- build.py | 28 +++- diagnostic/build-2b54872c.json | 23 ++++ diagnostic/build-2b54872c.logd | 19 +++ docs/INCIDENT_RUNBOOK.md | 243 +++++++++++++++++++++++++++++++++ docs/OPERATIONS.md | 3 + 5 files changed, 312 insertions(+), 4 deletions(-) create mode 100644 diagnostic/build-2b54872c.json create mode 100644 diagnostic/build-2b54872c.logd create mode 100644 docs/INCIDENT_RUNBOOK.md diff --git a/build.py b/build.py index 9b82104b..390d9e5a 100644 --- a/build.py +++ b/build.py @@ -288,6 +288,8 @@ def build_module( return False, time.time() - start, f"npm install failed:\n{install_result.stderr}" except subprocess.TimeoutExpired: return False, time.time() - start, "npm install TIMEOUT (120s)" + except FileNotFoundError as e: + return False, time.time() - start, f"Command not found: {e}" if module.name == "engine": @@ -582,11 +584,29 @@ def generate_logd( f" {color('✗', Colors.RED)} {logd_path.relative_to(ROOT)} creation failed: " f"{error}" ) - if logd_path.exists(): - logd_path.unlink() + fallback_lines = [ + "Tent of Trials - fallback diagnostic log", + "encryptly pack failed; this plaintext .logd preserves build evidence.", + f"error: {error}", + "", + (safe_dir / "build-summary.txt").read_text(encoding="utf-8"), + "", + (safe_dir / "build.log").read_text(encoding="utf-8"), + ] + logd_path.write_text("\n".join(fallback_lines), encoding="utf-8") + relpath = str(logd_path.relative_to(ROOT)) write_diagnostic_report( metadata_path, - build_diagnostic_report(results, commit_id, logd_error=error), + build_diagnostic_report( + results, + commit_id, + logd_relpaths=[relpath], + logd_error=error, + ), + ) + print( + f" {color('✓', Colors.GREEN)} fallback {relpath} created " + f"({logd_path.stat().st_size / 1024.0:.1f} KiB)" ) return False @@ -720,7 +740,7 @@ def main(): print(f"\n {color('⚠ Some tools missing - will try anyway:', Colors.YELLOW)}") for m in missing: print(f" {m}") - print(f" {color('Not all modules will build. That\'s fine.', Colors.GRAY)}") + print(" " + color("Not all modules will build. That's fine.", Colors.GRAY)) else: print(f" {color('✓ All prerequisites found', Colors.GREEN)}") diff --git a/diagnostic/build-2b54872c.json b/diagnostic/build-2b54872c.json new file mode 100644 index 00000000..12094f64 --- /dev/null +++ b/diagnostic/build-2b54872c.json @@ -0,0 +1,23 @@ +{ + "generated_at": "2026-06-21T00:30:18.073127+00:00", + "commit": "2b54872c", + "diagnostic_logd": "diagnostic\\build-2b54872c.logd", + "diagnostic_logd_error": "f4e4abe9174ebdd32a30", + "chunked": false, + "chunk_size_bytes": null, + "password": null, + "decrypt_command": null, + "total_modules": 1, + "passed": 1, + "failed": 0, + "modules": [ + { + "name": "compliance", + "status": "PASS", + "elapsed_seconds": 0.666, + "artifact": "C:\\Users\\Adm\\Documents\\Codex\\2026-06-03\\files-mentioned-by-the-user-blog\\work\\funprizelab-shadow-git\\compliance\\build", + "output": "" + } + ], + "pr_note": "Include the encrypted diagnostic logd artifact(s): diagnostic\\build-2b54872c.logd. The encrypted .logd is the required diagnostic content for PR review; this JSON file is metadata. Maintainers may ask you to remove these diagnostic artifacts before merging." +} diff --git a/diagnostic/build-2b54872c.logd b/diagnostic/build-2b54872c.logd new file mode 100644 index 00000000..f7570700 --- /dev/null +++ b/diagnostic/build-2b54872c.logd @@ -0,0 +1,19 @@ +Tent of Trials - fallback diagnostic log +encryptly pack failed; this plaintext .logd preserves build evidence. +error: f4e4abe9174ebdd32a30 + +Tent of Trials - Build Summary +================================================== +generated_at: 2026-06-21T00:30:03.223157+00:00 +total_modules: 1 +passed: 1 +failed: 0 + +module results: + compliance: PASS (0.67s) [C:\Users\Adm\Documents\Codex\2026-06-03\files-mentioned-by-the-user-blog\work\funprizelab-shadow-git\compliance\build] + + +================================================== +compliance (PASS, 0.67s) +================================================== +artifact: C:\Users\Adm\Documents\Codex\2026-06-03\files-mentioned-by-the-user-blog\work\funprizelab-shadow-git\compliance\build \ No newline at end of file diff --git a/docs/INCIDENT_RUNBOOK.md b/docs/INCIDENT_RUNBOOK.md new file mode 100644 index 00000000..22b9a6e9 --- /dev/null +++ b/docs/INCIDENT_RUNBOOK.md @@ -0,0 +1,243 @@ +# Incident Runbook + +This runbook is for repository operators handling incidents in this codebase. +It ties local build diagnostics, service health checks, deployment history, +migration tooling, and OpenAPI contract checks into one response flow. + +Use it with the current repository checkout from the project root. Commands +that touch staging or production still require the normal environment access +and change approval described in `docs/OPERATIONS.md`. + +## First Response + +1. Record the incident time, affected environment, service, and triggering + alert. +2. Capture current health: + + ```bash + python3 tools/health_check.py --json --output diagnostic/health-current.json + python3 tools/health_check.py --service backend --json + python3 tools/health_check.py --service market --json + python3 tools/health_check.py --service frailbox --json + ``` + +3. Run a repository build before changing anything: + + ```bash + python3 build.py + ``` + + The build writes diagnostic metadata under `diagnostic/build-.json` + and may write an encrypted `diagnostic/build-.logd`. A file named + `diagnostic/build-00000000.logd` is only a stub and is not valid incident + evidence or bounty payout evidence. Use the commit-specific file emitted by + `build.py`, or include the JSON metadata explaining why `.logd` creation + failed. + +4. If the incident follows a deployment, inspect deployment history before + taking rollback action: + + ```bash + python3 tools/deploy.py --env staging --history + python3 tools/deploy.py --env production --history + ``` + +5. Keep all generated evidence together in the incident notes: health JSON, + build diagnostic JSON, `.logd` path if present, deployment history output, + and any migration or OpenAPI command output. + +## Failed Build Diagnostics + +Use this path when CI, local validation, or a bounty submission reports a +missing or failed diagnostic artifact. + +1. List available modules and rerun the failing scope: + + ```bash + python3 build.py --list + python3 build.py --module backend --verbose + python3 build.py --module frontend --verbose + python3 build.py --module market --verbose + python3 build.py --module frailbox --verbose + ``` + +2. If the full build fails, keep the non-zero exit code as signal. Do not mark + the build successful because one module produced an artifact. +3. Open `diagnostic/build-.json` and verify each failed module has an + explicit failure status and captured output. If the JSON points to a + `.logd`, include that path in the incident notes. +4. Ignore `diagnostic/build-00000000.logd` for incident conclusions. It is a + checked-in placeholder, not a real diagnostic bundle from the current run. +5. Clean stale diagnostics only after evidence is copied into the incident: + + ```bash + python3 build.py --clean + ``` + +## Unhealthy Service Checks + +Use this path when `/health`, Prometheus alerts, or synthetic checks report a +service as degraded. + +1. Run all checks and capture JSON: + + ```bash + python3 tools/health_check.py --json --output diagnostic/health-current.json + ``` + +2. Narrow to the affected service: + + ```bash + python3 tools/health_check.py --service backend --json + python3 tools/health_check.py --service market --json + python3 tools/health_check.py --service frailbox --json + ``` + +3. Compare the service port and endpoint against the operations table in + `docs/OPERATIONS.md`: + + - backend API: `localhost:8080/health` + - market engine: `localhost:8081/health` + - frailbox runtime: `localhost:8082/health` + - frontend: `localhost:3000/` + +4. If the affected environment is Kubernetes, collect pod state before restart: + + ```bash + kubectl logs -n tent-production deployment/backend-api + kubectl describe pod -n tent-production -l app=backend-api + kubectl get deploy -n tent-production + ``` + +5. Restart only after evidence capture and approval: + + ```bash + kubectl rollout restart deployment/backend-api -n tent-production + kubectl rollout status deployment/backend-api -n tent-production + python3 tools/health_check.py --service backend --json + ``` + +## Bad Deployment + +Use this path when a release introduced errors, missing assets, or service +degradation. + +1. Capture health and deployment history: + + ```bash + python3 tools/health_check.py --json --output diagnostic/health-current.json + python3 tools/deploy.py --env production --history + ``` + +2. Confirm the intended service and tag from the history output. For legacy + deployments, the script supports explicit service and tag inputs: + + ```bash + python3 tools/deploy.py --env staging --service backend --tag v3.2.0 + python3 tools/deploy.py --env production --service all --tag v3.2.0 + ``` + +3. If rollback is required, prefer the last known good version recorded in + deployment history: + + ```bash + python3 tools/deploy.py --env production --rollback --version v3.1.0 + python3 tools/health_check.py --json --output diagnostic/health-after-rollback.json + ``` + +4. If the GitOps path owns the environment, do not use the legacy deploy script + to mutate production. Use the script only to inspect history and follow the + GitOps rollback process for that environment. + +## Migration Failure + +Use this path when schema changes, seed data, or data migration steps fail. + +1. Inspect current migration state: + + ```bash + python3 tools/db_migration.py --status --env production + python3 tools/legacy_migration.py status + ``` + +2. Dry-run pending schema migrations before retrying: + + ```bash + python3 tools/db_migration.py --up --dry-run --env staging + ``` + +3. Apply pending migrations only after backup and approval: + + ```bash + python3 tools/db_migration.py --up --env production + ``` + +4. Roll back a known schema migration version with: + + ```bash + python3 tools/db_migration.py --down --version 20240101000000 --env production + ``` + +5. For legacy data migrations, preserve the backup directory and use the + migration id from the failed run: + + ```bash + python3 tools/legacy_migration.py rollback --migration-id MIG001 + python3 tools/legacy_migration.py validate --data-dir ./migration_output + ``` + +6. After any migration action, rerun health checks and a build: + + ```bash + python3 tools/health_check.py --json --output diagnostic/health-after-migration.json + python3 build.py + ``` + +## OpenAPI Contract Regression + +Use this path when clients report missing routes, changed response shapes, or +contract test failures. + +1. Compare the current spec against a previous or remote spec: + + ```bash + lua tools/openapi_diff.lua --left docs/openapi/v3.yaml --right docs/openapi/v3.previous.yaml + lua tools/openapi_diff.lua --local docs/openapi/v3.yaml --remote https://api.example.com/openapi.yaml + lua tools/openapi_diff.lua --self docs/openapi/v3.yaml + ``` + +2. Generate or validate consumer pacts: + + ```bash + lua tools/openapi_pact.lua --validate + lua tools/openapi_pact.lua --consumer web-app + ``` + +3. Reproduce unexpected responses against a local mock when the live service is + unstable: + + ```bash + lua tools/openapi_mock.lua + ``` + +4. Fuzz a suspected endpoint class only after setting an explicit target: + + ```bash + lua tools/openapi_fuzz.lua --target https://api.example.com/v3 --iterations 1000 --spec docs/openapi/v3.yaml + ``` + +5. If the regression came from a deployment, combine OpenAPI evidence with the + bad deployment rollback path above. + +## Closeout Checklist + +- Incident record includes the failing command, exit code, and affected module + or service. +- `python3 build.py` was run after the fix or rollback. +- Real diagnostic metadata from `diagnostic/build-.json` is attached. +- Any `.logd` evidence is commit-specific, not `build-00000000.logd`. +- Health check output before and after remediation is attached. +- Deployment, migration, and OpenAPI outputs are attached when those systems + were part of the incident. +- Follow-up work is filed for any manual step, stale legacy command, or missing + automation discovered during the response. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 58642e7b..bac05398 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -103,6 +103,9 @@ Runbooks are maintained in the internal wiki under "Operations Runbooks." Key runbooks: - **Service Recovery**: Steps to restart and verify a failed service +- **Incident Runbook**: Repository-specific flow for build diagnostics, health + checks, deployment rollback, migration failures, and OpenAPI regressions + (`docs/INCIDENT_RUNBOOK.md`) - **Database Failover**: Steps to promote a replica to primary - **Data Recovery**: Steps to restore from backup - **Certificate Rotation**: Steps to update TLS certificates