Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 31 additions & 12 deletions .codex/skills/dev-instance/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
name: dev-instance
description: Run the current worktree as a production-shaped local dev instance — core, Slack, web UI, admin, portal, on a real Pi LLM + Postgres — reachable in Slack as your own bot. Each developer uses their own set of Slack apps from their own machine's pool store, so many worktrees (yours and a teammate's) can run reachable at once without colliding. Use when asked to /dev-instance, "spin this up so I can QA it in Slack", or "let me test your branch end to end".
description: Run the current worktree as a production-shaped local dev instance — core, optional Slack, web UI, admin, portal, on a real Pi LLM + Postgres. Each developer can use Slack apps from their own machine's pool store, while browser-only instances need no Slack configuration. Use when asked to /dev-instance, "spin this up so I can QA it", or "let me test your branch end to end".
---

# dev-instance

`dev-instance` runs the current worktree as a full, production-shaped stack on your
machine and makes it reachable in Slack as one of _your_ bots. It is the way to QA a
machine, with Slack enabled by default and optional for browser-only QA. It is the way to QA a
branch end to end: real LLM turns, a real sandbox, a real local Postgres (empty by
default; opt in to prod data), and the real Slack/web/admin surfaces.

Expand All @@ -15,6 +15,7 @@ every command accepts `--json` for machine-readable output):

```bash
bash scripts/dev-instance.sh up
bash scripts/dev-instance.sh up --no-slack
bash scripts/dev-instance.sh status
bash scripts/dev-instance.sh down
bash scripts/dev-instance.sh doctor
Expand All @@ -23,17 +24,19 @@ bash scripts/dev-instance.sh restart [child]
bash scripts/dev-instance.sh logs [child] [-f]
```

`npm run dev-instance`, `npm run dev-instance:status`, `npm run dev-instance:down`, and
`npm run dev-instance`, `npm run dev-instance:no-slack`, `npm run dev-instance:status`, `npm run dev-instance:down`, and
`npm run dev-instance:doctor` are equivalent. The Codex-visible skill copy lives at
`.codex/skills/dev-instance/SKILL.md`; keep the two skill descriptions equivalent.

## What `up` Starts

`up` claims one free Slack app slot from **this machine's** pool store (see "Slack reach"
below), then spawns a **per-slot supervisor daemon** that owns the production-shaped stack:
below), then spawns a **per-slot supervisor daemon** that owns the production-shaped stack.
With `--no-slack` or `DEV_INSTANCE_NO_SLACK=1`, it claims only a local port slot and needs
no pool configuration:

- core API + workers
- Slack Socket Mode plugin (connected as the claimed app's bot)
- Slack Socket Mode plugin when Slack is enabled (connected as the claimed app's bot)
- web UI surface
- admin surface
- portal front door proxying `/web-ui/` and `/admin/`
Expand All @@ -42,7 +45,7 @@ The supervisor restarts crashed children with backoff, waits for a port to actua
before respawning (no more EADDRINUSE), health-probes everything every 10s, and writes a
heartbeat so slot reclaim can tell "actively in use" from "abandoned".

**`up` only prints success after proving the bot is reachable**: the Slack socket must be
When Slack is enabled, **`up` only prints success after proving the bot is reachable**: the Slack socket must be
the app's _only_ connection (`num_connections == 1`, read from the hello frame) and — when
the slot has a `CANARY_CHANNEL` — a posted canary message must arrive back over that same
socket. If another machine/worktree holds a connection to the app (the classic "boots LIVE
Expand All @@ -52,12 +55,27 @@ means a stale Slack app; `up` flags and rotates past that too.

**Re-running `up` on a live instance is a reload, not a no-op**: it re-reads your shell
env, dev.env, and `.env`, diffs against what the children are running, and does a rolling
restart + re-verification when anything changed (`--force` to restart regardless,
restart plus Slack re-verification when enabled and anything changed (`--force` to restart regardless,
`--rotate` to move to a different Slack app).

Open the portal URL printed by the CLI. Direct web/admin URLs are also printed for
debugging, but the portal URL is the prod-like path.

## Browser-only mode

Use browser-only mode when Slack is outside the test scope:

```bash
bash scripts/dev-instance.sh up --no-slack
bash scripts/dev-instance.sh doctor --no-slack
```

This starts the same core, local Postgres, sandbox, web UI, admin, portal, supervisor,
health checks, reloads, and teardown flow. It omits Slack credentials, Socket Mode,
canaries, exclusivity checks, and the pool requirement. `DEV_INSTANCE_NO_SLACK=1` is the
environment equivalent. Switching an existing worktree between modes requires `down`
followed by `up` with the desired mode.

## Sandbox: local Docker by default

The agent's `execute` sandbox runs as a **local Docker container** (`SANDBOX_BACKEND=local`)
Expand Down Expand Up @@ -146,14 +164,15 @@ DEV_INSTANCE_ALLOW_MOCK=1 bash scripts/dev-instance.sh up
DEV_INSTANCE_ALLOW_MEMORY=1 bash scripts/dev-instance.sh up
DEV_INSTANCE_WATCH=0 bash scripts/dev-instance.sh up
DEV_INSTANCE_RECLAIM_STALE=0 bash scripts/dev-instance.sh up
DEV_INSTANCE_NO_SLACK=1 bash scripts/dev-instance.sh up
```

## Env Discovery

The launcher reads values from, in priority order: exported shell env, the machine-global
`~/.config/qm/dev.env`, your login shell (for a model credential exported there), and this
worktree's `.env` (seeded from the main checkout in linked worktrees). Slack pool tokens
default to `~/.config/qm/slack-pool`.
worktree's `.env` (seeded from the main checkout in linked worktrees). When Slack is enabled,
pool tokens default to `~/.config/qm/slack-pool`.

When a cloud sandbox backend is configured it also validates that provider's access at
startup, refreshes a stale provider token from the provider CLI's own logged-in session
Expand All @@ -162,15 +181,15 @@ self-API calls can reach your local core. None of that runs on the default local

## After Startup

Report the slot, portal URL, Slack handle, and log directory. To test Slack-specific
Report the slot, portal URL, log directory, and Slack handle when enabled. To test Slack-specific
behavior, DM the printed `@<handle>` (on Alice's machine that's one of `@bot1 … @bot10`)
in `example.slack.com`; for admin and web behavior, open the printed portal URL. Tear down
with `bash scripts/dev-instance.sh down` when QA is finished.

## Troubleshooting

**Start with `dev doctor` (or `doctor --json`).** It runs the checks that used to take a
debugging session by hand — socket exclusivity (`num_connections`), a live canary
**Start with `dev doctor` (or `doctor --json`; pass `--no-slack` before the first browser-only boot).** It runs the checks that used to take a
debugging session by hand — socket exclusivity (`num_connections`) and a live canary when Slack is enabled,
round trip, env/git drift since boot, per-child health and restart counts, stale leases,
port squatters, machine-wide token orphans, Docker daemon — and prints a ranked diagnosis
with a remedy per finding. `doctor --fix` applies the safe ones (child restarts).
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"start": "node --env-file-if-exists=.env src/index.ts",
"dev": "SHUTDOWN_DRAIN_MS=2000 node --env-file-if-exists=.env --watch src/index.ts",
"dev-instance": "bash scripts/dev-instance.sh up",
"dev-instance:no-slack": "bash scripts/dev-instance.sh up --no-slack",
"dev-instance:status": "bash scripts/dev-instance.sh status",
"dev-instance:down": "bash scripts/dev-instance.sh down",
"worker": "node --env-file-if-exists=.env src/runs/worker-main.ts",
Expand Down
71 changes: 49 additions & 22 deletions scripts/dev/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ function parseCli() {
follow: { type: "boolean", short: "f", default: false },
fix: { type: "boolean", default: false },
sandbox: { type: "string", default: "auto" },
"no-slack": { type: "boolean", default: false },
"no-watch": { type: "boolean", default: false },
org: { type: "string" },
},
Expand All @@ -74,13 +75,13 @@ const command = positionals[0] ?? "up";
const store = poolStore();

const commandOptions: Record<string, readonly string[]> = {
up: ["json", "force", "strict", "rotate", "sandbox", "no-watch", "org"],
up: ["json", "force", "strict", "rotate", "sandbox", "no-slack", "no-watch", "org"],
down: ["json"],
status: ["json"],
restart: ["json"],
canary: ["json"],
logs: ["follow"],
doctor: ["json", "fix"],
doctor: ["json", "fix", "no-slack"],
};

const devServiceNames = [...CHILD_ORDER, "web-ui"];
Expand Down Expand Up @@ -115,6 +116,7 @@ function emitJson(payload: unknown): void {
}

const orgId = opts.org ?? process.env.DEV_INSTANCE_ORG_ID ?? "acme";
const slackEnabled = !opts["no-slack"] && process.env.DEV_INSTANCE_NO_SLACK !== "1";
const devCallerEnv = (): Record<string, string> => ({ ...callerEnvSnapshot(), DEV_INSTANCE_ORG_ID: orgId });

async function legacyTeardown(lease: LeaseInfo): Promise<void> {
Expand Down Expand Up @@ -175,6 +177,15 @@ function claimNext(exclude: Set<string>): string | null {
return null;
}

function claimNextLocal(exclude: Set<string>): string | null {
for (let i = 1; i <= 99; i++) {
const slot = `pool${i}`;
if (exclude.has(slot)) continue;
if (claimSlotLock(slot, store)) return slot;
}
return null;
}

function renderPhase(e: BootPhaseEvent): void {
if (opts.json || e.event !== "phase") return;
let mark = "…";
Expand All @@ -188,7 +199,7 @@ function renderPhase(e: BootPhaseEvent): void {
async function bootOnSlot(slot: string, worktree: string, branch: string): Promise<BootResult> {
const ports = slotPorts(slot);
const lock = lockDir(slot, store);
const tokens = slotTokens(slot, store);
const tokens = slackEnabled ? slotTokens(slot, store) : null;

writeFileSync(
join(lock, "meta"),
Expand All @@ -200,6 +211,7 @@ async function bootOnSlot(slot: string, worktree: string, branch: string): Promi
`web_port=${ports.web}`,
`admin_port=${ports.admin}`,
`portal_port=${ports.portal}`,
`slack=${slackEnabled ? "1" : "0"}`,
"booting=1",
`owner_pid=${process.pid}`,
`created_epoch=${nowEpoch()}`,
Expand All @@ -208,11 +220,13 @@ async function bootOnSlot(slot: string, worktree: string, branch: string): Promi
].join("\n"),
);

const swept = await sweepSlackTokenOrphans(tokens.appToken, new Set(), (m) => out(m));
if (swept.swept.length) out(`swept ${swept.swept.length} orphaned process(es) holding ${slot}'s Slack app token`);
if (tokens) {
const swept = await sweepSlackTokenOrphans(tokens.appToken, new Set(), (m) => out(m));
if (swept.swept.length) out(`swept ${swept.swept.length} orphaned process(es) holding ${slot}'s Slack app token`);
}

const callerEnv = devCallerEnv();
const canaryChannel = tokens.canaryChannel || callerEnv.DEV_INSTANCE_CANARY_CHANNEL || "";
const canaryChannel = tokens?.canaryChannel || callerEnv.DEV_INSTANCE_CANARY_CHANNEL || "";
writeFileSync(
join(lock, "boot-spec.json"),
JSON.stringify(
Expand All @@ -225,6 +239,7 @@ async function bootOnSlot(slot: string, worktree: string, branch: string): Promi
sandbox: opts.sandbox as "local" | "sprites" | "auto",
canaryChannel,
strict: opts.strict,
slack: slackEnabled,
},
null,
2,
Expand Down Expand Up @@ -256,7 +271,9 @@ async function bootOnSlot(slot: string, worktree: string, branch: string): Promi
};
}

out(`booting on slot ${slot} (@${tokens.handle || `agent-${slot}`})...`);
out(
tokens ? `booting on slot ${slot} (@${tokens.handle || `agent-${slot}`})...` : `booting on local slot ${slot}...`,
);
let result: BootResult | null = null;
await streamBootEvents(sock, (e) => {
renderPhase(e);
Expand All @@ -270,15 +287,18 @@ function printSuccess(result: BootResult, branch: string): void {
const lock = lockDir(result.slot, store);
const meta = readMeta(lock);
out("");
const hasSlack = result.slackEnabled !== false;
out(
`[ok] dev instance up -- slot ${result.slot} (VERIFIED: socket exclusive${result.canary ? `, canary ${result.canary.rttMs}ms round trip` : ", delivery unverified -- no canary channel"})`,
hasSlack
? `[ok] dev instance up -- slot ${result.slot} (VERIFIED: socket exclusive${result.canary ? `, canary ${result.canary.rttMs}ms round trip` : ", delivery unverified -- no canary channel"})`
: `[ok] dev instance up -- slot ${result.slot} (Slack disabled)`,
);
out(` branch : ${branch}`);
out(` portal : http://localhost:${ports.portal} -> prod-style front door: the assistant at / and /admin`);
out(
` core : http://localhost:${ports.core} (org=${orgId}, session_store=${meta.session_store}, run_store=${meta.run_store})`,
);
out(` slack : @${result.handle} -> mention it in example.slack.com to test`);
if (hasSlack) out(` slack : @${result.handle} -> mention it in example.slack.com to test`);
out(` web : http://localhost:${ports.portal}/ (direct: http://localhost:${ports.web})`);
out(` admin : http://localhost:${ports.portal}/admin/ (direct: http://localhost:${ports.admin})`);
out(` logs : ${lock}/{core,web,admin,portal,supervisor}.log`);
Expand Down Expand Up @@ -327,9 +347,10 @@ async function cmdUp(): Promise<number> {
const excluded = new Set<string>();
const waitMax = Number(process.env.DEV_INSTANCE_WAIT || 120);
for (let attempt = 1; attempt <= 3; attempt++) {
let slot = claimNext(excluded);
if (!slot && (await reclaimReclaimable())) slot = claimNext(excluded);
if (!slot && waitMax > 0 && attempt === 1) {
const claim = (): string | null => (slackEnabled ? claimNext(excluded) : claimNextLocal(excluded));
let slot = claim();
if (!slot && (await reclaimReclaimable())) slot = claim();
if (!slot && slackEnabled && waitMax > 0 && attempt === 1) {
out("");
out(`all pool apps are in use by other worktrees -- waiting up to ${waitMax}s for a free slot.`);
out(` this is normal contention, not an error. held now: ${takenSummary(store)}`);
Expand All @@ -338,14 +359,16 @@ async function cmdUp(): Promise<number> {
await sleep(5000);
waited += 5;
await reapStale();
if (await reclaimReclaimable()) slot = claimNext(excluded);
if (!slot) slot = claimNext(excluded);
if (await reclaimReclaimable()) slot = claim();
if (!slot) slot = claim();
}
}
if (!slot) {
emitJson({ ok: false, reason: "no free pool slot", held: takenSummary(store) });
out(
`no free pool app. Another worktree holds each one -- 'dev down' one of them, add a poolN.env, or raise DEV_INSTANCE_WAIT.`,
slackEnabled
? `no free pool app. Another worktree holds each one -- 'dev down' one of them, add a poolN.env, or raise DEV_INSTANCE_WAIT.`
: `no free local slot. Another worktree holds every slot -- 'dev down' one of them and retry.`,
);
return EXIT.noFreeSlot;
}
Expand Down Expand Up @@ -404,15 +427,15 @@ async function cmdDown(): Promise<number> {
return EXIT.ok;
}
const slot = mine.slot;
const tokens = slotTokens(slot, store);
const tokens = mine.meta.slack === "0" ? null : slotTokens(slot, store);
await teardownLease(mine);
const residue: string[] = [];
for (const [name, port] of Object.entries(slotPorts(slot))) {
if (name === "supervisor" || name === "prodProxy") continue;
const holders = portHolders(port);
if (holders.length) residue.push(`port ${port} (${name}) still held by pid(s) ${holders.join(",")}`);
}
const swept = await sweepSlackTokenOrphans(tokens.appToken, new Set(), (m) => out(m));
const swept = tokens ? await sweepSlackTokenOrphans(tokens.appToken, new Set(), (m) => out(m)) : { swept: [] };
if (residue.length) {
emitJson({ ok: false, slot, residue });
out(`[!] down completed with residue:\n ${residue.join("\n ")}`);
Expand All @@ -435,15 +458,19 @@ async function cmdStatus(): Promise<number> {
}
})();
const rows: Record<string, unknown>[] = [];
for (const slot of listSlots(store)) {
const leases = listLeases(store);
const slots = [...new Set([...listSlots(store), ...leases.map((lease) => lease.slot)])].sort(
(a, b) => Number(a.slice(4)) - Number(b.slice(4)),
);
for (const slot of slots) {
const lock = lockDir(slot, store);
const ports = slotPorts(slot);
const flag = readSlotFlag(slot, store);
if (!existsSync(lock)) {
rows.push({ slot, state: flag && slotFlagged(slot, store) ? `flagged(${flag.reason})` : "free", ports });
continue;
}
const lease = listLeases(store).find((l) => l.slot === slot);
const lease = leases.find((l) => l.slot === slot);
if (!lease) continue;
const sock = resolveSocketPath(lock);
if (await supervisorReachable(sock)) {
Expand Down Expand Up @@ -511,7 +538,7 @@ async function cmdStatus(): Promise<number> {
}
const taken = rows.filter((r) => r.state !== "free" && !String(r.state).startsWith("flagged")).length;
console.log("");
console.log(`${taken} taken / ${rows.length - taken} free / ${rows.length} pool apps total`);
console.log(`${taken} taken / ${rows.length - taken} free / ${rows.length} slots total`);
console.log("live = supervised + verified. Reclaim never touches a slot with a fresh supervisor heartbeat.");
return EXIT.ok;
}
Expand Down Expand Up @@ -597,10 +624,10 @@ async function main(): Promise<number> {
case "logs":
return await cmdLogs();
case "doctor":
return await runDoctor({ json: opts.json, fix: opts.fix, store });
return await runDoctor({ json: opts.json, fix: opts.fix, store, slack: slackEnabled });
default:
console.error(
"usage: dev [up|down|status|restart|canary|logs|doctor] [--json] [--force] [--rotate] [--strict] [--sandbox local|sprites|auto] [--no-watch] [--org id] [--fix]",
"usage: dev [up|down|status|restart|canary|logs|doctor] [--json] [--force] [--rotate] [--strict] [--sandbox local|sprites|auto] [--no-slack] [--no-watch] [--org id] [--fix]",
);
return EXIT.usage;
}
Expand Down
Loading