fix(tests): stop test cleanup from deleting the production exo config - #186
Conversation
Root cause of the recurring "agent tasks wipe my API tokens" incidents: clean_test_dbs() in scripts/run-tests.sh ran `rm -f` on the GLOBAL per-user app dirs — including ~/Library/Application Support/exo, the packaged app's REAL user data — before and after every `npm test`. That deleted the production exo-config.json (anthropicApiKey, exaApiKey, ollamaCloud.apiKey, hostler.apiKey, prompts, all settings), forcing a from-scratch re-setup whenever the app couldn't silently resurrect it. The cleanup predates the .dev-data isolation (May 2026); since dev/test state moved to .dev-data/, the global cleanup could ONLY ever destroy production data. Changes: - scripts/run-tests.sh: clean_test_dbs() now only cleans the project-local .dev-data/ (worker demo DBs + test config). No global paths, by construction. - tests/e2e/undo-send.spec.ts: resetTestEnvironment() cleans .dev-data/data instead of the four global app dirs. - tests/unit/no-global-data-dirs.spec.ts (new): regression guard — fails if any file in scripts/ or tests/ references a global per-user app-data dir. - src/main/data-dir.ts + src/main/index.ts: EXO_USER_DATA_DIR override, honored even when packaged. A locally-built .app shares productName (and therefore userData) with the real install, so packaged smoke tests were writing Chromium profile state, demo DBs, logs, and electron-store migrations into production. - tests/packaged/smoke.spec.ts: launches with EXO_USER_DATA_DIR pointing at project-local .packaged-test-data/ (gitignored). - .github/workflows/ci.yml: fix EXO_PACKAGED_BINARY path (dist/ -> release/, matching electron-builder's configured output). The packaged smoke job has been silently skipping on CI because the binary path never existed. - docs/LOCAL_DEVELOPMENT.md: correct packaged-binary paths and document the userData isolation requirement. - CLAUDE.md: hard rules — cleanup code must be project-local; never launch a packaged binary without EXO_USER_DATA_DIR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Greptile SummaryThis PR fixes a production data wipe:
Confidence Score: 5/5Safe to merge — all cleanup code now targets project-local paths only, the EXO_USER_DATA_DIR override is validated before use, and the packaged smoke tests are properly isolated. The core fix (re-rooting cleanup to .dev-data/) is straightforward and verifiably correct. The new override mechanism is a small leaf module with a single absolute-path validation, and it's tested both in unit tests and end-to-end via smoke.spec.ts. The regression guard is thoughtfully layered — app-dir name patterns remain the primary defense even where the $HOME shell pattern has a minor gap. tests/unit/no-global-data-dirs.spec.ts — the
|
| Filename | Overview |
|---|---|
| scripts/run-tests.sh | clean_test_dbs() rewritten to only target .dev-data/ (project-local), and EXO_USER_DATA_DIR is unset at script entry to prevent stale exports from corrupting per-worker isolation. |
| src/main/user-data-override.ts | New leaf module that reads EXO_USER_DATA_DIR and validates it is absolute; shared by data-dir.ts and logger.ts to avoid drift. |
| src/main/data-dir.ts | getDataDir() now checks getUserDataOverride() first, honoring EXO_USER_DATA_DIR even in packaged builds; resolution order well-documented. |
| src/main/index.ts | app.setPath("userData") is now triggered by either is.dev OR EXO_USER_DATA_DIR being set; adds a loud warn log when the override is active to surface stale exports. |
| tests/unit/no-global-data-dirs.spec.ts | New regression guard scanning tracked files in scripts/, tests/, benchmarks/ for global app-dir references; the |
| tests/packaged/smoke.spec.ts | Smoke test now creates .packaged-test-data/ as isolated userData, cleans it at beforeAll for a deterministic start, and verifies the override took effect via app.getPath("userData"). |
| tests/e2e/undo-send.spec.ts | resetTestEnvironment() now targets .dev-data/data instead of four global per-user app dirs; correctly uses __dirname via fileURLToPath for ESM compatibility. |
| tests/e2e/launch-helpers.ts | env is now constructed explicitly and EXO_USER_DATA_DIR is deleted from it before launch, preventing a stale export from collapsing per-worker isolation. |
| tests/unit/data-dir.spec.ts | New describe block verifies EXO_USER_DATA_DIR: absolute path is returned verbatim, relative path throws; beforeEach/afterEach correctly save and restore the env var. |
| .github/workflows/ci.yml | EXO_PACKAGED_BINARY corrected from dist/ to release/ (electron-builder actual output dir); fixes the silent CI skip that kept packaged smoke tests from ever running. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[App / Test starts] --> B{EXO_USER_DATA_DIR set?}
B -- Yes --> C[getUserDataOverride\nvalidate absolute path]
C -- invalid/relative --> D[throw Error\nfail loudly]
C -- valid --> E[use override path\n.packaged-test-data/]
B -- No --> F{app.isPackaged?}
F -- No / dev --> G[use .dev-data/\nproject-local]
F -- Yes / packaged --> H[use app.getPath userData\nreal install dir]
subgraph run-tests.sh / launch-helpers.ts
I[unset EXO_USER_DATA_DIR\nbefore any test launch]
end
subgraph smoke.spec.ts
J[rmSync + mkdirSync\n.packaged-test-data/]
K[launch binary with\nEXO_USER_DATA_DIR set]
L[assert app.getPath userData\n== USER_DATA_DIR]
J --> K --> L
end
subgraph no-global-data-dirs.spec.ts
M[git ls-files scripts/ tests/ benchmarks/]
N{any forbidden\nglobal path pattern?}
M --> N
N -- Yes --> O[test FAILS]
N -- No --> P[test PASSES]
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[App / Test starts] --> B{EXO_USER_DATA_DIR set?}
B -- Yes --> C[getUserDataOverride\nvalidate absolute path]
C -- invalid/relative --> D[throw Error\nfail loudly]
C -- valid --> E[use override path\n.packaged-test-data/]
B -- No --> F{app.isPackaged?}
F -- No / dev --> G[use .dev-data/\nproject-local]
F -- Yes / packaged --> H[use app.getPath userData\nreal install dir]
subgraph run-tests.sh / launch-helpers.ts
I[unset EXO_USER_DATA_DIR\nbefore any test launch]
end
subgraph smoke.spec.ts
J[rmSync + mkdirSync\n.packaged-test-data/]
K[launch binary with\nEXO_USER_DATA_DIR set]
L[assert app.getPath userData\n== USER_DATA_DIR]
J --> K --> L
end
subgraph no-global-data-dirs.spec.ts
M[git ls-files scripts/ tests/ benchmarks/]
N{any forbidden\nglobal path pattern?}
M --> N
N -- Yes --> O[test FAILS]
N -- No --> P[test PASSES]
end
Reviews (4): Last reviewed commit: "fix(tests): repair doubly-escaped guard ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
✅ Pre-PR verification — PASS
Agentic verification — summaryAgentic verification — verify-diff
Summarycategory=F. This PR is a safety/infra fix: (1) Agentic verification — literal trace[2026-07-15T19:49:12.304Z] text: Good — no console errors, API available. Let me verify the {"configLoaded":false,"error":"TypeError: window.api.settings.getConfig is not a function"}[2026-07-15T19:49:17.172Z] tool#9: mcp__chrome-devtools__evaluate_script {"topLevel":["_debugLog","gmail","analysis","drafts","compose","emails","style","contacts","search","settings","memory","sender","onboarding","accounts","sync","prefetch","backgroundSync","splits","snippets","theme","auth","extensions","archiveReady","snooze","network","scheduledSend","calendar","attachments","agent","defaultMailApp","updates","outbox","find","usage"],"settingsKeys":["get","set","validateApiKey","validateOllamaKey","getPrompts","setPrompts","onPromptsChanged","removePromptsChangedListener","getEA","setEA","exportLogs","validateGithubToken","testOpenclawConnection"]}[2026-07-15T19:49:21.352Z] tool#10: mcp__chrome-devtools__evaluate_script {"configLoaded":true,"hasAnthropicKey":false,"configKeys":["success","data"]}[2026-07-15T19:49:26.856Z] tool#11: mcp__chrome-devtools__evaluate_script {"settingsSuccess":true,"settingsDataKeys":["maxEmails","model","modelConfig","dryRun","analysisPrompt","draftPrompt","enableSenderLookup","senderLookupProvider","syncDraftsToGmail","theme","inboxDensity","undoSendDelay","sendAndArchive","showExoBranding","autoDraft","keyboardBindings","configVersion","posthog","lastSelectedAccountId"],"accountsCount":1,"firstAccountEmail":"exoemailtest@gmail.com"}[2026-07-15T19:50:00.820Z] text: Settings load successfully ( {"emailsApi":["archive","batchArchive","archiveThread","trash","batchTrash","setStarred","setRead","getThread","search","searchRemote","blockSender","unblockSender","listBlockedSenders"],"syncApi":["init","start","stop","now","status","setInterval","getEmails","getSentEmails","prefetchBodies","onNewEmails","onNewSentEmails","onStatusChange","onEmailsRemoved","onEmailsUpdated","onDraftsRemoved","onActionFailed","onActionSucceeded","onSyncProgress","removeAllListeners"]}[2026-07-15T19:50:11.777Z] tool#14: mcp__chrome-devtools__evaluate_script {"emailsFromDB":"object","firstEmailId":null,"exoErrors":[]}[2026-07-15T19:50:32.154Z] text: I have sufficient evidence. The app is fully operational: settings loaded from DB ( The non-runtime changes ( {"verdict":"pass","summary":"category=F. This PR is a safety/infra fix: (1) |
Multi-agent review of the prod-data-protection PR surfaced real gaps in
the protections themselves:
- gmail-client: skip migrateOldConfigIfNeeded() when EXO_USER_DATA_DIR is
set — the override made newDir never equal ~/.config/exo, so a packaged
smoke run on a machine with legacy config would copy real OAuth
tokens/credentials out of production into the scratch dir.
- packaged smoke: assert the userData redirect actually engaged
(app.getPath("userData") === override) — previously nothing failed if
the override silently regressed and the suite wrote to production.
- packaged smoke: a set-but-missing EXO_PACKAGED_BINARY now throws
instead of skipping — the silent skip is how the dist/->release/ gap
stayed green in CI indefinitely.
- packaged smoke: wipe .packaged-test-data/ before each run so it stays
a deterministic first-run test (also raised by Greptile).
- data-dir/logger: share EXO_USER_DATA_DIR resolution via a leaf helper
(user-data-override.ts) so validation can't drift between the two
resolvers; logger honors the override for import-time log writes.
- index: log loudly when the override is active — a leftover export
otherwise looks like total data loss with zero evidence.
- launch-helpers/run-tests.sh: strip/unset EXO_USER_DATA_DIR for e2e —
a leftover export would make all parallel workers share one data dir.
- no-global-data-dirs guard: scan tracked files via git ls-files (kills
machine-local false positives from untracked scratch files), add
homedir()/$HOME/segment-joined/Windows patterns, cover benchmarks/.
- data-dir.spec: behavior tests for the override (absolute honored,
relative throws).
- undo-send.spec: move __dirname below imports (Greptile).
- CLAUDE.md/data-dir header: document EXO_USER_DATA_DIR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review summary (multi-agent /review + /reviewloop)Ran 5 parallel review passes (testing, maintainability, security specialists + red team + adversarial) over the diff, plus Greptile triage. Codex cross-model pass was unavailable (broken local install). Fixed in 31e1fab
Considered, not done (deliberate)
Notes
|
…rod-token-wipe # Conflicts: # scripts/run-tests.sh
Greptile caught that the shell-escaped macOS path pattern in no-global-data-dirs.spec.ts was doubly-escaped: the regex source ended up as `Application\\\\ Support/exo` (two literal backslashes), so the guard variant advertised for `Application\ Support/exo` never matched anything. Halve the backslash count so the regex matches one literal backslash. Add a self-test that asserts every FORBIDDEN pattern matches its canonical bad example — a pattern that matches nothing is a silently dead guard, and this is exactly how this one shipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Fixes the recurring "agent tasks wipe my API tokens" incidents. Forensics (prod-app logs + filesystem birth-times + agent-session transcripts) traced every wipe to one line:
clean_test_dbs()inscripts/run-tests.shranrm -fon the global per-user app dirs — including~/Library/Application Support/exo, the packaged app's real user data — before and after everynpm test/npm run test:e2e, from any worktree. That deleted the productionexo-config.json(anthropicApiKey, exaApiKey, ollamaCloud.apiKey, hostler.apiKey, prompts, EA settings), forcing a from-scratch re-setup. It re-occurred twice on 2026-07-15 alone (test runs at 10:13 and 10:21 local, confirmed via prod-log error fingerprints).The Gmail OAuth tokens were never touched — their same-second mtime updates are the app's routine hourly access-token refresh. The wiped "API tokens" were the settings-store keys.
The cleanup predates the
.dev-data/isolation (May 2026). Since dev/test state moved to.dev-data/, the global cleanup could only ever destroy production data.Changes
scripts/run-tests.sh—clean_test_dbs()now cleans only the project-local.dev-data/(worker demo DBs + test config). No$HOME-derived paths, by construction. Comment documents the incident.tests/e2e/undo-send.spec.ts—resetTestEnvironment()cleans.dev-data/datainstead of the four global app dirs.tests/unit/no-global-data-dirs.spec.ts(new) — regression guard: fails if any file underscripts/ortests/references a global per-user app-data dir (Application Support/exo,Application Support/Electron,.config/exo,.config/Electron).src/main/data-dir.ts+src/main/index.ts— newEXO_USER_DATA_DIRoverride (absolute path, validated), honored even when packaged. Rationale: a locally-built .app has the same productName as the real install, so it resolves the same userData dir — the packaged smoke tests were writing Chromium profile state, demo DBs, logs, and electron-store config migrations into production.tests/packaged/smoke.spec.ts— launches the packaged binary withEXO_USER_DATA_DIRpointing at project-local.packaged-test-data/(gitignored)..github/workflows/ci.yml—EXO_PACKAGED_BINARY: dist/linux-unpacked/exo→release/linux-unpacked/exo. electron-builder's configured output isrelease/(package.jsonbuild.directories.output), so the packaged smoke job has been silently skipping on CI (the spec skips when the binary path doesn't exist).docs/LOCAL_DEVELOPMENT.md— corrected packaged-binary paths (dist/→release/) and documented the userData-isolation requirement.CLAUDE.md— hard rules from the incident: cleanup code must target project-local paths only; never launch a locally-built packaged binary withoutEXO_USER_DATA_DIR.Outside this PR (defense in depth, already live locally)
npm test/run-tests.shin worktrees whoserun-tests.shstill carries the old cleanup (every open branch is dangerous until it merges this), and denies destructive shell commands referencing the prod dir.Testing
npm test): 1468 passed — integration + e2e + unit, including the new regression guard.npm run typecheck,npm run lint: clean.Notes for review
resetTestEnvironment()intentionally keeps the same worker-scoped semantics, just re-rooted to.dev-data/data.EXO_USER_DATA_DIRenv override is a deliberate escape hatch in packaged builds; a local attacker with env control already has user-level file access, so it grants nothing new.🤖 Generated with Claude Code
Pre-PR verdict: PASS
full6d2db48