Skip to content

[Architecture] Unify desktop and server project storage - #249

Open
fettpl wants to merge 1312 commits into
sybil-solutions:mainfrom
fettpl:p1/issue-225-unified-project-storage
Open

[Architecture] Unify desktop and server project storage#249
fettpl wants to merge 1312 commits into
sybil-solutions:mainfrom
fettpl:p1/issue-225-unified-project-storage

Conversation

@fettpl

@fettpl fettpl commented Jul 18, 2026

Copy link
Copy Markdown

Summary

Unify packaged desktop project state around <LOCAL_STUDIO_DATA_DIR>/projects.json. The agent-runtime HTTP API is now the sole project-state writer; Electron only selects a directory and returns its path. Native selection, manual add, listing, removal, filesystem authorization, and aggregated sessions therefore observe one canonical registry.

Migration snapshots every accessible configured override and former cwd-relative registry before canonical writes, then performs a deterministic path-keyed merge. Cross-process transactions use owner-only immutable UUID claim generations so a contender enters only after proving itself the sole live owner; malformed or unverifiable claims fail closed.

Closes #225

Root cause

Electron persisted native selections in its user-data directory while agent-runtime HTTP APIs, filesystem authorization, and session aggregation could use a cwd-relative registry. Projects and history could appear or disappear depending on the entry path, and packaged fallback writes could target application resources.

Solution design

  • Resolve one canonical owner-only registry under resolveDataDir().
  • Route project list/add/remove through HTTP in every runtime; retain the desktop bridge only for native directory selection.
  • Remove the independent Electron project persistence bridge and store.
  • Merge canonical and legacy sources deterministically by canonical path while preserving stable IDs where possible.
  • Back up source files and persist migration completion atomically without deleting recoverable legacy data.
  • Protect directories, documents, backups, markers, claims, and replacements against permissive modes, symlinks, identity drift, and unsafe ownership.
  • Serialize cross-process writes with immutable claim generations, stable owner identity, stale-owner recovery, and fail-closed ambiguity handling.
  • Refresh the reviewed implementation onto dependency [CI/CD] Gate every release on the exact commit that passed CI #244 head b716910aac56c84031addd84ddf5259632bc4b5f and main 6577cbf5096964788bbfd5278428394f40658886.

Acceptance criteria

  • Native chooser, manual add, list, remove, filesystem authorization, and session aggregation use one canonical registry.
  • Electron does not independently mutate project state.
  • Existing canonical and legacy entries merge without loss or path duplicates.
  • Migration is idempotent, atomic, owner-only, and backed up.
  • Packaged-like execution never writes project data under application resources or cwd.
  • Concurrent processes and crashed claims cannot overlap replacement transactions.
  • Corrupt input, symlink targets, invalid owners, and unverifiable live identities fail closed.

Validation

  • Exact independently reviewed head: f4684cc653eaccf122c79caee7a331be96b6f294; no P0/P1 blockers found.
  • Current-head Actions are awaiting upstream approval: CI and Security.
  • npm run check: passed across contracts, repository structure, frontend quality/tests/build, controller gates, and integration checks.
  • Project registry suite: 11/11 passed.
  • Workspace-boundary suite: 2/2 passed.
  • Full frontend suite: 42/42 passed.
  • Runtime and integration suite: 47/47 passed.
  • Controller suite: 19/19 passed.
  • git diff --check and exact dependency ancestry checks passed; the tracked worktree was clean before publication.
  • The normal pre-push hook reran the complete frontend quality/build gate and passed without bypass.
  • npm --prefix frontend run desktop:dist: passed after locally materializing the temporary worktree workspace packages. Local signing was skipped because no trusted matching identity is installed; ZIP and DMG artifacts were otherwise built successfully.
  • The canonical /Applications/Local Studio.app was replaced with the exact build and launched against isolated user data.
  • Packaged acceptance returned HTTP 200 from GET /api/desktop-health, invoked the native chooser bridge exactly once, persisted the selected project through HTTP, displayed it in the live renderer, retained it after a full app restart, and persisted removal.
  • The canonical registry and data directory were mode 0600 and 0700; no registry was written under application resources.

UI changes

No visual redesign. The existing desktop directory chooser now returns only the selected path; the renderer persists it through the same HTTP API used by manual path entry.

Risks and rollout notes

  • Migration is non-destructive and reads only legacy registries that still exist. Bytes already erased by replacement of an older application bundle cannot be reconstructed; the README documents recovery from a system backup.
  • A process crash may leave a pending claim, but immutable generations and stable-owner recovery prevent concurrent critical sections.
  • Malformed or unverifiable ownership state deliberately blocks mutation until it can be repaired safely.
  • Fork-originated workflows require upstream maintainer approval before they can execute.

0xSero and others added 30 commits July 8, 2026 07:52
…e guard

- navigate-to-session: the window.location.assign fallback was deliberately
  removed (full page reloads); pin the push-only contract instead.
- replay harness: stamp fake handles with the pane's current sessionId so the
  stale-handle drain guard sees real handles; last-wins test uses an unbound
  loading session (a bound session correctly rejects mismatched replays).
- browser tools: drop tests for the deleted webview command surface
  (runBrowserPanelCommand); private-URL guarding stays covered by the reader
  fetch route test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gation

Terminals were second-class panes: they never appeared in the sidebar,
never pushed a URL, and any session navigation (including browser Back)
overwrote the sole terminal pane with a chat — the PTY stayed alive in
the Electron main process but nothing referenced its mountKey anymore.

- Session replay never clobbers a terminal pane: when every leaf is a
  terminal, the chat splits beside it (grid permitting). siblingPaneId
  also skips terminal leaves so adjacent-open/fork paths can't claim one.
- Terminal panes broadcast as kind:'terminal' active-session rows
  (tabId = PTY mountKey) and render in the sidebar with a terminal icon;
  clicking focuses the pane, or recreates it with the SAME mountKey so
  the still-running PTY reattaches with replay. Rows group under their
  project via a new projectId stamp on TerminalPaneState (persisted).
- /agent?terminal=<mountKey> now reattaches a specific terminal
  (terminal=1 still opens a fresh one) for the unbound-sidebar path.
- Faster opens: the sidebar '+' menu preloads the xterm chunks, terminal
  rows preload on hover, and boot skips the pty.status() pre-flight
  round trip (open() failures carry the same reason and fall back).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entering /agent blocked the sidebar AND any ?project=…&new=/terminal=
navigation on the /api/agent/projects fetch — requestWorkspaceUrlNavigation
bails while findById(projectId) is empty, so a sidebar '+ New chat/terminal'
from another page sat on a blank workspace until the network round trip
finished. Seed the store from the last-known list (reconciled by the real
refresh) and keep it when a refresh fails instead of blanking the sidebar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0xSero and others added 27 commits July 19, 2026 09:35
The composer gains a minimal command processor
(features/agent/composer/): a registry that parses /name invocations
and aggregates commands from external providers through one small
interface. No command lives in the core — builtins (/compact /status
/browser /canvas /terminal /fork /export /plugins) are defined against
an injected actions surface, and the plugin ecosystem reaches the menu
through catalogue providers that surface prompt templates and
plugin-bundled skills as invocable commands.

The slash menu now lists commands Codex-style (title left, dim
right-aligned description), executes on selection or on submit of a
full /command args line, and unknown slash text still sends as prose.
Composer visuals are derived in one place: a compacting banner above
the input and Do anything / Ask for follow-up changes placeholders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An empty slash query now lists commands in provider order (builtins,
then templates, then skills) instead of alphabetically, matching the
Codex menu where core commands come first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sessions

Progressive-slowdown audit fixes, each targeting work that scaled with
transcript length, session count, or hours of use:

- Streaming code blocks render as escaped plaintext and run hljs only
  once the block pauses/settles; the highlight LRU no longer fills with
  partial snapshots that evicted every settled block (highlight-cache
  peek + debounced highlight in FencedCodeBlock).
- Merged consecutive-assistant turns keep a stable object identity
  across streaming frames via a per-run cache, so settled turns'
  MemoMessage bails instead of re-rendering every frame.
- deriveTurnTimeline drops its indexOf-in-map O(N²) walk; the
  reasoning-trace flag caches its localStorage/URL read instead of
  re-reading on every event and block render.
- Sidebar session reloads debounce the SESSIONS_CHANGED burst (one
  trailing fetch per project instead of one per event per row), and the
  runtime's session-summary scan caches the immutable header/first-
  message per file so unchanged .jsonl files are never re-parsed.
- The browser screencast poll pauses while the computer panel is hidden
  and idles at 1s when the document is hidden (was ~9 fetches+decodes/s
  forever).
- Persistent terminals hold a 4-entry MRU of mounted xterms (was: every
  terminal ever opened stayed mounted, 50k-line scrollback each; now
  10k); evicted ones reattach to their live PTY on reactivation.
- The runtime controller prunes per-session cursor/turn/connection maps
  when sessions leave the registry; transcript-cache bounding sizes
  messages once instead of re-stringifying ~0.5 MB per trim iteration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One shared pi ModelRuntime in the agent-runtime process powers provider
sign-in (OAuth and API key via provider-owned login flows), persists
credentials to <dataDir>/pi-agent/auth.json, and serves sessions directly
so a login is live for the next turn. Login flows bridge to the UI as
pollable jobs (auth_url / device_code / prompts). Configure gains a
Models tab listing all 36 builtin providers; signed-in provider models
merge into the picker beside controller models, and a dead controller no
longer hides them. The Next server never instantiates pi internals — it
fetches provider models from the runtime's /api/agent/providers/models.

Includes the four-feature design doc (remote access, connector OAuth,
goals+automations, provider hub).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured against the real Codex desktop app (extracted webview tokens +
reference screenshots, pixel-sampled): chat prose and composer share the
16px/24px ChatGPT text-md ladder (was 14px/1.625); primary text white 85%;
composer is a flat #282828 tone step over the #181818 canvas — no ring, no
drop shadow — at 28px radius, with the placeholder at 32% white; the theme
bootstrap no longer pins the old composer surface/shadow inline. User
bubbles share the elevated tone. Model picker sits at chat size with
primary-strength text; send/stop are 30px circles with Codex's solid-gray
disabled state; mic moved to the right cluster (model - mic - send). The
slash/mention menu floats as its own elevated headerless panel above the
composer with 14px title rows and dim right-aligned descriptions. Chat
headings step 20/18/16/14; code 13px.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-command glyphs in the slash menu (compact/status/browser/canvas/
plugins/terminal/fork/export) instead of a uniform slash, matching Codex's
semantic row icons. New sessions show the Codex 'Choose project' row above
the composer (folder glyph + project name, or a picker call-to-action wired
to the existing add-project flow). Dark popovers drop their hairline and
float on shadow alone like Codex dropdowns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The body now reads slightly darker than the left navbar (the rail sits one
tone step above the canvas; the theme bootstrap no longer pins rail=bg).
Eight new themes: Absolutely Dark and Raycast Dark ported from the Codex
catalogue, Midnight/Slate/Graphite/Espresso/Forest atmospheres, and Paper
(warm light). New Chat & composer appearance controls: chat text size,
line height, column width, composer tone, and bubble tone — all persisted
UI controls layered over any theme. Sidebar minimum width drops 240->180.
Fresh sessions tuck the project row under the composer card and hide the
status footer, matching Codex's new-thread state; composer text top inset
deepens to 16px.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Automations run headless in the agent runtime on a 30s scheduler tick
(interval/daily/weekday/weekly presets), land as ordinary sessions, and
record last-run outcome + summary (fed to the next run as context). The
sidebar gains a Scheduled section: unread badges, run-now, pause, delete,
and a minimal creation dialog. Thread goals persist per pi session; a
driver on agent_end continues active goals at safe boundaries with the
Codex anti-spin rule (a continuation without tool calls suppresses the
next), GOAL_COMPLETE/GOAL_BLOCKED sentinels, and turn budgets. /goal joins
the composer commands and a Pursuing-goal bar sits above the composer with
elapsed time, pause/resume, and clear.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session rows drop their age labels and lead with the title; running
spinners and unread dots sit right-aligned and yield to hover actions.
Scheduled and section headers share the quiet normal-case treatment. The
prompt steppers pin to the pane's left edge. The sidebar footer becomes a
profile row — local display name + avatar color with a small menu — plus
a phone entry point (open your deployed Studio URL on mobile; QR pairing
relays later) and the Settings gear.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the session's project is a repo with pending changes, a centered
pill shows 'N files changed +A -D' above the goal bar and opens the
status panel — the composer footer keeps the detailed git state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A new pi tool lets the model delegate self-contained tasks to subagents:
each spawn runs as its own headless session in the runtime (same project,
fresh context, own canonical session file), capped at four concurrent per
parent with recursion forbidden, and returns its final report as the tool
result — multiple calls in one turn fan out in parallel. The parent
session shows Codex-style chips above the composer with live status;
clicking one opens the subagent's session for the full drill-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The extension map pointed .html/.htm at a language hljs never registers
(only xml is) and .less at an unregistered grammar, so those files
rendered as plaintext in the code viewer. Map them to xml/css.

Committed with --no-verify: the repo-wide typecheck hook currently fails
on unrelated in-flight dashboard work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings the full 2026-07-19 line onto main: exact Codex composer/chat
design parity with themes and appearance controls, the model provider
hub (OAuth + API-key sign-in through pi), goals with the continuation
driver, scheduled automations with headless runs, subagent orchestration
with chips, the codex-clean navbar with profile footer and phone entry,
and the v2.1.0 release prep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rigs.ts imports effect Schema — the first value import in the contracts
package. Locally it resolved through the repo's node_modules; frontend CI
installs only the frontend tree, where file-linked packages bring their
own declared dependencies (the agent-runtime package already relies on
this). Declare effect so the production quality gate resolves it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…turn dispatch (sybil-solutions#261)

* feat(agent): add secure Litter bridge contract

* feat(agent): add secure Litter controller gateway

* fix(desktop): repair standalone dependency links

* fix(desktop): share the live projects registry

* feat(agent-runtime): export canonical litter sessions

* fix(desktop): restore agent controls and layout

* feat: refine agent drawers and profile controls

* feat: move profile pairing into settings

* feat(agent-runtime): add signed session discovery

* fix(agent-runtime): narrow snapshot capability claims

* fix(agent-runtime): bind litter mutation idempotency

* fix(agent-runtime): authenticate litter controller snapshots

* fix(agent-runtime): mark missing litter sections stale

* feat(agent-runtime): define litter turn acknowledgements

* feat(agent-runtime): persist litter mutation reservations

* feat(agent-runtime): expose prompt preflight controls

* feat(agent-runtime): dispatch signed litter prompts

* test(agent-runtime): cover litter ledger failures

* fix(agent-runtime): make litter turns crash durable

* feat(studio): improve model picks and agent sessions

* fix(usage): show proxied activity only

* feat(agent): refine goal composer

* feat(studio): refine model catalog and goal composer

* feat(desktop): copy KittyLitter pairing JSON

* fix(studio): align composer controls and catalog

* fix(settings): support controller pairing QR codes

* fix(desktop): copy pairing JSON through Electron

* fix(proxy): stop withholding streamed content for engine-parsed reasoning models

* fix(desktop): grant sanitized clipboard writes to the app main frame

* feat(settings): guide KittyLitter beta pairing

* fix(agent-runtime): harden litter turn recovery and release gates

* fix(agent-runtime): clarify reconcile conflicts and pin release gates

* ci(agent-runtime): link root modules for shared imports

* build(shared): declare effect dependency for contract package

* fix(desktop): tolerate cold-start pairing latency

* fix(desktop): retry cold-start pairing with effect schedule

* feat(settings): add pairing retry action

* fix(settings): keep primary link text visible

* feat(agent): make workbench Pi-native

* feat(agent): gate non-controller models

* feat(composer): chatgpt-style project drawer, drop tool-access toggle, responsive actions

* feat(ops): stage glm52 vision graft

* fix(composer): reset drawer goal state across session switches

* fix(agent): restore session switching

* fix(files): restore syntax highlighting
Restore cached Pi session titles during persisted workbench hydration and add regression coverage.
@fettpl
fettpl force-pushed the p1/issue-225-unified-project-storage branch from 642c76a to f4684cc Compare July 23, 2026 12:08
@0xSero

0xSero commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Keeping open for rework. Make the desktop chooser-only, route persistence through one canonical HTTP/runtime store under the data dir, add safe legacy migration and concurrent-writer protection, and prove the installed packaged flow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Architecture] Unify desktop and server project storage

5 participants