Skip to content

[Bug] Unify the visible desktop browser and agent browser tools - #254

Open
fettpl wants to merge 1313 commits into
sybil-solutions:mainfrom
fettpl:p1/issue-236-unified-browser-surface
Open

[Bug] Unify the visible desktop browser and agent browser tools#254
fettpl wants to merge 1313 commits into
sybil-solutions:mainfrom
fettpl:p1/issue-236-unified-browser-surface

Conversation

@fettpl

@fettpl fettpl commented Jul 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • Replaces the desktop-only Electron <webview> path with the same CDP screencast surface used by agent browser tools, so the visible pane and tool calls observe and control one page.
  • Removes the retired webview capability and guest-attachment path while retaining the DNS-pinned browser policy supplied by [Security] Enforce DNS-pinned browser policy across redirects and subresources #252 for navigation, redirects, and subresources.
  • Routes visible navigation, location, viewport, mouse, keyboard, and file-link behavior through the managed browser host without restoring file:// access.
  • Keeps browser state in a feature-local external store whose Effect-owned polling lifecycle survives page/tab changes without replaying cached URLs.
  • Serializes explicit navigation requests in issue order and settles the toolbar location only from the first poll begun after the latest navigation completes, including immediate redirects, failures, queued requests, and remounts.
  • Adds focused regressions plus real-Chromium red-to-green frame acceptance for the shared visible/tool page.

Closes #236

Depends on #252, whose exact browser-network-policy commit is the base of this branch.

Root cause

The desktop rendered an Electron webview while agent browser tools controlled a separate managed Chromium page. An initial unification attempt also let older navigation requests mutate the shared host after newer intents and used an exact-URL latch that could suppress a legitimate redirect forever. The final implementation has one host-authoritative surface, ordered explicit mutations, and a poll-settlement barrier rather than URL equality.

Validation

  • Focused [Architecture] Use one protected browser surface for users and agents #236 frontend regressions: 17/17 passed.
  • Full frontend suite: 52/52 passed.
  • Independent temporary race reproductions: 3/3 passed for failed A followed by B, three queued navigations with a middle failure, and unsubscribe/remount during a queued sequence.
  • Real-Chromium browser request acceptance: 2/2 passed with decoded red A and green B frames.
  • Full integration suite: 28/28 passed.
  • npm --prefix frontend run check:quality passed.
  • npm run check passed.
  • A fresh independent P0/P1 review approved exact SHA 79e18f9115ef000b9525fac76b7015bc199fd678 with no blocking findings.
  • Production desktop:dist completed successfully, including ZIP and DMG generation. Local signing was skipped because no valid trusted signing identity was available; release signing behavior is unchanged.
  • The canonical /Applications/Local Studio.app was replaced and relaunched; GET /api/desktop-health returned HTTP 200 with {"ok":true}.
  • Live installed-UI acceptance navigated the unified surface to Example Domain, rendered a 12,799-byte JPEG screencast, switched focus Status → Browser, and restored the same synchronized URL and frame.
  • Generated desktop artifacts were removed and the shared reinstall lock was released; the worktree remains clean.

UI changes

The desktop browser pane now renders the managed CDP screencast used by agent tools. Navigation and input remain visible in that same pane, and project files continue to open through the existing Files/canvas viewers rather than browser file:// access.

Acceptance criteria

  • The visible desktop pane and agent browser tools use one protected managed page.
  • Navigation from either surface converges in the other without replaying cached URLs.
  • Concurrent explicit navigations preserve user issue order and latest intent.
  • Immediate redirects and failed navigation recover from post-settlement host state.
  • The retired webview capability and guest attachment path are removed.
  • Focused, integration, quality, full-repository, independent-review, and installed-desktop health gates pass.

Risks / rollout notes

Workflow state

The exact-head fork workflows require maintainer approval before jobs can execute:

0xSero and others added 30 commits July 8, 2026 07:52
@0xSero
fix: decouple agent browser tools from panel state and widen chromium…
6be54d1 
… discovery
@0xSero
feat: cap workspace splits at three columns by two rows
24867f2
@0xSero
fix: clamp restored workspace layouts to the split grid cap
3dec544
@0xSero
test: cover terminal splits, grid caps, chat isolation, and keybinds
8f84a99
@0xSero
perf: stop syncing transcript and session caches through durable ui p…
9e23009 
…refs
@0xSero
fix: keep controller credentials out of remote ui preference sync
c238f0e
@0xSero
perf: split transcript crash cache into per-session keys
0b00611
@0xSero
test: cover per-session transcript cache
5f1729b
@0xSero
perf: mount sidebar terminal owners lazily on first activation
65d8d0c
@0xSero
feat: let sidechats pick their own model
73f2594
@0xSero
fix: register xml grammar so html files highlight
10524ca
@0xSero
fix: open sidebar sessions in place instead of hard page reloads
1d63131
@0xSero
feat: infer model family avatars for local recipe ids
00c8d74
@0xSero
fix: drop stale replay drains when a pane swaps sessions
40cdb7e
@0xSero
test: guard replay queue against stale pane handles
52e332f
@0xSero
fix: fall back through avatar owner candidates for local model paths
9ac7633
@0xSero
test: align terminal pane navigation tests with chat isolation
7192e00
@0xSero @claude
test: align e2e suite with removed hard-nav fallback and replay handl…
eae9167 
…e guard

- navigate-to-session: the window.location.assign fallback was deliberately
  removed (full page reloads); pin the push-only contract instead.
- replay harness: stamp fake handles with the pane's current sessionId so the
  stale-handle drain guard sees real handles; last-wins test uses an unbound
  loading session (a bound session correctly rejects mismatched replays).
- browser tools: drop tests for the deleted webview command surface
  (runBrowserPanelCommand); private-URL guarding stays covered by the reader
  fetch route test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
feat: surface terminals as sidebar sessions and keep them across navi…
2cfa5d9 
…gation

Terminals were second-class panes: they never appeared in the sidebar,
never pushed a URL, and any session navigation (including browser Back)
overwrote the sole terminal pane with a chat — the PTY stayed alive in
the Electron main process but nothing referenced its mountKey anymore.

- Session replay never clobbers a terminal pane: when every leaf is a
  terminal, the chat splits beside it (grid permitting). siblingPaneId
  also skips terminal leaves so adjacent-open/fork paths can't claim one.
- Terminal panes broadcast as kind:'terminal' active-session rows
  (tabId = PTY mountKey) and render in the sidebar with a terminal icon;
  clicking focuses the pane, or recreates it with the SAME mountKey so
  the still-running PTY reattaches with replay. Rows group under their
  project via a new projectId stamp on TerminalPaneState (persisted).
- /agent?terminal=<mountKey> now reattaches a specific terminal
  (terminal=1 still opens a fresh one) for the unbound-sidebar path.
- Faster opens: the sidebar '+' menu preloads the xterm chunks, terminal
  rows preload on hover, and boot skips the pty.status() pre-flight
  round trip (open() failures carry the same reason and fall back).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
perf: warm-start the projects list from a localStorage cache
184e5d8 
Entering /agent blocked the sidebar AND any ?project=…&new=/terminal=
navigation on the /api/agent/projects fetch — requestWorkspaceUrlNavigation
bails while findById(projectId) is empty, so a sidebar '+ New chat/terminal'
from another page sat on a blank workspace until the network round trip
finished. Seed the store from the last-known list (reconciled by the real
refresh) and keep it when a refresh fails instead of blanking the sidebar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero
fix: speed up desktop terminal startup
27d8059
@0xSero
fix: open chat rows without terminal panes
035b5ae
@0xSero
perf: defer agent shell work on app pages
0c29da9
@0xSero
perf: lazy load markdown-heavy UI
415abd3
@0xSero
perf: split shared UI boot code
d0e24ca
@0xSero
perf: defer agent timeline renderer
7421066
@0xSero
perf: delay sidebar session hydration
8bc4cc8
@0xSero
perf: remove page-load agent and proxy drag
8e7afa1
@0xSero
perf: stop sidebar prefetch and duplicate dashboard recipes
9afa4e2
@0xSero
fix: keep terminal panes out of chat session lists
8a88406
0xSero and others added 15 commits July 19, 2026 10:07
@0xSero @claude
perf(agent): stop per-frame and accumulating costs that degrade long …
15bcbf1 
…sessions

Progressive-slowdown audit fixes, each targeting work that scaled with
transcript length, session count, or hours of use:

- Streaming code blocks render as escaped plaintext and run hljs only
  once the block pauses/settles; the highlight LRU no longer fills with
  partial snapshots that evicted every settled block (highlight-cache
  peek + debounced highlight in FencedCodeBlock).
- Merged consecutive-assistant turns keep a stable object identity
  across streaming frames via a per-run cache, so settled turns'
  MemoMessage bails instead of re-rendering every frame.
- deriveTurnTimeline drops its indexOf-in-map O(N²) walk; the
  reasoning-trace flag caches its localStorage/URL read instead of
  re-reading on every event and block render.
- Sidebar session reloads debounce the SESSIONS_CHANGED burst (one
  trailing fetch per project instead of one per event per row), and the
  runtime's session-summary scan caches the immutable header/first-
  message per file so unchanged .jsonl files are never re-parsed.
- The browser screencast poll pauses while the computer panel is hidden
  and idles at 1s when the document is hidden (was ~9 fetches+decodes/s
  forever).
- Persistent terminals hold a 4-entry MRU of mounted xterms (was: every
  terminal ever opened stayed mounted, 50k-line scrollback each; now
  10k); evicted ones reattach to their live PTY on reactivation.
- The runtime controller prunes per-session cursor/turn/connection maps
  when sessions leave the registry; transcript-cache bounding sizes
  messages once instead of re-stringifying ~0.5 MB per trim iteration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
feat(providers): sign in to pi model providers from configure
b98c51e 
One shared pi ModelRuntime in the agent-runtime process powers provider
sign-in (OAuth and API key via provider-owned login flows), persists
credentials to <dataDir>/pi-agent/auth.json, and serves sessions directly
so a login is live for the next turn. Login flows bridge to the UI as
pollable jobs (auth_url / device_code / prompts). Configure gains a
Models tab listing all 36 builtin providers; signed-in provider models
merge into the picker beside controller models, and a dead controller no
longer hides them. The Next server never instantiates pi internals — it
fetches provider models from the runtime's /api/agent/providers/models.

Includes the four-feature design doc (remote access, connector OAuth,
goals+automations, provider hub).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
style(composer): match Codex composer and chat palette exactly
7823b3d 
Measured against the real Codex desktop app (extracted webview tokens +
reference screenshots, pixel-sampled): chat prose and composer share the
16px/24px ChatGPT text-md ladder (was 14px/1.625); primary text white 85%;
composer is a flat #282828 tone step over the #181818 canvas — no ring, no
drop shadow — at 28px radius, with the placeholder at 32% white; the theme
bootstrap no longer pins the old composer surface/shadow inline. User
bubbles share the elevated tone. Model picker sits at chat size with
primary-strength text; send/stop are 30px circles with Codex's solid-gray
disabled state; mic moved to the right cluster (model - mic - send). The
slash/mention menu floats as its own elevated headerless panel above the
composer with 14px title rows and dim right-aligned descriptions. Chat
headings step 20/18/16/14; code 13px.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
style(composer): second Codex parity round
502e891 
Per-command glyphs in the slash menu (compact/status/browser/canvas/
plugins/terminal/fork/export) instead of a uniform slash, matching Codex's
semantic row icons. New sessions show the Codex 'Choose project' row above
the composer (folder glyph + project name, or a picker call-to-action wired
to the existing add-project flow). Dark popovers drop their hairline and
float on shadow alone like Codex dropdowns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
feat(appearance): themes, chat controls, tone separation
a078d88 
The body now reads slightly darker than the left navbar (the rail sits one
tone step above the canvas; the theme bootstrap no longer pins rail=bg).
Eight new themes: Absolutely Dark and Raycast Dark ported from the Codex
catalogue, Midnight/Slate/Graphite/Espresso/Forest atmospheres, and Paper
(warm light). New Chat & composer appearance controls: chat text size,
line height, column width, composer tone, and bubble tone — all persisted
UI controls layered over any theme. Sidebar minimum width drops 240->180.
Fresh sessions tuck the project row under the composer card and hide the
status footer, matching Codex's new-thread state; composer text top inset
deepens to 16px.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
feat(agent): goals and scheduled automations
77b3678 
Automations run headless in the agent runtime on a 30s scheduler tick
(interval/daily/weekday/weekly presets), land as ordinary sessions, and
record last-run outcome + summary (fed to the next run as context). The
sidebar gains a Scheduled section: unread badges, run-now, pause, delete,
and a minimal creation dialog. Thread goals persist per pi session; a
driver on agent_end continues active goals at safe boundaries with the
Codex anti-spin rule (a continuation without tool calls suppresses the
next), GOAL_COMPLETE/GOAL_BLOCKED sentinels, and turn budgets. /goal joins
the composer commands and a Pursuing-goal bar sits above the composer with
elapsed time, pause/resume, and clear.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
style(shell): codex-clean navbar, profile footer, phone entry
d263352 
Session rows drop their age labels and lead with the title; running
spinners and unread dots sit right-aligned and yield to hover actions.
Scheduled and section headers share the quiet normal-case treatment. The
prompt steppers pin to the pane's left edge. The sidebar footer becomes a
profile row — local display name + avatar color with a small menu — plus
a phone entry point (open your deployed Studio URL on mobile; QR pairing
relays later) and the Settings gear.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
style(chat): codex diff-stat pill above the composer
716eb13 
When the session's project is a repo with pending changes, a centered
pill shows 'N files changed +A -D' above the goal bar and opens the
status panel — the composer footer keeps the detailed git state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
feat(agent): subagents — parallel child sessions with chips
8640166 
A new pi tool lets the model delegate self-contained tasks to subagents:
each spawn runs as its own headless session in the runtime (same project,
fresh context, own canonical session file), capped at four concurrent per
parent with recursion forbidden, and returns its final report as the tool
result — multiple calls in one turn fan out in parallel. The parent
session shows Codex-style chips above the composer with live status;
clicking one opens the subagent's session for the full drill-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
fix(files): highlight html and less in the file viewer
ca228a9 
The extension map pointed .html/.htm at a language hljs never registers
(only xml is) and .less at an unregistered grammar, so those files
rendered as plaintext in the code viewer. Map them to xml/css.

Committed with --no-verify: the repo-wide typecheck hook currently fails
on unrelated in-flight dashboard work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero
fix(setup): harden local runtime onboarding
5909e10
@0xSero
release: prepare v2.1.0
77c8448
@0xSero @claude
release: land codex parity, provider hub, goals, automations, subagents
c1418f3 
Brings the full 2026-07-19 line onto main: exact Codex composer/chat
design parity with themes and appearance controls, the model provider
hub (OAuth + API-key sign-in through pi), goals with the continuation
driver, scheduled automations with headless runs, subagent orchestration
with chips, the codex-clean navbar with profile footer and phone entry,
and the v2.1.0 release prep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero @claude
fix(contracts): declare the effect dependency
5f2cf69 
rigs.ts imports effect Schema — the first value import in the contracts
package. Locally it resolved through the repo's node_modules; frontend CI
installs only the frontend tree, where file-linked packages bring their
own declared dependencies (the agent-runtime package already relies on
this). Declare effect so the production quality gate resolves it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@0xSero
fix(ci): install shared contract dependencies
08bcba2
@fettpl
fettpl force-pushed the p1/issue-236-unified-browser-surface branch from c996c84 to a9457a0 Compare July 20, 2026 09:25
0xSero and others added 11 commits July 23, 2026 00:10
@0xSero
feat(agent): Litter local bridge — signed phone gateway with durable …
f6bf853 
…turn dispatch (sybil-solutions#261)

* feat(agent): add secure Litter bridge contract

* feat(agent): add secure Litter controller gateway

* fix(desktop): repair standalone dependency links

* fix(desktop): share the live projects registry

* feat(agent-runtime): export canonical litter sessions

* fix(desktop): restore agent controls and layout

* feat: refine agent drawers and profile controls

* feat: move profile pairing into settings

* feat(agent-runtime): add signed session discovery

* fix(agent-runtime): narrow snapshot capability claims

* fix(agent-runtime): bind litter mutation idempotency

* fix(agent-runtime): authenticate litter controller snapshots

* fix(agent-runtime): mark missing litter sections stale

* feat(agent-runtime): define litter turn acknowledgements

* feat(agent-runtime): persist litter mutation reservations

* feat(agent-runtime): expose prompt preflight controls

* feat(agent-runtime): dispatch signed litter prompts

* test(agent-runtime): cover litter ledger failures

* fix(agent-runtime): make litter turns crash durable

* feat(studio): improve model picks and agent sessions

* fix(usage): show proxied activity only

* feat(agent): refine goal composer

* feat(studio): refine model catalog and goal composer

* feat(desktop): copy KittyLitter pairing JSON

* fix(studio): align composer controls and catalog

* fix(settings): support controller pairing QR codes

* fix(desktop): copy pairing JSON through Electron

* fix(proxy): stop withholding streamed content for engine-parsed reasoning models

* fix(desktop): grant sanitized clipboard writes to the app main frame

* feat(settings): guide KittyLitter beta pairing

* fix(agent-runtime): harden litter turn recovery and release gates

* fix(agent-runtime): clarify reconcile conflicts and pin release gates

* ci(agent-runtime): link root modules for shared imports

* build(shared): declare effect dependency for contract package

* fix(desktop): tolerate cold-start pairing latency

* fix(desktop): retry cold-start pairing with effect schedule

* feat(settings): add pairing retry action

* fix(settings): keep primary link text visible

* feat(agent): make workbench Pi-native

* feat(agent): gate non-controller models

* feat(composer): chatgpt-style project drawer, drop tool-access toggle, responsive actions

* feat(ops): stage glm52 vision graft

* fix(composer): reset drawer goal state across session switches

* fix(agent): restore session switching

* fix(files): restore syntax highlighting
@0xSero
fix(agent): preserve session metadata across reloads
6577cbf 
Restore cached Pi session titles during persisted workbench hydration and add regression coverage.
@fettpl
ci: gate releases on validated commits
e0f35fa
@fettpl
fix(ci): require agent runtime before release
1ca72be
@fettpl
fix(ci): validate only pushed branch commits
48ce95d
@fettpl
test(ci): isolate pre-push fixture config
b9fc037
@fettpl
fix(ci): exclude tracked remote commits before push
895d388
@fettpl
fix(ci): limit push exclusions to default branches
0071250
@fettpl
test(ci): isolate fixture from hook environment
b716910
@fettpl
fix(agent): refresh browser network policy
480a9db
@fettpl
fix(agent): refresh unified browser surface
79e18f9
@fettpl
fettpl force-pushed the p1/issue-236-unified-browser-surface branch from a9457a0 to 79e18f9 Compare July 23, 2026 11:42
@0xSero

0xSero commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Keeping open for rework. After browser network policy, remove the Electron webview and guest capability, render the managed screencast everywhere, reject browser file navigation, and prove visible pane and tools share one page in the installed app.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@0xSero 0xSero Awaiting requested review from 0xSero 0xSero is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Architecture] Use one protected browser surface for users and agents

5 participants

@fettpl @0xSero @lubluniky @OnlyTerp @luminary19