Derive the personal drive DID from the agent key - #1272
Draft
joepio wants to merge 41 commits into
Draft
Conversation
cursor
Bot
changed the title
joepio
force-pushed
the
cursor/deterministic-personal-drive-cdb1
branch
2 times, most recently
from
August 16, 2026 15:49
120939d to
f01685c
Compare
Existing personalDrive pointers and random-DID homes stay ordinary workspaces. A best-effort list union is enough; do not keep the pointer authoritative. Co-authored-by: joepmeindertsma <joepmeindertsma@gmail.com>
The personal drive is a fact about an identity, not a pointer. Same Ed25519 key now mints the same did:ad: subject (fixed GenesisCert), repeat genesis for that subject merges, and extra drives are listed on the derived home. The Agent personalDrive field is a deprecated cache; readers that know the derivation ignore it. A previous random-DID home is unioned onto the derived drive when visible. linkPersonalDrive no longer waits on getResource after a failed agent fetch — that hung unit tests for 10s on an error stub. Co-authored-by: joepmeindertsma <joepmeindertsma@gmail.com>
CI never reached the tests on this branch: `@tomic/lib`'s lint step failed with two `@stylistic(padding-line-between-statements)` errors, and the pipeline stops at the first bad step. - `genesis.ts`: blank line between the end of `personalDriveSubject` and the `FLAG_HAS_STATE_HASH` doc comment. - `store.personal-drive.test.ts`: blank line before the `postCommit` stub assignment. Verified with the same binary and config CI uses (`oxlint -c ../.oxlintrc.json ./src`, then `oxfmt --check`): the two errors reproduce on the pre-fix files and both gates are clean after.
Migration only looked at `personalDrive` on the Agent *resource*. That pointer is absent whenever the server holding the account never wrote one — the ordinary shape of a self-hosted account whose drives were made ad hoc — and then an upgrading user's old home was never unioned onto the derived drive at all. The secret carries the other half. `initialDrive` is decoded from it into `Agent.initialDrive` (`CryptoProvider.decodeSecret` -> `Agent`), and unlike the resource pointer it travels with the key rather than with any one server's data, so it survives signing in on a device that has never seen the old drive. `maybeMigrateOldPersonalDrive` now takes a candidate list and unions each in turn, deduped against the derived subject. Candidates are migrated individually so one unreachable old home doesn't strand the other. Test: an Agent resource with no `personalDrive` and `initialDrive` naming the old home — its `drives` and `favorites` still land on the derived drive. Fails without the change (1 failed | 5 passed), passes with it. `@tomic/lib`: 40 files, 237 tests. oxlint, oxfmt and tsc clean.
A pre-DID account keeps its drives on the server it is migrating away from (atomicdata.dev), while the client already points somewhere else — a desktop node on localhost, or a self-hosted server. `isAdoptableDriveSubject` only kept subjects matching the current `serverUrl`, so every one of those drives was filtered out, `adoptLegacyDriveList` returned on the empty list, and the user arrived at their new home with nothing. The identity half of the same migration had already worked, which is what made this hard to see: the name came across from the legacy Agent while the drive list silently did not. Same-origin stays the rule; the legacy Agent's own origin is now also accepted. That origin is not a third party — it is the account being migrated from, which the user authenticated against and which we just fetched the list out of. This deliberately does not relax the case `legacy-drive-adoption.test.ts` exists for: a stale entry naming `http://localhost:9883` is still dropped, so a hosted app cannot be steered into issuing requests at the signed-in user's own machine. Those six spec cases are unchanged and still pass. Test: a legacy list mixing an `atomicdata.dev` drive, a `staging.` one and a `localhost` one — only the drive sharing the legacy Agent's origin is adopted. Fails without the change. `@tomic/lib`: 40 files, 238 tests. oxlint, oxfmt and tsc clean.
Both live only on the secret, and the secret is read exactly once — when it is pasted at sign-in. `agentStorage` persisted the keypair and subject and dropped the rest, so every launch after that rehydrated an Agent with `legacySubject === undefined`. `adoptLegacyAgentIdentity` returns on its first line when that is missing, so for an already-signed-in user the pre-DID migration never ran at all: no legacy drive list, no name, nothing — and no error either, since it is fire-and-forget. It only ever worked in the session where the secret was entered, which reads as "the migration does nothing" from the outside. `initialDrive` was lost the same way, which disables the other migration source added in c640a68. Both are now stored alongside the key and restored on load, on the non-extractable path and the readable fallback alike. The keypair-only overload of `saveAgentToIDB` re-stores them rather than blanking them, but only when the subject matches — a different account's values must not leak across.
`browser`'s root `build` orders this correctly — `--filter "@tomic/lib" run build` first, then everything that imports it. The Tauri build never goes through it: `tauri.conf.json` calls `pnpm -C browser/data-browser build:tauri`, which ran `vite build` against whatever `dist/` happened to be lying around. Vite consumes `@tomic/lib` from its built `dist`, not from source, so editing `lib/src` and building the app produced an app without the edit. Nothing failed: the source is right, `vitest` is green because it runs `src`, and the bundle builds fine — only the running app is stale. Today that cost several build-and-test rounds on three commits' worth of changes that were never in the binary being tested. CI cannot catch this. It builds every package from scratch in dependency order on a clean checkout, so `dist` is always fresh there; the failure exists only where the artifact survives between runs, which is exactly the local machine. The fix therefore has to live at the entry points that bypass the root script. `dev:tauri` gets the same treatment: `vite.config.ts` documents that the lib is "built in watch mode (tsup) alongside this dev server", but nothing starts that watcher, so a dev session silently served a stale `dist` too. Building once up front does not replace the watcher, but it does mean a session starts from the current source.
Every determinism test here builds its Agent from `JSCryptoProvider`. The app does not: in a secure context it stores a non-extractable `CryptoKey` and signs through `SubtleCryptoProvider`. A subject that was stable under one provider and not the other would pass CI and mint a fresh personal drive on every sign-in in the product, so the guarantee was only as good as its coverage of the provider actually used. It holds: stable across two Agents from one secret, stable across repeated calls, and equal to the JS provider's answer for the same key. Recorded because a real account was minting a new "My drive" on each sign-in and the crypto was the obvious suspect. It is not the cause — these tests rule it out, so the next look belongs on the server's repeat-genesis merge (the drive renders as "Server error", and `ensurePersonalDrive` is the only caller that names a drive "My drive").
A real account ended a single sign-in holding three drives named "My drive", each with a different subject, an empty "My drives" list and a personal drive rendering "Server error". `setAgent` fires `adoptLegacyAgentIdentity` and forgets it, and an app sets an agent more than once while booting — a node's own agent, then the signed-in one, then a rehydrate. Every pass reached `ensurePersonalDrive` and made a drive. The adopted drive list lands on whichever pass ran last, so the user sees an empty workspace beside a pile of orphans. Keyed on the agent rather than a bare flag, so signing out and into another account still migrates, and added before the first await so concurrent callers collapse onto one run instead of racing. What this is not: three suspects were eliminated by test first, and those tests are kept. - Derivation is stable, including under `SubtleCryptoProvider`, the provider the app actually uses (d7d0632). - `ensurePersonalDrive` is idempotent, including after the in-memory cache is dropped, which is what a reload looks like. - The server accepted every write: 42 commits, no rejection, so repeat-genesis merging was not failing either. Test: three `adoptLegacyAgentIdentity` calls for one agent, two of them concurrent, must produce exactly one legacy fetch. Fails without the guard (1 failed | 3 passed). `@tomic/lib`: 42 files, 245 tests. oxlint, oxfmt and tsc clean.
The private drive rendered as:
Could not open did:ad:Jn1MGZ4_2yArDrvrkpK2IPHl-b0BHRxYdiDnj_hd0vDc…
Resource not found. DID Resource … not found locally
Which should be impossible for a derived subject, and that is the tell.
`fetchPersonalDriveSubject` returned `agent.personalDriveSubject()` — pure
arithmetic on the key. It cannot fail and it answers just as readily for a
drive that has never existed, so callers linked at a subject the store had
nothing for. Nothing on that path creates it: `ensurePersonalDrive`'s only
caller was the legacy adoption, which does not run for every account or every
sign-in.
The subject is a fact about the identity. The resource is not, and only one of
the two comes for free.
It now materializes and returns that. `ensurePersonalDrive` is idempotent and
a repeat genesis merges, so this writes once per account and afterwards
returns what is already there. When materializing fails (offline, or the
server refuses the write) it falls back to the derived subject rather than to
a stale pointer — a later read can still resolve it once the drive exists.
Also adds a test over a real pre-DID account's 53-entry `drives` list, taken
verbatim, with the client on a desktop node rather than that account's server
— the case the legacy path exists for. The 27 subjects on the account's own
origin are adopted; the 15 on a sibling subdomain and the 11 naming
`localhost` are not.
`@tomic/lib`: 43 files, 246 tests. oxlint, oxfmt and tsc clean.
Reverts the code change in 363197b. The real-account test from that commit is kept; it was not implicated. Making `fetchPersonalDriveSubject` create the drive replaced one visible error with a much worse silent one. It is called from render and effect paths, so it runs repeatedly rather than once per sign-in, and a real account ended up with hundreds of drives named "My drive" filling the switcher. The important part is that they had DISTINCT `did:ad:` subjects. Repeated calls were supposed to converge: `ensurePersonalDrive` looks up the derived subject and returns the existing resource, and a repeat genesis merges server-side. Distinct subjects mean neither happened, so **`ensurePersonalDrive` is not idempotent in the running app**, whatever `personal-drive-idempotence.test.ts` shows with a mocked `postCommit` and a warm resource map. The test and the product disagree, and the product is right. That is now the thing to fix, and it is the same defect behind the three "My drive"s seen earlier — this change only turned three into hundreds by calling the path more often. Until it converges, nothing should call `ensurePersonalDrive` outside an explicit, once-per-sign-in flow. The original complaint stands and is unfixed: a derived subject that has never been materialized renders as "not found locally". The answer is to materialize it exactly once on a deliberate path, not lazily from a resolver that render code calls.
A `did:ad:` subject IS an Ed25519 signature over the resource's genesis certificate. RFC 8032 makes that signature deterministic, but determinism is a property of the *implementation*, not something WebCrypto promises — and WKWebView randomises the nonce. So `personalDriveSubject()` returned a different DID on every call, the reuse check in `createDrive` looked for a subject that had never existed, and each miss minted another drive. One session produced 411 personal drives from a single byte-identical certificate: 411 distinct signatures, all valid. Derive once from the raw private key with noble (matching `ed25519_dalek` on the server) and cache it on the Agent. It cannot be recomputed later, because the stored keypair is non-extractable — that is the whole point of `signBytes` — so the value is computed at sign-in while the secret is in hand. `createDrive` now pins the drive to that subject instead of re-signing the certificate, which was a second instance of the same bug: a correct derivation was discarded because `newResource` minted the subject through the same non-deterministic signer. Where neither a cached subject nor a deterministic signer is available, throw rather than sign. An unreproducible subject is worse than none: minting under it is exactly what produced the 411 drives, silently. Also here, from the same investigation: - Only the socket for the app's own server may report connection state. A client holds one socket per origin, and adopted drives add origins the app does not depend on. Their sockets fail and retry forever, and each failure marked the whole store disconnected — so the app showed "Working offline" and queued writes while its own server answered in under a millisecond. - `recordServerVersionFromWsProtocol` was called with its arguments swapped, so the capability map was keyed by the literal string `atomicdata-ws.v2` rather than by origin; and it compared against `atomicdata-ws.v0.1`, a name the client stopped sending when it moved to v2. - Migration fetches are bounded and say why they failed. They previously returned in silence, so a migration that never ran was indistinguishable from one that found nothing to do — and the once-per-agent guard means it is not retried. Tests use a deliberately randomising signer: node's WebCrypto Ed25519 *is* deterministic, so a test built on a real provider passes no matter how broken the product is. That is why this survived a green suite. The connection-scope test was checked to fail without its fix, not merely to pass with it.
…e server Three things the desktop app got wrong once the personal drive became a derived subject rather than a stored pointer. **Nothing created the drive.** The sign-in flow calls `fetchPersonalDriveSubject`, which *computes* the subject and hands it to the router as the navigation target — but no path ever materialised it. The app navigated to a DID that had never been written, which is the "Server error" / "not found locally" home drive. Sign-in now creates it, once, deliberately and bounded. Not lazily from a resolver called during render: letting whichever component asked first decide when a drive gets written is how a bad derivation turned into hundreds of them. **A failed derivation moved the whole app to another server.** `Store.setDrive` reads an http(s) drive subject as a SERVER ORIGIN — it sets `serverUrl` and does not even set `this.drive`. `initialDrive` travels with an old secret and is an http URL on the server the account is migrating *away from*, so the fallback handed the session to a pre-0.40 server that cannot authenticate this agent or serve any of its `did:ad:` resources. Measured on one run: the webview issued 38 requests to its own server and then none, while it retried a foreign socket forever. The fallback now only accepts a DID or a same-origin subject. **Boot re-applied it.** Restoring the last session went through the same call, so a single switch pinned the app to that server on every future launch with no obvious way back. Boot now refuses a stored foreign drive and says so. Deliberate switching — the drive switcher, Open by URL — is untouched. The server should move when the user picks another server's drive, not as a side effect of signing in. The agent's derived personal-drive subject is persisted beside the agent in IndexedDB, exactly as `legacySubject` and `initialDrive` already are: it is computed from the raw key at sign-in and the stored keypair cannot reproduce it.
…stale peer eviction
Three bugs in the live path, each hiding the next. Symptom: two paired nodes
bulk-sync fine on connect and then never exchange another change, while both
UIs show "Connected".
**A stale connection deregistered its replacement.** `LIVE_PEERS` is keyed by
node id. A reconnect installs a new entry under that same key, and the old
connection's loops tear down a moment later — removing the entry the NEW
connection had just installed:
[live] registered peer 6041773d78f9 (new=false)
[live] read error from 6041773d78f9: connection lost
[live] removed peer 6041773d78f9 <- evicts the live connection
The node is then left with no peer to broadcast to, and nothing recovers it
until the next reconnect. Entries now carry a connection generation and only
the connection that installed one may remove it. A deliberate reconnect still
evicts whoever is current, via an explicit variant.
**Echo suppression could not work.** It set a global `AtomicBool` around an
import so the push loop would skip re-broadcasting. But the push loop is a
separate task consuming a broadcast channel: it may not be scheduled until
after the flag is cleared, at which point it sends anyway. Suppression by luck.
The source is now stamped on the event instead. `add_resource_opts` reads the
importing peer while the write is still on the stack and puts it in
`DbEvent::Changed::source_id`; the push loop skips exactly that peer. No timing
assumption.
**And the suppression was applied to one case out of three.** It only ran when
the imported subject was this node's own agent — so the device whose agent it
is stays quiet, the peer for whom it is a stranger's agent re-sends it, and a
drive (nobody's own agent) echoes on both sides. Two idle nodes traded 355
frames in 58 seconds, ~8.6KB each, ~50KB/s. Measured after the fix: 6 frames.
Also here: exempt a repeat genesis from the Loro causality guard. A device that
renamed its home drive rejected its own stashed genesis forever — the intent
says `name = "My drive"`, stored state says the chosen name, they do not match,
and the outbox retried every 30s for as long as the app stayed open. A repeat
genesis is legitimate (`repeat_genesis_is_mergeable`); its creation defaults
losing to whatever the drive has become is the expected outcome, not evidence
of an unseeded client.
Tests: a stale connection cannot evict its replacement, a forced reconnect
still can, and a repeat genesis whose every value loses is accepted. Each was
checked to fail against the previous code. `sync::iroh_e2e` is unchanged at
10 passed / 3 failed — those three fail identically before this commit.
`remove_live_peer` now refuses when a newer connection has replaced the entry — correct for a connection retiring itself, wrong for the user pressing Disconnect, which means "drop this device" regardless of which connection currently holds it. Adds an explicit unconditional variant for that intent. Caught by compiling the server crate: the previous commit only checked `atomic_lib`, and this is a `pub` signature the server calls.
**Search never saw peer-synced data.** Indexing lives in `Handler<CommitMessage>`, which only runs for changes a commit produced. A peer sync writes straight through `add_resource_opts`, so 49 resources arrived and produced zero INDEXING events. The *query* index is updated, so the drive listed them fine — it was only search that could not see them, which reads as "my data did not arrive" to anyone who searches first. `Handler<ExternalChange>` now indexes too, off the actor thread. **"Synced 1 resource" when 49 had just landed.** `/iroh-sync` returned only `count` (what we pulled) while `pushed` was computed and dropped. A pass that sends 49 and receives 1 is not "1 resource synced". Both directions are now returned and shown — "sent 49, received 1" — with "Already up to date" when nothing moved. **A paired device card that could not tell working from broken.** It showed a name and a "Connected" pill, nothing else, while the server card beside it showed volume, last-sync and node ID. `KnownPeer.last_synced` was already tracked and simply not reported; it now is, alongside the node ID (click to copy, same affordance as the server card). "Connected" describes a socket; "synced 4 minutes ago" describes whether data actually moves. Volume per peer is still not shown — nothing tracks bytes or counts per peer, and inventing a number here would be the same failure in a new place.
`Db::setup` switched from `create_drive` to `ensure_personal_drive`, and with it lost the `set_active_drive` that `create_drive` performs as part of creating one. Every call afterwards failed with "No drive set. Call setup() first." Materializing a home and switching to it are deliberately separate — that split is why `ensure_personal_drive` does not set it, and why boot can call it without hijacking the session. Setup is the act that means both, so it says so. This is what CI has been failing on since the personal-drive change landed: 7 of 9 tests in `flutter/rust` `api::simple`, including `a_canvas_syncs_to_a_second_device_through_the_bridge`. All 9 pass now.
Two of the same person's machines synced nothing. The serving node refused every
subject because the peer authenticates as its own node agent — a stranger to the
drive — and the only remedy was hand-writing an ACL entry naming that agent, per
device and per drive, with nothing in the UI asking for it. Meanwhile the status
said "In sync".
Pairing is already an authenticated choice by the owner, so treat it as the
authority. `collect_readable_snapshots` now takes the dialled peer's node id: if
the owner deliberately paired with it, it is served what THIS node can read.
Why `known_peers` is a sound basis: only the initiator records a peer
(`add_known_peer` runs after we dial and the remote says HELLO). The accept side
deliberately does not — "the local user never chose to sync with this peer". So
the list means "nodes this user dialled", which is exactly pairing.
Deliberately narrow:
- It does not widen what is served. A paired replica gets exactly the subjects
this node can read, never more.
- It does not touch the relayed WebSocket path, which has no dialled node
identity to stand in for the choice. Rights only there.
- An unpaired node id changes nothing — the test asserts that first, then pairs
and asserts the difference.
Note for the record: peer sync does NOT carry signed commits. `SyncPushEntry` is
`{ subject, loro_bytes }` — raw CRDT state, no signature, no author. So a peer
cannot be treated as a courier relaying self-authorising writes; its identity is
the only credential on the wire, which is why the receive-side write check stays.
Making relaying credential-free would need signatures on the peer wire first.
Completes the device card. It asked three questions — when it last synced, how
much, and which node — and only two were answerable: nothing recorded volume per
peer, so the card said "Paired device" and left the user unable to tell a link
carrying data from one being refused every subject behind a green "Connected".
`KnownPeer` now keeps what the last completed sync moved each way, written where
`last_synced` already was, and reported on `/server` beside it:
Mac.home Connected
Paired device · synced 2 minutes ago · sent 49, received 1
Per-sync rather than a lifetime total, deliberately. A running counter has to
survive re-pairs, store resets and partial syncs, and quietly becomes fiction
the first time one of those is missed. These two describe one pass and are
checkable against the figure that pass reported — which is the property the rest
of this status work has been about.
The drive-usage figure on the server card ("53 resources · 212 KB") is not a
substitute: it measures the drive, so two peers sharing one would both display
it and neither number would say anything about either link.
Two machines syncing the same drive could not see each other's cursors. `EPHEMERAL: u8 = 0x40` was reserved in the peer protocol and referenced nowhere — never sent, never handled. Presence was entirely client-to-server WebSocket, fanned out to the subscribers of one server, so a browser on the hosted node and a desktop app on localhost were separate islands: drive state crossed, presence did not. The frame carries the drive, the originating agent, and an opaque Loro `EphemeralStore` payload. The agent travels with it because a peer link is node-to-node while presence is per-agent: one node may relay several people's cursors, and the receiver needs to know whose it is. Presence is unlike everything else on this link, and the handling reflects that: - **It never reaches the store.** Handled first in the read loop and returned from immediately — every path below it ends in a write, and cursor positions merged into the CRDT would be persisted and synced forever. It travels on its own channel (`Db::subscribe_ephemeral`) rather than `DbEvent`, whose consumers all write or index. - **Gated on read, not write.** It discloses who is looking at what, so a peer that cannot read the drive receives none of it — but it authors nothing, so the write checks do not apply. - **Echo-suppressed.** A frame relayed IN carries no websocket address, which is what stops it being sent back out. At cursor frequency an echo would saturate the link far faster than resource changes could (see the 355-frames-in-58s storm fixed earlier on this branch). - **Droppable.** A small channel, and a lagging subscriber logs and continues instead of erroring. A stale cursor is better than a delayed document, and presence failing must never affect drive sync. - **Bounded.** 64KB per payload, refused rather than parsed beyond that: presence skips every check a write faces, so an oversized frame is either a bug or an attempt to move real data down it. Tests: a two-node test asserts presence crosses AND that the receiver's resource count is unchanged — checked to fail (timeout) with the handler disabled. Plus codec round-trip, oversized refusal, and truncation. Also fixes the peer volume figures from the previous commit: `last_sent` / `last_received` were captured and never inserted into the reported propvals, so the card had nothing to show. The compiler said so via an unused-variable warning and I pushed it anyway.
A peer connection can die on one side while the other believes it is still up. Observed: the accepting node saw the read error at 11:38:19, the initiator not until 11:53:11 — fifteen minutes in which - the peer stayed in `live_peer_ids`, so `auto_connect` skipped redialling it (it only dials peers it thinks are disconnected), - every local change was broadcast into a socket nobody was reading, - and nothing surfaced: no error, no status change, the UI still said "Connected". Which is why sync looked intermittent rather than broken. Anything written inside a genuinely-live window arrived; anything written into a half-open one vanished. A chat message landed and a table's rows did not, purely on timing — and they appeared the moment the link was rebuilt, 16 resources at once. Fixed on both loops of a live connection: - The read loop now bounds its wait (`LIVENESS_TIMEOUT`, 35s). Silence is treated as a dead link, and the loop's existing teardown removes the peer — which hands it straight to `auto_connect`'s 30s retry, a path that already worked. Nothing new needed to reconnect; the gap was noticing. - The write loop sends a `KEEPALIVE` (0x41) whenever it has nothing else to send, so the far side can tell an idle link from a gone one. Payload-free and never answered: receiving it is the whole point. The hazard here is the opposite failure — tearing down healthy connections that simply have nothing to say — so the test idles a real two-node link past the keepalive interval with no traffic, asserts the peer is still registered, and then syncs a resource across it to show it is genuinely alive rather than merely listed. The timeout is deliberately several keepalives wide so a couple of dropped probes cannot kill a working link.
The liveness timeout added in the previous commit assumed the far side sends keepalives. Against a peer that does not — any build from before that commit — silence is normal, and treating it as death turns a perfectly good idle link into a 35-second reconnect loop. Not hypothetical: deploying the two sides minutes apart produced exactly that, 10 teardowns and 10 full reconnects against the node that had not been updated yet, with zero read errors. It stopped the moment both ran the same build. So the timeout only counts as a liveness signal once the peer has actually sent a KEEPALIVE. Before that, a timeout logs and keeps waiting: - new peer ↔ new peer: keepalives flow, silence means dead, half-open links are caught in seconds — the behaviour this was written for. - new peer ↔ old peer: no keepalive ever arrives, the timeout never fires, and the connection behaves exactly as it did before. Support is inferred by observation rather than negotiated, which keeps the handshake untouched and degrades in the safe direction: the worst case for an un-upgraded peer is the old bug, not a new one.
The previous commit wired only half of presence. There are two ephemeral channels and they are not interchangeable: - `LORO_EPHEMERAL_UPDATE` — ephemeral state for one subject (cursors inside a document), fanned out to that subject's subscribers. - `PRESENCE_UPDATE` — drive-scoped presence, "who is in this drive", fanned out to the drive's presence subscribers. Only the first was relayed, so the avatars — which are the second — still stopped at the server boundary. Frames now carry which channel they belong to, and re-enter through the same one on the far side. Inbound drive presence gets its own actor message rather than reusing `PresenceUpdate`. That handler requires a sender address and checks the sender is a subscriber, which is where the drive read gate happens for local clients — precisely the check that should not be faked for relayed traffic. A peer's presence has already passed its own read gate in the sync loop and has no local connection to attribute it to, so `RemotePresenceUpdate` fans out to every subscriber with nobody to exclude, and the local gate stays intact for local senders.
Ground truth for a link that was live and exchanging data: the desktop's own server reported `live = true` with a current `lastSeen`. What the two Sync pages showed: - The paired-device card: a static "Paired" pill with no live state at all, so nothing there could say whether the connection was up. - "synced 5 hours ago" — that card reads `lastSync` from `localStorage['atomic-peers']`, which only updates when the user presses "Sync now". The server's own answer was current. - "not synced yet" on the always-on node — `mark_peer_synced` only ran on the dialling side, so the node that is always dialled INTO never recorded a sync no matter how much data flowed. That one was mine, from the commit that added these figures. So: the accepting side records syncs too, and the paired-device card takes its status from this device's own server instead of a local record that updates only on a button press. The counts are now `Option` rather than defaulting to 0. The two sides genuinely know different things — the dialling side tallies both directions, while the accepting side answers frames through the engine and only counts what came in — and reporting an unknown figure as 0 is the same class of lie the rest of this work removes. Absent stays absent, and the UI omits what it does not know.
`ephemeralStore.apply()` was unguarded. Cursor positions reference Loro containers, so a peer editing a document this device has not caught up on names containers the local doc does not have, and Loro throws "The container does not exist in the doc" — uncaught, once per keystroke of someone else's typing. Rare while presence stayed inside one server, because both clients had the same document state. Routine once presence crosses peer links: it travels on its own channel and does not wait for content, and content commits are debounced while cursors are instant. Dropping the frame is the right handling. Presence is a snapshot of now, so there is nothing to replay — the next update after the document catches up applies cleanly. Throwing achieved nothing except noise.
Remote carets never appeared between two paired nodes, and the receiving browser logged "The container does not exist in the doc" on every keystroke the other side typed. Only half of a collaborative edit was crossing. The client has two channels and they are easy to mistake for one: `broadcastLoroEphemeralUpdate` carries cursor positions, `broadcastLoroSyncUpdate` carries the characters. `broadcast_ephemeral` had exactly two callers — the ephemeral and presence handlers. `Handler<LoroSyncUpdate>` fanned out to local websocket subscribers and stopped there. A caret pointing into text the receiving document has never heard of is exactly what Loro refuses to place. Document ops now relay as a third `ephemeral_kind` on the existing frame rather than a new frame type. Two things differ from presence, switched on the kind byte: admission is on write rather than read, because these are somebody else's characters appearing in a document and a peer that may only read has no business putting them in front of an editor; and the payload ceiling is 1MB rather than 64KB, because a paste is a single op and the presence limit would drop exactly the edits most worth relaying. Nothing is written to the store on receipt. Relayed ops reach open editors and become durable only if a local user saves, which produces a signed commit under that user's own identity — the same trust model two browsers on one server already have, now spanning two nodes. Measured as an A/B on one variable, same page instance on both sides, toggling only this relay. With it off, an edit reaches the peer's store in ~190ms (`imported update`) and never reaches the peer's open editor — still absent twelve seconds later, while a caret rendered throughout, which is what rules out a dropped websocket. With it on, the text appears in ~2.5s and the remote caret renders against it. So the earlier framing of this as a timing race was wrong: a save puts an edit on the other node's disk and no further, and live collaboration between two nodes did not work in an open window at all. Worth re-testing "table rows do not sync" against this, since it has the same shape. One thing this does not fix, recorded as M9b: the channel is deltas with no gap recovery, so a client that misses one op queues every later delta as pending and silently stops updating until it reloads.
…thout a human Verifying two-node collaborative editing needs someone typing in one window and watching another. That made the sync work slow to check and, twice, made me report a fix as working on evidence that turned out to be a coincidence. Registers `tauri-plugin-mcp-bridge` in debug builds, which exposes the webview over a WebSocket: open a document, type into it, read the DOM back. With it, the `DOC` relay in the previous commit could be A/B tested against a real browser on the paired node instead of argued from frame sizes in a log. Bound to `127.0.0.1` rather than the plugin's default `0.0.0.0`. The channel is unauthenticated and can execute JS in the webview, so the default offers that to everyone on the subnet. `withGlobalTauri` is the other half the bridge needs, and it is static config that `debug_assertions` cannot gate. Rather than ship it, it lives in `tauri.dev.conf.json` and is opted into with `cargo tauri dev --config tauri.dev.conf.json`. This app runs with `csp: null` and renders data from drives the user may not control, so exposing the Tauri API to page content in release builds would widen a surface that is already wide.
… (M10) Tested because "adding things to a table does not sync" had the same shape as the document bug fixed in the previous commits, and might have shared a cause. It shares the shape, not the fix. Row added on the desktop with the table open on both nodes: instant locally, absent on the peer's open page after 18 seconds, present after a reload. So it crosses the link and lands in the store, and the open page never learns. A table row is a child resource created by a commit, not an op inside the table's own Loro document, so it travels the commit -> UPDATE -> ExternalChange route rather than the live document channel. Recorded with what is established (the notification is sent) separately from what is not (whether the client can act on it, given query-update frames were retired in favour of snapshot-carrying UPDATE frames).
The first version of this finding said a peer's new child resources never reach an open page, on the strength of a websocket hook that captured zero frames while a positive control on the same page and socket captured two. The measurement was real; the conclusion was wrong. The test account had never had a drive set — it was created through an invite — so `store.getDrive()` was the server root and `subscribeToDrive()` subscribed to that. `Handler<ExternalChange>` then correctly declined to fan out, because the row's drive is not within the subscribed one. Zero frames was the server working as intended on a wrongly subscribed connection, not a bug. The sidebar showing `/` instead of the drive name, and a 500 on `/search?parents=https://atomic.ontola.io`, were both visible at the time and should have been the first thing checked. With the drive set correctly the same test shows something narrower and real: the peer's page learns the row exists — it renders, and the footer count goes 2 to 3 — but the cell is empty until a reload, which then shows the value. So membership propagates live and content does not. Recorded with the two candidate explanations kept apart rather than collapsed into a story: either no UPDATE frame is sent for the row subject itself (the parent table's own change alone would move the count), or one is sent and the client stores it somewhere the table cell does not read.
…(M11) The webview is ready before the embedded server binds, so any fetch in that window fails and the app settles on "Could not reach the server / Offline: resource not available locally" — and stays there. Measured: server bound at 16:23:40 answering in 1.2ms, webview still showing the error two minutes later. Only the Retry button clears it. Visible constantly while working on sync because `cargo tauri dev` watches `lib/` and `server/` as well as `desktop/`, so every Rust edit restarts the app. That is a dev-loop annoyance, but the packaged app boots the same way, so a cold start can strand a user on a dead-end error for a resource that is seconds from being available. Also records that the message misdescribes the fault: "Offline" and "Reconnect to fetch" point at the network, when the machine, the server and the on-disk resource are all fine. Same family as M8 — a transient condition recorded as a permanent verdict.
… wrong thing Adds the measured mechanism behind the stuck offline state. The websocket connect fails once at restart (`close code=1006 opened=false`) and is never retried — that is the last WS line in the log, while the server bound seconds later and has answered in ~1ms since. Dispatching `online` by hand produced zero new sockets and no repeated failures appear either, so this is not slow backoff, it is no reconnect at all after a connect that never opened. Also records why Retry does not help: it re-issues the fetch, not the connection, and the fetch then queues on a socket whose handshake never completed. Per `websockets.ts`'s own comment, `ws.fetch`'s REQUEST_TIMEOUT only starts after auth, so a stalled handshake hangs every fetch indefinitely. That produces a third state beyond loaded and failed — "Still loading... hasn't loaded after 15 seconds", where the 15s is a UI notice rather than a request timeout, and nothing underneath ever gives up. Notes the misleading signal that an HTTP keep-alive to the same port stays ESTABLISHED throughout, so the app looks reachable while insisting it is offline.
`Handler<ExternalChange>` reads its payload straight out of `Tree::LoroSnapshots` and sent it with `HAS_COMMIT_ID | PUSH`, omitting `flags::SNAPSHOT` — unlike the normal push path in `web_sockets.rs`, which sets it. The commit path a few lines below correctly omits the flag because it carries a commit's delta; this one carries full state. Correct on its own terms, and the client honours the flag on its pending-GET branch. It does NOT fix the bug it was written for, and the planning note records that plainly: deployed and verified on the wire (frames arrive `flags: 5`, SNAPSHOT|PUSH), and a peer's new table row still renders with an empty cell. That test also produced the finding that redirects the whole investigation. Reading the receiving page's store directly shows the row present with its name, `isA` set, `loading: false` and no error, while the cell renders blank. The data crosses the link and imports cleanly; the table does not re-render. It is a render bug, and both server-side theories chased so far were aimed at the wrong layer.
…dering M9b was written from a synthetic test where the control run deliberately left a client behind. It reproduces in ordinary use: two clients on the paired nodes, same document open, one had typed a line the other never received. The receiving editor showed `awd` where the sender had `awdawdawad oawdinawiodawoi dn`, and a reload pulled the full text immediately — so the server had it throughout and only the open editor was stuck. Records the detail that makes this legible from a user's side: the other client's cursor kept rendering in the diverged editor the whole time. Presence is stateless and cannot fall behind; content is a delta stream that can. Both channels up, one silently stale, which is why it reads as "presence works but content does not". Also adds M12: a newly created resource sorts to the top of the drive tree instead of the bottom. Logged mainly because it first looked like the resource had gone missing on the device that created it — it had not; the store holds it with no error and it is in the DOM, just not where the eye expects.
…it silently A collaborator's text stopped arriving in an open document and never came back. Measured on two paired nodes: the sender had `awdawdawad oawdinawiodawoi dn`, the receiver kept showing `awd`, and a reload pulled the full text immediately — so the server had it throughout and only the open editor was stuck. The sender's cursor kept blinking in that stale editor the whole time, which is what made it read as "presence works but content doesn't": presence is stateless and cannot fall behind, content is a delta stream that can. The cause was an unappliable delta being reported as success. `applyIncoming` failed a resource whose import left ops pending only when it had no `isA` — the "already has usable content" case fell through to `return 'applied'` and stamped `lastCommit`. So one delta whose base ops never arrived parks in Loro as pending, every later delta parks behind it, and the document quietly stops being live with no error and no indicator. Now that case asks the server for full state. `forceOverride` replaces rather than merges, which is what closes the gap; the existing guards still refuse to clobber unsaved local edits or a pending outbox, and the stale-but-readable document is kept rather than blanked, so a failed repair costs nothing. This is the mid-session equivalent of what `SYNC_VV` already does at connect time. It deliberately does NOT stamp `lastCommit` for an unapplied commit. Claiming it would make the echo-dedup at the top of the same method drop the very fetch issued to repair the gap — the fix would have been a no-op that still looked like it worked. Tests cover the shape that actually reproduces it: a seed the receiver has, a withheld commit, then deltas exported `from` a version it never reached. Exporting a fresh doc as `update` does not reproduce it — with no prior version it carries every op from the start and applies cleanly, which is how the first draft of these tests passed against the unfixed code. Three of the four fail without the fix.
…ssion's A row created on a paired node reached the other device's store complete — `name` set, `isA` set, `loading: false`, no error — and never appeared in the table. Measured on the receiving page: `collection.totalMembers` 8 against `aria-setsize` 5. `memberCount` is frozen at the count the collection had when it first became ready. That freeze is deliberate and load-bearing: it stops a materialising session row from flipping `TableNewRow` → `TableRow`, which used to remount it mid-edit and drop keystrokes. Rows below the frozen index render as collection members, rows above come from `newRowSubjects`. A row from a peer is neither. It grows the collection but not the baseline and is not one of this session's drafts, so nothing ever draws it. Same for a second tab on the same drive. Now the baseline accounts for what this session actually contributed — each materialised draft adds a member while still rendering from `newRowSubjects` — and anything beyond that raises it. It only ever raises: shrink stays with `decrementMemberCount` and the existing clamp, and session rows keep their `_new:` key through the index shift, so the no-remount property that motivated the freeze is preserved. Telling those two apart needs to know whether a placeholder was ever aliased to a real subject, which was private to the store. Adds `Store.isAliased`, with tests for the three cases that matter: an unpersisted draft, a materialised one, and a subject that arrived from elsewhere. Also fixes a typecheck error I committed in the previous change: the gap-recovery test referenced `core.classes.document`, which does not exist. I had run the tests but not `pnpm typecheck`, and vitest does not typecheck. Not verified end-to-end. The reasoning and the unit tests hold, but this needs two live clients to confirm, and today has already produced two fixes that were correct in themselves and did not move the symptom.
M10a: a row created on the paired node now appears on the other device with its name, live, no reload (aria-setsize 9 to 10, footer 8 to 9). Also records that the earlier reading of this symptom was wrong: five named rows plus one empty row against a footer count of eight meant the new rows were not rendered at all, and the empty row was the trailing placeholder. Reading 'empty cell' instead of 'missing row' aimed two hours of work at the wrong layer. M9b: an unappliable delta injected at a live resource now returns 'invalid' rather than 'applied', logs the catch-up fetch, leaves the document intact, and does not claim the commit it never applied. That last part is load-bearing -- the echo-dedup would otherwise drop the repair fetch.
The desktop app boots its webview before its embedded server binds, so the first websocket connect fails. It then sat on "Offline" indefinitely, next to a server answering in ~1ms, with every fetch hanging and only a reload clearing it. `openPromise` only ever resolved. A socket that dies before opening leaves it pending forever, and `authenticate()` awaits it while holding `isAuthenticating` — whose `finally` sits downstream of that await. The flag stays set for the life of the client, so when the retry loop's socket opens, its `authenticate()` takes the `if (this.isAuthenticating) await this.authPromise` branch onto the dead promise and waits forever. Auth never completes, `reportConnected(true)` never fires, and `ws.fetch` hangs because `REQUEST_TIMEOUT` only starts after auth. It now rejects on close-before-open, so the stuck auth settles and the flag clears. A derived `.catch` absorbs the unhandled rejection for the common case where nobody is awaiting yet; awaiters still see it. Verified end-to-end on the same cold-start race: before, Offline indefinitely with the error screen; after, Connected with the drive loaded, no reload. Both tests fail without the fix and the second times out at 5002ms, which is the deadlock itself. The planning note is corrected too. It claimed the reconnect never runs — it does, with correct backoff, and a socket killed after opening recovers fine. One run that appeared to prove otherwise was the known double-instance bug: three app instances alive, the oldest holding the redb lock, so there was genuinely no server to reach.
A table created in the sidebar rendered FIRST rather than last, which reads as the resource having gone missing when you look for it at the bottom of a list whose shape you know. It was in the store and in the DOM the whole time, just not where the eye expects. `sortOrder` and `createdAt` share a number space on purpose — drag-and-drop mints a fractional key BETWEEN two neighbours' keys and the server sorts by the same fallback, so those two are comparable by construction. A member carrying neither fell back to its array index, which is not in that space at all: an index of 3 against timestamps around 1.7e12 sorts to the very front. Measured on the affected drive, `RelayTableTest` and `Tekenign` had neither property. Keyless members now inherit the preceding member's key, so they stay where the server put them (it already returns members in `createdAt` order) with the index tie-break preserving their relative order. Ones before the first known key inherit that key rather than sorting as zero. The ordering moved to `@tomic/lib` as `orderChildren` because `@tomic/react` has no test setup and this rule is worth pinning: five tests cover the keyless cases plus the drag-and-drop case that must keep working. Verified in the running app: the tree now ends "Meetings | Tekenign | Ontology | RelayTableTest" where it previously began "RelayTableTest | Tekenign".
…ER_URL `FRONTEND_URL` and `SERVER_URL` are both env-overridable, but the `storageState.origins` list they feed was hardcoded to the default ports. Run the suite anywhere else — a second checkout, a port that isn't already taken by a dev server — and the origin under test silently loses `viewTransitionsDisabled`, which the config sets specifically to stop view transitions making the suite flaky. The tests then fail on timing rather than on anything real. The two env values are now derived into origins ahead of the fixed entries, which stay for the default local setup and for dagger's `atomic.localhost`. Found while setting up a run on a non-default port: the checkout under test could not use 6747, since that was already serving a different branch.
…m the catalogs Two things the rebase onto develop surfaced. `develop` independently used M9-M12 while this branch was open, for entirely different findings — echo storm, search indexing, sync status, presence over peer links. This branch's own M9/M10/M11 collided with three of them, so the note carried two M9s, two M10s and two M11s. This branch's set moves to M13-M16 and keeps its cross-references; develop's keep the numbers they were published under. The locale catalogs had merge-conflict markers baked in as translatable strings — `msgid "App>>>>>>> Stashed changes"`, `msgid "Remove======="` and seven more per language. `origin/develop` has none of these, so they came in on this branch: the known failure mode where two wuchale extractors race on `src/locales/*.po`. Stripped, one language at a time with nothing else writing; msgid/msgstr counts stay balanced at 1589 per file.
joepio
force-pushed
the
cursor/deterministic-personal-drive-cdb1
branch
from
August 16, 2026 19:27
96ac3c2 to
22ebab5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Implements the plan in
planning/deterministic-personal-drive.md.What this does
The personal drive is a fact about an identity, not a pointer. The same Ed25519 key now mints the same
did:ad:subject from a fixedGenesisCert(created_at: 0, nonce = SHA-256(atomic-personal-drive-v1)[..16]). Repeat genesis for that subject merges via Loro instead of being rejected. Extra drives are listed on the derived home.personalDriveon the Agent is a deprecated cache for older clients. Readers that know the derivation ignore it (fetchPersonalDriveSubjectcomputes the DID). If a previous random-DID home is visible, itsdrives/favorites/sharedWithMelists are unioned onto the derived drive and the old home is kept as an ordinary workspace.linkPersonalDriveno longer waits ongetResourceafter a failed agent fetch. That hung unit tests for 10s on an error stub, and it overwrote the old pointer before migration could read it.Tests
All green on this revision:
@tomic/lib: 40 files, 236 testsatomic_libgenesis: 12 passedatomic_libcommit: 25 passed, includingrepeat_personal_drive_genesis_mergesDb::setup/ensure_personal_drive/ extra-drive listingNew coverage:
createDrive({ personal: true })uses the derived DID and certNot covered yet: Flutter
create_drivestill mints a random DID. E2E sign-in on a second machine with the old machine offline.Checklist
planning/deterministic-personal-drive.md,TESTING_COVERAGE.md)