Skip to content

Decommission the in-memory graph backend; SQLite becomes the only store - #473

Merged
zzet merged 34 commits into
mainfrom
refactor/sqlite-only-graph-backend
Aug 9, 2026
Merged

Decommission the in-memory graph backend; SQLite becomes the only store#473
zzet merged 34 commits into
mainfrom
refactor/sqlite-only-graph-backend

Conversation

@zzet

@zzet zzet commented Aug 5, 2026

Copy link
Copy Markdown
Owner

What

Retires the in-memory graph backend and makes the SQLite store the only persistence layer. *graph.Graph survives in exactly two demoted roles: the indexer's cold-index staging shadow, and the fixture for the existing test corpus — a new fence test (internal/graph/newfence_test.go, AST-based) pins graph.New() to the sanctioned staging sites so it cannot drift back into service as a backend.

Why the staging shadow stays: measured on this repository (3,994 files, 1.56M edges), a cold index into sqlite takes 6m34s with the shadow and did not finish in 35 minutes without it (killed mid-resolver; per-call disk reads). On rails (4.5k files): 43s vs 99s. The shadow is load-bearing; the backend option was not.

Removed

  • --backend memory selection in serverstack.OpenBackend — the memory spellings now fail fast with guidance (--backend sqlite --backend-path <path> for an ephemeral store). An empty backend name still defaults to sqlite; silently falling back would have written an "ephemeral" store into ~/.gortex/store/store.sqlite.
  • Both graph-snapshot subsystems (dead code on the sqlite default — every entry point was gated on .(*graph.Graph)): the daemon's gob+gzip snapshot (cmd/gortex/daemon_snapshot.go, ~1.3k LOC with its consumers) and the one-shot commit-keyed FileStore snapshot in cmd/gortex/mcp.go, plus the graph half of internal/persistence.
  • The boot shape-degradation guard — it only ever armed from the gob snapshot load, so it has been a no-op on the sqlite default all along. A store-sourced replacement can be built later from the persisted repo index state.
  • The last production .(*graph.Graph) behaviour forks (daemon warmup, warm-restart routing in multi.go, explain_change_impact).

Converted to real stores

  • gortex mcp without a daemon now runs on a private per-process temp sqlite store (guarded so it can never touch the daemon's shared store; stale temp stores are reaped at startup). Accepted trade-off: without the snapshot cache, every daemonless one-shot invocation cold-indexes the repository. --no-cache becomes a hidden no-op shim like the other retired MCP flags.
  • Eval harnesses and the bench trio run against temp sqlite stores; the recall harness's lexical row now measures the store-native FTS the daemon actually serves.
  • gortex init intake dry-run no longer constructs a store at all.
  • pkg/gortex (breaking): New(opts ...Option) (*Engine, error), Close() error required, WithStorePath added. The break is deliberate — a compile error instead of a silently leaked sqlite handle + WAL goroutine per engine.

Fixed on the way (pre-existing sqlite bugs this removal would have exposed)

  • The direct (non-shadow) cold path never wrote symbol_fts — every large-repo cold index shipped with degraded symbol search. Now populated on both the direct and streaming paths, with a regression test.
  • Seven sqlite cursor helpers returned silently truncated results on mid-scan driver errors (rows.Err() unchecked / swallowed Query errors).
  • Attribute-only ReindexEdges entries were dropped by sqlite but persisted by the memory store; converged on persisting, with the receipt preload aligned to match.

Conformance

The store contract now specifies what was previously implicit, green on both implementations: the persist round-trip, detached-vs-aliasing read semantics (declared per backend), adjacency ordering (sqlite's ORDER BY is plan-locked as index-served, including the degraded plan during bulk load), reentrancy from iterator callbacks, and mutation receipts.

Compatibility notes

  • docs/versioning.md scopes SemVer to the MCP tool API and the CLI. This PR rejects a value of a kept flag (not a flag removal), shims the one removed flag, and keeps every MCP tool signature. Two response-payload details changed: explain_change_impact no longer emits the community-couplings detail (the dedicated coupling analysis covers it; the zero-impact warning/caveat annotations are preserved and now armed on every backend), and the snapshot_loaded readiness phase (name kept) no longer carries a snapshot_repos count.
  • BENCHMARK.md: the fixture row was re-measured against sqlite; the token-efficiency and daemon-latency tables are marked as measured on the retired backend, pending re-measurement.

Follow-ups (deliberately out of scope)

  • Partition mixed reindex batches so attribute-only entries stop disqualifying the resolved-conversion UPDATE fast path.
  • Liveness guard for terminal-stamp clears racing file eviction.
  • Delete the now-caller-less vector snapshot API (SetSkipVectorBuild / ImportVectorIndex / ExportVectorIndex).
  • rows.Err() sweep of the remaining cursor family (analysis/vector/purge helpers).
  • A bulk-staged direct cold path — the prerequisite for ever deleting the staging type itself.

Validation

  • go test -race -count=1 ./... green (142 packages), before and after the review-fix commits. The one flake it surfaced — the perf fixture's latency budgets, still calibrated for the retired backend — was recalibrated against measured sqlite figures with ~13-18x headroom.
  • golangci-lint run — 0 issues.
  • Dual-backend conformance suite, the staging fence (proven to catch aliased-import evasion), and new regression tests each proven to fail without their fix.
  • Three independent adversarial review passes over the full diff; every confirmed finding fixed or listed above as a follow-up.

zzet added 25 commits August 6, 2026 01:11
The edge and node scan helpers ended their rows.Next() loop without
checking rows.Err(), so a driver failure part-way through a cursor was
indistinguishable from a clean exhaust: callers received a truncated
slice and treated it as the complete result. The raw-SQL siblings also
dropped the statement-level Query error on the floor entirely.

Every one of these helpers now checks rows.Err() and routes both it and
the Query error through panicOnFatal, which keeps degrading a teardown
race to an empty result while raising a genuine storage failure. A
single row that will not decode stays non-fatal and is skipped, so one
corrupt row still costs only that row. The qualified-name lookup's
missing-index case now asserts the raised error instead of an empty map.
A cold index that gets an in-memory staging shadow writes the backend's
symbol FTS while draining that shadow to disk. When the shadow is
refused — too many files, too many bytes, or admission denied — parsing
runs straight against the disk store, and nothing on that path writes
the FTS, because graph mutations do not maintain it. Symbol search was
therefore degraded on exactly the large repositories that refuse the
shadow, and on every streaming-flush parse.

The direct path now rebuilds its repository's FTS documents once the
index completes, streaming nodes through the store's bounded scoped
projection so a repository that did not fit in RAM is not materialised
here either. Admission and token derivation reuse the same helpers every
other FTS writer uses, so the corpus is identical whichever path built
it.
The two Store implementations answer a read differently and nothing said
so: the in-memory staging graph hands back its own live pointers, so an
in-place field assignment is already durable, while the SQLite store
decodes each row into a fresh struct and drops any mutation the caller
does not hand back through a write.

RunConformance now takes a Semantics value so each backend declares which
side it is on, and two subtests hold it there — one asserting an
unpersisted mutation is dropped (or kept) as declared on both the edge and
the node axis, one exercising the persistence capabilities that make an
attribute change stick under either semantics.
…mantics

Conformance now covers four things the Store contract left open: that a
predicate iterator's yield body may read back into the store, that more
concurrent readers than the SQLite pool holds connections for queue rather
than fail, that mutation receipts describe a window exactly or admit they
are incomplete, and what the empty repository prefix means on each read.

Took the convergent fix for ordering: the four whole-graph and adjacency
statements now declare an explicit order. It is free — each clause leads
with the column the chosen index already sorts on and breaks ties on the
rowid every index carries, so no plan grows a sorter, and a plan lock now
holds them to that. This matters because callers that take the first edge
matching a predicate would otherwise pick a different call site depending
on which index the planner chose. Whole-graph reads stay unordered by
contract: the staging graph walks hash-map shards and cannot promise a
sequence.

Also converged the batched reindex on the staging store's payload
behaviour. An entry whose identity did not move was dropped as nothing to
do, which silently discarded the resolver's terminal-stamp clears and left
those edges excluded from later resolution; such an entry is now a mutation
whose old and new keys are equal, and the existing simulator writes only
when the row actually differs.
The batch transaction journal derived its directory by taking the parent
of the daemon snapshot path, which coupled an unrelated durability
journal to the snapshot file layout. Add daemon.StateDir(), which
resolves the daemon runtime state directory the same way the socket and
PID file paths do, and have the journal join its subdirectory onto that
directly.

The resolved location is unchanged; only the derivation is.
The daemon's gob+gzip snapshot only ever served the in-memory backend:
every save and load entry point sat behind a *graph.Graph type assertion,
so on the sqlite default nothing wrote or read it. Graph rows, per-file
mtimes, contract records and the vector index all live in the store
already, so the save-on-shutdown hook, the ten-minute snapshotter, the
warm-start replay and the per-repo/contract/vector collectors go with it,
along with GORTEX_DAEMON_SNAPSHOT and the snapshot path helpers.

Prior file mtimes now come only from the backend's FileMtime sidecar
table; an empty result means the store has never seen the repo, which
routes it through a full cold index exactly as the missing-snapshot case
used to.

The boot shape-degradation guard is deleted rather than ported. It
compared a reloaded repo's live counts against a baseline that was only
ever populated by the snapshot load, so it has been a no-op on the sqlite
default all along and dies with the snapshot. A store-sourced replacement
is possible later from the persisted repo index state.

The wire-contract fingerprint test keeps its graph.Node / graph.Edge
cases — those types are still persisted row by row, and a field the
backend never learns to write still reads back zero — and drops the three
snapshot-only wire types.
The daemonless `gortex mcp` path kept a second, independent graph
snapshot: a file store it replayed nodes and edges from on startup and
wrote back on shutdown, plus the --no-cache flag that only existed to
turn it off. It is gone, and so is the flag.

The accepted consequence, stated plainly: without the snapshot cache,
every daemonless one-shot invocation cold-indexes the repository from
scratch. Run the daemon if you want an index that survives the process.

`gortex repos` loses its snapshot-store fallback with it — that store was
only ever written by this path, so the repo_index_state rows the daemon
writes are now the sole freshness source and a repo without one reports
as never indexed. Its tests move to seeding those rows.
The Store interface, its gob+gzip FileStore implementation, the no-op
store, and the (repo, branch) snapshot-slot key existed only to give the
retired graph snapshot a place on disk. Nothing calls them any more.

The sidecar database and the notes / memories / savings / frecency /
keyword / combo managers in this package are untouched — they are a
separate store with their own live callers. The gob type registrations
that rode along in the file store's init covered graph node and edge Meta
only; nothing left in this package encodes an interface value, and the
parser crash pool registers the same set for the binaries that still need
it.
The embedded MCP server used to run on an in-process graph. It now
defaults to sqlite like every other lifecycle, backed by a per-process
temp file that `gortex mcp` creates and removes on shutdown.

That path has to be explicit. The one-shot server takes no store lock,
and an empty backend path resolves to the shared store under ~/.gortex —
so a second, unsynchronised writer would land on the daemon's database.
The constructor now refuses a one-shot with no backend path instead of
silently sharing, and tests pin both the refusal and the temp location.
The intake dry run classifies files against the corpus-admission gates
and reports aggregate buckets; it parses nothing and writes nothing. It
still built a whole graph store just to satisfy the indexer constructor.

DryRunIntake is now a package-level function taking the index config, the
extension registry and a logger — the only state the gates read. It runs
on a walk-only indexer with no store, resolver or search backend behind
it, so `gortex init --dry-run-intake` allocates none of that. The
callers, including the two gate tests, pass config and registry directly.
Every eval subcommand built its own in-memory graph, which no longer
matches anything users run: the daemon serves a SQLite store, and the
indexer picks its search backend from what the store implements. All
four now open a throwaway SQLite store in a temp dir and remove it on
exit.

For `eval recall` this deliberately changes what the lexical row
measures: the text backend is now the store's own full-text index
rather than an in-process BM25 build. That is the retrieval stack the
daemon actually serves, so the row is now reproducible by a user; the
old number described a code path no deployment took. The row still
isolates lexical ranking, since the engine is pointed at the pure text
side even when embeddings are on.

For `eval embedders` the measurement is unchanged in substance — it
scores the vector index, and each variant still gets a fresh index
built from its own embeddings, now owned by the store and reached
through the same delegation the daemon uses. `eval pack` and
`eval-server` only ever needed a working store.

Widens eval.NewHandler to graph.Store to match server.NewHandler.
The perf, daemon-latency and token-efficiency harnesses each built an
in-memory graph, so every published number described a configuration
no deployment runs. All three now index into a throwaway SQLite store
and remove it when the run ends. Cold-index, search and handler
latencies therefore include real store I/O and the store's native
full-text search, which is what a daemon serves.

The perf harness's DB-size column was a calibrated per-node/per-edge
byte guess; it now sums the actual store directory, write-ahead log
included, so the column reports what the index weighs on disk. The
published fixture row and its column notes are re-measured to match.

Graph reads in these harnesses moved off whole-corpus scans and onto
per-kind iteration and the store's own counters — walking every node
to find two samples or to tally edges is one round-trip per node on a
disk backend.

The cold-index regression harness under bench/index-perf keeps its
in-memory graph: it exists to guard the parse-and-resolve path that
feeds the indexer's staging shadow, its committed baseline still
holds, and putting disk I/O under a 15% wall-clock gate would only
add variance.
The embedding engine now backs its graph with the SQLite store instead of
an in-memory graph, so it holds a database handle and a write-ahead-log
checkpointer that have to be released. New returns an error because
opening the store can fail, and the new Close is required: it closes the
store and removes the temp directory the zero-config constructor creates.

WithStorePath opts into a store that outlives the process, so an embedder
can index once and query the same graph on later runs.
The zero-blast-radius classifier was gated behind an in-memory graph, so on
the default disk-backed store an agent only saw a vague "gaps may exist"
string. Its evidence path is a handful of indexed point lookups per input
symbol, which every backend serves cheaply, so the gate is gone and the
per-symbol caveat now arms everywhere.

The cross-community coupling score went the other way: it materialised the
whole edge table once per community pair, which an interactive pre-edit gate
cannot afford. Impact now reports only the affected communities; the
dedicated coupling analysis remains the place to ask for the numbers.
The server stack now opens the sqlite store for every lifecycle. An empty
backend name still resolves to it, so the daemon and the one-shot embedded
server keep working unchanged, but the in-memory names are refused with an
error that names the replacement: --backend sqlite with a throwaway
--backend-path reproduces the old ephemeral behaviour. Falling back silently
would write a caller who asked for a scratch store into the shared database.

With one backend left, the name check no longer varies by lifecycle: the
per-lifecycle default and the is-it-sqlite predicate collapse into a single
validation that runs before the store lock is taken, and the unused HTTP
lifecycle constant goes with them (the daemon's HTTP surface has always run
as the daemon lifecycle).

*graph.Graph is untouched and still implements the whole Store contract — it
remains the indexer's cold-index staging buffer and the fixture the rest of
the tree's tests build on.
Warm restart routed reconciliation through a store-type fork: an
in-memory graph always took the full-tree incremental pipeline, while a
durable store took the changed-file census and its scoped/full-retrack
routes. The daemon only ever restores into the durable store, so the
fork was dead in production and the census route is now unconditional.

Two reconcile tests encoded the memory-only semantics; they now run
against a file-backed sqlite store and carry enough files that a single
offline change stays on the scoped route instead of escalating to a
whole-repo re-track.
The in-memory Store is no longer something a deployment can select. It
survives for two reasons: the indexer stages a cold index in it before
draining into the durable store, which is several times faster than
writing every node and edge through, and tests use it as a cheap fixture.

A new test walks the tracked non-test sources and fails if any file
outside the indexer's staging path constructs it, with a message telling
the next reader to take a real store instead. The doc comment on the
constructor and the shadow-swap site now say the same thing in prose.
The docs still described persistence as a gob+gzip snapshot written at
shutdown behind a pluggable backend. None of that is true: the SQLite
store is the graph, the daemon writes into it as it indexes and queries
it in place, and a restart reopens it and re-indexes only what changed.

Rewrites the affected sentences in the architecture diagrams and data
flow, the persistence and feature blurbs, the daemon-stop help lines,
and the one-shot server's story — that path now runs against a private
store deleted at exit. The in-memory graph appears only where it is
real: the staging buffer a cold index drains into the store.
…probe

A zero blast radius reached the agent with no safety annotation at all
when every input symbol classified as "has real incoming usage edges":
the per-symbol caveat list came back empty and nothing replaced it. Fall
back to the plain uncertainty warning in that case, so an empty impact
result always carries one or the other.

The caveat path also tested unresolved same-name candidates by reading
their whole in-edge list. Those placeholders are hubs for common method
names, so the existence check materialised thousands of edges with their
Meta blobs. Probe through the in-degree counting capability when the
backend offers it, and keep the direct edge read as the fallback.
--no-cache was deleted outright rather than shimmed. Editor mcp.json
configs are never migrated and cobra hard-errors on an unknown flag, so
a stale config would stop starting the server. Re-register it hidden and
inert alongside the other retired flags, each of which now carries its
own reason in the deprecation note.

The one-shot server's private store directory was removed only by defer,
so a SIGKILL stranded a store that can run to gigabytes. Sweep the temp
root for directories older than the TTL before allocating a new one.

Also correct --cache-dir's help and the ledger comment: the flag moves
the notes and feedback side stores, not the graph.
…t preload

The two node cursors behind the repo-scoped enumerations still ended on a
bare loop exit, so a driver failure part-way through a repo-wide node
scan handed the caller a silently truncated slice. Check rows.Err() the
way their siblings already do.

The receipt preload also still skipped entries whose identity does not
move, but the mutation builder stopped skipping them: such an entry
carries payload and now produces a write. Its source node was therefore
never preloaded, so an edge with no file of its own recorded a write
with no exact file and pushed the receipt to incomplete, falsely
widening the incremental resolution frontier for the whole window.
…ics honestly

The construction fence matched the literal "graph.New()", so an aliased
or dot import walked straight past it. Resolve the import's local name
per file with go/parser and match the New symbol through it; a reference
counts even when it is not immediately called, since taking the function
value constructs the same store.

The read-ordering conformance seeded out-edges in ascending line order,
so the in-memory store passed the "ordered by line" assertion by
replaying its append slice. Seed out of line order and have each backend
declare the order it returns: sorting the staging graph's adjacency
would tax the indexer's hot path for an order no staging caller reads,
so the honest fix is the declaration, not a sort. In-edge order within a
kind is insertion order on both backends — the suite claimed line order
and never tested it.

The adjacency plan locks also only held with every index present. Pin
the degraded plan a read gets while a bulk load has the dense edge
indexes dropped, so the per-call sort there is a stated trade-off.
The XDG paragraph put the graph store under XDG_CACHE_HOME; it resolves
under XDG_DATA_HOME, and --cache-dir moves only the side stores. The
project overview still described the graph as in-memory.

The daemon-latency numbers were measured on the retired in-memory
backend while the harness now builds a SQLite store, so both places they
are published say so rather than reading as current.
The perf harness now indexes into a temp SQLite store, and the fixture
smoke test kept the budgets it was given when the harness ran against an
in-memory graph. Measured under -race the fixture lands at search p95
~40ms against a 100ms gate and impact p95 ~27ms against a 50ms one — a
margin thin enough that running the package alongside the rest of the
suite trips the gate on a contended machine, which is what a full
-race ./... run does.

Raise both to 500ms, an order of magnitude over the observed figures.
The gates are there to catch a pathological regression, not to enforce a
latency; the published claim is checked by the strict-mode path with its
own budgets. The measured row is logged so the next recalibration starts
from a number instead of a guess.
Each of these guards bails out with Fatal and then dereferences the
pointer it just proved non-nil. Fatal does end the test, but static
analysis treats the testing helper as an ordinary call, so the
dereference reads as reachable with a nil pointer and the lint job
rejects it. Return explicitly after the bail-out.

The same shape appears in five more places than the ones reported, all
with the identical fix, so they are closed together rather than one CI
round at a time.

@peterkc peterkc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the complete SQLite-only change at the current head. The overall direction is coherent, but I found two P1 and four P2 correctness or lifecycle gaps.

Five findings have focused failing reproductions. The signal-shutdown finding was confirmed through execution-path inspection and independent review. I recommend resolving these before merge.

The existing focused tests, race tests, vet, diff check, and GitHub checks remain green. Details and suggested regression tests are inline.

Comment thread cmd/gortex/mcp.go
Comment thread internal/indexer/indexer.go Outdated
Comment thread internal/graph/store_sqlite/store.go Outdated
Comment thread cmd/gortex/repos_cmd.go Outdated
Comment thread cmd/gortex/repos_cmd.go Outdated
Comment thread cmd/gortex/daemon.go Outdated
zzet added 4 commits August 8, 2026 19:51
Directory age never proved orphanhood. The embedded MCP server's sqlite
store writes land in files INSIDE its temp directory and never bump the
directory's own mtime, so a `gortex mcp` session that had been serving
for longer than the TTL looked exactly like debris a SIGKILL left behind
— and the next launch's reaper deleted its database out from under it.

Each embedded store now takes an advisory lock inside its own directory
and holds it for the process lifetime; the reaper removes a candidate
only after non-blockingly acquiring that same lock. Occupancy is proven
by the kernel, which drops the lock even on SIGKILL, so an abandoned
store is still reaped. The age check stays as a cheap pre-filter in
front of the lock probe, not as the liveness test.

A store whose lock cannot be taken fails the allocation instead of
running unprotected: an unlockable store is indistinguishable from
debris and would be deleted under a live session.
Rebuilding a repository's symbol corpus reset it and then appended
chunks, each in its own transaction. The wipe therefore committed before
the first document was written, so anything that ended the rebuild
part-way left the repository holding whatever chunks happened to land: a
truncated corpus whose misses are indistinguishable from real ones and
that nothing repairs short of another full index. A 2,500-document
corpus rebuilt with a failure on the second chunk came back with 2,048
documents and durable false negatives for the rest.

The store now offers ReplaceSymbolFTS, which runs the wipe and every
document a producer streams through one transaction. The producer emits
bounded chunks, so neither side retains a whole-repository slice, and a
producer error, a failing chunk, or a failing commit rolls the whole
replacement back with the previous corpus intact. The chunk writer is
shared with the incremental batch path so the two cannot drift in how
they derive ownership or reuse docids.

The producer runs while the store holds its writer, which is safe to
read against on disk (separate reader pool) but not on an in-process
database, where one handle serves both. That case is refused with a
named error rather than deadlocking on its own transaction.

The shadow drain's reset-then-append is left as it is. It interleaves
node writes with FTS writes, so it cannot run inside a transaction that
holds the writer for its duration — and it has nothing to protect: the
repository's rows are evicted immediately before the drain begins, so
there is no prior corpus that a failure could destroy.
The materialising cursor scans surfaced driver errors but stepped over
any row they could not decode. A corrupt edge meta blob therefore made
queryEdgesSQL return normally with that edge simply absent, and the
caller had no way to tell the short slice from a genuinely small result
— the edge reads as a relationship the graph does not contain. The same
held for the node scan and for the light projection, whose failure shape
is a promoted scalar that will not convert rather than a bad blob.

Decode failures now go through panicOnFatal and end the scan, matching
how these loops already treat a driver failure. A teardown-race read
still degrades to what was materialised; everything else is raised.

This retires the skip-tolerant contract, which one test encoded
directly: TestGetNodesByQualNamesFailsClosedOnDecodeQueryAndClosedStore-
Errors asserted that a single undecodable row was skipped and the rest
of the lookup returned. It now expects the raise. Skipping was never
safe for that lookup in particular — its result is a map, and a dropped
entry is indistinguishable from "no such qualified name", which is the
one answer a resolver acts on irreversibly.
Every failure to read the freshness store degraded to an empty map, so
`gortex repos` answered a store full of invalid bytes with a successful
run reporting each repo as never indexed. Corruption, permission
failures and schema errors were all indistinguishable from "nothing has
been indexed yet" — a confident claim about the repos made without
having looked, which sends the user to re-index work already done.

ReadRepoIndexStates now tolerates exactly the two conditions that really
do mean nothing is recorded: no store file, and a store predating the
repo_index_state table. Everything else is returned, and `gortex repos`
fails with the path it could not read instead of printing a listing it
has no evidence for.
zzet added 5 commits August 8, 2026 20:18
`gortex repos` reads index freshness straight out of the graph store,
but only ever looked at the platform default. A daemon started with
--backend-path writes its repo_index_state rows somewhere else, so every
repo that daemon had indexed came back reported as never indexed, and
there was no way to correct it — the command took no path of its own.

The daemon now records the choices an out-of-band CLI cannot discover in
a small runtime-state file beside its PID file: the PID that wrote it and
the resolved store path. It shares the PID file's lifetime and a reader
ignores a record whose process is gone, so a killed daemon cannot route
anyone at a store nothing has open.

`gortex repos` resolves the store as --backend-path, else the running
daemon's recorded store, else the platform default — and now registers
--backend-path, which existed only as an internal variable, with the
order documented in the command help. The server stack publishes the
store file it actually opened rather than each caller re-deriving it.
The sqlite close chain hung off the controller's shutdown hook, which
only a control-socket stop reaches. A SIGINT or SIGTERM is handled
inside the daemon server: it shuts the listener down directly, Serve
returns, and the controller never hears about it — so a signalled daemon
skipped watcher shutdown, the savings flush, and the final WAL
checkpoint outright and left the store as the process found it. That is
the ordinary way a foreground daemon ends.

Both exits now converge on one sync.Once-guarded teardown: the
controller keeps it as its hook, and it is deferred around the serve
loop so whichever exit happens runs it. A daemon signalled while a stop
is already in flight reaches both paths and still closes the store once.
Later callers get the same error the run produced rather than a
misleading nil, since whoever exits second is often the one reporting.

The watcher stop is passed to the installer explicitly so the ordering
it exists for — quiesce the watchers before the backend closes — is
visible at the call site rather than buried in the chain.
…graph-backend

* origin/main: (83 commits)
  State what the startup barrier actually guarantees
  Update CONTRIBUTING.md
  Report a repository the daemon is not actually watching
  Let the watcher observe its own startup handshake on macOS
  csharp: review round — claim self-typed-field recursion, stamp cross-repo exact-type origins
  csharp: stop member calls from binding to the calling method itself
  Write down where the user, not Gortex, is carrying the risk
  Describe the boundaries the code actually enforces
  Stop the HTTP surfaces from being reachable by anyone who can route to them
  Confine the generator tools' output paths, and keep the root set fixed
  Refuse a git revision that git would read as an option
  Stop diff handlers falling back to the daemon's own working directory
  Apply the overbroad-root refusal to ScopeForCWD's own containment arm
  Refuse to bind a session to a root too broad to be anyone's project
  Keep the embedded MCP fallback out of the directory it launched from
  Bind an MCP session opened above its repos to the repos it contains
  fix(indexer): announce a point patch whose graph mutation already landed
  Verify the user-state sandbox on the platform it exists for
  Fail cmd/gortex when a test writes to the real user state
  Route every home-isolating test through the shared sandbox
  ...

# Conflicts:
#	docs/onboarding.md
The store-path resolution added its own checkBackend call, but the
constructor already rejects an unusable backend name at the top — before
the store lock is taken and before any path is resolved, which is the
whole point of doing it there. The second call could only ever agree
with the first.
…graph-backend

* origin/main:
  Bump version to v0.63.2
  docs: refresh the contributing guide and invite newcomers to Discord

# Conflicts:
#	CONTRIBUTING.md
@zzet

zzet commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Before/after timing A/B (single machine darwin/arm64, single sequential runs, isolated HOME per arm; main = 6a2905e, branch = 1239492):

measurement main branch
daemon warm start, rails (ready / enriched) 1.29s / 2.69s 1.24s / 2.71s
daemon cold start, rails (ready / enriched) 30.4s / 95.7s 21.8s / 79.1s
cold index (TestBackendBench, sqlite): 313-file repo 931ms 924ms
cold index: rails (4,530 files) 51.2s 45.3s
cold index: gortex source (4,061 files, 1.59M edges) 7m18.8s 5m29.9s

Warm start is the number that matters day-to-day and it is dead even; store sizes are identical. Cold-index runs put the branch at parity or better everywhere — I would not claim the branch is faster from single runs (run order favors the second arm's filesystem cache), only that there is no regression. Node/edge counts match across arms per corpus (gortex corpus differs by 29 edges of 1.59M, from enrichment pick order).

The one accepted regression stands as designed: daemonless gortex mcp re-launches pay a full cold index (previously snapshot replay) — for a repo of the 300-file class that is under a second of indexing; rails-class repos pay tens of seconds per launch, which is the trade documented in the PR description.

@zzet
zzet merged commit 78a79ea into main Aug 9, 2026
12 checks passed
@zzet
zzet deleted the refactor/sqlite-only-graph-backend branch August 9, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants