Skip to content

fix(engines): pre-add v0.20 + v0.26.3 forward-reference columns in bootstrap #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brandonlipman wants to merge 1 commit into from
Closed

fix(engines): pre-add v0.20 + v0.26.3 forward-reference columns in bootstrap #682

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 42 additions & 8 deletions src/core/pglite-engine.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,12 @@ export class PGLiteEngine implements BrainEngine {
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
* - `content_chunks.search_vector` + `parent_symbol_path` + `doc_comment`
* + `symbol_name_qualified` columns (indexed by `idx_chunks_search_vector`
* and `idx_chunks_symbol_qualified`) — v0.20 Cathedral II
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5
* - `mcp_request_log.agent_name` + `params` + `error_message` columns
* (indexed by `idx_mcp_log_agent_time`) — v0.26.3
*
* **Maintenance contract:** when a future migration adds a column-with-index
* or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method
Expand Down Expand Up @@ -245,7 +250,13 @@ export class PGLiteEngine implements BrainEngine {
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='symbol_name') AS symbol_name_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='language') AS language_exists
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='language') AS language_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='content_chunks' AND column_name='search_vector') AS search_vector_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema='public' AND table_name='mcp_request_log') AS mcp_log_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='mcp_request_log' AND column_name='agent_name') AS agent_name_exists
`);
const probe = rows[0] as {
pages_exists: boolean;
Expand All @@ -257,17 +268,22 @@ export class PGLiteEngine implements BrainEngine {
chunks_exists: boolean;
symbol_name_exists: boolean;
language_exists: boolean;
search_vector_exists: boolean;
mcp_log_exists: boolean;
agent_name_exists: boolean;
};

const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
const needsLinksBootstrap = probe.links_exists
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
&& (!probe.symbol_name_exists || !probe.language_exists || !probe.search_vector_exists);
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
// v0.26.3 (v33): idx_mcp_log_agent_time in PGLITE_SCHEMA_SQL needs agent_name col.
const needsMcpLogBootstrap = probe.mcp_log_exists && !probe.agent_name_exists;

// Fresh installs (no tables yet) and modern brains both no-op.
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap) return;

console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');

Expand Down Expand Up @@ -305,14 +321,19 @@ export class PGLiteEngine implements BrainEngine {
}

if (needsChunksBootstrap) {
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
// surface (language, symbol_name, symbol_type, start_line, end_line).
// The bootstrap only adds the two columns the schema blob's partial
// indexes reference (idx_chunks_symbol_name, idx_chunks_language).
// v26 runs later via runMigrations and adds the rest idempotently.
// v26 (content_chunks_code_metadata) adds symbol_name + language; v27
// (Cathedral II) adds parent_symbol_path + doc_comment +
// symbol_name_qualified + search_vector. PGLITE_SCHEMA_SQL has indexes
// (idx_chunks_search_vector, idx_chunks_symbol_qualified) that need the
// v27 columns to exist before they run. v26 + v27 run later via
// runMigrations and are idempotent.
await this.db.exec(`
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS parent_symbol_path TEXT[];
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS doc_comment TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name_qualified TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS search_vector TSVECTOR;
`);
}

Expand All @@ -325,6 +346,19 @@ export class PGLiteEngine implements BrainEngine {
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
`);
}

if (needsMcpLogBootstrap) {
// v33 (admin_dashboard_columns_v0_26_3) adds agent_name + params +
// error_message to mcp_request_log. PGLITE_SCHEMA_SQL's
// `CREATE INDEX idx_mcp_log_agent_time ON mcp_request_log(agent_name,...)`
// crashes without agent_name. v33 runs later via runMigrations and is
// idempotent (and also handles backfill).
await this.db.exec(`
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS agent_name TEXT;
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS params JSONB;
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS error_message TEXT;
`);
}
}

async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
Expand Down
49 changes: 42 additions & 7 deletions src/core/postgres-engine.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,12 @@ export class PostgresEngine implements BrainEngine {
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
* - `content_chunks.search_vector` + `parent_symbol_path` + `doc_comment`
* + `symbol_name_qualified` columns (indexed by `idx_chunks_search_vector`
* and `idx_chunks_symbol_qualified`) — v0.20 Cathedral II
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5
* - `mcp_request_log.agent_name` + `params` + `error_message` columns
* (indexed by `idx_mcp_log_agent_time`) — v0.26.3
*
* Keep this in sync with the PGLite version; covered by
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
Expand All @@ -183,6 +188,9 @@ export class PostgresEngine implements BrainEngine {
chunks_exists: boolean;
symbol_name_exists: boolean;
language_exists: boolean;
search_vector_exists: boolean;
mcp_log_exists: boolean;
agent_name_exists: boolean;
}[]>`
SELECT
EXISTS (SELECT 1 FROM information_schema.tables
Expand All @@ -202,20 +210,28 @@ export class PostgresEngine implements BrainEngine {
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'search_vector') AS search_vector_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'mcp_request_log') AS mcp_log_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'mcp_request_log' AND column_name = 'agent_name') AS agent_name_exists
`;
const probe = probeRows[0]!;

const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
const needsLinksBootstrap = probe.links_exists
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
&& (!probe.symbol_name_exists || !probe.language_exists || !probe.search_vector_exists);
// v0.26.5: pages_deleted_at_purge_idx in SCHEMA_SQL crashes if the column
// doesn't exist yet. Migration v34 also adds it, but bootstrap runs first.
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
// v0.26.3 (v33): idx_mcp_log_agent_time in SCHEMA_SQL needs agent_name col.
const needsMcpLogBootstrap = probe.mcp_log_exists && !probe.agent_name_exists;

if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap) return;

console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');

Expand Down Expand Up @@ -253,13 +269,19 @@ export class PostgresEngine implements BrainEngine {
}

if (needsChunksBootstrap) {
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
// surface. The bootstrap only adds the two columns the schema blob's
// partial indexes reference (idx_chunks_symbol_name, idx_chunks_language).
// v26 runs later via runMigrations and adds the rest idempotently.
// v26 (content_chunks_code_metadata) adds symbol_name + language; v27
// (Cathedral II) adds parent_symbol_path + doc_comment +
// symbol_name_qualified + search_vector. The schema blob has indexes
// (idx_chunks_search_vector line 141, idx_chunks_symbol_qualified
// line 142) that need the v27 columns to exist before they run.
// v26 + v27 run later via runMigrations and are idempotent.
await conn.unsafe(`
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS parent_symbol_path TEXT[];
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS doc_comment TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name_qualified TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS search_vector TSVECTOR;
`);
}

Expand All @@ -272,6 +294,19 @@ export class PostgresEngine implements BrainEngine {
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
`);
}

if (needsMcpLogBootstrap) {
// v33 (admin_dashboard_columns_v0_26_3) adds agent_name + params +
// error_message to mcp_request_log. SCHEMA_SQL's
// `CREATE INDEX idx_mcp_log_agent_time ON mcp_request_log(agent_name,...)`
// crashes without agent_name. v33 runs later via runMigrations and is
// idempotent (and also handles backfill).
await conn.unsafe(`
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS agent_name TEXT;
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS params JSONB;
ALTER TABLE mcp_request_log ADD COLUMN IF NOT EXISTS error_message TEXT;
`);
}
}

async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
Expand Down
27 changes: 27 additions & 0 deletions test/schema-bootstrap-coverage.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,22 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
// v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_language
// ON content_chunks(language) WHERE language IS NOT NULL`.
{ kind: 'column', table: 'content_chunks', column: 'language' },
// v0.20+ Cathedral II — forward-referenced by `CREATE INDEX
// idx_chunks_search_vector ON content_chunks USING GIN(search_vector)`.
{ kind: 'column', table: 'content_chunks', column: 'search_vector' },
// v0.20+ Cathedral II — forward-referenced by `CREATE INDEX
// idx_chunks_symbol_qualified ON content_chunks(symbol_name_qualified)`.
{ kind: 'column', table: 'content_chunks', column: 'symbol_name_qualified' },
// v0.20+ Cathedral II — populated by update_chunk_search_vector trigger;
// present in PGLITE_SCHEMA_SQL CREATE TABLE definition.
{ kind: 'column', table: 'content_chunks', column: 'parent_symbol_path' },
{ kind: 'column', table: 'content_chunks', column: 'doc_comment' },
// v0.26.5 — forward-referenced by `CREATE INDEX pages_deleted_at_purge_idx
// ON pages (deleted_at) WHERE deleted_at IS NOT NULL`.
{ kind: 'column', table: 'pages', column: 'deleted_at' },
// v0.26.3 (v33) — forward-referenced by `CREATE INDEX idx_mcp_log_agent_time
// ON mcp_request_log(agent_name, created_at DESC)`.
{ kind: 'column', table: 'mcp_request_log', column: 'agent_name' },
];

test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
Expand Down Expand Up @@ -90,11 +103,25 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in

DROP INDEX IF EXISTS idx_chunks_symbol_name;
DROP INDEX IF EXISTS idx_chunks_language;
DROP INDEX IF EXISTS idx_chunks_search_vector;
DROP INDEX IF EXISTS idx_chunks_symbol_qualified;
DROP TRIGGER IF EXISTS chunk_search_vector_trigger ON content_chunks;
DROP FUNCTION IF EXISTS update_chunk_search_vector;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS language;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS parent_symbol_path;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS doc_comment;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name_qualified;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS search_vector;

DROP INDEX IF EXISTS pages_deleted_at_purge_idx;
ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at;

DROP INDEX IF EXISTS idx_mcp_log_agent_time;
DROP INDEX IF EXISTS idx_mcp_log_time_agent;
ALTER TABLE mcp_request_log DROP COLUMN IF EXISTS agent_name;
ALTER TABLE mcp_request_log DROP COLUMN IF EXISTS params;
ALTER TABLE mcp_request_log DROP COLUMN IF EXISTS error_message;
`);

// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
Expand Down