Code chunks land in DB with NULL language / symbol_name / symbol_type across all languages

[rayers]

Symptoms

After gbrain sync --strategy code walks a multi-language repo (1882 source files: Java + Python + C + TypeScript + JavaScript + bash + CSS + HTML + YAML + JSON, etc.), the chunks land in content_chunks with the chunk text correctly tagged in the body (e.g., [Java] dview/.../DstreamView.java:5-5 import java.awt.C), but the language, symbol_name, symbol_type, and symbol_name_qualified columns are NULL on essentially every chunk.

code-def <symbol> returns 0 hits as a result — the query filters on symbol_name which is unpopulated. code-refs still works because it's a text-scan over chunk_text.

Concrete numbers from a freshly-synced nvr repo (~31k chunks)

After gbrain sync --source <id> --strategy code finishes successfully, plus the autopilot daemon doing a few maintenance cycles plus a manual gbrain extract all:

Source language (inferred from slug)	chunks	with language	with symbol_name	with symbol_type
java	16,663	0	0	0
python	10,541	0	0	0
c	3,024	0	0	0
javascript	692	0	0	0
c-header	461	0	0	0
bash	234	0	0	0
json	117	104	0	104
html	59	0	0	0
yaml	52	0	0	0
css	50	0	0	0
cpp	14	0	0	0
typescript	1	0	0	0

JSON's 104 chunks with symbol_type are the only language with consistent metadata; everything else is empty.

The chunker code path looks plausible:

• src/core/chunkers/code.ts:542-579 calls extractSymbolName(node) and normalizeSymbolType(node.type) to populate metadata.
• src/core/import-file.ts:127, 470 reads c.metadata.symbolName and writes to the symbol_name column.
• The chunkable-node Sets at src/core/chunkers/code.ts:296-300 for Java look right (method_declaration, class_declaration, interface_declaration, etc.).

So either the chunker is computing metadata but not persisting (some pipeline disconnect), or the chunker isn't actually invoking the language grammar for code files (silent fallback to a non-AST chunker), or a downstream process (extraction, autopilot maintenance) rewrites chunks and strips metadata.

Reproduction

1. Register a code source pointing at a real multi-language repo:

   gbrain sources add my-code --path /path/to/repo --federated

2. Set strategy=code in the source's stored config (until sync --strategy code dropped on first sync via performFullSync #767's PR lands, this is needed to make sync walk code files):

   UPDATE sources SET config = '{"federated":true,"strategy":"code"}' WHERE id='my-code';

3. First sync:

   cd /path/to/repo
   gbrain sync --source my-code --strategy code --no-embed

4. Verify chunks landed:

   SELECT COUNT(*) FROM content_chunks WHERE chunk_text LIKE '[Java] %';
   -- many
   SELECT COUNT(*) FROM content_chunks WHERE language = 'java' OR symbol_name IS NOT NULL;
   -- zero or near-zero

5. Confirm code-def is empty for known symbols:

   gbrain code-def <a-class-name-from-the-repo>
   # {"symbol": "...", "count": 0, "results": []}

Suspected interaction with gbrain extract all / autopilot

A pre-extract snapshot showed __init__: 325, os: 133, stdMsg: 102 populated in Python symbol_name (from gbrain stats with strategy filter). After running gbrain extract all plus a few autopilot maintenance cycles, those counts dropped to single digits or zero. Suspect the extract / autopilot path may be rewriting content_chunks rows (e.g., for backlink reconciliation or timeline extraction) without preserving the original symbol metadata.

Environment

• gbrain v0.30.1 (dffb607) with PR fix(sync+embed+extract): code-symbol ingest + graph extraction (#767 + #769 + extract follow-ups) #768 applied locally (the strategy fix; doesn't touch chunkers or import-file)
• Engine: Postgres 16.13 (pgvector + pg_trgm) via Docker
• bun 1.3.10
• macOS 26.3.1 (build 25D771280a)

Why I'm filing as an observation rather than a fix

Root-causing this requires instrumenting the chunker output before DB write to see whether c.metadata.symbolName is populated on the way in (chunker bug vs persistence bug) and tracing the extract / autopilot paths to see if either rewrites chunks without metadata. The chunker pipeline is yours; faster for you to triage than for me to debug-explore. Happy to run any diagnostic queries or share more data.

[rayers]

Root-caused this and bundled the fix into #768 so #767 + #769 ship together (they're complementary halves of the same code-ingest path — #767 makes chunks exist, #769 keeps their metadata alive).

Root cause (more precise than the original report's "extract / autopilot suspicion"):

embed.ts has three re-upsert call sites — embedOnePage, embedAll non-stale, embedAllStale (autopilot common path). All three load chunks via getChunks() (which DOES return the metadata) and then map them to a fresh ChunkInput with only 5 fields:

const updated: ChunkInput[] = chunks.map(c => ({
  chunk_index: c.chunk_index,
  chunk_text: c.chunk_text,
  chunk_source: c.chunk_source,
  embedding: embeddingMap.get(c.chunk_index),
  token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));

The 8 metadata fields (language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified) get dropped. Then upsertChunks's ON CONFLICT clause overwrites them from EXCLUDED unconditionally — the asymmetric outlier vs embedding/embedded_at/model/embedding_image, which already use a chunk_text-gated CASE pattern. So every embed pass nukes them to NULL.

Not the chunker, not extract, not autopilot rewrites — the embed path itself.

Fix in #768 (commit 2be8040):

1. New preserveCodeMetadata(loaded, base) helper threading metadata through all three re-upsert sites in embed.ts.
2. Defense-in-depth: applied the chunk_text-gated CASE pattern to all 8 metadata columns in both postgres-engine.ts and pglite-engine.ts upsertChunks. Symmetric with the embedding column's existing pattern.
3. Three regression tests in test/embed.serial.test.ts covering --stale / --all / --slugs. Each fails on pre-fix code (8 fields come back undefined) and passes on the patched code.

Backfill needed after deploy: gbrain sync --strategy code --force --source <id> per code source, since re-embed alone never produces metadata — only the chunker does. Existing NULL chunks stay NULL until re-chunked.

Repo I confirmed on: 4875 code pages × ~10 chunks/page on a real gbrain DB, all NULL metadata. code-def returned 0 hits for every nvr/cloudapi/dragon symbol I tried.

[marshadcm]

Confirmed reproduction on gbrain 0.35.1.0 (clean upgrade from 0.18.2, monorepo of 1949 TS/JS files). Adding a third independent repro since #768 has sat 8 days without review and the bug is actively biting downstream — happy to test the patch if helpful.

Specific scenario (slightly different from the two upstream reports — symbols did exist briefly, then got nuked mid-embed):

# Fresh upgrade 0.18.2 → 0.35.1.0
gbrain apply-migrations --yes   # all 17 migrations clean

# First-time code ingest — symbols get populated correctly
gbrain sync --strategy code --source <code-source>
gbrain code-def validateSchemeAccess
# → returns 1 result, file=packages/api/src/trpc.ts ✅

# Embed pass starts (11981 stale chunks)
gbrain embed --stale
# … runs for a while, hits OpenAI TPD cap mid-way, stops at 8967/11981 …

gbrain code-def validateSchemeAccess
# → returns 0 results ❌
gbrain code-def TRPCError
# → returns 0 results ❌
gbrain code-def export   # any DEF_TYPES symbol
# → returns 0 results ❌

Re-running gbrain reindex-code --source <id> --yes --force claims 1949 reindexed, 0 failed but doesn't restore the symbol_name column — confirms @rayers's analysis that reindex-code only re-embeds (it doesn't re-chunk via tree-sitter).

Workaround we landed on: re-run gbrain sync --strategy code --source <id> to repopulate symbols, then don't run embed until #768 lands. That gives us working code-def/code-refs/code-callers at the cost of no semantic search on the code corpus.

Repro signal that might be useful for embed.serial.test.ts: the metadata loss is gradual — you can watch code-def return progressively fewer results as embed batches land. So even a partial/interrupted embed run (like one capped by 429s) corrupts metadata for the chunks it did touch. The existing 3 regression tests presumably cover this implicitly, but worth confirming.

Lock-in cost is real: we shipped CLAUDE.md guidance (`gbrain code-def` / `code-refs` over Grep) and it works for ~5 min then dies. PR #768 merging would close the loop for at least one downstream user immediately.