fix(review): retry a context-length-overflowed RAG chunk truncated instead of dropping it

[JSONbored]

Summary

• Fixes GITTENSORY-D (518 occurrences over 2 weeks, marked "regressed").
• CHUNK_CHARS is a character-count proxy for bge-m3's real 8192-token context budget. Dense/minified content has a worse char-to-token ratio than typical code, so a chunk can still exceed the model's actual token limit despite staying under the character budget — the documented, previously-diagnosed cause of production ai_embed_http_400s (see the existing #4996/#5046 comments in src/review/rag.ts).
• embedTexts already isolates a bad item so its batch siblings survive (a prior fix), but the bad item itself was always dropped entirely — losing that chunk's RAG signal outright.
• This gives embedSingleText exactly one additional retry, specifically for a detected context-length overflow (narrow regex match on the provider's own error text, so this never fires for an unrelated error), at a conservatively truncated length (4000 chars — safe even at a pessimistic ~1 char/token ratio for very dense content). A truncated-but-present vector still carries retrieval signal, which beats losing the chunk's context entirely.
• Deliberately does NOT touch CHUNK_CHARS itself — bge-m3 was chosen specifically for its large context window to minimize vector count per file (documented design goal), and shrinking the general chunk budget to marginally reduce a rare edge case would work against that for the 99%+ case that already embeds fine.

Scope

• In wantedPaths (src/, test/)
• No secrets/wallets/hotkeys/trust-scores/reward values
• No site/, CNAME, **/lovable/**
• No CHANGELOG.md edit

Validation

• npx vitest run test/unit/rag.test.ts — 98/98 pass, including 4 new tests covering: truncate-retry succeeds, truncate-retry also fails (bounded to exactly one retry, no infinite loop), a non-context-length error does NOT retry, and a short context-length-erroring text does NOT retry (nothing smaller to try)
• npx vitest run test/unit/rag.test.ts --coverage --coverage.include='src/review/rag.ts' — 100% branch coverage on every line this PR touches (confirmed via a before/after diff of the coverage report; the one pre-existing uncovered branch is unrelated and unchanged)
• npx tsc --noEmit — clean on changed files
• npm audit --audit-level=moderate — 0 vulnerabilities
• git diff --check — clean

Safety

• No secrets in code/comments/tests
• Narrow, opt-in retry (only fires for the one failure mode it can fix); every other error path is unchanged

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.05%. Comparing base (a6c8f1d) to head (b37f5fe).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5724   +/-   ##
=======================================
  Coverage   95.05%   95.05%           
=======================================
  Files         581      581           
  Lines       46164    46169    +5     
  Branches    14810    14811    +1     
=======================================
+ Hits        43880    43885    +5     
  Misses       1523     1523           
  Partials      761      761           

Flag	Coverage Δ
shard-1	43.43% <20.00%> (-0.54%)	⬇️
shard-2	35.96% <100.00%> (+0.25%)	⬆️
shard-3	32.16% <20.00%> (-0.13%)	⬇️
shard-4	31.14% <20.00%> (-1.88%)	⬇️
shard-5	32.38% <20.00%> (+0.78%)	⬆️
shard-6	44.84% <20.00%> (+0.37%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/review/rag.ts	99.60% <100.00%> (+<0.01%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-14 07:59:54 UTC}

2 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds a single bounded retry to embedSingleText: on a detected context-length-overflow error (narrow regex match) for a text longer than 4000 chars, it truncates and retries once instead of dropping the chunk. The recursive call passes the truncated text back into embedSingleText, and since the retry text length is exactly 4000 (not > 4000), the guard correctly prevents further recursion, avoiding an infinite loop. Tests cover success, failure-after-retry, non-matching error types, and short texts, and appear to exercise the real embedTexts→embedSingleText path rather than a fabricated scenario.

Nits — 4 non-blocking

• The regex in isEmbedContextLengthError (src/review/rag.ts) matches on `String(error)`, which relies on the error's message format staying stable across provider/adapter changes — worth a comment noting this is coupled to src/selfhost/ai.ts's error string shape (partially already done).
• The console.warn debug logging at src/review/rag.ts:360 is intentional structured logging consistent with the existing pattern in this file, not a debug leftover, but confirm the external brief's flag isn't pointing at something else.
• Consider extracting the magic number 4000 vs CONTEXT_OVERFLOW_RETRY_CHARS comment into a single source of truth if other retry-length constants are added later (not currently needed).
• If bge-m3's error format ever changes, the regex silently stops matching and chunks fall back to being dropped — a low-cardinality metric/log distinguishing 'context-overflow detected but retry also failed' vs 'error didn't match any known pattern' could aid future debugging.

Concerns raised — review before merging

• No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
• Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue — Link the relevant issue (for example `Closes #123`) before opening the PR.

Signal	Result	Evidence
Code review	❌ 2 blockers	1 reviewer
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 45 registered-repo PR(s), 37 merged, 321 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 45 PR(s), 321 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.
Improvement	✅ Minor	risk: clean · value: minor — Code changes are accompanied by test evidence.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, JavaScript, MDX, Shell, Solidity
• Official Gittensor activity: 45 PR(s), 321 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Explain no-issue PR.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @gittensory ask <question> answers contribution-quality Q&A with source citations and freshness.
• @gittensory chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review