Skip to content

⚡ Bolt: optimize context building with safe caching #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SuvenSeo wants to merge 1 commit into
base: master
Choose a base branch
from
Open

⚡ Bolt: optimize context building with safe caching #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## 2026-06-04 - Dangerous Hoisting in Agentic Loops
**Learning:** Hoisting context construction (e.g., `getFullPrompt`) outside a tool execution loop in an agentic system is dangerous. Tools often modify state (tasks, memory), and if the context is not refreshed, the agent sees a stale world state, leading to redundant tool calls or failure to recognize progress.
**Action:** Keep context construction inside the loop if state changes are expected. Optimize via short-TTL caching of static components instead of hoisting.

## 2026-06-04 - Knowledge Reranking Cache
**Learning:** Semantic reranking of knowledge candidates is expensive (LLM call). Caching results based on a composite key of normalized message + sorted keywords provides significant latency reduction for repetitive queries or multi-turn tool loops.
**Action:** Use `getCache`/`setCache` with composite keys for LLM-backed processing steps.
9 changes: 9 additions & 0 deletions .jules/sentinel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## 2026-06-03 - [Timing Attack Hardening]
**Vulnerability:** Standard equality comparison (===) for secrets/tokens leaked length information via timing.
**Learning:** Even if the input is hashed, comparing variable-length inputs can still leak information if not handled carefully.
**Prevention:** Hash both inputs with SHA-256 before using `timingSafeEqual` to ensure fixed-size buffers and constant-time comparison.

## 2026-06-04 - [Cache-Based Denial of Service]
**Vulnerability:** Caching empty results or error states on database failures can effectively "break" a feature for all users until the cache expires.
**Learning:** Indiscriminate caching of function outputs can transform a transient error into a persistent outage.
**Prevention:** Only call `setCache` when the operation is confirmed successful. Avoid caching "empty" results unless they are explicitly verified as the correct logical state.
190 changes: 52 additions & 138 deletions frontend/src/lib/services/context.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,7 @@ function setCache(key, val, ttlMs) {
const TTL_5MIN = 5 * 60 * 1000;
const TTL_1MIN = 1 * 60 * 1000;
let hasWarnedMissingKnowledgeFts = false;
const EPISODE_FETCH_LIMIT = 40;
const BACKGROUND_EPISODE_LIMIT = 8;
const SESSION_BREAK_MS = 90 * 60 * 1000;
const CONTEXT_NOISE_PATTERNS = [
/^_?📋 \d+ task\(s\) logged/i,
/^_?⏰ \d+ reminder\(s\) set/i,
/^_?💡 \d+ idea\(s\) captured/i,
/^_?🧠 \d+ memory update\(s\)/i,
/^unknown command:/i,
/^❌ usage:/i,
/^⚠️ all ai providers are currently down/i,
/i(?:'| a)m not going to engage in this conversation anymore/i,
/^end of conversation/i,
/^🔴 .*test message/i,
];

const GREETING_ONLY_PATTERN = /^(hi|hii+|hello+|helloo+|hey+|heyy+|yo+|sup+|hola+|good (morning|afternoon|evening)|what'?s up|whats up|how are you)\W*$/i;
const RESPONSE_GUARDRAILS = `NON-NEGOTIABLE RESPONSE RULES:
- Follow explicit user instructions first (especially reset/delete/clear requests).
Expand All @@ -42,10 +28,6 @@ const RESPONSE_GUARDRAILS = `NON-NEGOTIABLE RESPONSE RULES:
- Keep the tone natural and collaborative; avoid robotic scripts and repeated fallback wording.
- Before finalizing your answer, self-check for contradiction with the latest user message and fix it.
- If unsure, ask one clarifying question instead of guessing.`;
const MEANINGFUL_HINTS = [
'task', 'remind', 'deadline', 'due', 'exam', 'project',
'meeting', 'decide', 'decision', 'priority', 'plan', 'commit',
];

async function rerankKnowledgeSemantically(userMessage, candidates) {
if (!candidates || candidates.length <= 1) return (candidates || []).slice(0, 5);
Expand Down Expand Up @@ -86,6 +68,12 @@ ${candidateText}`);
}

async function fetchRelevantKnowledge(userMessage, keywords) {
// Use a composite key for knowledge cache based on normalized message and keywords
const normalizedMsg = userMessage.toLowerCase().trim().replace(/\s+/g, ' ');
const cacheKey = `knowledge:${normalizedMsg}:${[...keywords].sort().join(',')}`;
const cached = getCache(cacheKey);
if (cached) return cached;
Comment on lines +71 to +75

const ftsQuery = keywords.join(' | ');
const baseQuery = supabase
.from('knowledge_base')
Expand All @@ -97,84 +85,49 @@ async function fetchRelevantKnowledge(userMessage, keywords) {
const { data: ftsRows, error: ftsError } = await baseQuery
.textSearch('fts', ftsQuery, { type: 'plain', config: 'english' });

if (!ftsError) {
return rerankKnowledgeSemantically(userMessage, ftsRows || []);
}
let results = [];
let shouldCache = false;

const message = (ftsError.message || '').toLowerCase();
const isMissingFts = ftsError.code === '42703' || message.includes('fts');
if (!isMissingFts) {
console.error('[Context] knowledge FTS query failed:', ftsError.message);
return [];
}

if (!hasWarnedMissingKnowledgeFts) {
console.warn('[Context] knowledge_base.fts missing; falling back to keyword ILIKE search.');
hasWarnedMissingKnowledgeFts = true;
}

const fallbackFilter = keywords
.map(k => `content.ilike.%${k.replace(/,/g, ' ')}%`)
.join(',');

const { data: fallbackRows, error: fallbackError } = await supabase
.from('knowledge_base')
.select('content, source, created_at')
.neq('source', 'secure_note')
.or(fallbackFilter)
.order('created_at', { ascending: false })
.limit(12);

if (fallbackError) {
console.error('[Context] knowledge fallback query failed:', fallbackError.message);
return [];
}

return rerankKnowledgeSemantically(userMessage, fallbackRows || []);
}

function compressVerboseContent(content = '') {
const text = content.replace(/\s+/g, ' ').trim();
if (!text) return '';
if (!ftsError) {
results = await rerankKnowledgeSemantically(userMessage, ftsRows || []);
shouldCache = true;
} else {
const message = (ftsError.message || '').toLowerCase();
const isMissingFts = ftsError.code === '42703' || message.includes('fts');
if (!isMissingFts) {
console.error('[Context] knowledge FTS query failed:', ftsError.message);
// Don't cache on unexpected DB errors
} else {
if (!hasWarnedMissingKnowledgeFts) {
console.warn('[Context] knowledge_base.fts missing; falling back to keyword ILIKE search.');
hasWarnedMissingKnowledgeFts = true;
}

if (text.startsWith('[Image analysis]')) {
return `[image summary] ${text.replace('[Image analysis]', '').trim().slice(0, 180)}...`;
}
if (text.startsWith('[Document:')) {
const title = text.slice(0, text.indexOf(']') + 1);
const body = text.slice(text.indexOf(']') + 1).trim();
return `${title} ${body.slice(0, 160)}...`;
}
if (text.includes('[URL content from')) {
const start = text.indexOf('[URL content from');
const end = text.indexOf(']', start);
const label = end > start ? text.slice(start, end + 1) : '[URL content]';
return `${label} ${text.slice(0, 130)}...`;
}
if (text.length > 550) {
return `${text.slice(0, 220)}...`;
const fallbackFilter = keywords
.map(k => `content.ilike.%${k.replace(/,/g, ' ')}%`)
.join(',');

const { data: fallbackRows, error: fallbackError } = await supabase
.from('knowledge_base')
.select('content, source, created_at')
.neq('source', 'secure_note')
.or(fallbackFilter)
.order('created_at', { ascending: false })
.limit(12);

if (fallbackError) {
console.error('[Context] knowledge fallback query failed:', fallbackError.message);
} else {
results = await rerankKnowledgeSemantically(userMessage, fallbackRows || []);
shouldCache = true;
}
}
}
return text;
}

function scoreEpisodeForContext(episode) {
const text = (episode.content || '').toLowerCase();
let score = episode.role === 'user' ? 2 : 1;
if (text.length < 24) score -= 1;
if (text.startsWith('[image analysis]') || text.startsWith('[document:') || text.includes('[url content from')) {
score -= 2;
}
for (const hint of MEANINGFUL_HINTS) {
if (text.includes(hint)) score += 1;
if (shouldCache) {
setCache(cacheKey, results, TTL_5MIN);
}
return score;
}

function isContextNoiseEpisode(episode) {
if (!episode || !episode.content) return true;
const text = (episode.content || '').replace(/\s+/g, ' ').trim();
if (!text) return true;
return CONTEXT_NOISE_PATTERNS.some((pattern) => pattern.test(text));
return results;
}

function isGreetingOnlyMessage(message = '') {
Expand All @@ -186,44 +139,6 @@ function isGreetingOnlyMessage(message = '') {
return GREETING_ONLY_PATTERN.test(text);
}

function selectConversationLines(episodes = []) {
if (!episodes.length) return [];

const ordered = [...episodes].sort(
(a, b) => new Date(a.created_at || 0).getTime() - new Date(b.created_at || 0).getTime()
);

let sessionStart = ordered.length - 1;
for (let i = ordered.length - 1; i > 0; i--) {
const cur = new Date(ordered[i].created_at || 0).getTime();
const prev = new Date(ordered[i - 1].created_at || 0).getTime();
if (!cur || !prev || Number.isNaN(cur) || Number.isNaN(prev)) continue;
if ((cur - prev) > SESSION_BREAK_MS) break;
sessionStart = i - 1;
}

const background = ordered.slice(0, sessionStart);
const currentSession = ordered.slice(sessionStart);

const selectedBackground = background
.map(ep => ({ ep, score: scoreEpisodeForContext(ep) }))
.sort((a, b) => b.score - a.score)
.slice(0, BACKGROUND_EPISODE_LIMIT)
.map(x => x.ep)
.sort((a, b) => new Date(a.created_at || 0).getTime() - new Date(b.created_at || 0).getTime());

const lines = [];
for (const ep of selectedBackground) {
if (isContextNoiseEpisode(ep)) continue;
lines.push(`[${ep.role}] ${compressVerboseContent(ep.content)}`);
}
for (const ep of currentSession) {
if (isContextNoiseEpisode(ep)) continue;
lines.push(`[${ep.role}] ${(ep.content || '').replace(/\s+/g, ' ').trim()}`);
}
return lines;
}

/**
* Build the full dynamic context for the AI brain.
* Called before every Groq API call to inject current state.
Expand All @@ -246,14 +161,13 @@ async function buildContext(userMessage = '') {
const cachedCore = getCache('core_memory');
const cachedPatterns = getCache('patterns');
const cachedIdeas = getCache('ideas');
const cachedWorking = getCache('working_memory');

const promises = [
// Always fresh: episodic memory window for smart context selection
supabase.from('episodic_memory').select('role, content, created_at').order('created_at', { ascending: false }).limit(EPISODE_FETCH_LIMIT),
// Always fresh: open tasks
// Always fresh: open tasks (crucial for agent tool feedback loop)
supabase.from('tasks').select('id, title, description, deadline, priority, status, follow_up_count, tier').in('status', ['open', 'snoozed']).order('priority', { ascending: true }),
// Cached 1 min: working memory
supabase.from('working_memory').select('key, value, expires_at').or(`expires_at.is.null,expires_at.gt.${now.toISOString()}`),
cachedWorking ? Promise.resolve({ data: cachedWorking }) : supabase.from('working_memory').select('key, value, expires_at').or(`expires_at.is.null,expires_at.gt.${now.toISOString()}`),
// Cached 5 min: core memory
cachedCore ? Promise.resolve({ data: cachedCore }) : supabase.from('core_memory').select('key, value').order('key'),
// Cached 5 min: patterns
Expand All @@ -263,7 +177,6 @@ async function buildContext(userMessage = '') {
];

const [
{ data: episodes },
{ data: tasks },
{ data: workingMemory },
{ data: coreMemory },
Expand All @@ -272,9 +185,10 @@ async function buildContext(userMessage = '') {
] = await Promise.all(promises);

// Update caches for slow-changing data
if (!cachedCore && coreMemory) setCache('core_memory', coreMemory, TTL_5MIN);
if (!cachedPatterns && patterns) setCache('patterns', patterns, TTL_5MIN);
if (!cachedIdeas && ideas) setCache('ideas', ideas, TTL_5MIN);
if (!cachedCore && coreMemory) setCache('core_memory', coreMemory, TTL_5MIN);
if (!cachedPatterns && patterns) setCache('patterns', patterns, TTL_5MIN);
if (!cachedIdeas && ideas) setCache('ideas', ideas, TTL_5MIN);
if (!cachedWorking && workingMemory) setCache('working_memory', workingMemory, TTL_1MIN);

// (History is now handled via the messages array in the chat completion call to prevent redundancy)

Expand Down