Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions recipes/typed-edge-classifier/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Typed Edge Classifier

![Community Contribution](https://img.shields.io/badge/OB1_COMMUNITY-Approved_Contribution-2ea44f?style=for-the-badge&logo=github)

**Created by [@sahwan11](https://github.com/sahwan11)**

> An Opus/Haiku hybrid LLM classifier that reads pairs of thoughts and writes typed reasoning edges (`supports`, `contradicts`, `evolved_into`, `supersedes`, `depends_on`, `related_to`) into the `thought_edges` table.

## What It Does
Expand All @@ -12,7 +16,7 @@ Walks candidate pairs of thoughts (pairs that share at least N entities via `tho
- [`schemas/typed-reasoning-edges/`](../../schemas/typed-reasoning-edges/) applied (this recipe writes to `thought_edges`)
- [`entity-extraction` schema (PR #197)](https://github.com/NateBJones-Projects/OB1/pull/197) applied — this is where candidate pairs come from (thoughts that share entities via `thought_entities`). You can skip this if you only ever pass explicit `--pair UUID_A,UUID_B`.
- Node.js 18+
- Anthropic API key
- An LLM API key — `OPENROUTER_API_KEY` (preferred — one key covers every OB1 recipe) or `ANTHROPIC_API_KEY` (direct, retained for back-compat)

## Credential Tracker

Expand All @@ -26,8 +30,10 @@ FROM YOUR OPEN BRAIN SETUP
Project URL: ____________ -> OPEN_BRAIN_URL
Service-role secret: ____________ -> OPEN_BRAIN_SERVICE_KEY

ANTHROPIC
API key: ____________ -> ANTHROPIC_API_KEY
LLM PROVIDER (pick ONE)
OpenRouter key: ____________ -> OPENROUTER_API_KEY (preferred)
-- OR --
Anthropic key: ____________ -> ANTHROPIC_API_KEY (direct)

COST CAP FOR FIRST RUN
Max USD: ____________ (recommend $1-2 for a dry run first)
Expand All @@ -38,14 +44,21 @@ COST CAP FOR FIRST RUN
## Steps

1. Copy `classify-edges.mjs` into a local directory you control (or clone this recipe's folder)
2. Set the three required environment variables:
2. Set the required environment variables. `OPEN_BRAIN_URL` and `OPEN_BRAIN_SERVICE_KEY` are always required; provide ONE LLM provider key:

```bash
export OPEN_BRAIN_URL="https://YOUR-PROJECT.supabase.co"
export OPEN_BRAIN_SERVICE_KEY="..." # service_role key — server-side only

# Option A — OpenRouter (preferred; one key works across every OB1 recipe)
export OPENROUTER_API_KEY="sk-or-v1-..."

# Option B — Anthropic direct (retained for back-compat)
export ANTHROPIC_API_KEY="sk-ant-..."
```

When OpenRouter is used, the default Anthropic model names (`claude-haiku-4-5-20251001`, `claude-opus-4-7`) are auto-prefixed with `anthropic/` so OpenRouter routes correctly. Pass an already-prefixed string via `--filter-model` / `--classify-model` to override. If both keys are set, OpenRouter wins (matches the priority order in `entity-extraction-worker`).

3. Run a **dry run** first with a small limit and a low cost cap:

```bash
Expand Down Expand Up @@ -231,17 +244,17 @@ For now: flag off by default, behavior documented, decision deferred to dev-revi

## Troubleshooting

**Issue: `Missing env vars: OPEN_BRAIN_URL, OPEN_BRAIN_SERVICE_KEY, ANTHROPIC_API_KEY`**
Solution: Export all three before running. The service-role key is required because the classifier writes to `thought_edges` directly via PostgREST; the anon key won't have permission. Never commit this key or paste it into any browser-facing app.
**Issue: `Missing env vars: OPEN_BRAIN_URL, OPEN_BRAIN_SERVICE_KEY, OPENROUTER_API_KEY or ANTHROPIC_API_KEY`**
Solution: Export `OPEN_BRAIN_URL` + `OPEN_BRAIN_SERVICE_KEY` plus one LLM provider key (either `OPENROUTER_API_KEY` or `ANTHROPIC_API_KEY`). The service-role key is required because the classifier writes to `thought_edges` directly via PostgREST; the anon key won't have permission. Never commit any of these keys or paste them into a browser-facing app.

**Issue: `Candidate sampling requires thought_entities (from schemas/entity-extraction/)`**
Solution: Either apply the `entity-extraction` schema (so this recipe has a pool to sample from), or skip sampling entirely by passing `--pair UUID_A,UUID_B` for each pair you want classified.

**Issue: Classifier returns `filter_rejected` for most pairs**
Solution: That's usually correct — most co-mentioning pairs don't have a reasoning relation. If you're sure there are real relations being missed, try `--no-hybrid` to send every pair to Opus directly. Be warned: cost goes up roughly 15-20x.

**Issue: `Anthropic claude-opus-4-7: 429` (rate limit)**
Solution: The classifier now retries 429 and 5xx responses automatically with exponential backoff + jitter (base 1s, doubles each attempt, capped at 60s, up to 5 retries per call). You will see `[classify-edges] Anthropic ... 429: retry N/5 in Nms` lines on each retry. If retries still run out, drop `--parallelism` to 1 or 2; sustained 429s usually mean the account-level rate limit is saturated, not a transient burst.
**Issue: `Anthropic claude-opus-4-7: 429` or `OpenRouter anthropic/claude-opus-4-7: 429` (rate limit)**
Solution: The classifier retries 429 and 5xx responses automatically with exponential backoff + jitter (base 1s, doubles each attempt, capped at 60s, up to 5 retries per call). You will see `[classify-edges] LLM ... 429: retry N/5 in Nms` lines on each retry. If retries still run out, drop `--parallelism` to 1 or 2; sustained 429s usually mean the account-level rate limit is saturated, not a transient burst.

**Issue: Duplicate-key errors on insert**
Solution: Should not occur in this recipe — the classifier calls the `thought_edges_upsert` RPC (from `schemas/typed-reasoning-edges/schema.sql`) which uses `INSERT ... ON CONFLICT DO UPDATE`. Repeat classifications of the same `(from, to, relation)` bump `support_count`, take the max confidence, and refresh the temporal bounds (GREATEST for `valid_until`, LEAST for `valid_from`, NULL-safe). If you see a duplicate-key error, the RPC is not installed — re-apply `schemas/typed-reasoning-edges/schema.sql`.
Expand Down
117 changes: 111 additions & 6 deletions recipes/typed-edge-classifier/classify-edges.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,15 @@
* REQUIRED ENV VARS
* OPEN_BRAIN_URL e.g. https://YOUR-PROJECT.supabase.co
* OPEN_BRAIN_SERVICE_KEY service_role key (server-side only!)
* ANTHROPIC_API_KEY sk-ant-...
*
* And ONE of (OpenRouter is preferred to match the rest of OB1's recipes):
* OPENROUTER_API_KEY sk-or-v1-... (routes to Anthropic models)
* ANTHROPIC_API_KEY sk-ant-... (direct, retained for back-compat)
*
* When using OpenRouter, the default models (claude-haiku-4-5-20251001
* and claude-opus-4-7) are auto-prefixed with "anthropic/". Pass an
* already-prefixed string (e.g. "anthropic/claude-haiku-4-5") via
* --filter-model / --classify-model to override.
*
* USAGE
* node classify-edges.mjs --dry-run
Expand Down Expand Up @@ -81,7 +89,14 @@ const PRICING = {
const _warnedUnknownPricing = new Set();

function estimateCost(model, inTokens, outTokens) {
const p = PRICING[model];
// Normalize "anthropic/claude-haiku-4-5" → "claude-haiku-4-5" so a
// model passed via --filter-model with the OpenRouter prefix still
// finds its pricing row. PRICING keys remain bare Anthropic names
// because OpenRouter passes Anthropic's per-token rates through with
// only a small surcharge — close enough for the --max-cost-usd cap
// (which is a soft pre-flight bound, not exact billing).
const key = normalizeModelForPricing(model);
const p = PRICING[key];
if (!p) {
if (!_warnedUnknownPricing.has(model)) {
_warnedUnknownPricing.add(model);
Expand Down Expand Up @@ -113,7 +128,7 @@ function assertPricingKnown(args) {
if (args.hybrid) used.add(args.filterModel);
used.add(args.singleModel || args.classifyModel);

const unknown = [...used].filter((m) => !PRICING[m]);
const unknown = [...used].filter((m) => !PRICING[normalizeModelForPricing(m)]);
if (unknown.length === 0) return;

if (args.noCostCap) {
Expand Down Expand Up @@ -241,9 +256,17 @@ function printHelp() {
function loadEnv() {
const env = process.env;
const missing = [];
for (const k of ["OPEN_BRAIN_URL", "OPEN_BRAIN_SERVICE_KEY", "ANTHROPIC_API_KEY"]) {
for (const k of ["OPEN_BRAIN_URL", "OPEN_BRAIN_SERVICE_KEY"]) {
if (!env[k]) missing.push(k);
}
// Need at least one LLM provider key. Prefer OpenRouter to match the
// multi-provider pattern in entity-extraction-worker (and so a single
// OPENROUTER_API_KEY can serve every recipe in OB1).
const hasOpenrouter = Boolean(env.OPENROUTER_API_KEY);
const hasAnthropic = Boolean(env.ANTHROPIC_API_KEY);
if (!hasOpenrouter && !hasAnthropic) {
missing.push("OPENROUTER_API_KEY or ANTHROPIC_API_KEY");
}
if (missing.length > 0) {
throw new Error(`Missing env vars: ${missing.join(", ")}`);
}
Expand All @@ -254,10 +277,42 @@ function loadEnv() {
return {
OPEN_BRAIN_URL: base,
OPEN_BRAIN_SERVICE_KEY: env.OPEN_BRAIN_SERVICE_KEY,
ANTHROPIC_API_KEY: env.ANTHROPIC_API_KEY,
OPENROUTER_API_KEY: env.OPENROUTER_API_KEY || "",
ANTHROPIC_API_KEY: env.ANTHROPIC_API_KEY || "",
LLM_PROVIDER: hasOpenrouter ? "openrouter" : "anthropic",
};
}

// ── LLM provider helpers ──────────────────────────────────────────────────
//
// Both providers accept the same conceptual call (system prompt + user
// message + max_tokens) but differ in payload shape, response shape, and
// auth headers. resolveModel/resolveProvider/normalizeModelForPricing
// keep that switch contained so callLlmOnce stays readable.

/**
* When the operator passes a bare Anthropic model name like
* "claude-haiku-4-5-20251001" but the active provider is OpenRouter,
* prefix it with "anthropic/" so OpenRouter routes correctly. Already-
* prefixed names ("anthropic/...", "openai/...", etc.) pass through.
*/
function resolveModel(model, provider) {
if (provider !== "openrouter") return model;
if (!model) return model;
if (model.includes("/")) return model;
return `anthropic/${model}`;
}

/**
* For PRICING lookup, the keys live as bare Anthropic model names.
* Strip a leading "anthropic/" prefix before consulting the table so a
* model passed as "anthropic/claude-haiku-4-5" still finds its row.
*/
function normalizeModelForPricing(model) {
if (!model) return model;
return model.startsWith("anthropic/") ? model.slice("anthropic/".length) : model;
}

// ── Supabase REST client ───────────────────────────────────────────────────

function sbClient(env) {
Expand Down Expand Up @@ -424,6 +479,17 @@ function backoffDelayMs(attempt) {
}

async function callAnthropicOnce(env, model, system, userMsg, maxTokens) {
// Provider router. OpenRouter takes priority when both keys are set
// (matches entity-extraction-worker preference order). The shared
// retry policy in callAnthropic treats 429 + 5xx as retryable for
// both providers, which their public docs confirm.
if (env.LLM_PROVIDER === "openrouter") {
return callOpenRouterOnce(env, model, system, userMsg, maxTokens);
}
return callAnthropicDirectOnce(env, model, system, userMsg, maxTokens);
}

async function callAnthropicDirectOnce(env, model, system, userMsg, maxTokens) {
const res = await fetch("https://api.anthropic.com/v1/messages", {
method: "POST",
headers: {
Expand Down Expand Up @@ -455,6 +521,45 @@ async function callAnthropicOnce(env, model, system, userMsg, maxTokens) {
};
}

async function callOpenRouterOnce(env, model, system, userMsg, maxTokens) {
// OpenRouter speaks OpenAI-flavored chat completions. The Anthropic
// "system" parameter becomes the first message with role:"system";
// response.choices[0].message.content carries the text; usage uses
// prompt_tokens / completion_tokens. Auto-prefix bare Anthropic
// model names with "anthropic/" so callers don't have to.
const routedModel = resolveModel(model, "openrouter");
const res = await fetch("https://openrouter.ai/api/v1/chat/completions", {
method: "POST",
headers: {
"authorization": `Bearer ${env.OPENROUTER_API_KEY}`,
"content-type": "application/json",
},
body: JSON.stringify({
model: routedModel,
max_tokens: maxTokens,
messages: [
{ role: "system", content: system },
{ role: "user", content: userMsg },
],
}),
});
if (!res.ok) {
const body = await res.text();
const err = new Error(`OpenRouter ${routedModel}: ${res.status} ${body.slice(0, 400)}`);
err.status = res.status;
err.retryable = shouldRetryAnthropicStatus(res.status);
throw err;
}
const body = await res.json();
const raw = body?.choices?.[0]?.message?.content?.trim() ?? "";
const usage = body?.usage || {};
return {
raw,
inTokens: usage.prompt_tokens || 0,
outTokens: usage.completion_tokens || 0,
};
}

async function callAnthropic(env, model, system, userMsg, maxTokens) {
let lastErr;
for (let attempt = 0; attempt <= ANTHROPIC_RETRY_MAX; attempt++) {
Expand All @@ -468,7 +573,7 @@ async function callAnthropic(env, model, system, userMsg, maxTokens) {
}
const delay = backoffDelayMs(attempt);
console.warn(
`[classify-edges] Anthropic ${model} ${e.status || "network"}: retry ${attempt + 1}/${ANTHROPIC_RETRY_MAX} in ${delay}ms`,
`[classify-edges] LLM ${model} ${e.status || "network"}: retry ${attempt + 1}/${ANTHROPIC_RETRY_MAX} in ${delay}ms`,
);
await new Promise((r) => setTimeout(r, delay));
}
Expand Down
Loading