[FEATURE] Library Mode

[nbbaier]

Note: This is more of a big idea than a feature request. I'd love to use this issue to noodle on the idea more!

Problem Statement

I really love this tool and want to expand my use of it outside of the CLI. Right now, supertag-cli can't be used programmatically within other typescript applications, beyond calling the CLI from a subprocess.

Proposed Solution

Refactor to create a core library that the CLI can then be based on.

Alternatives Considered

Using the cli from a subprocess, which is limited.

Use Case

While I do not have a specific use case in mind at the moment, I think it opens up broader opportunities to use the excellent base that has been created here in other apps.

Example Commands

N/A

Additional Context

I've already had claude, copilot, and amp begin work on this idea on a fork. Really early still!

Are you willing to contribute?

• I'd be willing to submit a PR for this feature
• I'd be willing to help test this feature

[nbbaier]

Also, tbc @jcfischer, happy to take the lead here!

[nbbaier]

Note: This is an exploration written by amp - you can see the thread where it comes from here

SDK Architecture Analysis

Problem Statement

The supertag-cli package currently bundles all functionality into a single CLI application. This creates limitations for users who want to:

1. Use supertag programmatically — Import the query engine, indexer, or search functions into their own Bun/TypeScript projects without needing the CLI binary
2. Build custom integrations — Create applications on top of the core logic without duplicating functionality
3. Embed the functionality — Use supertag capabilities in contexts where a CLI isn't appropriate (scripts, automation, custom tools)

The goal is to understand what architectural changes would be needed to split this into:

• A core SDK/library package containing all business logic, installable via npm/bun
• A CLI package that is purely argument parsing + calling the SDK

---

Current Architecture Analysis

High-Level Structure

The codebase has three entry points:

Entry Point	File	Purpose
Main CLI	src/index.ts	Commander-based CLI with global config
Export CLI	src/export/index.ts	Playwright browser automation
MCP Server	src/mcp/index.ts	AI tool integration via JSON-RPC

Module Categories

Command Layer (CLI-coupled)

• src/commands/*.ts — Each file defines one Commander command
• src/commands/helpers.ts — CLI-specific utilities (resolveDbPath, checkDb, formatJsonOutput)

These modules:

• Import DB, config, and services directly
• Perform argument validation via process.exit(1)
• Handle all human-facing formatting (console.log, emojis, tables)
• Make decisions based on CLI flags (--json, --pretty, --format)

Core/Business Modules (largely UI-agnostic)

• src/db/* — Database operations, indexer, query builder, migrations
• src/config/* — Configuration manager, workspace resolver, paths
• src/query/* — TanaQueryEngine, UnifiedQueryEngine, parser
• src/services/* — Aggregation, schema, graph traversal, node builder
• src/embeddings/* — TanaEmbeddingService, content filters, semantic search
• src/utils/* — Logger, errors, formatters, projections

These modules generally do not depend on Commander, process.argv, or CLI-specific formatting. They use fs, bun:sqlite, drizzle-orm, and resona (LanceDB).

The MCP Layer as a Model

The MCP tools in src/mcp/tools/*.ts are significant because they already behave like SDK functions:

// Example: src/mcp/tools/search.ts
// - Uses TanaQueryEngine, resolveWorkspaceContext
// - Returns structured JS objects, not formatted strings
// - No process.exit, no console.log

This layer demonstrates that the core logic is already extractable — the MCP tools are thin adapters over business logic that return data rather than printing it.

Where Coupling Exists

The CLI commands in src/commands/*.ts exhibit these coupling patterns:

1. Process lifecycle coupling

   if (!dbExists) {
     console.error("Database not found");
     process.exit(1);
   }

2. Output formatting in business logic

   // In handleFtsSearch
   if (outputOpts.pretty) {
     printPrettyTable(results);
   } else if (format === 'json') {
     console.log(JSON.stringify(results));
   }

3. CLI option interpretation

   const limit = parseInt(String(options.limit ?? '20'));
   const includeAncestor = options.ancestor !== false && !options.raw;

4. Business logic embedded in commands

   • search.ts contains FTS index management, date filter building, ancestor resolution
   • embed.ts contains Ollama endpoint communication, content filtering, maintenance flows

---

Exploration: What Would Need to Change

Modules That Would Move to SDK

Almost everything under these directories is SDK material:

Directory	Contents	Notes
src/db/*	Indexer, query builder, migrations, field queries	Core data layer
src/config/*	ConfigManager, workspace resolver, paths	Already parameterizable
src/query/*	Query engines, parser, types	Pure logic
src/services/*	Aggregation, schema, graph, node builder	Business services
src/embeddings/*	Embedding service, content filters, search	Semantic layer
src/types/*	Type definitions	Shared types

Modules That Would Stay in CLI

Module	Reason
src/commands/*.ts	Commander definitions, argument parsing
src/utils/format.ts	Emojis, tables, CLI tips
src/utils/output-formatter.ts	Table/CSV/JSON formatting for terminal
src/utils/output-options.ts	CLI output mode resolution
CLI portions of commands/helpers.ts	CLI error messages, process.exit wrappers

The Refactoring Pattern

For each command, the pattern would be:

Before (coupled):

// src/commands/search.ts
export function createSearchCommand() {
  return new Command('search')
    .action(async (query, options) => {
      const dbPath = resolveDbPath(options);
      if (!checkDb(dbPath)) process.exit(1);
      
      const results = await withQueryEngine({ dbPath }, async (ctx) => {
        return ctx.engine.searchNodes(query, { limit: options.limit });
      });
      
      if (options.json) {
        console.log(JSON.stringify(results));
      } else {
        printPrettyTable(results);
      }
    });
}

After (decoupled):

// @supertag/sdk/search.ts
export async function searchFts(opts: FtsSearchOptions): Promise<SearchResult[]> {
  return withQueryEngine({ dbPath: opts.dbPath }, async (ctx) => {
    return ctx.engine.searchNodes(opts.query, { limit: opts.limit });
  });
}

// @supertag/cli/commands/search.ts
export function createSearchCommand() {
  return new Command('search')
    .action(async (query, options) => {
      const dbPath = resolveDbPath(options);
      if (!checkDb(dbPath)) process.exit(1);
      
      const results = await searchFts({ dbPath, query, limit: options.limit });
      
      if (options.json) {
        console.log(JSON.stringify(results));
      } else {
        printPrettyTable(results);
      }
    });
}

The SDK function is pure: no process.exit, no console output, just input → output.

---

Constraints Discovered

Runtime Dependency: Bun-Only

The codebase uses bun:sqlite directly throughout:

import { Database } from "bun:sqlite";

This means:

• The SDK would require Bun as runtime — It cannot run under Node.js without significant refactoring
• No binary npm dependencies for installation — bun:sqlite is built into Bun, so npm install works cleanly
• Node.js support would require abstracting the DB driver — This is a large effort (XL) involving interface definitions and alternative implementations

For v1, a Bun-only SDK is the pragmatic choice. Node.js support could be added later by:

1. Creating a SqlDatabase interface
2. Implementing BunSqlDatabaseFactory and NodeSqlDatabaseFactory
3. Refactoring all DB usage to go through the interface

Note: I think one could also do something like moving to a local turso db

LanceDB/Resona Dependency

The embeddings system uses resona (LanceDB wrapper), which has its own native/WASM artifacts. This is already handled cleanly and doesn't block SDK extraction, but it's another reason to document Bun as the required runtime.

Error Handling Shift

Current pattern:

if (error) {
  console.error("Something went wrong");
  process.exit(1);
}

SDK pattern:

if (error) {
  throw new StructuredError("ERROR_CODE", "Something went wrong", { details });
}

The CLI would catch these and convert to user-friendly output. This is already partially implemented via StructuredError and handleMcpError.

---

Conclusion

Feasibility

Splitting into SDK + CLI is feasible and relatively straightforward because:

1. Core logic is already separated — The src/db, src/query, src/services, and src/embeddings modules are largely UI-agnostic
2. MCP tools provide a template — They already function as SDK-like APIs returning structured data
3. The coupling is shallow — It's primarily in the command files, not deep in the business logic

Recommended Architecture

packages/
  supertag-sdk/           # @supertag/sdk
    src/
      index.ts            # Main exports
      search.ts           # searchFts, searchSemantic, searchTagged
      query.ts            # runQuery, runAggregate
      nodes.ts            # createNode, batchCreate
      embeddings.ts       # generate, search, maintain
      sync.ts             # indexLatest, status, cleanup
      workspaces.ts       # list, get, add, remove
      db/                 # Re-exported from current src/db
      config/             # Re-exported from current src/config
      ...
    package.json

  supertag-cli/           # supertag-cli (with bin: supertag)
    src/
      index.ts            # CLI entry point
      commands/*.ts       # Commander definitions only
      utils/format.ts     # CLI formatting utilities
    package.json

  supertag-mcp/           # @supertag/mcp (with bin: supertag-mcp)
    src/
      index.ts            # MCP server entry
      tools/*.ts          # Thin wrappers calling SDK
    package.json

Key Decisions

Decision	Rationale
Bun-only for v1	Avoids large refactoring effort; Node support can come later
MCP tools as API template	They already demonstrate the right patterns
Monorepo structure	Keeps packages in sync, simplifies development
SDK exposes both high-level and low-level APIs	Power users can access TanaQueryEngine directly

What This Enables

Once implemented, users could:

import { searchFts, createNode } from '@supertag/sdk';

// Programmatic search
const results = await searchFts({
  workspace: 'main',
  query: 'meeting notes',
  limit: 10,
});

// Programmatic node creation
await createNode({
  workspace: 'main',
  supertag: 'task',
  name: 'Review Q4 metrics',
  fields: { status: 'todo', priority: 'high' },
});

This is the same functionality as supertag search "meeting notes" and supertag create task "Review Q4 metrics", but accessible without spawning a subprocess.

---

Appendix: Effort Estimates

Step	Effort	Description
Create SDK entrypoint with re-exports	S (<1h)	Export existing modules through src/sdk/index.ts
Port MCP tool logic to SDK facades	M (1-3h)	Extract core functions from MCP tools
Refactor example CLI commands	M (1-3h)	Update search, query, aggregate to use SDK
Split into monorepo packages	L (1-2d)	Create package structure, fix imports
Documentation and types	M (1-3h)	Document runtime requirements, add examples
Node.js support (optional, future)	XL (>2d)	Abstract DB driver, add Node implementation

[jcfischer]

Thanks for this thorough analysis, @nbbaier — both the issue write-up and the Amp exploration are excellent. The architectural assessment is spot-on.

I've reviewed your fork's copilot/add-headless-library-mode branch and the src/lib.ts approach of re-exporting existing modules is the right starting point. The codebase was designed with this separation in mind — the MCP tools already function as SDK-like adapters, and the core modules (src/db/, src/query/, src/services/) are largely UI-agnostic.

A few things to be aware of as you continue:

Current state: Main is at v2.1.2 and 88 commits ahead of your fork's base. Notable additions since your fork:

• Unified workspace resolver (resolveWorkspaceContext()) — this is now the canonical way to resolve workspaces (Spec 052)
• Query builder utilities (buildPagination, buildOrderBy, etc.) — Spec 055
• Universal format options (6 output formats) — Spec 060
• Structured error system (StructuredError) — Spec 073
• MCP lite mode (Spec 096)

Your lib.ts already exports some of these, but you'll want to rebase and align with the current APIs.

Architecture alignment:

• The withDatabase/withQueryEngine RAII pattern is the right abstraction for database access
• resolveWorkspaceContext() should be the primary workspace entry point (not resolveWorkspace() directly)
• The TanaQueryEngine and UnifiedQueryEngine are the query layer — exposing these gives users the most power
• For Spec 073 errors: SDK consumers should get StructuredError instances they can catch and inspect

What I'd suggest for a v1 PR:

1. Start with a src/lib.ts re-export approach (as you have)
2. Add exports field to package.json for proper ESM resolution
3. Keep it Bun-only for now (the bun:sqlite dependency is fine as-is)
4. Include the examples directory — it's great for onboarding
5. Tests that import from the library entry point to verify exports work

Happy to review a PR whenever you're ready. Since you offered to lead, go for it — I'll provide feedback on the PR itself.