feat: add BM25 + semantic search artifacts

Author: jddunn

.gitignore

@@ -12,11 +12,11 @@ build/
 .next/
 out/
 
-# SQL Cache
+# SQL Cache (local/CI)
 .cache/
-*.db
-*.db-shm
-*.db-wal
+.cache/*.db
+.cache/*.db-shm
+.cache/*.db-wal
 
 # OS / Editor
 .DS_Store

CONTRIBUTING.md

@@ -0,0 +1,158 @@
+# Contributing to Frame Codex
+
+Welcome! This document is a high-level, human-friendly guide for contributing to **Frame Codex**.  
+For the full, step-by-step workflow, see [`docs/contributing/how-to-submit.md`](docs/contributing/how-to-submit.md).
+
+---
+
+## 1. What is Frame Codex?
+
+Frame Codex is a **data-only, markdown-first knowledge fabric** designed for AI systems:
+
+- **Fabric** – The entire repository: a collection of weaves.
+- **Weave** – Top-level knowledge universe (e.g. `weaves/frame/`, `weaves/wiki/`).
+- **Loom** – Any subdirectory inside a weave (topic/module, inferred from folders).
+- **Strand** – Individual markdown file at any depth inside a weave (atomic unit).
+
+There is **no UI** in this repo. The primary viewer lives at:
+
+- **Frame.dev Codex UI**: <https://frame.dev/codex>  
+- **Repo**: <https://github.com/framersai/frame.dev> (app that renders this Codex)
+
+---
+
+## 2. Recommended Contribution Path (Frame.dev UI)
+
+The easiest way to contribute is **inside the Codex UI** at [`frame.dev/codex`](https://frame.dev/codex):
+
+1. Open the Codex viewer and browse to the area you want to extend.
+2. Click **“Contribute”** in the toolbar (or use the contribution hotkey).
+3. Fill out:
+   - **Title** (required)
+   - **Summary** (20–300 chars, required)
+   - **Content** (markdown, required)
+   - **Weave + Loom** (optional – UI can suggest based on current path/content)
+   - **Tags, Difficulty, Subjects, Topics** (optional but encouraged)
+4. (Optional) Toggle **AI Enhancement** to let the pipeline refine tags and metadata.
+5. (Optional) Provide a **GitHub Personal Access Token (PAT)** to let the browser create a PR via API.
+6. Preview the generated markdown + frontmatter.
+7. Click **“Create Pull Request”**:
+   - With PAT → a PR is created via the GitHub API from your fork.
+   - Without PAT → GitHub’s web editor opens with the file pre-filled.
+
+### GitHub PAT Privacy
+
+- PAT is entered **only** into the Codex contribution modal in your browser.
+- It is stored **only in memory** in that tab while the modal is open.
+- It is **never** written to:
+  - `localStorage`
+  - `sessionStorage`
+  - IndexedDB / SQL cache
+  - Any Frame.dev backend
+- The token is sent **only** to GitHub’s API endpoints for:
+  - Forking `framersai/codex`
+  - Creating a branch + file
+  - Opening a pull request
+
+You can always skip the PAT and use the GitHub web editor flow instead.
+
+---
+
+## 3. Manual Git Workflow (Advanced)
+
+If you prefer the CLI, follow the traditional flow:
+
+1. **Fork & Clone**
+
+   ```bash
+   gh repo fork framersai/codex --clone
+   cd codex
+   ```
+
+2. **Create a Branch**
+
+   ```bash
+   git checkout -b add-my-content
+   ```
+
+3. **Add Content**
+
+   - Place files under a weave:
+
+     ```text
+     weaves/
+       [weave]/                # e.g. frame/, wiki/, technology/
+         weave.yaml
+         overview.md           # Strand at weave root
+         guides/               # Loom (folder inferred from path)
+           loom.yaml           # Optional
+           intro.md            # Strand
+           deep-dive/notes.md  # Nested loom/strand
+     ```
+
+   - See [`docs/contributing/submission-schema.md`](docs/contributing/submission-schema.md) for required fields.
+
+4. **Validate & Index**
+
+   ```bash
+   npm install
+   npm run validate
+   npm run index -- --validate
+   ```
+
+5. **Commit & Push**
+
+   ```bash
+   git add .
+   git commit -m "feat: add [your content title]"
+   git push origin add-my-content
+   ```
+
+6. **Open a PR**
+
+   ```bash
+   gh pr create --title "Add: [Your Content Title]" --body "Description of your contribution"
+   ```
+
+---
+
+## 4. Using the Frame.dev UI with Your Own Codex Repo
+
+You can reuse the Frame.dev Codex UI to render **any** Codex-style repository:
+
+1. **Fork `framersai/frame.dev`**.
+2. In your fork, create `.env.local` under `apps/frame.dev`:
+
+   ```bash
+   NEXT_PUBLIC_CODEX_REPO_OWNER=your-github-username-or-org
+   NEXT_PUBLIC_CODEX_REPO_NAME=your-codex-repo
+   NEXT_PUBLIC_CODEX_REPO_BRANCH=main
+   ```
+
+3. The viewer reads these in `components/codex/constants.ts` via `REPO_CONFIG`.
+4. Run the UI locally:
+
+   ```bash
+   cd apps/frame.dev
+   pnpm install
+   pnpm dev
+   ```
+
+5. Visit `http://localhost:3000/codex` to browse your own fabric.
+
+---
+
+## 5. Quality Guidelines
+
+- **Content**: Clear, self-contained strands with real knowledge (no placeholders).
+- **Metadata**: Fill in `title`, `summary`, `tags`, `difficulty`, `subjects`, `topics`.
+- **Structure**: Use headings, lists, and code blocks for readability.
+- **Licensing**: Contributions must be compatible with **CC-BY-4.0**.
+
+For full details, examples, and schema reference, see:
+
+- [`docs/contributing/how-to-submit.md`](docs/contributing/how-to-submit.md)
+- [`docs/contributing/submission-schema.md`](docs/contributing/submission-schema.md)
+- [`docs/openstrand-architecture.md`](docs/openstrand-architecture.md)
+
+

README.md

@@ -37,7 +37,7 @@ Frame Codex is a data-only knowledge repository designed to be the canonical sou
 - **Frame Codex**: Public markdown knowledge repository (this repo) - read-only, curated, version-controlled
 - **OpenStrand**: Full personal knowledge management platform at [openstrand.ai](https://openstrand.ai) - supports any file type (images, videos, PDFs, code), AI analysis, serialization to markdown, private workspaces, and advanced features
 
-**Schema**: Frame Codex follows the [OpenStrand schema specification](https://openstrand.ai/docs/schema) for weaves, looms, and strands. Looms are now inferred from folders (no `looms/` prefix required) and strands are any markdown files within a weave.
+**Schema**: Frame Codex follows the [OpenStrand schema specification](https://openstrand.ai/docs/schema) for weaves, looms, and strands. Looms are now inferred from folders (no `looms/` or `strands/` prefixes required) and strands are any markdown files within a weave.
 
 ## 🔄 Automated Indexing Workflow
 
@@ -132,11 +132,11 @@ Frame Codex uses [@framers/sql-storage-adapter](https://github.com/framersai/sql
 - Reduces indexing time from ~30s to ~2-5s on typical PRs (85-95% speedup)
 - Cache persists across workflow runs via GitHub Actions cache
 
-**Browser (IndexedDB):**
-- Caches fetched index data locally for offline access
-- Progressive sync with ETag-based updates
-- Instant repeat loads, no network requests for cached content
-- Quota: 50MB-1GB+ depending on browser
+**Browser (IndexedDB via Frame.dev Codex UI):**
+- Caches fetched Codex strands locally for faster reloads
+- SQL-backed cache lives entirely in your browser (IndexedDB/sql.js), never on Frame.dev servers
+- No secrets or tokens are ever stored in this cache—only public markdown content
+- Quota: 50MB–1GB+ depending on browser
 
 **Performance:**
 - First run: ~30s (full analysis, populates cache)
@@ -148,6 +148,23 @@ Frame Codex uses [@framers/sql-storage-adapter](https://github.com/framersai/sql
 SQL_CACHE_DISABLED=true  # Disable SQL caching (falls back to full indexing)
 ```
 
+### Search Data (BM25 + Semantic Embeddings)
+
+After building the main index, generate the search artifacts consumed by `frame.dev/codex`:
+
+```bash
+npm run index           # builds codex-index.json
+npm run build:search    # builds codex-search.json (BM25 + MiniLM embeddings)
+```
+
+`codex-search.json` contains:
+
+- **BM25 postings** for every token (term frequency per strand)
+- **Document metadata** (path, title, summary, weave/loom, doc length)
+- **Packed Float32 embeddings** (MiniLM-L6-v2, mean pooled, normalized) stored as base64
+
+These assets are completely static, so they can be hosted on GitHub Pages or any CDN. Frame.dev downloads them once and performs all ranking + semantic re-ranking in the browser (no server calls, no API keys).
+
 ## Repository Structure
 
 ```
@@ -188,11 +205,40 @@ Frame.dev and OpenStrand consume this content via:
 ```javascript
 // Example: Fetch a strand (file at any depth inside a weave)
 const response = await fetch(
-  'https://raw.githubusercontent.com/framersai/codex/main/weaves/frame/openstrand/architecture.md'
+  'https://raw.githubusercontent.com/framersai/codex/main/weaves/frame/overview.md'
 );
 const content = await response.text();
 ```
 
+### Using Frame.dev as the Codex Viewer
+
+The primary UI for browsing Frame Codex lives at [`https://frame.dev/codex`](https://frame.dev/codex):
+
+- **Browse**: Tree + outline view with loom/strand badges
+- **Search**: NLP-enhanced client-side search (names + content, typo-tolerant)
+- **Bookmarks & History**: Stored locally in your browser
+- **Contribution Modal**: AI-assisted PR creation with optional GitHub PAT
+
+> Privacy: The Frame.dev Codex UI stores bookmarks, history, preferences, and SQL cache **only in your browser**.  
+> GitHub Personal Access Tokens (PATs), if you choose to provide one, are held only in memory while the contribution modal is open and are sent directly to GitHub—never to any Frame.dev backend and never written to localStorage/IndexedDB/SQL.
+
+### Pointing Frame Codex UI at Your Own Repository
+
+You can reuse the Frame.dev Codex viewer to render *any* GitHub-hosted Codex-style repository:
+
+1. **Fork `framersai/frame.dev`**
+2. In the `apps/frame.dev` app, configure the Codex repo via environment variables:
+
+   ```bash
+   # .env.local
+   NEXT_PUBLIC_CODEX_REPO_OWNER=your-github-username-or-org
+   NEXT_PUBLIC_CODEX_REPO_NAME=your-codex-repo
+   NEXT_PUBLIC_CODEX_REPO_BRANCH=main
+   ```
+
+3. The viewer reads these in `components/codex/constants.ts` (`REPO_CONFIG`) and will render your repository instead of `framersai/codex`.
+4. Deploy your fork (e.g., Vercel, Netlify, GitHub Pages) and you now have a hosted Codex UI for your own knowledge fabric.
+
 ### Building the Index
 
 ```bash