Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,18 @@ Format: [Semantic Versioning](https://semver.org/)

---

## [6.14.0] — 2026-06-07

### Added

- **`verify-ai-output` — Hallucination Guard prototype (Phase 1 MVP, #227, PR #228):**
- New command `sigmap verify-ai-output <answer.md> [--json]` flags fabricated claims in an AI answer against the real repository. Deterministic core — runs fully offline, no LLM.
- Three detectors: **fake-file** (referenced path absent on disk), **fake-import** (relative import does not resolve; bare import absent from `package.json` deps, with Node/Python builtins allow-listed and scoped packages handled), and **fake-symbol** (called function/class absent from the SigMap symbol index via `buildSigIndex`).
- Markdown report by default, `--json` for CI (`{ file, issues, summary }`). Exits `1` when any issue is found, `0` when clean.
- New modules `src/verify/parsers.js` (file/import/symbol/code-block extraction) and `src/verify/hallucination-guard.js` (`verify(answerText, cwd, opts)`); all external lookups are injectable so the core is unit-testable.

---

## [6.13.0] — 2026-06-05

### Added
Expand Down
3 changes: 3 additions & 0 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ To ensure proper attribution:

We welcome contributions! See [Contributing](./docs/CONTRIBUTING.md) for guidelines.

### Recent Contributors (v6.14.0)
- **@manojmallick** — feat: `verify-ai-output` Hallucination Guard prototype — deterministic fake-file / fake-import / fake-symbol detectors, markdown + `--json`, offline (#227, PR #228)

### Recent Contributors (v6.13.0)
- **@manojmallick** — feat: line anchors for JavaScript + member-level anchors (TS & JS); index-mode token cut 4.6% → 32–42% on real repos; overhead-aware token budget (#223, PR #224)

Expand Down
51 changes: 49 additions & 2 deletions docs-vp/guide/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ head:
content: "SigMap CLI Reference — every command and flag with examples"
- - meta
- property: og:description
content: "All 39 SigMap commands and flags documented with examples. ask, plan, bench, judge, validate, roots, history, --ci, --cost, --coverage, --watch, --diff, --mcp, --report, --health, weights --export/--import and more."
content: "All 40 SigMap commands and flags documented with examples. ask, plan, bench, judge, validate, roots, history, --ci, --cost, --coverage, --watch, --diff, --mcp, --report, --health, weights --export/--import and more."
- - meta
- property: og:url
content: "https://manojmallick.github.io/sigmap/guide/cli"
Expand All @@ -19,7 +19,7 @@ head:
content: "SigMap CLI Reference — every command and flag with examples"
- - meta
- name: twitter:description
content: "All 39 SigMap commands and flags documented with examples. ask, plan, bench, judge, validate, history, --ci, --cost, --coverage, --watch, --diff, --mcp, --report, --health, weights --export/--import and more."
content: "All 40 SigMap commands and flags documented with examples. ask, plan, bench, judge, validate, history, --ci, --cost, --coverage, --watch, --diff, --mcp, --report, --health, weights --export/--import and more."
- - meta
- name: twitter:image:alt
content: "SigMap CLI Reference"
Expand Down Expand Up @@ -51,6 +51,7 @@ If you are new to the product, start with the workflow pages first:
| `ask "<query>" --since <ref>` | Delta context: restrict ranked output to files changed since a git ref |
| `plan "<goal>"` | Analyze change impact and plan modifications — returns files grouped by confidence |
| `judge --response <f> --context <f>` | Rule-based groundedness scoring for LLM responses |
| `verify-ai-output <answer.md>` | Hallucination Guard — flag fake files, imports, and symbols in an AI answer (deterministic, offline) |
| `validate` | Validate config and coverage; optional query symbol check |
| `learn` | Boost, penalize, or reset learned file ranking weights |
| `weights` | Show learned file multipliers or emit them as JSON |
Expand Down Expand Up @@ -294,6 +295,52 @@ Exit code `0` = pass, `1` = fail. Use in CI to gate on response quality.

---

## verify-ai-output

Hallucination Guard (prototype). Scans an AI answer (markdown or plain text) and flags claims that do not match the repository: fake file paths, unresolvable imports, and function/class symbols that are not in the SigMap index. Fully deterministic — runs offline, no LLM API.

```bash
sigmap verify-ai-output ai-answer.md
sigmap verify-ai-output ai-answer.md --json
```

```
[sigmap] ✗ ai-answer.md — 3 issues found
fake-file: 1 fake-import: 1 fake-symbol: 1

L6 [Fake file] File not found on disk: src/extractors/nonexistent.js
L10 [Fake import] Import does not resolve: ./src/totally/madeup
L4 [Fake symbol] Symbol not found in repo index: magicallyFix()
```

Three deterministic detectors:

| Detector | Flags |
|----------|-------|
| `fake-file` | A referenced path that is not present on disk |
| `fake-import` | A relative import that does not resolve, or a bare package absent from `package.json` dependencies (Node/Python builtins and scoped packages are allow-listed) |
| `fake-symbol` | A called function/class (`` `name()` ``) absent from the SigMap symbol index (`buildSigIndex`) |

JSON output (`--json`) for CI:

```json
{
"file": "ai-answer.md",
"issues": [
{ "type": "fake-file", "value": "src/ghost.js", "line": 6, "message": "File not found on disk: src/ghost.js" }
],
"summary": { "total": 1, "byType": { "fake-file": 1, "fake-import": 0, "fake-symbol": 0 }, "clean": false, "symbolsIndexed": 288 }
}
```

| Option | Description |
|--------|-------------|
| `--json` | Emit machine-readable `{ file, issues, summary }` instead of the markdown report |

Exit code `0` = clean (no hallucinations), `1` = at least one issue found. Use in CI to gate AI-generated patches or answers before they are trusted.

---

## learn

Manual feedback loop for the ranker. Learned weights live in `.context/weights.json` and are always local to the repo.
Expand Down
18 changes: 14 additions & 4 deletions docs-vp/guide/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Roadmap
description: SigMap version history and roadmap. From v0.0 to v6.13.0, with recent releases adding demand-driven Surgical Context (get_lines MCP tool, --mode index, --since) and line anchors for JavaScript plus per-member anchors.
description: SigMap version history and roadmap. From v0.0 to v6.14.0, with recent releases adding the verify-ai-output Hallucination Guard prototype, demand-driven Surgical Context (get_lines MCP tool, --mode index, --since), and line anchors for JavaScript plus per-member anchors.
head:
- - meta
- property: og:title
Expand All @@ -22,7 +22,7 @@ head:

Fifty-eight versions shipped. MIT open source from day one.

**Stats:** 96.5% overall token reduction · 741 tests passing · 29 languages · 17-language source resolver · 0 npm deps
**Stats:** 96.5% overall token reduction · 914 tests passing · 29 languages · 17-language source resolver · 0 npm deps

## Token reduction by version

Expand Down Expand Up @@ -806,9 +806,19 @@ Widens line-anchor coverage so demand-driven retrieval actually pays off. The **

---

## Current milestone — v6.14+ (Hallucination Guard)
### v6.14.0 — Hallucination Guard prototype (`verify-ai-output`) ✓ (2026-06-07)

v6.0–v6.13.0 shipped graph-boosted retrieval with dependency-aware scoring, incremental signature cache, weights sharing, native tool instructions across all 7 adapters, MCP auto-wire for 10 AI tools, native tool registration, docs trust sync, intelligent source root detection, intent-aware retrieval with signal transparency, cross-session context memory with impact planning, JVM project structure auto-detection, enhanced monorepo JVM support, 2-hop graph boost with hub suppression, session-aware context carry-forward with safe change planning, segmented benchmarks with answer usefulness evaluation, monorepo workspace-scoped retrieval, R language support with S4 patterns, Python AST extraction for complex signatures, open-source agent/local LLM integration guides, complete Python import detection for accurate Python blast radius, line anchors on signatures (Surgical Context Phase 1), demand-driven retrieval with the `get_lines` MCP tool, `--mode index`, and `--since` delta context (Surgical Context Phase 2), and JavaScript + per-member line anchors (Phase 2.1). Next: **`verify-ai-output` (Hallucination Guard)** — a deterministic verifier that flags fake files, imports, and symbols in an AI answer against the live symbol index and file map — line anchors for the remaining extractors (Java, Go, Rust, C#, …), and performance optimizations for very large monorepos (>50K files).
The first headline verification command. `sigmap verify-ai-output <answer.md>` scans an AI answer and flags claims that do not match the repository, composing existing primitives (file map, import resolvers, symbol index) into a deterministic, offline check — no LLM. Three detectors ship in this prototype: **fake-file** (path absent on disk), **fake-import** (relative import that does not resolve, or a bare package missing from `package.json` deps — Node/Python builtins and scoped packages allow-listed), and **fake-symbol** (a called function/class absent from the SigMap symbol index). Markdown report by default, `--json` for CI; exits `1` on any issue, `0` when clean. All external lookups are injectable so the core is unit-testable.

**Tags:** `verify-ai-output` · `hallucination guard` · `fake-file` · `fake-import` · `fake-symbol` · `deterministic` · `offline` · `issue #227` · `PR #228`

**Impact:** new command surface for trust/verification; 65 integration test files pass (13 new). Foundation for the reliable MVP (closest-match suggestions, `fake-test-file`/`fake-npm-script`, 5-repo precision proof).

---

## Current milestone — v6.15+ (Hallucination Guard — Reliable MVP)

v6.0–v6.13.0 shipped graph-boosted retrieval with dependency-aware scoring, incremental signature cache, weights sharing, native tool instructions across all 7 adapters, MCP auto-wire for 10 AI tools, native tool registration, docs trust sync, intelligent source root detection, intent-aware retrieval with signal transparency, cross-session context memory with impact planning, JVM project structure auto-detection, enhanced monorepo JVM support, 2-hop graph boost with hub suppression, session-aware context carry-forward with safe change planning, segmented benchmarks with answer usefulness evaluation, monorepo workspace-scoped retrieval, R language support with S4 patterns, Python AST extraction for complex signatures, open-source agent/local LLM integration guides, complete Python import detection for accurate Python blast radius, line anchors on signatures (Surgical Context Phase 1), demand-driven retrieval with the `get_lines` MCP tool, `--mode index`, and `--since` delta context (Surgical Context Phase 2), JavaScript + per-member line anchors (Phase 2.1), and the **`verify-ai-output` (Hallucination Guard) prototype** — a deterministic verifier that flags fake files, imports, and symbols in an AI answer against the live symbol index and file map. Next: the **Hallucination Guard reliable MVP** — closest-match "did you mean?" suggestions, `fake-test-file` and `fake-npm-script` detectors, and a 5-repo precision proof — plus line anchors for the remaining extractors (Java, Go, Rust, C#, …), and performance optimizations for very large monorepos (>50K files).

---

Expand Down
4 changes: 2 additions & 2 deletions docs-vp/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ features:

<div style="max-width:840px;margin:0 auto;padding:18px 24px 0;text-align:center">
<div style="display:inline-flex;flex-wrap:wrap;gap:.5rem;justify-content:center;background:var(--vp-c-brand-soft,#ede9fe);border:1px solid rgba(124,106,247,.25);border-radius:999px;padding:.55rem .9rem;font-size:.9rem;color:var(--vp-c-text-1)">
<span><strong>Release:</strong> v6.13.0</span>
<span><strong>Release:</strong> v6.14.0</span>
<span>·</span>
<span>Line anchors for JavaScript + per-member anchors</span>
<span>verify-ai-output — Hallucination Guard prototype</span>
</div>
<div style="margin-top:.4rem;display:inline-flex;flex-wrap:wrap;gap:.5rem;justify-content:center;background:var(--vp-c-default-soft,#f3f4f6);border:1px solid rgba(0,0,0,.08);border-radius:999px;padding:.55rem .9rem;font-size:.9rem;color:var(--vp-c-text-2)">
<span><strong>Benchmark:</strong> sigmap-v6.13-main</span>
Expand Down
4 changes: 2 additions & 2 deletions gen-context.js
Original file line number Diff line number Diff line change
Expand Up @@ -6154,7 +6154,7 @@ __factories["./src/mcp/server"] = function(module, exports) {

const SERVER_INFO = {
name: 'sigmap',
version: '6.13.0',
version: '6.14.0',
description: 'SigMap MCP server — code signatures on demand',
};

Expand Down Expand Up @@ -8978,7 +8978,7 @@ const path = require('path');
const os = require('os');
const { execSync } = require('child_process');

const VERSION = '6.13.0';
const VERSION = '6.14.0';
const MARKER = '\n\n## Auto-generated signatures\n<!-- Updated by gen-context.js -->\n';

function requireSourceOrBundled(key) {
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "sigmap",
"version": "6.13.0",
"version": "6.14.0",
"description": "Zero-dependency AI context engine — 97% token reduction. No npm install. Runs on Node 18+.",
"main": "gen-context.js",
"exports": {
Expand Down
2 changes: 1 addition & 1 deletion packages/cli/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "sigmap-cli",
"version": "6.13.0",
"version": "6.14.0",
"description": "SigMap CLI wrapper — thin adapter for programmatic CLI invocation",
"main": "index.js",
"keywords": [
Expand Down
2 changes: 1 addition & 1 deletion packages/core/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "sigmap-core",
"version": "6.13.0",
"version": "6.14.0",
"description": "SigMap core library — zero-dependency code signature extraction, retrieval, and security scanning",
"main": "index.js",
"keywords": [
Expand Down
2 changes: 1 addition & 1 deletion src/mcp/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ const { readContext, searchSignatures, getMap, createCheckpoint, getRouting, exp

const SERVER_INFO = {
name: 'sigmap',
version: '6.13.0',
version: '6.14.0',
description: 'SigMap MCP server — code signatures on demand',
};

Expand Down
2 changes: 1 addition & 1 deletion version.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "6.13.0",
"version": "6.14.0",
"benchmark_date": "2026-06-05",
"benchmark_id": "sigmap-v6.13-main",
"languages": 31,
Expand Down
Loading