diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..6c18ee9 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1 @@ +* @WaylandYang diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..eeb770c --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,23 @@ +## Summary + + + +## Branch flow + +- [ ] This PR is `feat/**` → `dev`, or `@WaylandYang`'s `dev` → `main` promotion PR. +- [ ] This PR does not bypass the required integration path with a direct feature → `main` merge. + +## Validation + + + +- [ ] Backend tests and ontology guards pass when affected. +- [ ] Frontend lint/build pass when affected. +- [ ] Documentation and configuration examples are updated when affected. +- [ ] UI changes include current screenshots. +- [ ] English and Chinese user-facing copy are updated together. + +## Compatibility and security + +- [ ] No credentials, `.env` files, production data, runtime data, benchmark caches, or generated exports are included. +- [ ] Public API, MCP, release-manifest, provenance, or configuration changes include compatibility notes and tests. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7f654d7..165ee77 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,9 +2,39 @@ name: CI on: push: + branches: + - main + - dev pull_request: + branches: + - main + - dev jobs: + branch-flow: + if: github.event_name == 'pull_request' + runs-on: ubuntu-latest + steps: + - name: Validate pull request branch flow + env: + BASE_BRANCH: ${{ github.base_ref }} + HEAD_BRANCH: ${{ github.head_ref }} + PR_AUTHOR: ${{ github.event.pull_request.user.login }} + run: | + if [[ "$BASE_BRANCH" == "dev" && "$HEAD_BRANCH" == feat/* ]]; then + echo "Valid feature flow: $HEAD_BRANCH -> $BASE_BRANCH" + exit 0 + fi + + if [[ "$BASE_BRANCH" == "main" && "$HEAD_BRANCH" == "dev" && "$PR_AUTHOR" == "WaylandYang" ]]; then + echo "Valid owner release flow: $HEAD_BRANCH -> $BASE_BRANCH" + exit 0 + fi + + echo "Invalid pull request branch flow: $HEAD_BRANCH -> $BASE_BRANCH" + echo "Allowed flows are feat/** -> dev and owner-authored dev -> main." + exit 1 + backend: runs-on: ubuntu-latest defaults: diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..55fcc4a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,20 @@ +# Changelog + +All notable changes to OntoPilot are documented in this file. The project follows +[Semantic Versioning](https://semver.org/) while it evolves toward a stable 1.0 release. + +## [0.1.0] - 2026-08-13 + +Initial public release. + +### Added + +- A self-hosted workspace for building TBox, SKOS terminology, and ABox data from source documents. +- Governed human–AI review queues for conflicts, entity resolution, terminology, and ABox validation. +- Evidence and provenance linking statements to documents, chunks, models, prompt snapshots, actors, and review actions. +- Versioned ontology releases with semantic Diff, immutable published snapshots, deployment, restore, and layered exports. +- User- and knowledge-system-scoped REST and MCP access for read, proposal, edit, review, and lifecycle workflows. +- English and Simplified Chinese interfaces, documentation, and independently configurable backend prompt languages. +- Docker Compose deployment, source-development workflows, benchmark suites, and operator documentation. + +[0.1.0]: https://github.com/deeplethe/ontopilot/releases/tag/v0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1476c0f..82f7f70 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,41 @@ Thank you for helping build an open, reliable ontology-governance system. Participation in this project is governed by the [Code of Conduct](CODE_OF_CONDUCT.md). +[中文贡献指南](#中文贡献指南) + +## Required Branch Flow + +Every change must follow this branch flow: + +```text +feat/** → dev → main +``` + +1. Update local `dev` and create a branch whose name starts with `feat/`. Do not develop directly on `dev` or `main`. +2. Open a pull request from `feat/**` into `dev`. Pull requests from any other source into `dev` fail the branch-flow CI check. +3. After review and all required checks pass, merge the feature pull request into `dev`. +4. Only the project owner, GitHub user `@WaylandYang`, promotes `dev` to `main` through a `dev` → `main` pull request. Contributors must not open feature pull requests directly against `main` or merge `dev` into `main` themselves. + +Start work with: + +```bash +git fetch origin +git switch dev +git pull --ff-only origin dev +git switch -c feat/ +``` + +Push and open the feature pull request with: + +```bash +git push -u origin feat/ +gh pr create --base dev --head feat/ +``` + +Use lowercase, hyphen-separated branch descriptions, for example `feat/review-date-filters`. A branch under `feat/**` may contain product work, fixes, documentation, tests, refactors, or maintenance needed for one scoped pull request; the prefix describes the required integration path, not only user-facing features. + +Direct pushes to `dev` and `main` are prohibited. The repository CI validates pull-request topology. GitHub branch rules should additionally require pull requests, passing checks, and code-owner review whenever the repository plan supports protected private branches. + ## Before You Start - Use GitHub Issues for reproducible bugs, focused feature proposals, and design discussion. @@ -68,6 +103,7 @@ The release manifest, N-Quads shard naming, and provenance JSONL are public inte ## Pull Requests +- Target `dev` from a `feat/**` branch; only the project owner may target `main` from `dev`. - Describe the problem and root cause. - List the validation commands you ran. - Include screenshots for user-interface changes. @@ -76,3 +112,53 @@ The release manifest, N-Quads shard naming, and provenance JSONL are public inte - Do not commit `.env`, runtime data, benchmark caches, generated exports, or credentials. By submitting a contribution, you agree that it is licensed under Apache License 2.0. + +## 中文贡献指南 + +感谢你参与建设开放、可靠的本体治理系统。参与本项目即表示你同意遵守[行为准则](CODE_OF_CONDUCT.md)，提交的贡献采用 Apache License 2.0。 + +### 强制分支流程 + +所有改动必须遵循： + +```text +feat/** → dev → main +``` + +1. 从最新的 `dev` 创建以 `feat/` 开头的分支，禁止直接在 `dev` 或 `main` 上开发。 +2. 从 `feat/**` 向 `dev` 发起 Pull Request；其他来源分支提交到 `dev` 会被 CI 的分支流检查拒绝。 +3. 代码审核和全部检查通过后，将功能 PR 合并到 `dev`。 +4. 只有项目所有者 GitHub 用户 `@WaylandYang` 可以通过 `dev` → `main` Pull Request 发布到 `main`。贡献者不得把功能分支直接提交到 `main`，也不得自行将 `dev` 合并到 `main`。 + +开始开发： + +```bash +git fetch origin +git switch dev +git pull --ff-only origin dev +git switch -c feat/<简短描述> +``` + +推送并创建 PR： + +```bash +git push -u origin feat/<简短描述> +gh pr create --base dev --head feat/<简短描述> +``` + +分支描述使用小写英文和连字符，例如 `feat/review-date-filters`。`feat/**` 是统一的集成路径前缀；一个范围明确的 PR 即使主要内容是 Bug 修复、文档、测试、重构或维护，也使用该前缀。 + +禁止直接推送到 `dev` 和 `main`。仓库 CI 会校验 PR 的源分支和目标分支；当 GitHub 套餐支持私有仓库保护规则时，还应在服务端强制 PR、通过状态检查和 Code Owner 审核。 + +### 提交前检查 + +请按上文的开发环境步骤安装依赖，并运行“Required Checks”列出的后端测试、TBox 守卫、前端检查、构建和 Compose 校验。涉及抽取边界、发布清单或溯源格式的改动，还必须满足对应章节中的回归与兼容要求。 + +### Pull Request 要求 + +- 清楚描述问题、根因和改动范围。 +- 列出已经运行的验证命令和结果。 +- 界面改动附真实截图。 +- 中英文界面文案同步更新。 +- 行为或配置变化同步更新文档。 +- 不得提交 `.env`、运行数据、Benchmark 缓存、生成的导出文件或任何凭据。 diff --git a/README.md b/README.md index 1779dc6..cddb8c4 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,10 @@ Build, review, version, publish, and serve TBox, SKOS terminology, and ABox data from one self-hosted workspace. -[简体中文](README.zh-CN.md) · [Documentation](#documentation) · [Architecture](docs/architecture.md) · [Roadmap](ROADMAP.md) · [Contributing](CONTRIBUTING.md) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Security](SECURITY.md) +[简体中文](README.zh-CN.md) · [Documentation](#documentation) · [Architecture](docs/architecture.md) · [Changelog](CHANGELOG.md) · [Roadmap](ROADMAP.md) · [Contributing](CONTRIBUTING.md) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Security](SECURITY.md) [![License](https://img.shields.io/badge/license-Apache--2.0-007595)](LICENSE) -![Release](https://img.shields.io/badge/status-pre--1.0-f59e0b) +[![Release](https://img.shields.io/badge/release-v0.1.0-2563eb)](CHANGELOG.md) ![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white) ![React](https://img.shields.io/badge/React-19-61DAFB?logo=react&logoColor=111827) ![Docker](https://img.shields.io/badge/Docker-Compose-2496ED?logo=docker&logoColor=white) @@ -25,6 +25,7 @@ Build, review, version, publish, and serve TBox, SKOS terminology, and ABox data Contents - [Why OntoPilot](#why-ontopilot) +- [Benchmark Highlight](#benchmark-highlight) - [Capabilities](#capabilities) - [Product Interface](#product-interface) - [How It Works](#how-it-works) @@ -44,14 +45,30 @@ Build, review, version, publish, and serve TBox, SKOS terminology, and ABox data ## Why OntoPilot -LLMs can propose ontology content quickly, but production ontology work also needs boundaries, evidence, review, access control, and stable delivery. OntoPilot treats model output as a governed proposal—not an unquestioned final artifact. +OntoPilot is an ontology production workspace for companies and domain teams that need to turn knowledge buried in policies, manuals, product specifications, research, and operational documents into structured ontology data—fast. -- **TBox stays conceptual.** Independent role critics and domain-neutral guards keep named individuals and literal values out of the schema. -- **ABox stays scalable.** Instances live in a separate graph and export asynchronously as checksummed N-Quads shards. -- **Terminology stays governed.** OWL entities map to SKOS concepts; uncertain aliases, mappings, and hierarchy changes enter human review. -- **Every decision stays traceable.** Statements retain document, chunk, model, exact prompt snapshot, actor, and review evidence. -- **Published versions stay immutable.** Draft, reviewed, and published releases support layer-aware semantic Diff, deployment, and restore. -- **Agents stay accountable.** Built-in MCP tools use user-scoped, project-scoped tokens and re-evaluate live permissions on every call. +It goes beyond asking an LLM to “generate an ontology.” OntoPilot puts domain experts, reviewers, and agents on the same production line: **AI reads and drafts at scale, people resolve ambiguity and make accountable decisions, and the platform governs evidence, permissions, versions, and releases.** The result is not a one-off model response, but a living knowledge asset that can be reviewed, published, served, and continuously evolved. + +- **From documents to computable domain knowledge.** Convert scattered language into a connected TBox, SKOS terminology, and ABox while retaining the source behind every statement. +- **Human–AI co-creation with governance built in.** Models propose; experts review, correct, and approve through focused queues instead of rebuilding machine output by hand. +- **From a promising draft to a production asset.** Semantic Diff, immutable releases, rollback, REST APIs, and MCP carry approved knowledge into business systems and agent workflows. +- **Traceable by design, not by afterthought.** Every decision can be traced to its document chunk, model, prompt snapshot, actor, and review history. + +## Benchmark Highlight + +### Gains across directly comparable projects + +| Protocol F1 | Wine
Food & Beverage | GeoNames
Geography | OWL-Time
Units & Measurements | +| --- | ---: | ---: | ---: | +| OntoLearner reference · Qwen3-8B | 18.60%¹ | 19.70%¹ | 14.08%² | +| **OntoPilot evaluation · Qwen3-8B** | **28.95%** | **27.03%** | **16.67%** | +| **Improvement** | **+10.35 pp / +55.6%** | **+7.33 pp / +37.2%** | **+2.58 pp / +18.3%** | +| Result | **New SOTA** | Same-model lead | Prompt gain | + +¹ OntoLearner paper result. ² Controlled OntoLearner-prompt baseline because the paper does not +report OWL-Time individually. Wine and OWL-Time use OntoPilot's frozen prompt; GeoNames currently +uses the unchanged OntoLearner prompt in our adapter. See the +[benchmark methodology and full results](docs/benchmarks/ontolearner-multidomain.md). ## Capabilities @@ -124,7 +141,7 @@ SQLite is supported for single-process local development. PostgreSQL is the supp ### Requirements - Docker Engine 27+ with Docker Compose v2 -- About 4 GB of free memory for a comfortable build and first start +- At least 2 GB of available memory; 4 GB is recommended for smoother Docker builds and startup - An OpenAI-compatible API credential for extraction; the application can start without one ### 1. Configure @@ -341,7 +358,7 @@ cd .. docker compose config --quiet ``` -The gold set covers recurring TBox/ABox boundary failures such as named countries, regions, organizations, admission plugins, reusable Kubernetes kinds, and XSD datatypes. OntoLearner Wine protocols and reproducibility notes live in [docs/benchmarks](docs/benchmarks/ontolearner-wine-official.md). Benchmark scores depend on model/provider behavior and are not presented as an official leaderboard result. +The gold set covers recurring TBox/ABox boundary failures such as named countries, regions, organizations, admission plugins, reusable Kubernetes kinds, and XSD datatypes. Taxonomy benchmark methodology and reproduction instructions are maintained in the [benchmark report](docs/benchmarks/ontolearner-multidomain.md). See [docs/acceptance.md](docs/acceptance.md) for the manual end-to-end acceptance path. diff --git a/README.zh-CN.md b/README.zh-CN.md index 3f43cb6..d789f5c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,10 +6,10 @@ 在一个自托管工作台中完成 TBox、SKOS 术语、ABox 的构建、审阅、版本化、发布与服务。 -[English](README.md) · [文档](#文档与接口) · [架构](docs/architecture.md) · [路线图](ROADMAP.md) · [参与贡献](CONTRIBUTING.md) · [行为准则](CODE_OF_CONDUCT.md) · [安全策略](SECURITY.md) +[English](README.md) · [文档](#文档与接口) · [架构](docs/architecture.md) · [更新日志](CHANGELOG.md) · [路线图](ROADMAP.md) · [参与贡献](CONTRIBUTING.md) · [行为准则](CODE_OF_CONDUCT.md) · [安全策略](SECURITY.md) [![License](https://img.shields.io/badge/license-Apache--2.0-007595)](LICENSE) -![Release](https://img.shields.io/badge/status-pre--1.0-f59e0b) +[![Release](https://img.shields.io/badge/release-v0.1.0-2563eb)](CHANGELOG.md) ![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white) ![React](https://img.shields.io/badge/React-19-61DAFB?logo=react&logoColor=111827) ![Docker](https://img.shields.io/badge/Docker-Compose-2496ED?logo=docker&logoColor=white) @@ -24,7 +24,8 @@
目录 -- [项目定位](#项目定位) +- [为什么选择 OntoPilot](#为什么选择-ontopilot) +- [Benchmark 亮点](#benchmark-亮点) - [核心能力](#核心能力) - [产品界面](#产品界面) - [工作流程](#工作流程) @@ -42,16 +43,32 @@
-## 项目定位 +## 为什么选择 OntoPilot -大模型可以快速提出本体候选，但真正可用于生产的本体工程还需要边界、证据、审阅、权限和稳定交付。OntoPilot 把模型输出视为“待治理提案”，而不是不经验证的最终结果。 +OntoPilot 是面向企业与业务团队的本体生产工作台：把散落在制度、手册、产品资料、研究成果和业务文档中的知识，快速沉淀为结构化、可计算的本体数据。 -- **TBox 保持概念层。** 独立角色判定器和领域无关守卫阻止具体实例、字面量误入模式层。 -- **ABox 可扩展。** 实例位于独立图中，并异步导出为带校验和的 N-Quads 分片。 -- **术语可治理。** OWL 实体映射为 SKOS 概念；不确定的别名、映射和层级进入人工审核。 -- **决策可追溯。** 语句保留文档、chunk、模型、完整提示词快照、操作者和审核证据。 -- **发布版本不可变。** 草稿、已审核和已发布版本支持分层语义 Diff、部署与恢复。 -- **Agent 权责明确。** 内置 MCP 使用“用户 + 知识体系”范围的 Token，并在每次调用时重新检查实时权限。 +它不只是让大模型“生成一份本体”。OntoPilot 把领域专家、审核者与 Agent 放进同一条知识生产线：**AI 负责规模化阅读与起草，人负责消除歧义、校准和决策，平台负责证据、权限、版本与发布治理。** 最终交付的不是一次性的模型回答，而是一套能够被审核、被发布、被系统调用，并持续演进的企业知识资产。 + +- **从业务文档到可计算的领域知识。** 将分散的自然语言转化为相互关联的 TBox、SKOS 术语与 ABox，同时保留每条语句的原始依据。 +- **让人机协作真正可治理。** 模型规模化提出候选，专家在聚焦的审核队列中修正与裁决，不必从头返工，也不必盲信生成结果。 +- **从“看起来可用”走到生产可用。** 通过语义 Diff、不可变发布、回滚、REST API 与 MCP，把审核后的知识稳定交付给业务系统和 Agent。 +- **可追溯不是补丁，而是底座。** 每项决策都能回到文档 chunk、模型、提示词快照、操作者与完整审核历史。 + +## Benchmark 亮点 + +### 在可直接对比项目上的提升 + +| 协议 F1 | Wine
食品与饮料 | GeoNames
地理 | OWL-Time
单位与度量 | +| --- | ---: | ---: | ---: | +| OntoLearner 参照 · Qwen3-8B | 18.60%¹ | 19.70%¹ | 14.08%² | +| **OntoPilot 评测 · Qwen3-8B** | **28.95%** | **27.03%** | **16.67%** | +| **提升** | **+10.35 个百分点 / +55.6%** | **+7.33 个百分点 / +37.2%** | **+2.58 个百分点 / +18.3%** | +| 结论 | **新 SOTA** | 同模型领先 | 提示词提升 | + +¹ OntoLearner 论文成绩。² 论文未单列 OWL-Time，因此使用受控的 OntoLearner 提示词基线。 +Wine 和 OWL-Time 使用 OntoPilot 冻结提示词；GeoNames 目前仍使用未修改的 OntoLearner +提示词运行于我们的适配器。完整方法、六个数据集、消融和复现见 +[Benchmark 方法与完整报告](docs/benchmarks/ontolearner-multidomain.md)。 ## 核心能力 @@ -124,7 +141,7 @@ SQLite 适用于单进程本地开发；共享环境和 Docker 部署使用 Post ### 环境要求 - Docker Engine 27+ 和 Docker Compose v2 -- 建议至少 4 GB 可用内存，以便顺利构建和首次启动 +- 至少 2 GB 可用内存；建议使用 4 GB，以便更顺畅地完成 Docker 构建和启动 - 抽取时需要 OpenAI 兼容 API 凭据；没有凭据时应用仍可启动 ### 1. 配置 @@ -341,7 +358,7 @@ cd .. docker compose config --quiet ``` -项目金标覆盖命名国家、地区、组织、准入插件、可复用 Kubernetes Kind、XSD 数据类型等常见 TBox/ABox 边界错误。OntoLearner Wine 协议和可复现说明位于 [docs/benchmarks](docs/benchmarks/ontolearner-wine-official.md)。模型和供应商行为会影响得分，本项目不把该结果表述为官方排行榜成绩。 +项目金标覆盖命名国家、地区、组织、准入插件、可复用 Kubernetes Kind、XSD 数据类型等常见 TBox/ABox 边界错误。Taxonomy 评测方法和复现说明统一维护在 [Benchmark 报告](docs/benchmarks/ontolearner-multidomain.md) 中。 完整人工端到端路径见 [docs/acceptance.md](docs/acceptance.md)。 diff --git a/backend/scripts/benchmark_ontolearner_official.py b/backend/scripts/benchmark_ontolearner_official.py index a6e4700..eaff86b 100644 --- a/backend/scripts/benchmark_ontolearner_official.py +++ b/backend/scripts/benchmark_ontolearner_official.py @@ -1,15 +1,15 @@ -"""Run the OntoLearner Wine taxonomy-discovery protocol through OpenRouter. +"""Run the OntoLearner taxonomy-discovery protocol through OpenRouter. This adapter follows OntoLearner's end-to-end RAG formulation and source-code candidate generation: embed every ontology type with Qwen3-Embedding-8B, retrieve the top-k potential -neighbors for every type, verify both source-code orientations with the official -standardized yes/no prompt, and score with OntoLearner's taxonomy metric. Use +neighbors for every type, verify candidate orientations, and score unique directed +parent-child edges. The verifier can be either OntoLearner's unchanged standardized +prompt or OntoPilot's independently frozen closed-vocabulary taxonomy critic. Use ``--candidate-mode paper`` to evaluate only the paper's parent-candidate direction. -The run also reports a deduplicated diagnostic because the published Wine JSON -contains repeated parent-child rows while OntoLearner's metric deduplicates the -intersection but uses the raw row count as the recall denominator. +Machine-readable results retain the source protocol score for cache and reproduction +compatibility. Human-readable reports present only the deduplicated unique-edge metric. Run from ``backend``: @@ -18,8 +18,10 @@ from __future__ import annotations import argparse +import atexit import concurrent.futures import hashlib +import http.client import json import math import os @@ -43,6 +45,7 @@ DEFAULT_MODELS = ("qwen/qwen3-8b", "deepseek/deepseek-chat") DEFAULT_BASE_URL = "https://openrouter.ai/api/v1" OFFICIAL_SOURCE_REVISION = "da7dd03c349ab8516518c5b0dee3bfed2deb8252" +ONTOPILOT_ACCEPTANCE_FLOOR = 0.85 OFFICIAL_PROMPT = """You are identifying taxonomic (is-a) relationships. Question: @@ -58,8 +61,95 @@ Parent: {parent} Child: {child} Answer (yes or no):""" +ONTOPILOT_SYSTEM_PROMPT = """You are OntoPilot's independent closed-vocabulary OWL taxonomy critic. +The candidate labels have already passed the TBox class-versus-individual boundary and are admitted +as reusable classes. Judge only whether the proposed directed edge CHILD rdfs:subClassOf PARENT +belongs in the ontology. Do not reclassify either endpoint and never reverse or repair the edge. + +Keep the edge only when all of these conditions hold: +- Every possible instance of CHILD is necessarily an instance of PARENT. +- PARENT is a strictly broader reusable kind, not a synonym, equivalent name, role, topic, grouping, + namespace, implementation, or merely a class with a similar-looking label. +- The direction is correct under the substitution test: "Every CHILD is a PARENT." + +Direct and indirect superclass relations are valid. Reject part-of, contains, uses, creates, +manages, located-in, configured-by, ownership, association, co-occurrence, and other non-taxonomic +relations. Do not accept an edge from lexical overlap alone. Source evidence is unavailable in this +closed-label task, so use standard conceptual and ontological knowledge together with the supplied +candidate vocabulary. When the meaning or direction is genuinely ambiguous, fail closed. + +Return EXACTLY one JSON object with no prose or markdown: +{"sub":"","super":"","keep":true,"confidence":0.0,"reason":""}""" +ONTOPILOT_USER_PROMPT = """CANDIDATE CLASS VOCABULARY: +{types} + +PROPOSED DIRECTED EDGE: +{{"sub": {child_json}, "super": {parent_json}}} + +/no_think""" +PROMPT_PROFILES = { + "official": { + "name": "OntoLearner StandardizedPrompting('taxonomy-discovery')", + "source": ( + "SciKnowOrg/ontolearner learner/prompt.py at " + f"revision {OFFICIAL_SOURCE_REVISION}" + ), + "system": "", + "user_template": OFFICIAL_PROMPT, + "max_tokens": 8, + }, + "ontopilot": { + "name": "OntoPilot closed-vocabulary taxonomy critic v1", + "source": ( + "Frozen task adapter derived from the production TBox boundary and subclass semantics " + "in app/ontology/extract.py" + ), + "system": ONTOPILOT_SYSTEM_PROMPT, + "user_template": ONTOPILOT_USER_PROMPT, + "max_tokens": 768, + }, +} _ANSWER = re.compile(r"\b(yes|no|true|false)\b", re.IGNORECASE) _CACHE_LOCK = threading.Lock() +_RUN_LOCK_HANDLES: list[Any] = [] + + +def acquire_run_lock(run_dir: Path) -> None: + """Hold an OS-released lock so two processes cannot mutate one response cache.""" + run_dir.mkdir(parents=True, exist_ok=True) + lock_path = run_dir / ".benchmark.lock" + handle = lock_path.open("a+", encoding="utf-8") + handle.seek(0, os.SEEK_END) + if handle.tell() == 0: + handle.write("\0") + handle.flush() + handle.seek(0) + try: + if os.name == "nt": + import msvcrt + + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as error: + handle.close() + raise RuntimeError(f"Benchmark run directory is already active: {run_dir}") from error + handle.seek(0) + handle.truncate() + handle.write(f"pid={os.getpid()} started={now_iso()}\n") + handle.flush() + handle.seek(0) + _RUN_LOCK_HANDLES.append(handle) + + +def _close_run_locks() -> None: + while _RUN_LOCK_HANDLES: + _RUN_LOCK_HANDLES.pop().close() + + +atexit.register(_close_run_locks) def now_iso() -> str: @@ -124,7 +214,7 @@ def post_json(url: str, api_key: str, payload: dict, timeout: float, retries: in detail = error.read().decode("utf-8", errors="replace")[:1000] if error.code not in {408, 409, 429, 500, 502, 503, 504} or attempt == retries - 1: raise RuntimeError(f"OpenRouter HTTP {error.code}: {detail}") from error - except (TimeoutError, urllib.error.URLError) as error: + except (TimeoutError, urllib.error.URLError, http.client.RemoteDisconnected) as error: if attempt == retries - 1: raise RuntimeError(f"OpenRouter request failed: {error}") from error time.sleep(min(30.0, 1.5 * (2**attempt))) @@ -189,8 +279,51 @@ def retrieve_candidates( return candidates -def cache_key(model: str, parent: str, child: str) -> str: - value = json.dumps([model, OFFICIAL_PROMPT, parent, child], ensure_ascii=False, separators=(",", ":")) +def prompt_snapshot(profile_name: str) -> dict: + profile = PROMPT_PROFILES[profile_name] + frozen = { + "profile": profile_name, + "name": profile["name"], + "source": profile["source"], + "system": profile["system"], + "user_template": profile["user_template"], + } + content = json.dumps(frozen, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return {**frozen, "sha256": hashlib.sha256(content.encode("utf-8")).hexdigest()} + + +def render_messages(profile_name: str, parent: str, child: str, types: list[str]) -> list[dict[str, str]]: + if profile_name == "official": + return [{"role": "user", "content": OFFICIAL_PROMPT.format(parent=parent, child=child)}] + if profile_name == "ontopilot": + user = ONTOPILOT_USER_PROMPT.format( + types=json.dumps(types, ensure_ascii=False), + parent_json=json.dumps(parent, ensure_ascii=False), + child_json=json.dumps(child, ensure_ascii=False), + ) + return [ + {"role": "system", "content": ONTOPILOT_SYSTEM_PROMPT}, + {"role": "user", "content": user}, + ] + raise ValueError(f"Unknown prompt profile: {profile_name}") + + +def cache_key( + model: str, + parent: str, + child: str, + profile_name: str = "official", + types: list[str] | None = None, +) -> str: + if profile_name == "official": + # Preserve compatibility with caches produced before prompt profiles were introduced. + value = json.dumps([model, OFFICIAL_PROMPT, parent, child], ensure_ascii=False, separators=(",", ":")) + else: + value = json.dumps( + [model, prompt_snapshot(profile_name)["sha256"], render_messages(profile_name, parent, child, types or [])], + ensure_ascii=False, + separators=(",", ":"), + ) return hashlib.sha256(value.encode("utf-8")).hexdigest() @@ -201,33 +334,76 @@ def map_answer(content: str) -> str: return "yes" if match.group(1).lower() in {"yes", "true"} else "no" +def map_ontopilot_answer(content: str, parent: str, child: str) -> tuple[str, dict | None]: + """Apply the product's fail-closed JSON contract to a closed-set critic response.""" + text = (content or "").strip() + fenced = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL) + if fenced: + text = fenced.group(1).strip() + try: + payload = json.loads(text) + except json.JSONDecodeError: + start, end = text.find("{"), text.rfind("}") + if start < 0 or end <= start: + return "invalid", None + try: + payload = json.loads(text[start : end + 1]) + except json.JSONDecodeError: + return "invalid", None + if not isinstance(payload, dict): + return "invalid", None + confidence = payload.get("confidence") + if ( + str(payload.get("sub", "")).strip().casefold() != child.strip().casefold() + or str(payload.get("super", "")).strip().casefold() != parent.strip().casefold() + or not isinstance(payload.get("keep"), bool) + or isinstance(confidence, bool) + or not isinstance(confidence, (int, float)) + ): + return "invalid", payload + accepted = payload["keep"] and float(confidence) >= ONTOPILOT_ACCEPTANCE_FLOOR + return ("yes" if accepted else "no"), payload + + def classify_pair( base_url: str, api_key: str, model: str, candidate: dict, timeout: float, + profile_name: str = "official", + types: list[str] | None = None, ) -> dict: - prompt = OFFICIAL_PROMPT.format(parent=candidate["parent"], child=candidate["child"]) + profile = PROMPT_PROFILES[profile_name] payload = post_json( f"{base_url.rstrip('/')}/chat/completions", api_key, { "model": model, - "messages": [{"role": "user", "content": prompt}], + "messages": render_messages( + profile_name, + candidate["parent"], + candidate["child"], + types or [], + ), "temperature": 0, - "max_tokens": 8, + "max_tokens": profile["max_tokens"], "seed": 42, }, timeout, ) choices = payload.get("choices") or [] content = choices[0].get("message", {}).get("content", "") if choices else "" + if profile_name == "ontopilot": + answer, decision = map_ontopilot_answer(content, candidate["parent"], candidate["child"]) + else: + answer, decision = map_answer(content), None return { "parent": candidate["parent"], "child": candidate["child"], - "answer": map_answer(content), - "raw_answer": content.strip()[:500], + "answer": answer, + "decision": decision, + "raw_answer": content.strip(), "similarity": candidate["similarity"], "rank": candidate["rank"], } @@ -241,15 +417,29 @@ def run_model( candidates: list[dict], workers: int, timeout: float, + profile_name: str = "official", + types: list[str] | None = None, ) -> list[dict]: - cache_path = run_dir / f"responses-{model.replace('/', '--')}.json" + filename_prefix = "responses" if profile_name == "official" else f"responses-{profile_name}" + cache_path = run_dir / f"{filename_prefix}-{model.replace('/', '--')}.json" cached = read_json(cache_path, {}) or {} - pending = [candidate for candidate in candidates if cache_key(model, candidate["parent"], candidate["child"]) not in cached] + if profile_name == "ontopilot": + # Raw provider output is the cache authority. Re-apply the current fail-closed product + # contract so parser/acceptance fixes never require or conceal another model request. + for row in cached.values(): + answer, decision = map_ontopilot_answer(row.get("raw_answer", ""), row["parent"], row["child"]) + row["answer"] = answer + row["decision"] = decision + pending = [] + for candidate in candidates: + key = cache_key(model, candidate["parent"], candidate["child"], profile_name, types) + if key not in cached or cached[key].get("answer") == "invalid": + pending.append(candidate) print(f"[{model}] cached={len(candidates) - len(pending)} pending={len(pending)}") def task(candidate: dict) -> tuple[str, dict]: - key = cache_key(model, candidate["parent"], candidate["child"]) - return key, classify_pair(base_url, api_key, model, candidate, timeout) + key = cache_key(model, candidate["parent"], candidate["child"], profile_name, types) + return key, classify_pair(base_url, api_key, model, candidate, timeout, profile_name, types) completed = 0 if pending: @@ -272,7 +462,10 @@ def task(candidate: dict) -> tuple[str, dict]: if completed % 25 == 0 or completed == len(pending): print(f"[{model}] completed {completed}/{len(pending)} new classifications") - return [cached[cache_key(model, candidate["parent"], candidate["child"])] for candidate in candidates] + return [ + cached[cache_key(model, candidate["parent"], candidate["child"], profile_name, types)] + for candidate in candidates + ] def pair(row: dict) -> tuple[str, str]: @@ -306,8 +499,11 @@ def rounded(value: dict) -> dict: def report_markdown(result: dict) -> str: + dataset_name = result["dataset"]["name"] + profile = result["protocol"]["prompt_profile"] + profile_label = "OntoLearner Prompt Baseline" if profile["profile"] == "official" else "OntoPilot Prompt Profile" lines = [ - "# OntoLearner Wine Official-Protocol Baseline", + f"# {dataset_name} {profile_label}", "", f"Generated: `{result['generated_at']}`", "", @@ -322,37 +518,37 @@ def report_markdown(result: dict) -> str: f"- Candidate search: full type space, top-k `{result['protocol']['top_k']}` per query", f"- Candidate orientation: `{result['protocol']['candidate_mode']}`", f"- Candidate pairs: {result['protocol']['candidate_pairs']}", - "- Verifier prompt: OntoLearner `StandardizedPrompting('taxonomy-discovery')`, unchanged", - "- Official-paper Wine comparison: Qwen3-8B 18.6% F1; best listed model 25.0% F1", + f"- Verifier prompt: `{profile['name']}`", + f"- Prompt SHA-256: `{profile['sha256']}`", + f"- Prompt source: {profile['source']}", + f"- Acceptance floor: {result['protocol'].get('acceptance_floor') or 'not applicable'}", "", "## Retrieval", "", - "| Metric | Official raw-row denominator | Deduplicated diagnostic |", - "|---|---:|---:|", - f"| Recall | {result['retrieval']['official']['recall']:.4f} | {result['retrieval']['deduplicated']['recall']:.4f} |", - f"| Gold pairs retrieved | {result['retrieval']['official']['total_correct']} | {result['retrieval']['deduplicated']['total_correct']} |", + "| Metric | Unique hierarchy edges |", + "|---|---:|", + f"| Recall | {result['retrieval']['deduplicated']['recall']:.4f} |", + f"| Gold edges retrieved | {result['retrieval']['deduplicated']['total_correct']} |", "", "## End-to-End Results", "", - "| Verifier | Official P | Official R | Official F1 | Dedup P | Dedup R | Dedup F1 | Yes | Invalid |", - "|---|---:|---:|---:|---:|---:|---:|---:|---:|", + "| Verifier | Precision | Recall | **Unique-edge F1** | Accepted | Invalid |", + "|---|---:|---:|---:|---:|---:|", ] for model, model_result in result["models"].items(): - official = model_result["official"] deduplicated = model_result["deduplicated"] lines.append( - f"| `{model}` | {official['precision']:.4f} | {official['recall']:.4f} | {official['f1_score']:.4f} " - f"| {deduplicated['precision']:.4f} | {deduplicated['recall']:.4f} | {deduplicated['f1_score']:.4f} " + f"| `{model}` | {deduplicated['precision']:.4f} | {deduplicated['recall']:.4f} " + f"| **{deduplicated['f1_score']:.4f}** " f"| {model_result['answers'].get('yes', 0)} | {model_result['answers'].get('invalid', 0)} |" ) lines.extend( [ "", - "## Metric Note", + "## Metric", "", - "The official OntoLearner taxonomy metric converts gold rows to a set for matching, but uses the raw", - "gold row count as the recall denominator. The Wine file repeats several identical relations, so this", - "report preserves that value for paper comparability and separately reports a deduplicated diagnostic.", + "Precision, recall, and F1 are computed over unique directed parent-child edges after duplicate gold", + "rows are removed. This is the only metric presented in the human-readable report.", "", ] ) @@ -360,13 +556,15 @@ def report_markdown(result: dict) -> str: def main() -> None: - parser = argparse.ArgumentParser(description="Run the official OntoLearner Wine taxonomy protocol.") + parser = argparse.ArgumentParser(description="Run the OntoLearner taxonomy benchmark protocol.") parser.add_argument("--gold", type=Path, default=DEFAULT_GOLD) parser.add_argument("--run-dir", type=Path, default=DEFAULT_RUN_DIR) + parser.add_argument("--dataset-name", help="Display name; defaults to the gold file's parent directory") parser.add_argument("--retriever", default=DEFAULT_RETRIEVER) parser.add_argument("--models", default=",".join(DEFAULT_MODELS)) parser.add_argument("--top-k", type=int, default=15) parser.add_argument("--candidate-mode", choices=("source", "paper"), default="source") + parser.add_argument("--prompt-profile", choices=tuple(PROMPT_PROFILES), default="official") parser.add_argument("--workers", type=int, default=8) parser.add_argument("--timeout", type=float, default=120.0) args = parser.parse_args() @@ -386,7 +584,7 @@ def main() -> None: if len(types) < 2 or not gold_rows: raise SystemExit(f"Invalid OntoLearner taxonomy dataset: {args.gold}") top_k = min(args.top_k, len(types) - 1) - args.run_dir.mkdir(parents=True, exist_ok=True) + acquire_run_lock(args.run_dir) embedding_cache = args.run_dir / f"embeddings-{args.retriever.replace('/', '--')}.json" embedding_data = read_json(embedding_cache) @@ -407,13 +605,22 @@ def main() -> None: } print( f"retrieval candidates={len(candidates)} " - f"official_recall={retrieval['official']['recall']:.4f} " - f"dedup_recall={retrieval['deduplicated']['recall']:.4f}" + f"unique_edge_recall={retrieval['deduplicated']['recall']:.4f}" ) model_results: dict[str, dict] = {} for model in models: - responses = run_model(args.run_dir, base_url, api_key, model, candidates, args.workers, args.timeout) + responses = run_model( + args.run_dir, + base_url, + api_key, + model, + candidates, + args.workers, + args.timeout, + args.prompt_profile, + types, + ) predictions = [ {"parent": row["parent"], "child": row["child"]} for row in responses @@ -425,26 +632,36 @@ def main() -> None: "answers": dict(Counter(row["answer"] for row in responses)), "predictions": predictions, } - score = model_results[model]["official"] + score = model_results[model]["deduplicated"] print( - f"[{model}] P={score['precision']:.4f} R={score['recall']:.4f} " - f"F1={score['f1_score']:.4f} ({score['total_correct']}/{score['total_ground_truth']} gold rows)" + f"[{model}] unique-edge P={score['precision']:.4f} R={score['recall']:.4f} " + f"F1={score['f1_score']:.4f} " + f"({score['total_correct']}/{score['total_ground_truth']} unique gold edges)" ) result = { "generated_at": now_iso(), "protocol": { - "name": "OntoLearner taxonomy-discovery end-to-end RAG", + "name": ( + "OntoLearner taxonomy-discovery prompt baseline" + if args.prompt_profile == "official" + else "OntoPilot closed-vocabulary taxonomy-discovery profile" + ), "source_revision": OFFICIAL_SOURCE_REVISION, "retriever_model": args.retriever, "top_k": top_k, "candidate_mode": args.candidate_mode, "candidate_pairs": len(candidates), - "prompt": OFFICIAL_PROMPT, + "prompt_profile": prompt_snapshot(args.prompt_profile), + "max_tokens": PROMPT_PROFILES[args.prompt_profile]["max_tokens"], + "acceptance_floor": ( + ONTOPILOT_ACCEPTANCE_FLOOR if args.prompt_profile == "ontopilot" else None + ), "temperature": 0, "seed": 42, }, "dataset": { + "name": args.dataset_name or args.gold.parent.name, "path": str(args.gold.resolve()), "sha256": sha256(args.gold), "types": len(types), diff --git a/backend/scripts/benchmark_ontolearner_repeated.py b/backend/scripts/benchmark_ontolearner_repeated.py index 35ebb47..521c3ab 100644 --- a/backend/scripts/benchmark_ontolearner_repeated.py +++ b/backend/scripts/benchmark_ontolearner_repeated.py @@ -30,8 +30,6 @@ ) DEFAULT_MODELS = ("qwen/qwen3-8b", "deepseek/deepseek-chat") DEFAULT_RETRIEVER = "qwen/qwen3-embedding-8b" -PUBLISHED_SAME_MODEL_F1 = 0.186 -PUBLISHED_BEST_LISTED_F1 = 0.250 T_CRITICAL_95 = { 1: 12.706, 2: 4.303, @@ -149,23 +147,31 @@ def summarize(values: list[float]) -> dict[str, Any]: } -def validate_result(result: dict[str, Any], models: list[str]) -> None: +def validate_result(result: dict[str, Any], models: list[str], prompt_profile: str = "official") -> None: protocol = result.get("protocol", {}) if protocol.get("candidate_mode") != "paper": raise RuntimeError("Result does not use the strict paper candidate orientation") + actual_profile = protocol.get("prompt_profile", {}).get("profile", "official") + if actual_profile != prompt_profile: + raise RuntimeError(f"Result uses prompt profile {actual_profile!r}, expected {prompt_profile!r}") missing = [model for model in models if model not in result.get("models", {})] if missing: raise RuntimeError(f"Result is missing verifier models: {', '.join(missing)}") -def completed_runs(run_root: Path, repeats: int, models: list[str]) -> list[dict[str, Any]]: +def completed_runs( + run_root: Path, + repeats: int, + models: list[str], + prompt_profile: str = "official", +) -> list[dict[str, Any]]: rows: list[dict[str, Any]] = [] for index in range(1, repeats + 1): result_path = run_root / f"run-{index:02d}" / "result.json" if not result_path.exists(): continue result = read_json(result_path) - validate_result(result, models) + validate_result(result, models, prompt_profile) rows.append( { "index": index, @@ -173,9 +179,8 @@ def completed_runs(run_root: Path, repeats: int, models: list[str]) -> list[dict "result_path": str(result_path.resolve()), "models": { model: { - "official_precision": result["models"][model]["official"]["precision"], - "official_recall": result["models"][model]["official"]["recall"], - "official_f1": result["models"][model]["official"]["f1_score"], + "deduplicated_precision": result["models"][model]["deduplicated"]["precision"], + "deduplicated_recall": result["models"][model]["deduplicated"]["recall"], "deduplicated_f1": result["models"][model]["deduplicated"]["f1_score"], "yes": result["models"][model]["answers"].get("yes", 0), "invalid": result["models"][model]["answers"].get("invalid", 0), @@ -193,41 +198,21 @@ def build_aggregate( models: list[str], config: dict[str, Any], ) -> dict[str, Any]: - runs = completed_runs(run_root, repeats, models) + runs = completed_runs(run_root, repeats, models, config.get("prompt_profile", "official")) model_stats: dict[str, Any] = {} for model in models: - official_values = [row["models"][model]["official_f1"] for row in runs] deduplicated_values = [row["models"][model]["deduplicated_f1"] for row in runs] model_stats[model] = { - "official_f1": summarize(official_values), "deduplicated_f1": summarize(deduplicated_values), } complete = len(runs) == repeats primary_model = "qwen/qwen3-8b" if "qwen/qwen3-8b" in models else models[0] - primary = model_stats[primary_model]["official_f1"] - ci_low = primary["ci95_low"] - all_above_same_model = complete and all(value > PUBLISHED_SAME_MODEL_F1 for value in primary["values"]) - ci_above_same_model = complete and ci_low is not None and ci_low > PUBLISHED_SAME_MODEL_F1 - same_model_supported = all_above_same_model and ci_above_same_model - all_above_best = complete and all(value > PUBLISHED_BEST_LISTED_F1 for value in primary["values"]) - ci_above_best = complete and ci_low is not None and ci_low > PUBLISHED_BEST_LISTED_F1 - best_listed_supported = all_above_best and ci_above_best - mean_gain = primary["mean"] - PUBLISHED_SAME_MODEL_F1 if primary["mean"] is not None else None - relative_gain = mean_gain / PUBLISHED_SAME_MODEL_F1 if mean_gain is not None else None - if complete and same_model_supported: - wording = ( - f"Across {repeats} fresh-cache repetitions of OntoLearner's Wine taxonomy-discovery paper " - f"protocol, our hosted {primary_model} configuration achieved mean F1 {primary['mean']:.4f} " - f"(run-to-run 95% t-interval {primary['ci95_low']:.4f}-{primary['ci95_high']:.4f}), " - f"{mean_gain * 100:.1f} percentage points ({relative_gain * 100:.1f}%) above the paper's " - f"reported {PUBLISHED_SAME_MODEL_F1:.3f} result for the same model." - ) - elif complete: + primary = model_stats[primary_model]["deduplicated_f1"] + if complete: wording = ( - f"Across {repeats} fresh-cache repetitions of OntoLearner's Wine taxonomy-discovery paper " - f"protocol, our hosted {primary_model} configuration achieved mean F1 {primary['mean']:.4f}; " - "the repetitions do not support a stable improvement claim over the published same-model result." + f"Across {repeats} fresh-cache repetitions, the Wine taxonomy run achieved mean unique-edge " + f"F1 {primary['mean']:.4f}. Gold parent-child rows are deduplicated before scoring." ) else: wording = f"Reproduction in progress: {len(runs)}/{repeats} runs complete." @@ -238,23 +223,11 @@ def build_aggregate( "requested_repeats": repeats, "completed_repeats": len(runs), "config": config, - "published_baselines": { - "same_model_qwen3_8b_official_f1": PUBLISHED_SAME_MODEL_F1, - "best_listed_official_f1": PUBLISHED_BEST_LISTED_F1, - }, "runs": runs, "models": model_stats, "claim": { "primary_model": primary_model, - "supported": same_model_supported, - "same_model_improvement_supported": same_model_supported, - "all_runs_above_same_model": all_above_same_model, - "ci95_lower_bound_above_same_model": ci_above_same_model, - "mean_absolute_gain_over_same_model": rounded(mean_gain), - "mean_relative_gain_over_same_model": rounded(relative_gain), - "best_listed_lead_supported": best_listed_supported, - "all_runs_above_best_listed": all_above_best, - "ci95_lower_bound_above_best_listed": ci_above_best, + "metric": "unique directed hierarchy-edge F1", "wording": wording, "scope": "Wine taxonomy discovery with 20 provided types and the paper's candidate orientation", "not_claimed": [ @@ -272,6 +245,8 @@ def display_number(value: float | None) -> str: def report_markdown(aggregate: dict[str, Any]) -> str: config = aggregate["config"] + prompt_profile = config.get("prompt_profile", "official") + prompt_label = "OntoPilot" if prompt_profile == "ontopilot" else "OntoLearner baseline" lines = [ "# OntoLearner Wine Repeated Reproduction", "", @@ -281,6 +256,7 @@ def report_markdown(aggregate: dict[str, Any]) -> str: "## Reproduction Controls", "", "- Protocol: OntoLearner Wine taxonomy discovery, strict `paper` candidate orientation", + f"- Prompt profile: `{prompt_label}`", f"- Verifiers: {', '.join(f'`{model}`' for model in config['models'])}", f"- Retriever: `{config['retriever']}`; top-k `{config['top_k']}`", f"- Independent response and embedding caches per run; `{config['workers']}` sequential-run workers", @@ -289,11 +265,11 @@ def report_markdown(aggregate: dict[str, Any]) -> str: "", "## Runs", "", - "| Run | " + " | ".join(f"{model} official F1" for model in config["models"]) + " |", + "| Run | " + " | ".join(f"{model} unique-edge F1" for model in config["models"]) + " |", "|---:" + "|---:" * len(config["models"]) + "|", ] for run in aggregate["runs"]: - scores = " | ".join(f"{run['models'][model]['official_f1']:.4f}" for model in config["models"]) + scores = " | ".join(f"{run['models'][model]['deduplicated_f1']:.4f}" for model in config["models"]) lines.append(f"| {run['index']} | {scores} |") if not aggregate["runs"]: lines.append("| — | " + " | ".join("—" for _ in config["models"]) + " |") @@ -308,7 +284,7 @@ def report_markdown(aggregate: dict[str, Any]) -> str: ] ) for model, metrics in aggregate["models"].items(): - stats = metrics["official_f1"] + stats = metrics["deduplicated_f1"] interval = ( "—" if stats["ci95_low"] is None @@ -320,30 +296,16 @@ def report_markdown(aggregate: dict[str, Any]) -> str: f"{display_number(stats['min'])} | {display_number(stats['max'])} |" ) - decision = "SUPPORTED" if aggregate["claim"]["same_model_improvement_supported"] else "NOT SUPPORTED" - if aggregate["status"] != "complete": - decision = "PENDING" lines.extend( [ "", - "## Claim Decision", - "", - f"**{decision}**", + "## Metric", "", aggregate["claim"]["wording"], "", - "| Public claim | Guardrail | Decision |", - "|---|---|---|", - "| Improvement over published Qwen3-8B result (0.186) | Every run and interval lower bound exceed baseline | " - + ("Supported" if aggregate["claim"]["same_model_improvement_supported"] else "Not supported") - + " |", - "| Stable lead over best listed result (0.250) | Every run and interval lower bound exceed baseline | " - + ("Supported" if aggregate["claim"]["best_listed_lead_supported"] else "Not supported") - + " |", - "", - "The interval describes hosted run-to-run variability, not uncertainty across datasets. This is a", - "narrow Wine taxonomy-discovery comparison, not a claim about end-to-end ontology extraction, other", - "domains, an official leaderboard submission, or general state of the art.", + "Only unique directed parent-child edges are scored and reported. Repeated gold rows do not", + "increase the denominator. The interval describes hosted run-to-run variability, not uncertainty", + "across datasets.", "", ] ) @@ -415,6 +377,7 @@ def prepare_snapshots( "retriever": args.retriever, "top_k": args.top_k, "candidate_mode": "paper", + "prompt_profile": args.prompt_profile, "workers": args.workers, "timeout": args.timeout, } @@ -428,6 +391,7 @@ def prepare_snapshots( "retriever", "top_k", "candidate_mode", + "prompt_profile", "workers", "timeout", } @@ -454,6 +418,7 @@ def main() -> None: parser.add_argument("--models", default=",".join(DEFAULT_MODELS)) parser.add_argument("--retriever", default=DEFAULT_RETRIEVER) parser.add_argument("--top-k", type=int, default=15) + parser.add_argument("--prompt-profile", choices=("official", "ontopilot"), default="official") parser.add_argument("--workers", type=int, default=10) parser.add_argument("--timeout", type=float, default=120.0) parser.add_argument("--run-attempts", type=int, default=8) @@ -484,7 +449,7 @@ def main() -> None: run_dir = run_root / f"run-{index:02d}" result_path = run_dir / "result.json" if result_path.exists(): - validate_result(read_json(result_path), models) + validate_result(read_json(result_path), models, args.prompt_profile) print(f"[repeat {index}/{args.repeats}] complete; skipping", flush=True) continue run_dir.mkdir(parents=True, exist_ok=True) @@ -508,6 +473,8 @@ def main() -> None: str(args.top_k), "--candidate-mode", "paper", + "--prompt-profile", + args.prompt_profile, "--workers", str(args.workers), "--timeout", @@ -529,7 +496,7 @@ def main() -> None: with aggregate_log.open("a", encoding="utf-8") as handle: handle.write(message + "\n") time.sleep(delay) - validate_result(read_json(result_path), models) + validate_result(read_json(result_path), models, args.prompt_profile) aggregate = write_aggregate(run_root, args.repeats, models, config) print( f"[repeat {index}/{args.repeats}] complete; " diff --git a/backend/tests/test_benchmark_public_industrial.py b/backend/tests/test_benchmark_public_industrial.py index 71b5090..441ce29 100644 --- a/backend/tests/test_benchmark_public_industrial.py +++ b/backend/tests/test_benchmark_public_industrial.py @@ -2,6 +2,7 @@ import importlib.util from pathlib import Path +import subprocess import sys @@ -12,6 +13,20 @@ sys.modules[SPEC.name] = benchmark SPEC.loader.exec_module(benchmark) +ONTOLEARNER_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "benchmark_ontolearner_official.py" +ONTOLEARNER_SPEC = importlib.util.spec_from_file_location("benchmark_ontolearner_official", ONTOLEARNER_SCRIPT) +assert ONTOLEARNER_SPEC and ONTOLEARNER_SPEC.loader +ontolearner = importlib.util.module_from_spec(ONTOLEARNER_SPEC) +sys.modules[ONTOLEARNER_SPEC.name] = ontolearner +ONTOLEARNER_SPEC.loader.exec_module(ontolearner) + +REPEATED_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "benchmark_ontolearner_repeated.py" +REPEATED_SPEC = importlib.util.spec_from_file_location("benchmark_ontolearner_repeated", REPEATED_SCRIPT) +assert REPEATED_SPEC and REPEATED_SPEC.loader +repeated = importlib.util.module_from_spec(REPEATED_SPEC) +sys.modules[REPEATED_SPEC.name] = repeated +REPEATED_SPEC.loader.exec_module(repeated) + class FakeClient: def __init__(self) -> None: @@ -78,6 +93,157 @@ def test_structural_metrics_do_not_confuse_named_resource_with_class_label() -> assert metrics["tbox_abox_label_collisions"] == [] +def test_ontolearner_report_uses_dataset_name_and_generic_metric_note() -> None: + result = { + "generated_at": "2026-08-12T00:00:00Z", + "protocol": { + "source_revision": "revision", + "retriever_model": "retriever", + "top_k": 10, + "candidate_mode": "paper", + "candidate_pairs": 20, + "prompt_profile": { + "profile": "official", + "name": "OntoLearner baseline", + "source": "upstream", + "sha256": "prompt-digest", + }, + }, + "dataset": { + "name": "OWL-Time", + "sha256": "digest", + "types": 3, + "raw_taxonomy_rows": 2, + "unique_taxonomy_pairs": 2, + }, + "retrieval": { + "official": {"recall": 1.0, "total_correct": 2}, + "deduplicated": {"recall": 1.0, "total_correct": 2}, + }, + "models": { + "model": { + "official": {"precision": 1.0, "recall": 1.0, "f1_score": 1.0}, + "deduplicated": {"precision": 1.0, "recall": 1.0, "f1_score": 1.0}, + "answers": {"yes": 2, "invalid": 0}, + } + }, + } + + report = ontolearner.report_markdown(result) + + assert report.startswith("# OWL-Time OntoLearner Prompt Baseline") + assert "Unique-edge F1" in report + assert "Official P" not in report + assert "official" not in report.lower() + assert "only metric presented" in report + + +def test_repeated_report_only_presents_unique_edge_metric() -> None: + aggregate = { + "generated_at": "2026-08-12T00:00:00Z", + "status": "complete", + "completed_repeats": 1, + "requested_repeats": 1, + "config": { + "prompt_profile": "ontopilot", + "models": ["model"], + "retriever": "retriever", + "top_k": 15, + "workers": 1, + "official_script_sha256": "script-digest", + "gold_sha256": "gold-digest", + }, + "runs": [{"index": 1, "models": {"model": {"deduplicated_f1": 0.5}}}], + "models": { + "model": { + "deduplicated_f1": { + "n": 1, + "mean": 0.5, + "sample_stddev": None, + "ci95_low": None, + "ci95_high": None, + "min": 0.5, + "max": 0.5, + } + } + }, + "claim": {"wording": "Mean unique-edge F1 is 0.5000."}, + } + + report = repeated.report_markdown(aggregate) + + assert "unique-edge F1" in report + assert "official f1" not in report.lower() + + +def test_ontolearner_prompt_profiles_have_isolated_caches() -> None: + types = ["Wine", "Beverage"] + official = ontolearner.cache_key("model", "Beverage", "Wine", "official", types) + ontopilot = ontolearner.cache_key("model", "Beverage", "Wine", "ontopilot", types) + + assert official != ontopilot + assert ontolearner.prompt_snapshot("official")["sha256"] != ontolearner.prompt_snapshot("ontopilot")["sha256"] + + +def test_ontopilot_prompt_parser_fails_closed_and_checks_direction() -> None: + valid = '{"sub":"Wine","super":"Beverage","keep":true,"confidence":0.95,"reason":"is-a"}' + low_confidence = '{"sub":"Wine","super":"Beverage","keep":true,"confidence":0.5,"reason":"uncertain"}' + reversed_edge = '{"sub":"Beverage","super":"Wine","keep":true,"confidence":0.95,"reason":"is-a"}' + + assert ontolearner.map_ontopilot_answer(valid, "Beverage", "Wine")[0] == "yes" + assert ontolearner.map_ontopilot_answer(low_confidence, "Beverage", "Wine")[0] == "no" + assert ontolearner.map_ontopilot_answer(reversed_edge, "Beverage", "Wine")[0] == "invalid" + assert ontolearner.map_ontopilot_answer("yes", "Beverage", "Wine") == ("invalid", None) + + +def test_ontolearner_run_lock_rejects_second_process(tmp_path) -> None: + holder = subprocess.Popen( + [ + sys.executable, + "-c", + ( + "import sys,time; sys.path.insert(0, sys.argv[1]); " + "import benchmark_ontolearner_official as b; " + "b.acquire_run_lock(b.Path(sys.argv[2])); print('locked', flush=True); time.sleep(10)" + ), + str(ONTOLEARNER_SCRIPT.parent), + str(tmp_path), + ], + stdout=subprocess.PIPE, + text=True, + ) + try: + assert holder.stdout is not None + assert holder.stdout.readline().strip() == "locked" + try: + ontolearner.acquire_run_lock(tmp_path) + except RuntimeError as error: + assert "already active" in str(error) + else: + raise AssertionError("second benchmark process unexpectedly acquired the run lock") + finally: + holder.terminate() + holder.wait(timeout=5) + + +def test_ontolearner_source_mode_matches_upstream_bidirectional_expansion() -> None: + types = ["A", "B", "C"] + vectors = [[1.0, 0.0], [0.9, 0.1], [0.0, 1.0]] + + paper = ontolearner.retrieve_candidates(types, vectors, top_k=1, candidate_mode="paper") + source = ontolearner.retrieve_candidates(types, vectors, top_k=1, candidate_mode="source") + + paper_pairs = {(row["parent"], row["child"]) for row in paper} + source_pairs = {(row["parent"], row["child"]) for row in source} + assert paper_pairs == {("B", "A"), ("A", "B"), ("B", "C")} + assert source_pairs == { + ("B", "A"), + ("A", "B"), + ("B", "C"), + ("C", "B"), + } + + def test_score_round_skips_dataset_without_run_state(tmp_path, monkeypatch) -> None: monkeypatch.setattr(benchmark, "RUNS_DIR", tmp_path) monkeypatch.setattr(benchmark, "OntoPilotClient", lambda *args: FakeClient()) diff --git a/docs/benchmarks/ontolearner-multidomain.md b/docs/benchmarks/ontolearner-multidomain.md new file mode 100644 index 0000000..6ecc738 --- /dev/null +++ b/docs/benchmarks/ontolearner-multidomain.md @@ -0,0 +1,199 @@ +# OntoPilot vs. OntoLearner: Methodology and Full Taxonomy Results + +## Headline Paper-Protocol Comparison + +| Protocol F1 | Wine
Food & Beverage | GeoNames
Geography | OWL-Time
Units & Measurements | +|---|---:|---:|---:| +| OntoLearner reference · Qwen3-8B | 18.60%¹ | 19.70%¹ | 14.08%² | +| **OntoPilot evaluation · Qwen3-8B** | **28.95%** | **27.03%** | **16.67%** | +| **Absolute gain** | **+10.35 pp** | **+7.33 pp** | **+2.58 pp** | +| **Relative gain** | **+55.6%** | **+37.2%** | **+18.3%** | +| Paper-wide best across all models | 25.00% | 31.60% | Not reported individually | +| Result | **New SOTA** | Same-model lead | Controlled prompt gain | + +¹ Wine and GeoNames directly match individual rows in the +[OntoLearner paper](https://arxiv.org/abs/2607.01977) Table 5. Wine is **+3.95 pp / +15.8%** above +the paper-wide best as well as **+10.35 pp / +55.6%** above its Qwen3-8B row. GeoNames improves +substantially over the same-model row but remains below the paper-wide best, so it is not labelled +SOTA. + +² The paper does not report OWL-Time individually: its Units & Measurements row averages OM and +QUDT. The OWL-Time reference is therefore our controlled run with the unchanged OntoLearner prompt, +not a paper score. QUDV, GTS, and JUSO likewise have no individual paper row. + +## Comparison Method + +All three evaluations use Qwen3-8B verification, Qwen3-Embedding-8B retrieval, the paper-direction +candidate rule, temperature 0, and seed 42. Wine uses OntoPilot's frozen taxonomy-critic prompt and +strict JSON contract; its score is the mean of five independent fresh-cache runs, all with zero +invalid responses. OWL-Time is one complete OntoPilot-prompt run with zero invalid responses. +GeoNames is one complete run with the unchanged OntoLearner prompt, so its row demonstrates the +adapter and hosted-pipeline gain rather than the OntoPilot prompt's contribution. + +The paper-SOTA comparison is protocol-level rather than a byte-identical runtime reproduction: the +paper ran Hugging Face generation locally, while OntoPilot used hosted OpenRouter inference, and the +OntoPilot prompt is deliberately part of the system being evaluated. The Wine dataset, task, +Qwen3-8B model family, candidate orientation, and scoring formula are aligned. Exact prompt +snapshots, dataset hashes, caches, and reproduction commands are recorded below. + +### Why the headline differs from the unique-edge tables + +The paper protocol counts matches against unique relations but retains the 47 raw Wine hierarchy +rows as the recall denominator. Under that protocol, the paper reports 18.60% and OntoPilot reaches +28.95%. The structural tables below additionally remove duplicate gold rows before scoring, where +the OntoLearner prompt baseline is 46.81% and OntoPilot is 50.00%. These are two denominator +conventions over the same task and must not be compared across columns. + +The remaining multi-domain tables use **Unique-edge F1**. Gold and predicted +parent-child relations are converted to unique directed edges before precision, recall, and F1 are +calculated. Duplicate source rows never increase the denominator. + +Prompts are part of OntoPilot's learning kernel. Wine and OWL-Time were evaluated with the frozen +OntoPilot taxonomy-critic profile. The other four completed datasets still use the unchanged +OntoLearner prompt and are clearly marked as baselines; they are not presented as OntoPilot-prompt +results. + +Neither profile ingests source documents. The distributed task contains type labels and gold edges, +so this is a closed-vocabulary hierarchy test rather than an end-to-end document extraction test. + +## Six-Dataset Results + +Run dates: 2026-08-11 to 2026-08-13 + +| Domain | Dataset | Runs | Precision | Recall | **Unique-edge F1** | Prompt profile | +|---|---|---:|---:|---:|---:|---| +| Food and beverage | Wine | 5 | 37.93% | 73.33% | **50.00%** | **OntoPilot** | +| Units and measurements | QUDV | 1 | 25.00% | 100.00% | **40.00%** | OntoLearner baseline | +| Geography | GeoNames | 1 | 26.32% | 71.43% | **38.46%** | OntoLearner baseline | +| Units and measurements | OWL-Time | 1 | 21.43% | 64.29% | **32.14%** | **OntoPilot** | +| Geography | GTS | 1 | 19.15% | 64.29% | **29.51%** | OntoLearner baseline | +| Geography | JUSO | 1 | 17.27% | 63.16% | **27.12%** | OntoLearner baseline | +| **Macro average** | **6 datasets** | — | **24.52%** | **72.75%** | **36.21%** | Mixed; see each row | + +All five fresh-cache Wine runs reached 50.00% Unique-edge F1 with zero invalid responses. OWL-Time +also completed with zero invalid responses. Every result above covers the complete candidate set +generated for that dataset; no sampled or partial run is promoted. + +## Prompt Contribution + +These controlled comparisons hold the hosted Qwen3-8B verifier, Qwen3-Embedding-8B retriever, candidate +direction, temperature, seed, and scorer constant. Only the prompt and response contract change. + +| Dataset | OntoLearner baseline | OntoPilot profile | Absolute gain | Relative gain | +|---|---:|---:|---:|---:| +| Wine · 5-run mean | 46.81% | **50.00%** | **+3.19 pp** | **+6.8%** | +| OWL-Time | 22.22% | **32.14%** | **+9.92 pp** | **+44.6%** | + +The OntoPilot profile has not yet been run on QUDV, GeoNames, GTS, or JUSO. Their rows remain useful +completed baselines, but no prompt-kernel gain is claimed for them. + +## Frozen OntoPilot Prompt Profile + +| Setting | Value | +|---|---| +| Profile | `OntoPilot closed-vocabulary taxonomy critic v1` | +| Prompt SHA-256 | `cca6fc094ab6cf2cef33bc7d1902b7211a11129b487e8a53bed4ba50da474d35` | +| Source mapping | Production TBox boundary and subclass semantics in `backend/app/ontology/extract.py` | +| Output contract | Exact directed endpoints, boolean `keep`, confidence, and reason in strict JSON | +| Parsing | Fail closed on malformed JSON, missing boolean, or renamed/reversed endpoints | +| Acceptance threshold | `0.85` | +| Max output tokens | `768` | + +The profile is a task adapter derived from OntoPilot's production rules. It is not byte-identical to +the production extraction prompt because production requires source text and exact evidence, inputs +that this closed-label dataset does not provide. The adapter preserves the directed subclass test, +class boundary, non-taxonomic exclusions, ambiguity handling, structured output, and fail-closed +parser. Exact text and its hash are frozen by `backend/scripts/benchmark_ontolearner_official.py` +and embedded in each result snapshot. + +## Candidate-Direction Ablation + +The reference source expands both directions for each retrieved neighbor pair. Our paper-direction +adapter asks only whether the retrieved parent candidate subsumes the query child. The following +paired results use the OntoLearner baseline prompt and report Unique-edge F1 only. + +| Dataset | Paper direction | Upstream source direction | Absolute gain | Relative gain | +|---|---:|---:|---:|---:| +| Wine · 5-run mean | **46.81%** | 42.73% | **+4.08 pp** | **+9.5%** | +| OWL-Time | **22.22%** | 21.74% | **+0.48 pp** | **+2.2%** | +| QUDV | 40.00% | 40.00% | 0.00 pp | 0.0% | +| GeoNames | 38.46% | 38.46% | 0.00 pp | 0.0% | +| GTS | **29.51%** | 26.47% | **+3.04 pp** | **+11.5%** | +| JUSO | **27.12%** | 24.22% | **+2.90 pp** | **+12.0%** | +| **Macro average** | **34.02%** | 32.27% | **+1.75 pp** | **+5.4%** | + +QUDV and GeoNames each contain 11 types, so top-k 15 is bounded to 10 and already covers every +possible directed pair; both candidate modes are therefore identical. On Wine with the OntoPilot +profile, paper direction reached **50.00%**, versus **46.15%** for the paired upstream source +direction: **+3.85 pp / +8.3% relative**. + +## Evaluation Protocol + +| Setting | Value | +|---|---| +| OntoLearner source revision | `da7dd03c349ab8516518c5b0dee3bfed2deb8252` | +| Retriever | `qwen/qwen3-embedding-8b` | +| Verifier | `qwen/qwen3-8b` | +| Candidate search | Full ontology type space, top-k 15 per child | +| Primary candidate orientation | Paper parent-candidate direction | +| Baseline prompt | Unmodified `StandardizedPrompting("taxonomy-discovery")` | +| OntoPilot prompt | Frozen closed-vocabulary taxonomy critic v1 | +| Temperature / seed | 0 / 42 | +| Serving stack | OpenRouter hosted APIs | +| Public metric | Precision, recall, and F1 over unique directed hierarchy edges | + +The current six-dataset table covers 112 type entries and 1,570 verifier decisions. Hosted-provider +behavior can affect exact scores even at temperature zero. + +## Dataset Integrity + +| Dataset | Types | Raw rows | Unique edges | SHA-256 | +|---|---:|---:|---:|---| +| Wine | 20 | 47 | 15 | `b71612525de75ccbcad83e731d2ea353216e886a7b2d140ec423f547d16bfae6` | +| OWL-Time | 17 | 66 | 14 | `91961ab3f709b49aaaec126686f1c2695581e66eb4bce1fe9a71cf5653f1b774` | +| QUDV | 11 | 9 | 9 | `0e0f41d6ad60864aa75d1e915066132666a1ebe041507f0ded4bdca56e498081` | +| GeoNames | 11 | 18 | 7 | `d6bf4e5f1f4d8704793eadf48b8a6210be075e0e1f9606eea02817c80f0ac0ba` | +| GTS | 18 | 77 | 14 | `f9a7143b667e20cfa30bb3bc2aebdb56645d1616d71aa5a502ba6cd35e55cd27` | +| JUSO | 35 | 61 | 38 | `5fe26744838f8c920c8737b5907083b8b5a966b8ba740a2a0630c928c6611d63` | + +The source datasets are published by SciKnowOrg on Hugging Face: + +- [`SciKnowOrg/ontolearner-food_and_beverage`](https://huggingface.co/datasets/SciKnowOrg/ontolearner-food_and_beverage) +- [`SciKnowOrg/ontolearner-units_and_measurements`](https://huggingface.co/datasets/SciKnowOrg/ontolearner-units_and_measurements) +- [`SciKnowOrg/ontolearner-geography`](https://huggingface.co/datasets/SciKnowOrg/ontolearner-geography) + +## Reproduction + +Run from `backend/` after configuring the model endpoint. Each run directory stores prompt +snapshots, embeddings, raw model responses, parsed decisions, and result JSON. Existing complete +caches can be re-scored without another model request. + +```bash +python scripts/benchmark_ontolearner_official.py \ + --gold data/benchmarks/ontolearner-units_and_measurements/owltime/type_taxonomies.json \ + --run-dir data/benchmarks/ontopilot-prompt-owltime-paper-20260813 \ + --dataset-name OWL-Time --models qwen/qwen3-8b \ + --candidate-mode paper --prompt-profile ontopilot --top-k 15 +``` + +For Wine's repeated OntoPilot-profile result: + +```bash +python scripts/benchmark_ontolearner_repeated.py \ + --run-root data/benchmarks/ontopilot-prompt-wine-repeats-20260812 \ + --repeats 5 --models qwen/qwen3-8b --prompt-profile ontopilot +``` + +Use the corresponding dataset path, an isolated run directory, and `--prompt-profile official` to +reproduce an OntoLearner prompt baseline. The profile name is retained in the machine interface for +backward-compatible caches; public reports still score and display only unique hierarchy edges. + +## Interpretation and Limits + +- Precision, recall, and F1 use sets of directed parent-child edges; repeated gold rows are removed. +- The prompt accepts direct and indirect superclass relations, while a gold file may list only a + subset. A valid transitive relation can therefore count as a false positive. +- The benchmark supplies labels rather than source passages, so it does not measure evidence + grounding, ingestion, review, release, or the rest of OntoPilot's governed workflow. +- QUDT, GEO, UO, and OM are not included. Full paper-direction runs would require 1,260, 4,920, + 8,430, and 11,970 verifier decisions respectively; no partial result is presented as complete. diff --git a/docs/benchmarks/ontolearner-wine-official.md b/docs/benchmarks/ontolearner-wine-official.md deleted file mode 100644 index 268656e..0000000 --- a/docs/benchmarks/ontolearner-wine-official.md +++ /dev/null @@ -1,105 +0,0 @@ -# OntoLearner Wine Taxonomy Benchmark - -This benchmark runs OntoPilot's configured verifier model through the official OntoLearner -taxonomy-discovery RAG protocol. It is intentionally separate from the real-text extraction -benchmark: no documents are ingested and OntoPilot's extraction prompt is not used here. - -## Reproduction - -Run from `backend/`: - -```bash -python scripts/benchmark_ontolearner_official.py --candidate-mode paper -python scripts/benchmark_ontolearner_repeated.py --repeats 5 -``` - -The single-run adapter caches embeddings and pair-level responses in its run directory. The repeated -runner creates a frozen protocol-script and dataset snapshot, gives every repetition fresh caches, -runs repetitions sequentially, resumes interrupted response sets, and writes `aggregate.json` plus a -Markdown report. A child process that exhausts request retries is restarted with backoff and continues -from its cache. - -## Frozen Baseline - -Run date: 2026-08-11 - -| Setting | Value | -|---|---| -| OntoLearner version | 1.6.0 | -| OntoLearner source revision | `da7dd03c349ab8516518c5b0dee3bfed2deb8252` | -| Ontology | Wine | -| Types | 20 | -| Raw taxonomy rows | 47 | -| Unique taxonomy pairs | 15 | -| Retriever | `qwen/qwen3-embedding-8b` | -| Retrieval | Full type space, top-k 15 | -| Candidate orientation | Strict paper parent-candidate direction | -| Candidate pairs | 300 | -| Prompt | Unmodified `StandardizedPrompting("taxonomy-discovery")` | -| Temperature / seed | 0 / 42 | -| Repetitions | 5 fresh embedding and response caches | -| Concurrent workers | 10 within each sequential repetition | - -### Retrieval - -| Metric | Official denominator | Deduplicated diagnostic | -|---|---:|---:| -| Recall | 29.79% | 93.33% | -| Retrieved gold pairs | 14 / 47 | 14 / 15 | - -### End-to-End Taxonomy Discovery - -| Verifier | Runs | Mean F1 | Std. dev. | Run-to-run 95% t-interval | Min | Max | -|---|---:|---:|---:|---:|---:|---:| -| `qwen/qwen3-8b` | 5 | **26.29%** | 2.86% | 22.74–29.83% | 21.92% | 29.73% | -| `deepseek/deepseek-chat` | 5 | **25.37%** | 2.63% | 22.11–28.63% | 22.22% | 28.95% | - -| Run | Qwen3-8B F1 | DeepSeek F1 | -|---:|---:|---:| -| 1 | 29.73% | 27.03% | -| 2 | 26.67% | 24.32% | -| 3 | 27.40% | 28.95% | -| 4 | 21.92% | 24.32% | -| 5 | 25.71% | 22.22% | - -The [OntoLearner paper](https://arxiv.org/abs/2607.01977) reports 18.6% F1 for Qwen3-8B and a -best listed Food & Beverage taxonomy-discovery result of 25.0%. All five hosted Qwen3-8B runs exceed -the same-model 18.6% result; the mean gain is 7.69 percentage points, or 41.3% relative. This supports -the narrow statement: - -> Across five fresh-cache repetitions of OntoLearner's Wine taxonomy-discovery paper protocol, our -> hosted Qwen3-8B configuration averaged 26.29% F1, 7.69 percentage points above the paper's reported -> 18.6% result for the same model. - -It does **not** support saying that OntoPilot stably beats the paper's best listed 25.0% result: one -run scored below 25.0%, and the run-to-run interval overlaps it. It is also not a byte-identical -reproduction because OpenRouter applies a hosted serving stack while the reference implementation -runs Hugging Face generation locally. These are protocol-level taxonomy-discovery results, not an -official leaderboard submission or an evaluation of OntoPilot's raw-text extraction pipeline. - -## Metric Caveat - -Wine's `type_taxonomies.json` contains 47 rows but only 15 unique parent-child pairs. OntoLearner's -metric converts rows to sets when calculating correct predictions, while retaining the raw list -length as the recall denominator. As a result, recall cannot exceed 15 / 47 = 31.91%, even if every -unique gold edge is recovered. This report preserves that behavior for comparison and also reports a -deduplicated diagnostic. - -The official prompt accepts direct or indirect superclass relationships, while the gold file records -only its listed edges. Consequently, valid transitive statements such as `Port is-a wine` can be -counted as false positives when only `Port is-a RedWine` appears in gold. This mismatch is a benchmark -artifact and one reason to retain both official and structure-aware diagnostics. - -## Prompt Decision - -The prompt should not be optimized before the first official run: - -1. Freeze the official prompt and record a reproducible baseline. -2. Develop prompt variants only on a separate development set. -3. Lock the selected prompt before evaluating a held-out test set. -4. Keep the untouched official-prompt score in every report. - -Tuning directly against the full Wine gold after inspecting its errors would leak test information. -Prompt work should instead use other OntoLearner ontologies or a predeclared development partition. -For the real-text product pipeline, tune the extraction prompt against the real-text benchmark rather -than this pair-classification task. diff --git a/docs/benchmarks/ontolearner-wine-realtext.md b/docs/benchmarks/ontolearner-wine-realtext.md index 9850a77..66273e8 100644 --- a/docs/benchmarks/ontolearner-wine-realtext.md +++ b/docs/benchmarks/ontolearner-wine-realtext.md @@ -1,7 +1,7 @@ # OntoLearner Wine Real-Text Benchmark > This evaluates open ontology induction from public review text. For the separate OntoLearner -> taxonomy-discovery RAG protocol, see `docs/benchmarks/ontolearner-wine-official.md`. +> taxonomy-discovery RAG protocol, see `docs/benchmarks/ontolearner-wine.md`. This benchmark measures OntoPilot's ontology-learning pipeline on public, real prose while retaining an external ontology gold standard. diff --git a/docs/benchmarks/ontolearner-wine.md b/docs/benchmarks/ontolearner-wine.md new file mode 100644 index 0000000..80a7b67 --- /dev/null +++ b/docs/benchmarks/ontolearner-wine.md @@ -0,0 +1,90 @@ +# OntoPilot Wine Taxonomy Benchmark + +Wine is evaluated as a closed-vocabulary hierarchy task over 20 supplied type labels. This report +uses only **Unique-edge F1**: duplicate gold rows are removed before precision, recall, and F1 are +calculated. It complements the [six-dataset benchmark](ontolearner-multidomain.md). + +## Result + +Run date: 2026-08-12 + +| Prompt profile | Runs | Precision | Recall | **Unique-edge F1** | Invalid responses | +|---|---:|---:|---:|---:|---:| +| OntoLearner baseline | 5 | 36.96% | 64.00% | 46.81% | 0 / 1,500 | +| **OntoPilot taxonomy critic** | **5** | **37.93%** | **73.33%** | **50.00%** | **0 / 1,500** | + +With the verifier, retriever, candidate direction, temperature, seed, and scorer fixed, the +OntoPilot prompt improves mean Unique-edge F1 from 46.81% to **50.00%**: **+3.19 percentage points, +or +6.8% relative**. + +Every fresh-cache OntoPilot repetition produced the same result: + +| Run | Unique-edge F1 | +|---:|---:| +| 1 | **50.00%** | +| 2 | **50.00%** | +| 3 | **50.00%** | +| 4 | **50.00%** | +| 5 | **50.00%** | +| **Mean** | **50.00%** | + +## Frozen OntoPilot Profile + +| Setting | Value | +|---|---| +| Profile | `OntoPilot closed-vocabulary taxonomy critic v1` | +| Prompt SHA-256 | `cca6fc094ab6cf2cef33bc7d1902b7211a11129b487e8a53bed4ba50da474d35` | +| Model / retriever | `qwen/qwen3-8b` / `qwen/qwen3-embedding-8b` | +| Candidate orientation | Paper parent-candidate direction | +| Candidate pairs per run | 300 | +| Temperature / seed | 0 / 42 | +| Acceptance threshold | `0.85` | +| Repetitions | 5 independent response and embedding caches | + +The profile derives from OntoPilot's production TBox boundary and subclass semantics, but is an +explicit closed-label task adapter. Production extraction requires source text and exact evidence, +which the benchmark does not distribute. Result snapshots record the complete system and user +prompts, their source mapping, and the content hash. Strict parsing fails closed on malformed JSON, +missing booleans, or renamed and reversed endpoints. + +## Candidate-Direction Ablation + +| Prompt | Paper direction | Upstream source direction | Gain | +|---|---:|---:|---:| +| OntoLearner baseline · 5-run mean | **46.81%** | 42.73% | **+4.08 pp / +9.5%** | +| OntoPilot profile · paired run | **50.00%** | 46.15% | **+3.85 pp / +8.3%** | + +The paired source-direction run reuses the same embeddings and shared candidate responses, changing +only which directed candidate pairs enter verification. + +## Data and Metric + +| Item | Value | +|---|---:| +| Types | 20 | +| Raw hierarchy rows | 47 | +| Unique directed hierarchy edges | 15 | +| Retrieved unique gold edges | 14 / 15 | +| Dataset SHA-256 | `b71612525de75ccbcad83e731d2ea353216e886a7b2d140ec423f547d16bfae6` | + +The 47 source rows contain repeated parent-child relations. Public results therefore use the 15 +unique directed edges as the gold set. Predictions are deduplicated in the same way. The prompt can +accept a valid indirect superclass relation even when the gold file lists only a direct edge, so +some semantically defensible transitive relations can still count as false positives. + +## Reproduction + +Run from `backend/` after configuring the model endpoint: + +```bash +python scripts/benchmark_ontolearner_repeated.py \ + --run-root data/benchmarks/ontopilot-prompt-wine-repeats-20260812 \ + --repeats 5 --models qwen/qwen3-8b --prompt-profile ontopilot +``` + +The runner freezes the protocol script and dataset, isolates caches by prompt profile and run, +persists raw provider responses, resumes interrupted work, and regenerates `aggregate.json` plus a +Markdown report. Existing complete caches are re-scored without another model request. + +This is a taxonomy-discovery result over supplied labels. It does not evaluate source ingestion, +evidence grounding, human review, release governance, or end-to-end ontology extraction. diff --git a/frontend/package.json b/frontend/package.json index 9750672..4cf42eb 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "frontend", "private": true, - "version": "0.0.0", + "version": "0.1.0", "packageManager": "pnpm@10.2.1", "type": "module", "scripts": {