ScholarVerse

ScholarVerse is a self-hosted academic research platform for literature, linguistics, and humanities work. The stack in this repo is built around evidence-first retrieval, citation traceability, asynchronous document intelligence, and open-source inference endpoints instead of proprietary hosted AI APIs.

What is here

• scholarverse-web/ Next.js 16 App Router frontend and server routes.
• migrations/ InsForge/PostgreSQL schema with pgvector-backed RAG tables, policies, and retrieval SQL.
• services/ingestion-worker/ FastAPI + Redis Queue ingestion service for OCR, metadata extraction, chunking, and callback delivery.
• docker-compose.yml Local orchestration for the web app, Redis, vLLM, optional Ollama, and the ingestion services.
• docs/architecture.md Production deployment notes and system flow.
• scholarverse-web/training/ Fine-tuning dataset export and supervised training helpers.

Core platform design

1. scholarverse-web handles the research workspace, streaming responses, admin surface, and upload entrypoint.
2. InsForge owns authentication, PostgreSQL, pgvector persistence, storage buckets, and chat/document metadata once the project is linked.
3. A self-hosted OpenAI-compatible endpoint, preferably vLLM, serves the primary research model and embedding model.
4. The ingestion worker performs OCR, parsing, semantic segmentation, duplicate detection, and callback delivery asynchronously through Redis-backed jobs.
5. Retrieval happens before generation. The app is designed to pass page-scoped source excerpts into the generation step and surface citations back to the user interface.

Local development

1. Copy .env.example to .env and fill the InsForge and model settings.

2. Install the web app dependencies:

   cd scholarverse-web
   npm install

3. Run the web app:

   npm run dev

4. Start the worker stack when you want document ingestion or self-hosted inference:

   docker compose up redis vllm ingestion-api ingestion-worker

5. Apply the database schema to your linked InsForge project:

   npx @insforge/cli whoami --json
   npx @insforge/cli current --json
   npx @insforge/cli db migrations up --all

Fine-tuning workflow

ScholarVerse now includes a practical prompt-to-training path:

1. Export grounded chat examples from the live corpus and saved conversations:

   cd scholarverse-web
   npm run finetune:export -- --output training/datasets/scholarverse-sft.jsonl

2. Train a ScholarVerse-specific model adapter:

   uv run training/train_scholarverse_sft.py \
     --dataset training/datasets/scholarverse-sft.jsonl \
     --model Qwen/Qwen2.5-32B-Instruct \
     --output-dir training/output/scholarverse-qwen-sft

3. Serve the tuned checkpoint from infrastructure you control and point SELF_HOSTED_CHAT_MODEL at it.

Deployment model

• Frontend: Vercel or InsForge-managed frontend deployment from scholarverse-web/
• Backend primitives: InsForge
• Self-hosted inference: dedicated GPU server running vLLM
• Async ingestion: containerized FastAPI + RQ worker on a compute service
• Redis: managed or self-hosted Redis for queueing

Current runtime behavior

The web app runs in a demo-backed mode until you provide real InsForge and self-hosted AI environment variables. That fallback is deliberate: it keeps the interface operable while you wire the actual infrastructure.