ChatVector

Open-Source Backend-First RAG Engine for Document Intelligence

ChatVector is an open-source Retrieval-Augmented Generation (RAG) engine for ingesting, indexing, and querying unstructured documents such as PDFs and text files.

Think of it as an engine developers can use to build document-aware applications — such as research assistants, contract analysis tools, or internal knowledge systems — without having to reinvent the RAG pipeline.

---

⭐ Star the repo to follow progress and support the project!

---

🔗 Quick Links

---

📌 Table of Contents

• What is ChatVector?
• ChatVector vs Frameworks
• Who is this for?
• Current Status
• Architecture Overview
• Quick Start
• Contributing
• License

---

🔎 What is ChatVector?

ChatVector provides a clean, extensible backend foundation for RAG-based document intelligence. It handles the full lifecycle of document Q&A:

• Document ingestion (PDF, text) with configurable chunking strategies
• Text extraction, cleaning, and semantic chunking
• Vector embedding and storage via pgvector
• Semantic retrieval with optional query transformations
• LLM-powered answer generation with cited responses
• Background processing queue with rate limiting and retry logic

The goal is to offer a developer-focused RAG engine you can deploy as a service and integrate via HTTP — not a polished end-user SaaS, and not a framework you have to assemble yourself.

---

ChatVector vs Frameworks

ChatVector is designed as a production-ready backend service, not a general-purpose framework. Here's how it compares:

Aspect	ChatVector (This Project)	General AI Framework (e.g., LangChain)
Primary Goal	Deliver a deployable backend service for document intelligence.	Provide modular components to build a wide variety of AI applications.
Out-of-the-Box Experience	A fully functional FastAPI service with logging, testing, rate limiting, and a clean API.	A collection of tools and abstractions you must wire together and productionize.
Architecture	Batteries-included, opinionated engine. Get a working system for one use case.	Modular building blocks. Assemble and customize components for many use cases.
Best For	Developers, startups, or teams who need a document Q&A API now and want to focus on their application layer.	Developers and researchers building novel, complex AI agents or exploring multiple LLM patterns from the ground up.
Path to Production	Short. Configure, deploy, and integrate via API. Built-in observability, rate limiting, and scaling patterns.	Long. Requires significant additional work on API layers, monitoring, deployment, and performance tuning.

---

👥 Who is this for?

ChatVector is designed for:

• Developers building document intelligence tools or internal knowledge systems
• Backend engineers who want a solid RAG foundation without heavy abstractions
• AI/ML practitioners experimenting with chunking, retrieval, and prompt strategies
• Open-source contributors interested in retrieval systems, embeddings, and LLM orchestration

---

🚀 Current Status

Phases 1–2.5 Complete | Phase 3 In Progress

Phases 1, 2, and 2.5 are complete. Phase 3 is actively underway — most backend quality and provider work has shipped, but production API-key enforcement is still in progress. See ROADMAP.md for the full breakdown.

What's working today:

Backend

• ✅ PDF and text document ingestion
• ✅ Configurable chunking strategies (fixed, paragraph, semantic)
• ✅ Vector embeddings + semantic search via pgvector
• ✅ Hybrid retrieval (PostgreSQL full-text + vector, RRF fusion)
• ✅ Baseline retrieval reranking (similarity + lexical overlap)
• ✅ Session-scoped and tenant-wide retrieval modes
• ✅ LLM-powered answers with source citations and relevance scores
• ✅ Query transformations (rewrite, expand, stepback)
• ✅ Configurable response personas and system prompt
• ✅ Session-based chat with persisted conversation history
• ✅ SSE streaming chat (/chat/stream)
• ✅ Background ingestion queue with rate limiting, retry, and DLQ
• ✅ Redis-backed ingestion queue (production default; in-memory fallback for local dev)
• ✅ Structured logging with request ID tracing
• ✅ Health checks with TTL caching on /status
• ✅ Per-IP rate limiting on all public endpoints
• ✅ Security headers, CORS hardening, input validation
• ✅ Production Compose config + GitHub Actions CI
• ✅ Pluggable LLM providers (Gemini, OpenAI, Ollama, Anthropic Claude)
• ✅ Pluggable embedding providers (Gemini, OpenAI, Ollama, Voyage AI)
• ✅ Mixed-provider configurations (e.g. Claude + Voyage)
• ✅ Response metadata: latency_ms and model on chat responses
• ✅ Python client SDK (core synchronous workflows)

Frontend Demo

• ✅ Document upload with live pipeline stage display and SSE progress
• ✅ Real-time ingestion status polling
• ✅ Full RAG chat with source citations and session sidebar
• ✅ SSE streaming chat, batch query demo, and live system status page
• ✅ Responsive design with dark developer aesthetic

Active Phase 3 work:

• 🚧 Production API-key authentication and strict tenant enforcement (plumbing scaffolded; validation not yet active)
• 🚧 Per-tenant rate limiting
• 🚧 Python SDK parity (sessions, streaming, retrieval scopes)
• 🚧 Node.js/TypeScript SDK (planned)
• 🚧 Query transformation debug metadata and retrieval inspection tooling

---

🧠 Architecture Overview

Backend Layer (Core)

• FastAPI — modern Python API framework
• Uvicorn — high-performance ASGI server
• slowapi — per-IP rate limiting
• Design goals: clarity, extensibility, resilience, and security by default

AI & Retrieval Layer

• Pluggable providers — Gemini, OpenAI, Ollama, Anthropic Claude (LLM), and Voyage AI (embeddings); mix and match independently
• Hybrid retrieval — vector similarity + PostgreSQL full-text search with RRF fusion
• Baseline reranking — deterministic similarity + lexical overlap reranker
• Retrieval scopes — session-scoped (default) or tenant-wide search
• Configurable chunking — fixed, paragraph, or semantic strategies
• Query transformations — rewrite, expand, or stepback before retrieval
• Response personas — default, concise, conversational, academic, technical

Data Layer

• PostgreSQL + pgvector — vector similarity search
• SQLAlchemy (development) / Supabase (production)
• Strategy pattern — swap backends without touching business logic

Reference Frontend (Non-Core)

• Next.js + TypeScript
• Full end-to-end demo — upload, ingest, chat, citations
• Not production-ready — exists to demonstrate and test backend capabilities

See ARCHITECTURE.md for full system design details.

---

🎯 Quick Start: Run in 5 Minutes

Prerequisites

• Docker & Docker Compose — Install Docker
• Google AI Studio API Key — Get Key

Backend Setup

1. Create backend/.env:

cp backend/.env.example backend/.env

Edit backend/.env and set your API key:

APP_ENV=development
LOG_LEVEL=INFO
GEN_AI_KEY=your_google_ai_studio_api_key_here
MAX_UPLOAD_SIZE_MB=10

Provider options: By default, ChatVector uses Google Gemini. To use OpenAI or Ollama instead, see backend/.env.example for all provider configuration variables.

2. Start the stack:

docker compose up --build

This starts Postgres with pgvector and the API with live reload.

3. Test the API:

• Root: http://localhost:8000
• Swagger UI: http://localhost:8000/docs

Try the endpoints:

1. POST /upload — upload a PDF, get a document_id and status_endpoint
2. GET /documents/{document_id}/status — poll ingestion stage and progress
3. POST /chat — ask questions using the document_id

---

Extra Docker Commands

Command	Purpose
docker compose up	Start containers without rebuilding
docker compose down	Stop containers, preserve data
docker compose down -v	Stop containers and delete all DB data
docker compose up --build	Rebuild containers after code changes
docker compose logs -f api	Follow API logs in real time
docker compose exec db psql -U postgres	Connect to Postgres directly

Or use the Makefile shortcuts — run make help to see all available commands.

---

Frontend Demo (Non-Core)

cd frontend-demo
npm install

# Create frontend env
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local

npm run dev

Frontend runs at http://localhost:3000

Or start backend + frontend together:

make dev

---

Python SDK

A synchronous Python client is available today. A Node.js/TypeScript SDK is planned for Phase 3.

pip install ./sdk/python

from chatvector import ChatVectorClient

with ChatVectorClient("http://localhost:8000") as client:
    doc = client.upload_document("report.pdf")
    client.wait_for_ready(doc.document_id, timeout=90)
    answer = client.chat("What are the key findings?", doc.document_id)
    print(answer.answer, answer.latency_ms, answer.model)
    for source in answer.sources:
        print(source.file_name, source.page_number, source.score)

Session management, streaming chat, and retrieval scope options are not yet exposed in the SDK. See sdk/python/README.md for details.

---

🤝 Contributing

High-impact contribution areas:

• Ingestion & indexing pipelines
• Retrieval quality & evaluation
• Chunking and query transformation strategies
• API design & refactoring
• Performance & scaling
• Security hardening
• SDK development
• Documentation & examples

Frontend contributions are welcome but considered non-core.

See CONTRIBUTING.md for details and Good First Issues to get started.

---

📄 License