GraphRAG Research Assistant

A Graph-Augmented Retrieval system that combines a Neo4j knowledge graph with ChromaDB vector search to answer questions over research documents. Evaluated against standard RAG using RAGAS — outperforms on faithfulness, context precision, and context recall.

---

RAGAS Evaluation Results

Evaluated on 30 questions across 3 AI papers (BERT, GPT-2, Attention Is All You Need).

Metric	GraphRAG	Standard RAG	Delta
Faithfulness	0.553	0.523	+0.030
Context Precision	0.650	0.294	+0.356
Context Recall	0.826	0.778	+0.048
Answer Relevancy	0.545	0.726	-0.181

GraphRAG wins on 3/4 metrics. The answer relevancy gap is structural: standard RAG's loose prompt allows the LLM to fill in gaps with parametric knowledge, while GraphRAG stays context-bound — which is why faithfulness and precision are higher.

---

Architecture

PDF Documents
      │
      ▼
┌─────────────────┐
│  DocumentParser │  (unstructured)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ SemanticChunker │  512-token chunks with overlap
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐ ┌──────────────┐
│  NER   │ │   Relation   │
│Extractor│ │  Extractor   │  (spaCy + LLM)
└───┬────┘ └──────┬───────┘
    │              │
    ▼              ▼
┌───────────┐  ┌───────────┐
│  Neo4j    │  │ ChromaDB  │
│ (graph)   │  │ (vectors) │
└─────┬─────┘  └─────┬─────┘
      │               │
      └──────┬────────┘
             ▼
    ┌─────────────────┐
    │  LangGraph Agent│
    │  ┌───────────┐  │
    │  │  Analyze  │  │  spaCy NER + graph entity lookup
    │  └─────┬─────┘  │
    │        ▼        │
    │  ┌───────────┐  │
    │  │  Retrieve │  │  graph traversal + vector search + RRF fusion
    │  └─────┬─────┘  │
    │        ▼        │
    │  ┌───────────┐  │
    │  │  Generate │  │  structured context prompt → llama3.1:8b
    │  └───────────┘  │
    └─────────────────┘
             │
             ▼
        FastAPI / Streamlit

Key Components

Component	File	Description
Document Parser	src/ingestion/parser.py	PDF/doc ingestion via unstructured
Semantic Chunker	src/ingestion/chunker.py	Section-aware 512-token chunking with overlap
NER Extractor	src/ingestion/ner_extractor.py	spaCy + LLM entity extraction and classification
Relation Extractor	src/ingestion/relation_extractor.py	LLM-based relation extraction (USES, BUILDS_ON, etc.)
Graph Store	src/storage/graph_store.py	Neo4j entity/relation storage and traversal
Vector Store	src/storage/vector_store.py	ChromaDB with nomic-embed-text embeddings
Graph Retriever	src/retrieval/graph_retriever.py	2-hop graph traversal + entity-specific vector search
Hybrid Ranker	src/retrieval/hybrid_ranker.py	Reciprocal Rank Fusion (RRF) over graph + vector results
Graph Agent	src/agent/graph_agent.py	LangGraph state machine: analyze → retrieve → generate
Ingestion Pipeline	src/ingestion/pipeline.py	End-to-end document ingestion
API	src/api/main.py	FastAPI query endpoint
UI	src/ui/app.py	Streamlit chat interface
Evaluator	src/tests/eval/run_eval.py	RAGAS evaluation vs standard RAG baseline

---

How It Works

1. Ingestion

Documents are parsed, chunked, and processed in parallel:

• Entities extracted via spaCy transformer model + LLM refinement, stored in Neo4j
• Relations extracted via LLM (AUTHORED_BY, USES, BUILDS_ON, PART_OF, etc.), stored as graph edges
• Chunks embedded with nomic-embed-text and stored in ChromaDB

2. Retrieval (the key difference from standard RAG)

At query time the agent runs a LangGraph state machine:

Analyze — spaCy NER on the question, then validates candidate terms against the live graph (catches technical terms spaCy misses, like "BERT" or "Transformer").

Retrieve — three-stage process:

1. Graph traversal (2 hops) from matched entities → structured triplets + evidence strings
2. Entity-specific vector searches per discovered entity (targeted prose chunks)
3. Primary semantic vector search on the full question

Results are fused with Reciprocal Rank Fusion (RRF). Chunks containing question keywords are promoted before fusion.

Generate — context is split into "Knowledge graph facts" and "Relevant passages" in the prompt, letting the LLM cross-reference structured and unstructured evidence.

3. Evaluation

run_eval.py runs both GraphRAG and standard RAG (plain vector search + LLM) over the same 30-question dataset and scores both with RAGAS.

---

Stack

Layer	Technology
LLM	Ollama — llama3.1:8b
Embeddings	nomic-embed-text via Ollama
Graph DB	Neo4j 5 (Community)
Vector DB	ChromaDB
NLP	spaCy en_core_web_trf
Agent	LangGraph
Evaluation	RAGAS
API	FastAPI
UI	Streamlit
Package manager	uv

---

Setup

Prerequisites

• Ollama installed and running
• uv installed
• Docker + Docker Compose

1. Pull models

ollama pull llama3.1:8b
ollama pull nomic-embed-text

2. Start databases

docker compose up -d

Neo4j: http://localhost:7474 (credentials: neo4j / graphrag123) ChromaDB: http://localhost:8001

3. Install dependencies

uv sync
uv run python -m spacy download en_core_web_trf

4. Ingest documents

uv run python -c "
from src.ingestion.pipeline import IngestionPipeline
pipeline = IngestionPipeline()
pipeline.ingest('src/data/sample_docs/1706.03762v7.pdf')  # Attention Is All You Need
pipeline.ingest('src/data/sample_docs/gpt2.pdf')
pipeline.ingest('src/data/sample_docs/bert.pdf')
"

5. Run the API

uv run uvicorn src.api.main:app --reload

6. Launch the UI

uv run streamlit run src/ui/app.py

---

Evaluation

Run the full RAGAS comparison (GraphRAG vs standard RAG):

uv run python -m src.tests.eval.run_eval

Output:

============================================================
Metric                      GraphRAG   Standard      Delta
============================================================
faithfulness                   0.553      0.523     +0.030
answer_relevancy               0.545      0.726     -0.181
llm_context_precision          0.650      0.294     +0.356
context_recall                 0.826      0.778     +0.048

---

Project Structure

graphrag/
├── docker-compose.yml
├── pyproject.toml
└── src/
    ├── agent/
    │   └── graph_agent.py       # LangGraph state machine
    ├── ingestion/
    │   ├── pipeline.py          # End-to-end ingestion
    │   ├── parser.py            # PDF/doc parsing
    │   ├── chunker.py           # Semantic chunking
    │   ├── ner_extractor.py     # Entity extraction (spaCy + LLM)
    │   └── relation_extractor.py # Relation extraction (LLM)
    ├── retrieval/
    │   ├── graph_retriever.py   # Graph traversal + entity vector search
    │   └── hybrid_ranker.py     # Reciprocal Rank Fusion
    ├── storage/
    │   ├── graph_store.py       # Neo4j interface
    │   └── vector_store.py      # ChromaDB interface
    ├── api/
    │   └── main.py              # FastAPI endpoints
    ├── ui/
    │   └── app.py               # Streamlit chat UI
    ├── tests/eval/
    │   └── run_eval.py          # RAGAS evaluation
    └── data/
        ├── sample_docs/         # PDF papers
        └── eval_datasets/
            └── eval_qa.json     # 30 Q&A pairs (1–3 hop)