CompeteLLM

A 56M-parameter language model trained from scratch in PyTorch, with two competing memory/retrieval systems benchmarked head-to-head — Standard RAG (System A) vs. an ancient-technique-inspired memory architecture (System B).

Background

This project was inspired by ancient human memory techniques — the Method of Loci (memory palace), Vedic recitation error-correction (jatapatha/ghanapatha), PAO (Person-Action-Object) encoding, Aboriginal songlines, Incan quipu, and research on expert chunking and synesthetic memory (Solomon Shereshevsky). These systems were engineered solutions to the same problems modern AI faces: durable storage, retrieval under noise, self-correction, and graceful forgetting.

The project asks: can these ancient principles be implemented as a machine memory layer that outperforms standard RAG?

Model

Architecture	Decoder-only Transformer (GPT-style)
Parameters	55,880,704 (~56M)
Layers	12 transformer blocks
Embedding dim	512
Attention heads	8 (64 dim each)
Feedforward dim	2048
Context length	256 tokens
Vocabulary	35,000 tokens (BPE via ByteLevelBPETokenizer)
Embeddings	Tied (input/output weight sharing)
Dropout	0.1
Hardware	Apple M3 Pro (MPS backend)
Framework	PyTorch

Training

• Dataset: WikiText-103 (raw, ~114M tokens after tokenization)
• Optimizer: AdamW (lr=3e-4, gradient clipping max_norm=1.0)
• Batch size: 32
• Epochs: 3 (checkpoint saved every 500 steps)
• Fine-tuning: SQuAD instruction fine-tuning (20k examples, 2 epochs, lr=5e-5) to teach QA format

Training results

Epoch	Val Loss	Perplexity
1	5.0975	163.61
2	4.4579	86.30
3	4.1659	64.45

For reference: GPT-2 Small (117M params) achieves ~18-29 perplexity on WikiText-103. N-gram models land around 100-200. Random baseline is ~35,000.

System A — Standard RAG

• Embeddings: sentence-transformers (all-MiniLM-L6-v2, 384-dim)
• Storage: flat vector store with cosine similarity search
• Retrieval: nearest-neighbor, no verification, no consistency check
• Generation: fine-tuned backbone model

Files: system_a/chunker.py, embedder.py, vector_store.py, retriever.py, generate.py

System B — Ancient-Technique Memory

Five components, each modeled on a specific historical memory technique:

1. Graph Index (Method of Loci) — chunks stored as nodes in a similarity graph with edges between related chunks (threshold 0.5); retrieval walks the graph from entry points for 2 hops, surfacing connected related content that flat search misses.
2. Redundant Store (Vedic jatapatha) — every chunk stored in 3 structural variants (original, second-half+first-half swap, every-other-word-group); a consistency score is the average pairwise cosine similarity between variants (0–1); chunks below 0.7 are flagged as uncertain — directly implementing the Vedic error-correction principle.
3. Concept Clusters (PAO system) — K-means clustering (20 clusters) groups chunks by topic; queries are first routed to the nearest cluster centroid, then searched only within that cluster, reducing search space by ~94% while maintaining precision.
4. Grounded Retriever (Songlines/Quipu) — every chunk gets an MD5 hash at index time; the hash is recomputed at retrieval time and compared, with a mismatch flagging the chunk as unverified; confidence levels (high/medium/low/unverified) are assigned per result; the system refuses to answer if no grounded results are found.
5. Memory Manager (Shereshevsky controlled forgetting) — activity scores (0–1) per chunk; retrieval boosts score (+0.2), time-decay reduces score periodically; chunks below min_score (0.3) are considered forgotten; prune() removes the lowest-scoring chunks when the total exceeds max_chunks (2000).

Files: system_b/graph_index.py, redundant_store.py, concept_clusters.py, grounded_retriever.py, memory_manager.py, generate.py

Benchmark results

15 questions across easy/medium/hard difficulty + 5 paraphrase pairs.

Metric	System A	System B
Retrieval Precision	0.6222	0.6222
Avg Response Time	3.32s	5.07s
Avg Consistency	0.0830	0.0543
Source Verification	None	MD5
Confidence Scoring	None	High/Med/Low
Refusal Capability	Never	Yes
Internal Consistency Check	None	0.85–0.92/chunk

Key finding

At 2,228-chunk scale, retrieval precision is identical since direct similarity search already finds all relevant content (graph walking adds value at 100k+ chunks). System B's advantages are architectural — verifiability, confidence, and refusal — which would show quantitatively at larger scale, with corrupted data, or under adversarial inputs.

Project structure

competellm/
├── data/                  # WikiText-103 raw + cached tokenized
├── tokenizers/            # BPE vocab.json + merges.txt
├── model/                 # config.py, architecture.py, train.py, evaluate.py, finetune.py
├── checkpoints/           # Saved model weights (gitignored — too large)
├── system_a/              # Standard RAG pipeline
├── system_b/              # Ancient-technique memory system
└── benchmark/             # run_benchmark.py + visualize.py + results/

Setup

• Python 3.12, PyTorch with MPS (Apple Silicon) or CUDA
• pip install torch transformers datasets tokenizers sentence-transformers numpy tqdm pandas

Run in order:

data/download_data.py
tokenizers/train_tokenizer.py
model/train.py
model/finetune.py
system_a/vector_store.py
system_b/graph_index.py + redundant_store.py + concept_clusters.py + grounded_retriever.py + memory_manager.py
benchmark/run_benchmark.py