ragfallback v2.0 — 10 silent RAG failure modes caught before they reach production

[irfanalidv]

Hey everyone 👋

Just shipped ragfallback v2.0 — a Python library that catches the RAG failures
that don't raise exceptions but quietly destroy answer quality in production.

The problem

Most RAG bugs are silent. Embedding dimension mismatches corrupt your index.
Bad chunks poison retrieval before a single query runs. Multi-hop questions always
fail single-shot retrieval. Stale indexes serve outdated data. Nothing crashes —
the system just returns wrong answers.

ragfallback gives you guardrails across the full pipeline: ingest → index →
retrieval → generation → evaluation.

10 failure modes covered

#	Failure	Module
1	Query mismatch → silent empty results	AdaptiveRAGRetriever + QueryVariationsStrategy
2	Embedding model switch corrupts index dimensions	EmbeddingGuard
3	Bad chunks poison retrieval	ChunkQualityChecker
4	Retrieved chunks overflow LLM context window	ContextWindowGuard
5	Keyword queries fail dense retrieval silently	SmartThresholdHybridRetriever
6	Primary retriever outage returns empty, no fallback	FailoverRetriever
7	Multi-step questions always fail single-shot RAG	MultiHopFallbackStrategy
8	Index serves stale data after document updates	StaleIndexDetector
9	Answer quality invisible in production	RAGEvaluator
10	Cross-boundary answers lost between adjacent chunks	OverlappingContextStitcher

Verified numbers on real public datasets

Ran examples/real_data_demo.py on 200 SQuAD Wikipedia passages:

RetrievalHealthCheck — 20 real Q&A probes:
  Hit rate   : 100%
  Avg latency: 25ms

RAGEvaluator — 10 real Q&A pairs, no LLM judge:
  Avg recall@k     : 100%
  Avg faithfulness : 79.5%

Install

# no API keys needed for core path
pip install ragfallback[chroma,huggingface,real-data]

LLM-dependent features (adaptive retrieval, multi-hop) accept any
LangChain-compatible LLM or a free Mistral key.

Looking for feedback on

1. Failure modes I've missed — what silent RAG bugs have you hit in production
   that aren't in the list of 10?
2. Vector store coverage — Pinecone and Weaviate are in the extras but
   untested end-to-end. Anyone using them?
3. Evaluation without an LLM judge — the heuristic faithfulness scorer is
   approximate by design. What thresholds are people actually using?

Repo: https://github.com/irfanalidv/ragfallback
PyPI: https://pypi.org/project/ragfallback/
CHANGELOG: https://github.com/irfanalidv/ragfallback/blob/main/CHANGELOG.md

Happy to review PRs or answer questions here.