Feat/rag

[sofiachavezb]

Summary

Adds the Retrieval-Augmented Generation (RAG) module to DashAI: a full
end-to-end pipeline to chat with your own documents, composed of four stages —
Document Loading → Chunking → Retrieval → Generation.

This PR introduces the complete backend (domain models, a dedicated service
layer, pure factories, a retriever hierarchy, embedding families, document
extractors, prompts and a background job) and the frontend (session setup
wizard, advanced configuration modals, document/prompt management and a chat
view with sources), plus the Alembic migrations, a comprehensive test suite and
architecture documentation.

Key capabilities:

• Chunking — Character, Token and Recursive-Character chunking models, with
  SHA-256 chunk-set caching to avoid re-chunking identical (documents, config)
  combinations.
• Retrieval — Sparse (TF-IDF, BM25), Dense (via any DenseEmbedding),
  Cross-Encoder re-rankers (17 SentenceTransformer models) and Composite
  retrievers (Sequential, Parallel, MMR). Retriever presets (Keyword / Semantic
  / Hybrid) and a visual composite-retriever tree builder on the frontend.
• Embeddings — BERT, DistilBERT, RoBERTa, E5, LaBSE, Instructor,
  Sentence-Transformers, OpenAI and more, exposed as schema-driven components.
• Generation — Any TextToTextGenerationTaskModel (local llama.cpp models
  and OpenAI-compatible remote APIs) driven by configurable, multi-language
  (en/es/pt/de/zh) prompts.
• Documents — Per-document extractors (PlainText, Pypdf, PyMuPDF,
  EasyOCR) with schema-driven configuration, on-demand extraction and a 1:1
  extracted-text cache.
• Strict validation — Recursive {component, params} schema validation for
  session creation and parameter updates, centralized in the validation service.

Type of Change

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

Changes (by file)

Backend — domain models (DashAI/back/models/RAG/)

• RAG_pipeline.py: RAGPipeline orchestration and typed RAGGenerationOutput.
• chunking_models/: BaseChunkingModel + Character/Token/Recursive models and factory.
• retrievers/: retriever hierarchy (sparse/, dense/, composite/, cross_encoder/),
  retriever_factory.py, enums.py, persistence.py.
• embeddings/: DenseEmbedding subclasses (BERT, DistilBERT, RoBERTa, E5, LaBSE,
  Instructor, Sentence-Transformer, OpenAI, FastText…).
• extractors/: PlainText, Pypdf, PyMuPDF, EasyOCR extractors.
• prompts/: generation + augmentation prompts and factories.
• documents/: BaseDocument, PDF/TXT documents, Chunk, DocumentFileType.
• exceptions/: unified RAG exception hierarchy.
• RAG_constants.py, RAG_models_factory.py, utils.py.

Backend — service layer (DashAI/back/services/RAG/)

• setup_service.py, session_validation_service.py, document_service.py,
  chunking_service.py, prompt_service.py, llm_service.py,
  retriever_db_service.py, embedding_storage_service.py,
  retriever_setup_service.py, cleanup_service.py, retriever_presets.py, utils.py.

Backend — endpoints / jobs / tasks / core

• api/api_v1/endpoints/: rag.py, documents.py, prompts.py,
  generative_session.py, generative_process.py, components.py.
• api/api_v1/schemas/RAG_prompt.py, document.py.
• job/RAG_job.py, job/generative_job.py, tasks/RAG_task.py.
• core/component_validation.py: generic recursive component-reference validation.
• initial_components.py: registers all RAG components (RAGPipeline, RAGTask,
  retrievers, embeddings, chunking models, extractors, prompts).

Backend — database (DashAI/alembic/versions/)

• New migrations for RAG tables, chunk-set architecture, composite retriever
  support, extractor table, parameters-hash columns and uniqueness constraints.

Frontend (DashAI/front/src/)

• pages/generative/RAGSession/: RAGSessionSetup wizard, RAGSessionPage,
  per-stage sections (ChunkingSection, RetrieverSection, GeneratorSection,
  PromptSection) and advanced modals (CompositeRetrieverBuilder,
  RetrieverConfigurationStep, NewPromptModal, …).
• components/generative/RAG/: document manager (DocumentSelector,
  DocumentTable, DocumentDetailPanel, DocumentsBar, preview/extractor modals),
  prompts (PromptSelectionTable, PromptParamsCard, PromptViewModal),
  summary and params panels, HighlightedTextarea, ragValidation.js.
• components/generative/: GenerativeChat, SourcesDisplay, DocumentReferencesModal.
• types/, utils/i18n/locales/*/generative.json (en/es/pt/de/zh), theme.js.

Docs & deps

• docs/RAG/: overview, backend/frontend architecture, execution flow, known
  limitations, document processing, future work, and README index.
• pyproject.toml: adds sentence-transformers, pypdf, pymupdf.
• uv.lock, yarn.lock, tsconfig.json.

Testing

• New backend test suite under tests/back/RAG/ (~28 files): pipeline unit,
  E2E job flow, session flow/validation, retriever configs, cross-encoder
  retrievers, extractors, chunking cache, prompts, LLM-service hash collision.
• Existing API tests updated in tests/back/api/.
• Run: pytest tests/back/RAG/ (in-memory SQLite, no setup needed).

Notes

• The RAGJob duplicates some GenerativeJob status helpers and the
  get_or_create_chunk_set() path is not concurrency-safe (SELECT-then-INSERT).
  See docs/RAG/05-known-limitations.md for the full list of accepted trade-offs.
• After schema changes, sqlite.db and ~/.DashAI/rag/ must be rebuilt from
  scratch (see the docs' maintenance section).
• FastTextEmbedding and the augmentation-prompt family are defined but not
  wired into the pipeline yet (see docs/RAG/07-future-work.md).

[cristian-tamblay]

Blocking: the plugin migration breaks upgrades for any user with an installed plugin

DashAI/alembic/versions/f928e0b5203d_rag_tables_added.py:24

batch_op.add_column(sa.Column('latest_version', sa.String(), nullable=False))
batch_op.drop_column('lastest_version')

This adds a NOT NULL column with no server_default. On SQLite, alembic runs
this in batch mode: it recreates the table and does
INSERT INTO _alembic_tmp_plugin ... SELECT ... FROM plugin. With any existing
row, that fails.

Reproduced on the real upgrade path (a database at develop's head
d5b3c8f2a041 with one installed plugin):

Running upgrade f06652057903 -> f928e0b5203d, RAG tables added
sqlalchemy.exc.IntegrityError: NOT NULL constraint failed: _alembic_tmp_plugin.latest_version

Migrations run on startup, so the app does not come up. The failure is also not
clean: it leaves alembic_version holding two rows (f06652057903 and
d5b3c8f2a041), so the database cannot go back to develop either:

ERROR [alembic.util.messaging] Can't locate revision identified by 'f06652057903'

Neither CI nor a fresh install catches this, because on an empty plugin table
the INSERT ... SELECT inserts nothing and passes.

Fix: pass server_default="" on the add_column, or add the column nullable,
backfill it, and only then make it NOT NULL. downgrade() at line 33 has the
same defect.

Two side notes on this migration: it is named "RAG tables added" but all it does
is rename lastest_version to latest_version on the plugin table, which is
unrelated to RAG and looks like it was picked up by accident. And downgrade()
reintroduces the typo.