Add data loading and chunking pipeline for document processing

[alok-kumar0421]

🚀 Feature Request

Problem
Currently, the project does not have a structured pipeline for loading documents and splitting them into chunks for downstream processing.

Proposed Solution
Implement a data ingestion module that:

• Loads documents (e.g., from GitHub or local files)
• Splits them into smaller chunks using a text splitter

Why is this needed?

• Required for embedding generation
• Improves retrieval performance
• Forms the base for RAG pipeline

Possible Approach

• Use LangChain document loaders
• Use RecursiveCharacterTextSplitter for chunking

Additional Context
This will be the first step towards building a full retrieval-based QA system.

[alok-kumar0421]

@yash-pouranik Hi,
I have created a diagram for the document loading and chunking pipeline.

Please let me know if this approach looks good. I would like to work on this under NSoC.

[yash-pouranik]

@alok-kumar0421
Hey thank you for pointing this out
Thats impressive approach. Just Some Points to go through.

Required Changes

• Do not merge all documents into a single list without structure
• Preserve document-level metadata (e.g., source file).
• This is important for traceability and better retrieval.
• Add MDX preprocessing step (important, bcoz If we directly use .txt or .pdf the document structure is losed so best is parse .mdx into .md than process)
• Since our docs are in .mdx, we need to strip JSX/components before chunking.
• Otherwise, embeddings will contain noise and hurt retrieval quality.
• Improve metadata
• In addition to filename and chunk index, include:
• section / heading (if possible)
• Adjust chunking strategy
• Increase chunk size (~700–900 tokens) for better context
• Ensure chunking does not break code blocks
• Define retrieval strategy
• Specify topK (e.g., 4–5 chunks) for similarity search

[yash-pouranik]

@alok-kumar0421
Also can u share which files will be affected or the new files/folder?

Suggested foler structure

apps/
└── python-service/ ❗ NEW
├── app/
│ ├── main.py
│ ├── api/
│ │ ├── routes/
│ │ │ ├── health.py
│ │ │ └── ai.py ❗ (/ai/ask later)
│ ├── core/
│ │ ├── config.py
│ │ └── security.py
│ ├── services/
│ │ └── rag/
│ │ ├── ingestion.py ❗ MAIN (loader + chunking)
│ │ ├── cleaner.py ❗ MDX cleaning
│ │ ├── chunker.py ❗ splitter logic
│ │ ├── embeddings.py
│ │ ├── vectorstore.py ❗ chromadb
│ │ └── retriever.py
│ └── models/
│ └── document.py ❗ schema (text + metadata)
├── requirements.txt
└── Procfile

[alok-kumar0421]

Hi @yash-pouranik,

I have implemented the basic document loading and chunking pipeline.

• Documents are loaded from folders (docs + internal)
• Metadata like source file and folder is added
• Chunking is done using RecursiveCharacterTextSplitter
• Embeddings are generated and stored in ChromaDB

Currently, it works end-to-end for ingestion and vector storage.

Next, I can work on:

• MDX cleaning
• Better metadata (headings/sections)
• Improving chunking strategy

Please let me know your feedback.

[yash-pouranik]

Create PR @alok-kumar0421

[yash-pouranik]

@alok-kumar0421 ??

[alok-kumar0421]

@yash-pouranik
i will create pr, now i were testing on my local file not oficially urBackend file after all testing i will create pr for all

[yash-pouranik]

So u mean u will add whole feature in one PR??
that will close ur 2 issues na?

[alok-kumar0421]

As I move the code of each item to the Urbanend website.
i will create pr for each

[alok-kumar0421]

@yash-pouranik mentor,
am i correct, if not please correct me.