Mumbai Swapnil Patil — RAG-to-MCP Submission

[swap002]

RAG-to-MCP — Submission PR

Name: Swapnil Patil
City / Group: Mumbai
Date: 2026-07-16
AI tool(s) used: Claude Code (Sonnet 5)

---

Submission Checklist

• uc-0a/agents.md — present and updated
• uc-0a/skills.md — present and updated
• uc-0a/classifier.py — runs without crash
• uc-0a/results_pune.csv — output present (no test_mumbai.csv exists in data/city-test-files/, so Pune was used — see note below)
• uc-rag/agents.md — present and updated
• uc-rag/skills.md — present and updated
• uc-rag/rag_server.py — not the stub, own implementation
• uc-mcp/agents.md — present and updated
• uc-mcp/skills.md — present and updated
• uc-mcp/mcp_server.py — passes 3 of 5 test_client.py checks (see UC-MCP section — the 2 failures are an environment limitation, not a server bug)
• 3+ commits with meaningful messages, one per UC
• All sections below filled

Note on city: the branch/participant name uses Mumbai, but data/city-test-files/ only ships test_pune.csv, test_hyderabad.csv, test_kolkata.csv, test_ahmedabad.csv (no Mumbai file). Used test_pune.csv for UC-0A.

Note on environment: per instructions for this exercise, no new packages were installed. sentence-transformers was already present; chromadb and google-generativeai were not, and were not installed. This means UC-RAG's build_index/retrieve_and_answer and UC-MCP's in-scope answer path could not be run end-to-end in this environment. Everything that doesn't require those two packages was verified directly (see below) — install chromadb + google-generativeai and set GEMINI_API_KEY to run the full pipeline.

---

UC-0A — Complaint Classifier

Which failure mode did you encounter first?

Severity blindness — a naive classifier that only pattern-matches the category taxonomy would classify "Deep pothole near bus stop. School children at risk during morning hours." (row PM-202402) as Standard priority, because "pothole" drives the category but nothing forces priority to escalate on the child/school signal.

Which enforcement rule fixed it? Quote from your agents.md:

"Priority must be Urgent if the description contains any of these severity keywords (case-insensitive, substring match): injury, child, school, hospital, ambulance, fire, hazard, fell, collapse. This check runs before any other priority logic and cannot be overridden by category."

Your commit message for UC-0A:

UC-0A Fix severity blindness: naive category-only matching would let an injury/child/school/hazard complaint fall through as Standard priority → added an explicit severity-keyword scan that forces Urgent independent of and prior to taxonomy classification, plus Other+NEEDS_REVIEW fallback for descriptions matching no taxonomy keyword

Verification checkpoints:

• All severity-signal rows (injury/child/school/hospital keywords) classified as Urgent — verified in results_pune.csv: rows PM-202402 (child/school), PM-202411 (hazard), PM-202420 (injury), PM-202446 (fell) are all Urgent.
• No invented categories outside the defined taxonomy — every row's category is one of the 10 allowed values.
• Justification column present and non-empty for every row.

---

UC-RAG — RAG Server

Which failure mode did you encounter? (chunk boundary / wrong retrieval / answer outside context)

Chunk boundary. HR Leave Policy clause 5.2 ("LWP requires approval from the Department Head and the HR Director. Manager approval alone is not sufficient.") is the exact multi-approver clause the README warns about — a fixed-size splitter with no sentence awareness can cut it mid-clause and leave one approver out of the retrieved context.

What chunking strategy did you use and why?

Sentence-accumulating chunking: split each document into sentences, then accumulate whole sentences into a chunk until adding the next sentence would exceed 400 tokens, at which point a new chunk starts. A chunk boundary can only ever fall between two sentences, never inside one. Verified directly (no chromadb needed) against the real policy documents: policy_hr_leave.txt → 2 chunks (max 399 tokens), and clause 5.2's two approvers ("Department Head" and "HR Director") land in the same chunk.

Did your system correctly refuse "What is the flexible working culture?"? (Should return refusal template — not in any document)

Not verified — requires the ChromaDB index to be built and a live LLM call, and chromadb/GEMINI_API_KEY were not installed/configured in this environment per instructions. The refusal path itself (retrieve_and_answer returns the refusal template without calling the LLM when no chunk clears the 0.6 threshold) is implemented and code-reviewed, not run end-to-end.

Did your system retrieve the correct document for "Can I use my personal phone for work files?"? (Should retrieve IT policy, not HR leave policy)

Not verified for the same reason (no chromadb installed to build/query the index). The retrieval code groups results by source document and never merges chunks from two documents into one claim, but the actual vector query was not executed in this environment.

Which enforcement rule in agents.md prevented answers outside retrieved context?

"The answer must be composed only from information present in the retrieved (post-threshold) chunks. Adding any claim, qualifier, or 'as is standard practice' statement not traceable to a retrieved chunk is a context breach." — enforced in code by building the LLM prompt exclusively from chunks that passed the similarity filter, and returning the refusal template (no LLM call at all) when nothing passes.

Your commit message for UC-RAG:

UC-RAG Fix chunk boundary: fixed-size splitting would cut HR clause 5.2's two approvers (Department Head, HR Director) across separate chunks → switched to sentence-accumulating chunking capped at 400 tokens; verified clause 5.2 stays intact in one chunk against the real policy documents

Verification checkpoints:

• At least 3 test queries return grounded answers — not verified, chromadb not installed in this environment.
• "What is the flexible working culture?" returns the refusal template — not verified, same reason.
• "Can I use my personal phone for work files?" retrieves IT policy, not HR leave policy — not verified, same reason.
• Chunking produces more than 1 chunk per document — verified directly: all 3 policy documents produce 2 chunks each.

---

UC-MCP — MCP Server

Paste your tool description from mcp_server.py TOOL_DEFINITION:

"Answers CMC staff questions about exactly three policy documents: the HR Leave Policy (leave entitlements, approvals, encashment), the IT Acceptable Use Policy (corporate/personal device rules, passwords, data handling), and the Finance Reimbursement Policy (travel, WFH equipment, training, mobile/internet reimbursement). Every answer is grounded in and cites the retrieved document chunks (source document + chunk index) — it never answers from general knowledge. Do NOT call this tool for questions outside these three documents (e.g. budget forecasts, hiring, org strategy, flexible-working culture, or anything not covered by HR leave, IT acceptable use, or finance reimbursement policy) — such questions return isError: true with a refusal message instead of a guessed answer."

Does it state the document scope explicitly?

Yes — names all three documents (HR Leave, IT Acceptable Use, Finance Reimbursement) and gives examples of what's out of scope.

Run result: python3 test_client.py --run-all (Paste the summary output)

tools/list — ✅ Tool description mentions scope
In-scope: 'Who approves leave without pay?' — ❌ FAIL (isError: True, "No module named 'chromadb'")
Cross-doc test: personal phone + work files — ❌ FAIL (isError: True, "No module named 'chromadb'")
Out-of-scope: 'What is the budget forecast for 2025?' — ✅ PASS — correctly refused (isError: True)
Unknown method → -32601 — ✅ PASS — expected error received

3 of 5 checks pass. The 2 failures are because chromadb is not installed in this environment (not installed per instructions for this exercise) — rag_server.py's query() raises ModuleNotFoundError, which query_policy_documents catches and correctly reports as isError: true rather than crashing, but no real answer is produced. This satisfies the minimum-pass bar ("passes at least one test_client.py test") but the in-scope answer path is unverified end-to-end.

Did the budget forecast question return isError: true?

Yes — verified, though for this environment it happens to hit the same chromadb import error as any other query rather than the intended "no chunk cleared 0.6" refusal path. Both paths correctly produce isError: true, but the out-of-scope-specific refusal logic itself is unverified until chromadb is installed.

In one sentence — why is the tool description the enforcement?

An agent decides whether to call the tool purely by reading its description before ever invoking it, so a vague description ("answers questions about policies") gives the agent implicit permission to call the tool for anything, while a scoped description that names the three documents and the refusal condition lets the agent self-filter out-of-scope questions before wasting a call.

Your commit message for UC-MCP:

UC-MCP Fix vague tool description: no stated scope would let an agent call query_policy_documents for any question, wasting calls and inviting hallucinated refusals → tool description now explicitly names the three covered documents and states refusal behavior for out-of-scope questions

Verification checkpoints:

• Tool description explicitly states document scope.
• Tool description states refusal behavior for out-of-scope queries.
• python3 test_client.py --run-all executes without connection error.
• Budget forecast question returns isError: true.

---

CRAFT Reflection

Which step of the CRAFT loop was hardest across all three UCs?

Check — validating UC-RAG and UC-MCP's actual retrieval/answer behavior was blocked by not installing chromadb/google-generativeai in this environment. I could verify the chunking logic and the JSON-RPC/error-handling contract directly (both are pure-Python and dependency-free), but the vector-similarity retrieval and LLM-grounded answer path could only be verified by code review, not by running it.

What did you add to agents.md manually that the AI did not generate?

The ordering guarantee that the severity-keyword check in UC-0A runs before and independently of category classification (so an urgent signal can never be masked by an uncertain category), and the UC-RAG rule that cross-document evidence must be grouped and cited per-source rather than merged into one undifferentiated context block.

One specific task in your real work where you will use R.I.C.E in the next 7 days:

Writing the enforcement rules for an internal support-ticket triage prompt, so that vague/ambiguous tickets are explicitly routed to a human-review queue instead of the LLM guessing a category with false confidence — the same failure mode as UC-0A's severity blindness.

[github-actions]

Hi there, participant! Thanks for joining our RAG-to-MCP Workshop!

We're reviewing your PR for the 3 Use Cases (UC-0A, UC-RAG, UC-MCP). Once your submission is validated and merged, you'll be awarded your completion badge!

Next Steps:

• Make sure all 3 UCs are finished.
• Ensure your commit messages match the required format.
• Fill out every section of the PR template.
• Good luck!