[Pune] Shubham Gurjar — RAG-to-MCP Submission

[shubhamg9109]

RAG-to-MCP — Submission PR

Name: Shubham Gurjar
City / Group: Pune
Date: 2026-07-22
AI tool(s) used: Claude Code

---

Submission Checklist

• uc-0a/agents.md — present and updated
• uc-0a/skills.md — present and updated
• uc-0a/classifier.py — runs without crash
• uc-0a/results_pune.csv — output present
• uc-rag/agents.md — present and updated
• uc-rag/skills.md — present and updated
• uc-rag/rag_server.py — not the stub, your implementation
• uc-mcp/agents.md — present and updated
• uc-mcp/skills.md — present and updated
• uc-mcp/mcp_server.py — passes at least one test_client.py test
• 3+ commits with meaningful messages, one per UC
• All sections below filled

---

UC-0A — Complaint Classifier

Which failure mode did you encounter first?

Ran a naive control classifier first (no severity keywords, no reason field, first-match category picking). It missed all four severity-signal complaints (school children at risk, electrical hazard, injury, elderly resident fell) — all classified Standard instead of Urgent — and confidently picked a single category even on a genuinely ambiguous row (a "Heritage street, lights out" complaint that matches both Heritage Damage and Streetlight keywords equally).

Which enforcement rule fixed it? Quote from your agents.md:

"Priority must be Urgent if the description contains any of these severity keywords (case-insensitive, substring match): injury, child, school, hospital, ambulance, fire, hazard, fell, collapse." and "If category cannot be determined confidently — zero category keywords match, OR two or more categories tie on keyword-match count — output category: Other and flag: NEEDS_REVIEW."

Your commit message for UC-0A:

UC-0A Fix severity blindness & false confidence: no keyword rule + naive category picks blended ambiguous cases -> added injury/child/school/hospital/ambulance/fire/hazard/fell/collapse Urgent triggers and category-tie detection that forces Other+NEEDS_REVIEW instead of guessing

Verification checkpoints:

• All severity-signal rows (injury/child/school/hospital keywords) classified as Urgent
• No invented categories outside the defined taxonomy
• Justification column present and non-empty for every row

---

UC-RAG — RAG Server

Which failure mode did you encounter?
(chunk boundary / wrong retrieval / answer outside context)

Wrong retrieval — but the deeper root cause I found was threshold miscalibration. With the README's literal single-chunk cosine similarity threshold of 0.6, the RAG server (and even the shipped "reference" stub_rag.py) refused all four reference queries, including "Who approves leave without pay?" — a question the HR document clearly answers. Root cause: all-MiniLM-L6-v2 cosine similarity between a natural-language question and a short policy clause rarely exceeds ~0.4, even for the objectively correct clause.

What chunking strategy did you use and why?

Sentence-aware accumulation up to 400 tokens (never splitting mid-sentence, per the chunk-boundary rule) — each document ends up as 2 chunks. I route retrieval by each document's average chunk score rather than any single chunk's score, since a single short clause can score deceptively low even when correct, while the document-level average is a stable signal. Recalibrated the relevance floor to 0.20 (empirically validated: in-scope documents averaged 0.29–0.37 on the four reference queries, the genuinely out-of-scope query topped out at 0.09 — a wide safety margin either side of 0.20).

Did your system correctly refuse "What is the flexible working culture?"?
(Should return refusal template — not in any document)

Yes — confirmed via --json output: "refused": true, all three documents' average scores fell between 0.03–0.09, well under the 0.20 floor.

Did your system retrieve the correct document for "Can I use my personal phone for work files?"?
(Should retrieve IT policy, not HR leave policy)

Yes — routes to policy_it_acceptable_use.txt only (both its chunks, average score 0.333), with zero HR chunks in the cited context. This was the case I had to specifically fix: a naive per-chunk top-k of 3 was letting a noisy IT header chunk beat the correct clause, and letting cross-document chunks leak in.

Which enforcement rule in agents.md prevented answers outside retrieved context?

"The answer must be generated from retrieved chunks only. The prompt sent to the LLM must state that only the provided context may be used, and must instruct the model to say so explicitly if the context does not contain the answer." — enforced in the prompt template inside retrieve_and_answer.

Your commit message for UC-RAG:

UC-RAG Fix wrong retrieval & threshold miscalibration: naive per-chunk cosine similarity let a noisy IT/HR header chunk outscore the correct clause and let 0.6 refuse every genuinely answerable query (even stub_rag.py) -> route by each document's average chunk score (stable signal), recalibrated threshold to 0.20 (empirically validated: in-scope docs 0.29-0.37, out-of-scope 0.09), added near-tie cross-document handling that answers per-document without blending, and switched llm_adapter.py to Claude (this participant's chosen provider)

Verification checkpoints:

• At least 3 test queries return grounded answers (cited from retrieved context)
• "What is the flexible working culture?" returns the refusal template (not a hallucinated answer)
• "Can I use my personal phone for work files?" retrieves IT policy, not HR leave policy
• Chunking produces more than 1 chunk per document (not whole-document embedding)

---

UC-MCP — MCP Server

Paste your tool description from mcp_server.py TOOL_DEFINITION:

"Answers questions about City Municipal Corporation (CMC) policy documents ONLY: the HR Leave Policy, IT Acceptable Use Policy, and Finance Reimbursement Policy. Returns an answer grounded in and citing the retrieved policy document chunks — never from general knowledge. Does NOT answer questions outside these three documents (for example budget forecasts, project timelines, staffing plans, or anything not covered by HR leave, IT acceptable use, or finance reimbursement rules) — those return isError: true with a refusal message instead of a guess."

Does it state the document scope explicitly?

Yes — names all three documents explicitly and states the refusal condition.

Run result: python3 test_client.py --run-all
(Paste the summary output)

tools/list: 1 tool returned, scope confirmed in description (✅).
"Who approves leave without pay?": isError false, answer returned (✅ PASS).
"Can I use my personal phone for work files?": isError false, answer returned, IT-only (✅ PASS).
"What is the budget forecast for 2025?": isError true, correctly refused (✅ PASS — correctly refused out-of-scope question).
Unknown method: JSON-RPC error -32601 Method not found (✅ PASS — expected error received).
All 5 checks passed.

Did the budget forecast question return isError: true?

Yes, confirmed.

In one sentence — why is the tool description the enforcement?

The tool description is the only thing an agent reads before deciding to call the tool, so a vague description ("answers policy questions") gives the agent implicit permission to call it for anything, while a scoped description with an explicit refusal condition lets the agent self-select away from out-of-scope questions before ever making the call.

Your commit message for UC-MCP:

UC-MCP Fix vague tool description & cold-start timeout: generic 'answers policy questions' gives no scope signal and lets agents call it for out-of-scope questions -> stated the exact HR/IT/Finance document scope and refusal behavior in TOOL_DEFINITION, implemented JSON-RPC 2.0 dispatch (tools/list, tools/call, -32601/-32602/-32700/-32603) always at HTTP 200, and added a startup warm-up call so the embedding model loads before accepting connections instead of on the first request (which exceeded test_client.py's 10s timeout); also added rag_server.query() so mcp_server.py uses the real UC-RAG implementation instead of silently falling back to stub_rag.py

Verification checkpoints:

• Tool description explicitly states document scope (which policies are covered)
• Tool description states refusal behavior for out-of-scope queries
• python3 test_client.py --run-all executes without connection error
• Budget forecast question returns isError: true (out of scope)

---

CRAFT Reflection

Which step of the CRAFT loop was hardest across all three UCs?

The Check step for UC-RAG, by far. The chunking, retrieval, and grounding logic all looked correct on inspection, but empirically the specified 0.6 similarity threshold refused every genuinely answerable reference query — including against the shipped stub_rag.py. It took directly measuring the actual embedding scores (not just trusting the spec's example number) to find that the real signal lived in per-document averages, not per-chunk maxima.

What did you add to agents.md manually that the AI did not generate?

The explicit calibration note in uc-rag/agents.md explaining why the enforcement threshold is 0.20 for all-MiniLM-L6-v2 rather than the README's literal 0.6 — with the actual measured scores from the four reference queries as evidence, so a reviewer can verify the number instead of just trusting it.

One specific task in your real work where you will use R.I.C.E in the next 7 days:

Writing the enforcement rules for an internal support-ticket triage prompt at work, where "escalate urgent tickets" has the same severity-blindness risk UC-0A surfaced — I'll write out the explicit keyword/condition list up front instead of trusting a vague instruction.

[github-actions]

Hi there, participant! Thanks for joining our RAG-to-MCP Workshop!

We're reviewing your PR for the 3 Use Cases (UC-0A, UC-RAG, UC-MCP). Once your submission is validated and merged, you'll be awarded your completion badge!

Next Steps:

• Make sure all 3 UCs are finished.
• Ensure your commit messages match the required format.
• Fill out every section of the PR template.
• Good luck!