Pune Zarrar Palekar — RAG-to-MCP Submission

[ZarrarPalekar]

RAG-to-MCP — Submission PR

Name: Zarrar Palekar
City / Group: Pune
Date: 2026-07-17
AI tool(s) used: Claude Code (Sonnet 5)

---

Submission Checklist

• uc-0a/agents.md — present and updated
• uc-0a/skills.md — present and updated
• uc-0a/classifier.py — runs without crash
• uc-0a/results_pune.csv — output present
• uc-rag/agents.md — present and updated
• uc-rag/skills.md — present and updated
• uc-rag/rag_server.py — not the stub, my own implementation
• uc-mcp/agents.md — present and updated
• uc-mcp/skills.md — present and updated
• uc-mcp/mcp_server.py — passes at least one test_client.py test (passes all 5)
• 3+ commits with meaningful messages, one per UC (5 total: 2×UC-0A, 2×UC-RAG, 1×UC-MCP)
• All sections below filled

---

UC-0A — Complaint Classifier

Which failure mode did you encounter first?

Taxonomy drift, then severity blindness. I ran a naive baseline first (free-text category guessing, no severity check) against test_pune.csv before writing the real classifier. It produced labels like "Pothole issue", "Lighting fault", and "General Civic Issue" instead of the fixed taxonomy — a different free-text label almost every time the underlying wording varied. It also left all 15 rows at Standard priority, including 4 rows containing explicit severity keywords (school/child, hazard, injury, fell).

Which enforcement rule fixed it? Quote from your agents.md:

"Category must be exactly one value from the fixed enum: Pothole, Flooding, Streetlight, Waste, Noise, Road Damage, Heritage Damage, Heat Hazard, Drain Blockage, Other. No synonyms, no new sub-categories, no free text."

"Priority must be Urgent if the description contains any of these severity keywords (case-insensitive, substring match): injury, child, school, hospital, ambulance, fire, hazard, fell, collapse. This rule overrides any other priority signal."

Your commit message for UC-0A:

UC-0A Fix taxonomy drift: naive classifier free-text labels ("Pothole issue", "Lighting fault") → restricted classify_complaint to the fixed 10-value category enum via leftmost-keyword matching, with Other+NEEDS_REVIEW fallback when nothing matches

UC-0A Fix severity blindness: 4 of 15 Pune complaints contained injury/child/school/hazard/fell keywords but stayed Standard priority → added SEVERITY_PATTERN regex that forces Urgent whenever a severity keyword is present, overriding all other priority signals

Verification checkpoints:

• All severity-signal rows (injury/child/school/hospital keywords) classified as Urgent — verified: PM-202402, PM-202411, PM-202420, PM-202446 all Urgent
• No invented categories outside the defined taxonomy — every row's category is one of the 10 allowed values
• Justification column present and non-empty for every row

---

UC-RAG — RAG Server

Which failure mode did you encounter?
(chunk boundary / wrong retrieval / answer outside context)

Chunk boundary, then wrong retrieval (as an over-refusal, the inverse of the usual "wrong document" version of this failure). I proved the chunk-boundary failure with a throwaway fixed-size 300-word chunker (no sentence awareness): it cut sentences mid-word — one chunk literally ended "...entitled to 26 weeks of paid maternity leave for the first two live", truncating "live births" — and separated HR clause 5.2's obligation from its 5.1 lead-in.

After fixing that with sentence-aware chunking, I hit a second, more interesting failure: even correctly-chunked, sentence-safe retrieval refused every single in-scope reference query at the README's literal 0.6 cosine-similarity threshold. I confirmed this wasn't a bug in my code — the repo's own reference stub_rag.py, using the same 400-token/0.6-threshold approach, reproduces the identical over-refusal. The root cause: all-MiniLM-L6-v2 rarely produces >0.6 raw cosine similarity between a short query and a paragraph-length passage, regardless of how relevant the passage is.

What chunking strategy did you use and why?

Section-aware chunking: each document is split on its own top-level numbered headings (e.g. "5. LEAVE WITHOUT PAY (LWP)"), so every chunk stays topically coherent instead of a generic max-token block spanning multiple unrelated sections. Any section that would still exceed 400 tokens falls back to sentence-safe accumulation. This produced 21 focused chunks across the 3 documents (vs. 6 with naive max-token packing) and measurably improved retrieval — e.g. "Who approves leave without pay?" scored 0.56 cosine similarity against the dedicated LWP section chunk, vs. 0.39 against a 400-word chunk blending five unrelated leave-policy sections.

Did your system correctly refuse "What is the flexible working culture?"?
(Should return refusal template — not in any document)

Yes — refused. Measured similarity for this query against every chunk in the index tops out at 0.22 (well below the 0.3 calibrated threshold), confirmed empirically before shipping the threshold change.

Did your system retrieve the correct document for "Can I use my personal phone for work files?"?
(Should retrieve IT policy, not HR leave policy)

Yes — all 3 top-k retrieved chunks are from policy_it_acceptable_use.txt only (sections 2, 3, 5: Corporate Devices, Personal Devices/BYOD, Data Handling). No HR chunks appear in the retrieved set, so there's nothing to blend.

Which enforcement rule in agents.md prevented answers outside retrieved context?

"The answer must use only information present in the retrieved chunks. Adding phrases like 'as is standard practice' or any claim not traceable to a retrieved chunk is a violation, even if it is factually plausible." — combined with the refusal-template rule, which is what actually gates generation: retrieve_and_answer never calls the LLM at all when no chunk clears the threshold, so there is no code path that can produce an unrooted answer for an out-of-scope question.

Your commit message for UC-RAG:

UC-RAG Fix chunk boundary: naive fixed-size 300-word splitting cut mid-sentence and even mid-word (verified: "...first two live" truncated inside "live births", and split clause 5.2's obligation from its 5.1 lead-in) → replaced with section-aware chunking that chunks on each document's numbered top-level headings (e.g. "5. LEAVE WITHOUT PAY"), falling back to sentence-safe packing only if a section exceeds 400 tokens, so a clause and its section never span two chunks

UC-RAG Fix wrong retrieval (over-refusal): literal 0.6 raw-cosine threshold from the README refused all 3 in-scope reference queries, including "Who approves leave without pay?" → measured actual score distributions (in-scope top scores 0.37-0.77, out-of-scope top scores 0.15-0.23 across all 21 chunks) and calibrated threshold to 0.3, which cleanly separates the two clusters; also fixed a distance-to-cosine conversion bug that was squaring an already-squared ChromaDB L2 distance

Verification checkpoints:

• At least 3 test queries return grounded answers (cited from retrieved context) — all 3 in-scope reference queries retrieve correct, correctly-attributed chunks
• "What is the flexible working culture?" returns the refusal template (not a hallucinated answer)
• "Can I use my personal phone for work files?" retrieves IT policy, not HR leave policy
• Chunking produces more than 1 chunk per document (not whole-document embedding) — 21 chunks across 3 documents (6–8 sections each)

---

UC-MCP — MCP Server

Paste your tool description from mcp_server.py TOOL_DEFINITION:

"Answers questions about CMC (City Municipal Corporation) HR Leave Policy, IT Acceptable Use Policy, and Finance Reimbursement Policy only. Returns an answer grounded in and citing the retrieved policy document chunks (document name + chunk index). For any question outside these three documents — budget forecasts, other departments, general HR advice not in these policies, etc. — returns a refusal (isError: true) instead of guessing. Call this tool only for questions about CMC HR leave, IT acceptable use, or finance reimbursement rules."

Does it state the document scope explicitly?

Yes — it names all three documents by their actual policy names (CMC HR Leave Policy, IT Acceptable Use Policy, Finance Reimbursement Policy), not a generic phrase like "company policies." I confirmed this mattered by first shipping the literal naive example from the README ("Answers questions about policies") and running test_client.py against it — it triggered the client's own built-in vagueness warning (⚠️ Tool description may be too vague) before I rewrote it.

Run result: python3 test_client.py --run-all
(Paste the summary output)

TEST: tools/list — discover available tools
✅ Tool description mentions scope

TEST: In-scope: 'Who approves leave without pay?'
isError: False → ✅ PASS — got an answer

TEST: Cross-doc test: personal phone + work files
isError: False → ✅ PASS — got an answer

TEST: Out-of-scope: 'What is the budget forecast for 2025?'
isError: True → ✅ PASS — correctly refused out-of-scope question

TEST: Unknown method → expect JSON-RPC error -32601
JSON-RPC Error: code=-32601 → ✅ PASS — expected error received

All 5 checks pass, no warnings.

Did the budget forecast question return isError: true?

Yes.

In one sentence — why is the tool description is the enforcement?

An agent decides whether to call query_policy_documents at all — and what it's allowed to expect back — purely by reading the description before ever seeing the code or the refusal logic behind it, so a vague description gives the agent no way to distinguish "in scope, will answer" from "out of scope, will refuse" until after it has already wasted a call.

Your commit message for UC-MCP:

UC-MCP Fix vague tool description: naive description "Answers questions about policies" gave no document scope — verified failure via test_client.py's own scope check, which flagged "⚠️ Tool description may be too vague" → rewrote description to name all three covered documents (CMC HR Leave, IT Acceptable Use, Finance Reimbursement) and state the refusal condition explicitly, plus implemented the full JSON-RPC 2.0 tools/list, tools/call, and error-code (-32700/-32601/-32602) handling in do_POST. Verified: all 5 test_client.py --run-all checks pass, including the budget-forecast question returning isError: true

Verification checkpoints:

• Tool description explicitly states document scope (which policies are covered)
• Tool description states refusal behavior for out-of-scope queries
• python3 test_client.py --run-all executes without connection error
• Budget forecast question returns isError: true (out of scope)

---

CRAFT Reflection

Which step of the CRAFT loop was hardest across all three UCs?

Check — specifically, distinguishing a genuine implementation bug from a genuine calibration problem. When UC-RAG's RAG pipeline refused every in-scope reference query at threshold 0.6, my first assumption was a bug in my retrieval code. It took cross-checking against the repo's own stub_rag.py reference implementation (which reproduces the identical over-refusal) and directly measuring cosine similarity distributions for in-scope vs. out-of-scope queries to confirm the threshold itself — not my code — was the actual defect. Enforcement rules copied verbatim from a spec are only as good as the assumptions baked into them; verifying those assumptions against real data was the hardest and most valuable step.

What did you add to agents.md manually that the AI did not generate?

The empirical calibration note in uc-rag/agents.md's similarity-threshold enforcement rule — the measured score ranges (in-scope 0.37–0.77, out-of-scope 0.15–0.23), the recalibrated value (0.3), and the explicit instruction to re-measure if the embedder or document set ever changes. A first-draft agents.md would have just repeated "threshold 0.6" from the README; the rule is only trustworthy once it reflects what the actual embedding model does on this actual data, not what the spec assumed it would do.

One specific task in your real work where you will use R.I.C.E in the next 7 days:

Any prompt or agent spec I write with a numeric default (a similarity threshold, a confidence cutoff, a retry limit) inherited from documentation or a template — I'll measure it against real data before trusting it, the same way the 0.6 similarity threshold here looked reasonable on paper but silently refused every legitimate query until I checked it against actual score distributions.

---

🤖 Built with Claude Code

[github-actions]

Hi there, participant! Thanks for joining our RAG-to-MCP Workshop!

We're reviewing your PR for the 3 Use Cases (UC-0A, UC-RAG, UC-MCP). Once your submission is validated and merged, you'll be awarded your completion badge!

Next Steps:

• Make sure all 3 UCs are finished.
• Ensure your commit messages match the required format.
• Fill out every section of the PR template.
• Good luck!