[Bangalore] Rahul Kini — AI-Code Sarathi Submission

[rahulkini31]

RAG-to-MCP — Submission PR

Name: Rahul Kini
City / Group: Bangalore
Date: 2026-08-23
AI tool(s) used: Claude Code

---

Submission Checklist

• uc-0a/agents.md — present and updated
• uc-0a/skills.md — present and updated
• uc-0a/classifier.py — runs without crash
• uc-0a/results_[city].csv — output present (using test_pune.csv — no Bangalore test file exists in data/city-test-files/)
• uc-rag/agents.md — present and updated
• uc-rag/skills.md — present and updated
• uc-rag/rag_server.py — not the stub, own implementation
• uc-mcp/agents.md — present and updated
• uc-mcp/skills.md — present and updated
• uc-mcp/mcp_server.py — passes all 5 test_client.py checks
• 3+ commits with meaningful messages, one per UC — see reviewer note below: this submission uses one combined commit, following resources/submission-guide.md's example format rather than the per-UC CRAFT formula in README/CONTRIBUTING/docs/guide.md. Those three docs disagree with submission-guide.md on both commit cadence and PR title text — flagging here rather than silently picking one.
• All sections below filled

---

UC-0A — Complaint Classifier

Which failure mode did you encounter first?

Severity blindness and taxonomy drift. A naive classifier has no fixed category enum and no explicit urgency keyword list, so it drifts on category spelling across similar complaints and won't reliably flag complaints mentioning children/schools/injury as Urgent unless told those words matter more than the complaint's age or reporter type.

Which enforcement rule fixed it? Quote from your agents.md:

"priority must be Urgent if the description contains any severity keyword (case-insensitive, word-boundary match): injury, child, school, hospital, ambulance, fire, hazard, fell, collapse — independent of category."

Your commit message for UC-0A:

Combined into the single session-submission commit — see "UC-0A UC-RAG UC-MCP Session submission — Rahul Kini" (full message in the commit body covers UC-0A's classifier work specifically).

Verification checkpoints:

• All severity-signal rows (injury/child/school/hospital keywords) classified as Urgent — verified: PM-202402 (school/children), PM-202411 (hazard), PM-202420 (injury), PM-202446 (fell)
• No invented categories outside the defined taxonomy
• Justification column present and non-empty for every row

---

UC-RAG — RAG Server

Which failure mode did you encounter?
(chunk boundary / wrong retrieval / answer outside context)

All three, at different points. Chunk boundary risk is structurally prevented by sentence-aware accumulation (never verified as a live failure since it's blocked by construction). Wrong retrieval and answer-outside-context were both observed directly: naive mode (no retrieval, full-document context) on "What is the flexible working culture?" produced a confident multi-paragraph synthesis stitching together unrelated clauses from all three documents — explicitly noting the term doesn't appear verbatim, then answering anyway instead of refusing.

What chunking strategy did you use and why?

Sentence-aware accumulation up to 400 tokens per chunk: sentences are added to the current chunk until the next one would exceed the limit, then a new chunk starts — so a chunk boundary always falls between sentences, never inside one (the fix for HR clause 5.2's two-approver requirement never being split across chunks).

Did your system correctly refuse "What is the flexible working culture?"?

Yes — returns the exact refusal template, versus naive mode's confident (wrong) synthesis quoted above.

Did your system retrieve the correct document for "Can I use my personal phone for work files?"?

Yes — retrieves and cites only policy_it_acceptable_use.txt (sections 3.1/3.2), never blending in HR leave policy, matching the reference table's requirement exactly.

Which enforcement rule in agents.md prevented answers outside retrieved context?

"Answer must use only information present in the retrieved chunks. Never add framing not in the source, e.g. 'as is standard practice in government organisations' — that phrase must never appear because it is never in the retrieved text." Combined with the generation prompt's explicit instruction to answer only from the provided context and never merge claims across documents.

Your commit message for UC-RAG:

Combined into the single session-submission commit — see "UC-0A UC-RAG UC-MCP Session submission — Rahul Kini" (full message in the commit body covers UC-RAG's threshold recalibration specifically).

Verification checkpoints:

• At least 3 test queries return grounded answers (cited from retrieved context) — ran all 4 reference queries, all pass
• "What is the flexible working culture?" returns the refusal template (not a hallucinated answer)
• "Can I use my personal phone for work files?" retrieves IT policy, not HR leave policy
• Chunking produces more than 1 chunk per document (not whole-document embedding) — 6 chunks total across 3 documents (2 each)

Note on the similarity threshold: README.md specifies 0.6. Measured against this corpus, all-MiniLM-L6-v2's raw cosine similarity for genuinely correct matches tops out around 0.35–0.39, while genuinely out-of-scope queries cap around 0.17–0.19 — a literal 0.6 threshold refuses every query in the reference table, including the ones required to return a cited answer. Recalibrated to 0.25 (documented in uc-rag/agents.md and inline in rag_server.py); all reference queries verified to pass at that value, including correctly refusing the out-of-scope ones.

---

UC-MCP — MCP Server

Paste your tool description from mcp_server.py TOOL_DEFINITION:

"Answers questions about CMC HR Leave Policy, IT Acceptable Use Policy, and Finance Reimbursement Policy only. Retrieves relevant document chunks and returns an answer grounded in and cited to those chunks (document name + chunk index) — never a claim outside the retrieved text. Questions outside these three documents (e.g. budget forecasts, general company culture, anything not covered by HR leave, IT acceptable use, or Finance reimbursement) return a refusal, not a guess."

Does it state the document scope explicitly?

Yes — names all three documents by their actual policy names, and states the refusal condition explicitly rather than leaving it implicit.

Run result: python3 test_client.py --run-all

All 5 checks passed:

• tools/list → 1 tool returned, ✅ description mentions scope
• "Who approves leave without pay?" → ✅ PASS, got an answer (both approvers cited)
• Cross-doc test (personal phone) → ✅ PASS, got an answer (IT-only, no blending)
• "What is the budget forecast for 2025?" → ✅ PASS, correctly refused (isError: true)
• Unknown method → ✅ PASS, JSON-RPC error -32601 received

Did the budget forecast question return isError: true?

Yes.

In one sentence — why is the tool description the enforcement?

An agent decides whether and how to call the tool purely from reading the description before any call happens, so a vague description ("answers questions about policies") is a scope agents will exceed, exactly like an unstated RICE enforcement rule.

Your commit message for UC-MCP:

Combined into the single session-submission commit — see "UC-0A UC-RAG UC-MCP Session submission — Rahul Kini" (full message in the commit body covers UC-MCP's server + tool description work specifically).

Verification checkpoints:

• Tool description explicitly states document scope (which policies are covered)
• Tool description states refusal behavior for out-of-scope queries
• python3 test_client.py --run-all executes without connection error
• Budget forecast question returns isError: true (out of scope)

---

CRAFT Reflection

Which step of the CRAFT loop was hardest across all three UCs?

Fix, specifically the part of Fix that isn't about the enforcement rule's wording but about whether the number in it is actually true for the real system. UC-RAG's 0.6 similarity threshold read as perfectly reasonable until it was run against real embeddings and real chunks — the fix wasn't "write a stricter rule," it was noticing the rule as written couldn't be satisfied by the actual tool it was governing, and recalibrating against evidence instead of assuming the documented number was correct.

What did you add to agents.md manually that the AI did not generate?

The threshold recalibration note in uc-rag/agents.md — the measured similarity ranges (0.35–0.39 for correct matches, 0.17–0.19 for out-of-scope ones) and the reasoning for landing on 0.25. That's not something a first-draft agents.md would contain at all; it only exists because the reference-verification table was actually run and the numbers were actually checked against it.

One specific task in your real work where you will use R.I.C.E in the next 7 days:

Setting up a retrieval-grounded internal Q&A tool at work — I'll run the naive full-context version first specifically to catch a threshold or scope assumption like this one before it ships, rather than trusting a plausible-sounding default.

---

Reviewer Notes (tutor fills this section)

Criterion	Score /4	Notes
RICE prompt quality
agents.md quality
skills.md quality
CRAFT loop evidence
Test coverage
Total	/20

Badge decision:

• Standard badge — meets pass threshold (score 11+/20 on this review, full rubric 22+/40)
• Distinction badge — meets distinction threshold (score 17+/20 on this review, full rubric 34+/40)
• Not yet — resubmit after addressing: _______________

[github-actions]

Hi there, participant! Thanks for joining our RAG-to-MCP Workshop!

We're reviewing your PR for the 3 Use Cases (UC-0A, UC-RAG, UC-MCP). Once your submission is validated and merged, you'll be awarded your completion badge!

Next Steps:

• Make sure all 3 UCs are finished.
• Ensure your commit messages match the required format.
• Fill out every section of the PR template.
• Good luck!