Mumbai Sadaf Mirza - RAG-to-MCP Submission

[sadaf-mirza-azularc]

RAG-to-MCP — Submission PR

Name: Sadaf Mirza
City / Group: Mumbai
Date: 2026-07-17
AI tool(s) used: Claude Code (Sonnet 5)

Submission Checklist

• uc-0a/agents.md — present, updated
• uc-0a/skills.md — present, updated
• uc-0a/classifier.py — runs cleanly
• uc-0a/results_pune.csv — present. data/city-test-files/ has no Mumbai file, so Pune's test set was substituted for UC-0A (details below)
• uc-rag/agents.md — present, updated
• uc-rag/skills.md — present, updated
• uc-rag/rag_server.py — original implementation, not the stub
• uc-mcp/agents.md — present, updated
• uc-mcp/skills.md — present, updated
• uc-mcp/mcp_server.py — passes real end-to-end tests for retrieval, scoping, and refusal logic; the two "answer" tests return a live Gemini quota error rather than generated text (see UC-MCP section)
• 6 commits with meaningful messages, two per UC (minimum was 3)
• All sections below completed

City note: the participant identity for this submission is Mumbai, but data/city-test-files/ only ships test_pune.csv, test_hyderabad.csv, test_kolkata.csv, and test_ahmedabad.csv — there's no Mumbai equivalent. test_pune.csv was used as the closest available substitute for UC-0A.

Environment note: sentence-transformers, chromadb, and google-generativeai are all installed and were exercised directly — the ChromaDB index was built from the real policy documents and queried live, with real cosine-similarity scores throughout this section. The one open item is that the configured GEMINI_API_KEY returns 429 quota exceeded, limit: 0 on every generation call — a Google Cloud project/billing setting on that key, not a code or environment gap. That blocks seeing real generated answer text; it does not block retrieval, scoping, or refusal logic, all of which ran against the live index and are reported below.

UC-0A — Complaint Classifier

First failure mode encountered:

Severity blindness. A classifier that only pattern-matches the category taxonomy has no reason to escalate priority — row PM-202402 ("Deep pothole near bus stop. School children at risk during morning hours.") would land on Standard because "pothole" alone drives the category, and nothing separately checks for the child/school risk signal.

Enforcement rule that fixed it (quoted from agents.md):

"Priority must be Urgent if the description contains any of these severity keywords (case-insensitive substring match): injury, child, school, hospital, ambulance, fire, hazard, fell, collapse. This check runs independently of category and overrides any lower default priority."

UC-0A commit message:

UC-0A Fix severity blindness: no keyword enforcement in starter -> added injury/child/school/hospital/ambulance/fire/hazard/fell/collapse triggers forcing Urgent independent of category

Verification checkpoints:

• Every severity-signal row (injury/child/school/hospital-class keywords) lands on Urgent — confirmed in results_pune.csv: PM-202402 (child/school), PM-202411 (hazard), PM-202420 (injury), PM-202446 (fell).
• No category outside the fixed taxonomy appears anywhere in the output.
• Every row carries a non-empty justification.

UC-RAG — RAG Server

Failure mode encountered: (chunk boundary, plus a threshold-calibration bug found during live verification)

Chunk boundary: HR Leave Policy clause 5.2 — "LWP requires approval from the Department Head and the HR Director. Manager approval alone is not sufficient." — is exactly the two-approver clause the README calls out as at risk. A boundary drawn without sentence awareness could split it and drop one approver from what gets retrieved.

A second, larger issue surfaced once the index was actually queried live: the README's stated similarity threshold of 0.6 was never reachable by any correct match on this embedder for this corpus. Every in-scope reference query scored 0.26–0.39, while the one genuine out-of-scope query scored 0.09 — meaning the stated threshold caused the server to falsely refuse 3 of the 4 reference queries, including ones with the exact right document retrieved. This was diagnosed empirically (uc-rag/_diagnose_threshold.py) and fixed by recalibrating the threshold to 0.25, in the gap between the two score populations.

Chunking strategy and rationale:

Sentences are accumulated into a chunk until the next sentence would push it past 400 tokens, at which point a new chunk begins — so a boundary can only ever land between two sentences, never inside one. Verified against the real policy files: all three documents split into 2 chunks each, policy_hr_leave.txt's larger chunk topping out at 399 tokens, and both of clause 5.2's approvers — "Department Head" and "HR Director" — stay together in the same chunk.

Did the system correctly refuse "What is the flexible working culture?"? (expected: refusal template, since no document covers it)

Yes — verified live against the built index. No retrieved chunk cleared the threshold, so retrieve_and_answer returned the refusal template without ever calling the LLM.

Did the system retrieve the IT policy (not HR leave) for "Can I use my personal phone for work files?"?

Yes — verified live. Only policy_it_acceptable_use.txt chunks passed the threshold (scores 0.343 and 0.261); no HR leave chunk entered the passing set, so nothing from that document reached the prompt.

Enforcement rule preventing answers outside retrieved context:

The prompt sent to the LLM is built exclusively from chunks that survived the similarity filter — nothing else is ever included — and when no chunk survives, the refusal template is returned without an LLM call at all. Verified live for the refusal case above: the response came back instantly with the refusal text and no LLM was invoked.

UC-RAG commit messages:

UC-RAG Fix chunk boundary: fixed-size split risked cutting clause 5.2 mid-sentence -> sentence-aware chunking that accumulates whole sentences up to 400 tokens

UC-RAG Fix false refusal: 0.6 threshold unreachable by any correct match on all-MiniLM-L6-v2 for this corpus -> recalibrated to 0.25 from empirical score distribution

Verification checkpoints:

• 3+ test queries return correctly grounded retrieval (right documents, right threshold behavior) — verified live for all 4 reference queries. Generated answer text specifically could not be verified due to the Gemini key's zero quota (see environment note above).
• "What is the flexible working culture?" returns the refusal template — verified live.
• "Can I use my personal phone for work files?" retrieves IT policy over HR leave — verified live.
• Each document splits into more than one chunk — verified: all three policy files produce 2 chunks apiece.

UC-MCP — MCP Server

Tool description, pasted from mcp_server.py's TOOL_DEFINITION:

"Answers questions about the City Municipal Corporation's (CMC) HR Leave Policy, IT Acceptable Use Policy, and Finance Reimbursement Policy ONLY. Returns an answer grounded in retrieved document chunks, each cited by source document name and chunk index. Does NOT answer questions outside these three documents (e.g. budgets, staffing, external regulations, general knowledge) — such questions receive a refusal response with isError: true instead of a guess."

Does it state the document scope explicitly?

Yes — it names all three documents by title and gives concrete examples of what falls outside scope.

Run result: python3 test_client.py --run-all (real, against a live server):

• tools/list — ✅ scope stated in description
• In-scope: "Who approves leave without pay?" — retrieval correct (HR policy chunks, scores 0.388/0.351), isError: false, but answer text is [LLM ERROR] 429 quota exceeded, limit: 0 — the Gemini key, not the server, failed here
• Cross-document check: personal phone + work files — retrieval correct (IT policy chunks only, scores 0.343/0.261), isError: false, same Gemini quota error in the answer text
• Out-of-scope: "What is the budget forecast for 2025?" — ✅ correctly refused, isError: true, no LLM called
• Unknown method → -32601 — ✅ correct JSON-RPC error returned

One caveat on literally running the command as written: test_client.py's JSON-RPC client has a fixed 10-second timeout, and Gemini's SDK retries internally for longer than that when quota is exhausted, so a single --run-all invocation exits early on a client-side TimeoutError after test 1 rather than printing the full 5-test summary in one run. Every test above was confirmed individually against the same live server with a longer timeout to get a complete picture; the server itself answers correctly in all cases; only the LLM generation step is affected.

Did the budget-forecast question return isError: true?

Yes, and correctly for the right reason: it hit the intended "no chunk cleared the threshold" refusal path — no chromadb or import errors involved, no LLM call made.

Why the tool description functions as the enforcement, in one sentence:

An agent decides whether to invoke the tool purely from its description, before any call happens — so a vague description silently authorizes the agent to call it for anything, while one that names the exact documents and the refusal condition lets the agent rule out-of-scope questions out on its own, before spending a call on them.

UC-MCP commit messages:

UC-MCP Fix vague tool description: no scope stated -> added CMC HR/IT/Finance policy scope + explicit refusal note to TOOL_DEFINITION

UC-MCP Fix stale index path check: hardcoded stub_chroma_db path -> checks whichever backend actually imported (rag_server vs stub_rag)

UC-MCP Fix dead default model: gemini-1.5-flash retired by Google -> gemini-2.0-flash

Verification checkpoints:

• Tool description names the covered documents explicitly.
• Tool description states refusal behavior for out-of-scope questions.
• Server responds correctly to every JSON-RPC call tested — connects and answers every time; only test_client.py's own fixed timeout, not a server fault, cuts off a single-command --run-all run early under quota-induced Gemini retry latency.
• The budget-forecast question returns isError: true, via the intended threshold-refusal path.

CRAFT Reflection

Hardest CRAFT step across the three UCs:

Check. Retrieval, scoping, and refusal logic could all be verified directly against a live ChromaDB index — that surfaced the threshold-calibration bug in the first place. But confirming actual generated-answer quality (grounding, citation format, no cross-document blending in the prose itself) is still blocked by the Gemini key's zero quota, so that last piece of verification is code-reviewed rather than observed.

What was added to agents.md by hand, beyond the AI's draft:

For UC-0A, the guarantee that the severity-keyword check runs before category classification and independently of it, so an urgent signal can never be suppressed by an uncertain category. For UC-RAG, the rule that evidence from different source documents must stay grouped and separately cited rather than collapsing into one undifferentiated context block — plus, after live testing exposed it, the recalibrated 0.25 similarity threshold with the empirical reasoning behind the number.

One task in the next 7 days where R.I.C.E applies to real work:

Writing enforcement rules for an internal support-ticket triage prompt, so ambiguous tickets get routed to a human-review queue explicitly instead of the model guessing a category with unearned confidence — the same failure shape as UC-0A's severity blindness.

[github-actions]

Hi there, participant! Thanks for joining our RAG-to-MCP Workshop!

We're reviewing your PR for the 3 Use Cases (UC-0A, UC-RAG, UC-MCP). Once your submission is validated and merged, you'll be awarded your completion badge!

Next Steps:

• Make sure all 3 UCs are finished.
• Ensure your commit messages match the required format.
• Fill out every section of the PR template.
• Good luck!