IRMA: Integrated Administrative & Welfare Guidance Multi-Agent System

Team: Yonsei University Data Science Lab (DSL) · NLP2 Modeling Team

Affiliation: Yonsei University Data Science Lab (DSL)

"One question, the right public service."

IRMA (Integrated Response for Multi-domain Administration) is a multi-agent LLM system that guides citizens through complaint procedures (민원) and subsidy / welfare programs (지원·복지) using a unified conversational interface.

By combining Supervisor → Knowledge → Action agent orchestration with Supabase semantic + text-index retrieval, phrase-first complaint ranking, and a FastAPI web UI, IRMA turns natural-language questions into structured service lookup and actionable answers grounded in a curated public-service database.

---

Project Overview

Item	Description
Title	IRMA — Integrated Administrative & Welfare Guidance
Team	DSL NLP2 Modeling Team (Yonsei University)
Objective	Answer citizen questions about complaints and subsidies with accurate, grounded multi-turn guidance
Core Components	Supervisor Agent · Knowledge Agent · Action Agent · NLPMappingTool (Supabase) · FastAPI Web UI
Data Sources	Complaint services (민원) · Subsidy / welfare programs (지원·복지)
LLM	OpenAI GPT-4.1-mini (parsing & answer generation)
Embeddings	OpenAI text-embedding-3-small (pgvector semantic search)

---

Pipeline Overview

1️⃣ Supervisor Agent — Intent Parsing & Clarification

• Parses each user utterance into a structured ParsedUserRequest (JSON).
• Classifies data source (COMPLAINT vs SUBSIDY), intent (lookup, eligibility, documents, fees, etc.), region, and user profile.
• Detects missing fields and issues follow-up questions when the query is ambiguous.
• Handles multi-turn context: service selection from a prior candidate list, keyword carry-over, and persona-aware query refinement.

📁 Key files

• agents/supervisor_agent.py
• config/prompts.py

The Supervisor acts as the router and dialog manager, deciding whether to clarify, search, or finalize.

---

2️⃣ Knowledge Agent — Database Retrieval

• Maps ParsedUserRequest to Supabase records via NLPMappingTool.
• Subsidy (SUBSIDY): semantic search (match_services RPC) + region-aware re-ranking.
• Complaint (COMPLAINT): phrase-first text-index search with tuned integer ranking weights.
• Returns an EvidencePack — ranked service candidates with match explanations and full detail fields (documents, application channels, etc.).

📁 Key files

• agents/knowledge_agent.py
• tools/nlp_mapping_tool.py
• supabase/match_services.sql

Knowledge retrieval is the evidence layer — every answer is anchored to concrete service records, not free-form hallucination.

---

3️⃣ Action Agent — Grounded Answer Generation

• Consumes ParsedUserRequest + EvidencePack and produces the final ActionResult.
• Multiple candidates (≥2): presents a numbered list and asks the user to choose.
• Single candidate / detail mode: cites DB fields (required documents, channels, eligibility) in natural language.
• Optionally enriches answers with Brave Search snippets when external context is needed (tools/external_search_tool.py).

📁 Key files

• agents/action_agent.py
• config/prompts.py

The Action Agent is the presentation layer — it converts structured evidence into citizen-friendly guidance.

---

4️⃣ Web Interface — FastAPI Chat UI

• Session-based chat API (POST /api/v1/chat) with cookie-backed session IDs.
• Interactive multi-candidate selection buttons when the pipeline returns multiple services.
• Optional Redis backend for persistent sessions across processes.
• Local launch: python -m web → http://127.0.0.1:8080

📁 Key files

• web/app.py, web/templates/index.html, web/static/

---

Environment Setup

Requirements

• Python 3.12+ recommended
• Supabase project with complaint & subsidy tables + match_services RPC
• OpenAI API key

Install & Run

cd IRMA_mk3/IRMA_mk2
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env               # fill in API keys
python -m web                      # web UI at http://127.0.0.1:8080

Terminal (CLI) mode:

python main.py

Environment Variables (.env)

Variable	Required	Description
OPENAI_API_KEY	Yes	LLM & embedding calls
SUPABASE_URL	Yes	Supabase project URL
SUPABASE_ANON_KEY or SUPABASE_SERVICE_ROLE_KEY	Yes	PostgREST access
BRAVE_API_KEY	No	External search fallback
REDIS_URL	No	Persistent session store (default: in-memory)

See .env.example for the full template. Never commit .env with real keys.

---

Demo Video

Final web UI demonstration: IRMA testing.mp4 (Google Drive)

---

Output (Evaluation)

IRMA was evaluated on curated gold sets covering complaint and subsidy queries with labeled service_id, data_source, intent, and follow-up expectations.

NLP Ranking Tuning (Knowledge retrieval)

Grid search over re-ranking integer weights on a 300-query evaluation set:

Metric	Best tuned result
Hit@1	50.7%
Hit@5	73.0%
Mean Top-1 F1	50.7%
Mean Top-k F1	25.8%

Key tuning insights:

• Phrase-first gating for complaint text-index search stabilizes ranking.
• Region-aware weights (region_metro_hit, region_district_hit) improve subsidy retrieval.
• Special-condition bonuses (military, youth) reduce false negatives on targeted programs.

End-to-End Pipeline Evaluation

Full Supervisor → Knowledge → Action runs were scored on:

Dimension	What was measured
Parsing accuracy	data_source, intent, follow-up need vs. gold labels
Retrieval accuracy	Gold service_id in top-k evidence
Dialog behavior	Correct follow-up questions vs. premature answers
Answer quality	Grounded citation of DB fields in final responses

Batch evaluation was performed via main.py --script with --trace-json for per-turn traces; summaries are generated by evaluation/trace_summary.py.

---

Repository Structure

IRMA_mk3/IRMA_mk2/
├── main.py                     # CLI & script-mode entry point
├── requirements.txt
├── .env.example                # Environment template (no secrets)
│
├── agents/                     # Multi-agent pipeline
│   ├── supervisor_agent.py     #   1. Parse & clarify
│   ├── knowledge_agent.py      #   2. Retrieve from DB
│   └── action_agent.py         #   3. Generate answer
│
├── tools/                      # Retrieval & schemas
│   ├── nlp_mapping_tool.py     #   Supabase search + re-ranking
│   ├── external_search_tool.py #   Brave Search (optional)
│   └── schemas.py              #   Pydantic models
│
├── infra/                      # Orchestration & state
│   ├── manager.py              #   run_turn() pipeline
│   ├── state_store.py          #   In-memory session/case store
│   ├── redis_state_store.py    #   Redis session store (optional)
│   ├── pii_masker.py           #   Phone-number masking
│   └── logging_tool.py         #   Structured JSON logs
│
├── config/
│   ├── prompts.py              #   Agent system prompts
│   └── settings.yaml           #   Model name, top_k, etc.
│
├── web/                        # FastAPI web UI
│   ├── app.py
│   ├── templates/index.html
│   └── static/                 #   Logos, avatars, OG image
│
├── supabase/
│   ├── match_services.sql      #   pgvector RPC definition
│   └── README.md               #   DB setup guide
│
└── evaluation/                 # Trace summary utilities (script mode)
    ├── trace_summary.py
    └── results_layout.py

---

Security & Privacy Notes

• No API keys or credentials are included in this repository.
• .env is listed in .gitignore; only .env.example with placeholders is shipped.
• User phone numbers are masked via infra/pii_masker.py before logging.
• Supabase credentials must be supplied by the evaluator at runtime.

---

Disclaimer

This project was conducted for academic and non-commercial purposes under the Yonsei University Data Science Lab (DSL). Public-service data is used strictly for research and demonstration. Generated guidance is informational only and does not constitute official government advice.

---

References

Frameworks & Libraries

• LangChain — langchain-ai/langchain
• FastAPI — tiangolo/fastapi
• OpenAI Python SDK — openai/openai-python
• Supabase PostgREST — supabase/postgrest-py

Models

• GPT-4.1-mini — OpenAI chat model for agent parsing & generation
• text-embedding-3-small — OpenAI embedding model for semantic service search

Search & Infrastructure

• pgvector — Vector similarity search in PostgreSQL
• Brave Search API — Optional external web search enrichment
• Redis — Optional session persistence for web deployment

Related Documentation

• supabase/README.md — Database schema and RPC setup
• web/app.py module docstring — Web server configuration options

---

Korean version: README_ko.md