Feat/evaluation#58
Conversation
…and related policies - Introduced new JSON datasets for residence permits, taxes, and work rights, including test cases for various personas. - Created expected documents and relevance labels for retrieval processes. - Defined domains, personas, and policies in YAML format to structure the evaluation framework. - Enhanced integration of information regarding immigration, taxation, and work rights for third-country nationals in Finland.
…g of context relevance
…data extraction - Added logging for assistant responses and queries in RAG orchestrator. - Implemented structured metadata extraction for documents in tracing utilities. - Introduced logging of retrieved document metadata and content previews in document retrieval service. - Added unit tests for metadata extraction and tracing functionalities.
|
Warning Review limit reached
Next review available in: 34 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (67)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
1 participant
Description
This PR solves #25 by implementing an evaluation harness. Introduces a comprehensive evaluation framework for Tapio’s RAG pipeline, designed to systematically measure and improve retrieval quality, answer correctness, safety, and conversational performance. The goal is to move beyond ad-hoc testing and establish a repeatable, production-grade evaluation setup that provides clear signals on system performance and regression.
Checklist
uv run pytestand all tests passuv run prek run --all-files(oruv run ruff check .anduv run ruff check . --fix) and addressed any issuesuv run mypy tapioanduv run pyrefly checkand addressed any issuesuv run pytest --cov=tapio)README.mdif this PR changes user-facing behaviorRelated issue
Additional context
At the core of this change are two complementary evaluation suites: DeepEval and Ragas. DeepEval is used for rubric-based evaluation of correctness, retrieval quality, and safety using structured scoring criteria, while Ragas focuses on retrieval and generation metrics such as faithfulness, answer relevancy, and context precision/recall. Together, they provide both qualitative and quantitative coverage of the system, ensuring that improvements (or regressions) can be detected reliably across different dimensions of the RAG pipeline.
To support this, the PR introduces structured datasets and a domain taxonomy covering key Finnish immigration topics such as Kela, DVV, residence permits, taxation, and work rights. These datasets are paired with retrieval anchors and relevance labels, enabling precise evaluation of whether the system is retrieving the correct documents. In addition, a conversational test framework has been added to evaluate multi-turn interactions, where the system must maintain context across turns and produce consistent, grounded responses.
Another major addition is LangSmith tracing, which provides visibility into the internal behavior of the RAG pipeline. Retrieval steps are now instrumented to log document metadata such as chunk IDs, source URLs, and content previews, making it easier to debug failures and understand why a particular answer was generated. This is fully integrated into the application lifecycle and can be configured via environment variables.