feat: 857 model2model communication #859

5piotr · 2025-11-17T10:54:54Z

Adding functionality to test agentic models with simulated human interaction

Agent that is being tested is specializing in hotel reservations. It is using hotel reservation api mock up. Human interaction is being simulated with second agent that has clearly defined goal/list of objectives. Evaluation is being done with third agent thats soul purpose is to decide if goal/objective is fulfilled

github-actions · 2025-11-18T14:05:34Z

Code Coverage Summary

Filename                                                                                                         Stmts    Miss  Cover    Missing
-------------------------------------------------------------------------------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
packages/ragbits-agents/src/ragbits/agents/__init__.py                                                               5       0  100.00%
packages/ragbits-agents/src/ragbits/agents/_main.py                                                                485     106  78.14%   61-63, 164, 170, 174-176, 179-180, 183-187, 190-191, 227, 274-277, 290-291, 358-359, 361, 477, 499, 662, 691, 704-722, 736-740, 745, 747, 749, 764-765, 777-782, 832, 836, 867-871, 883-888, 932, 947-948, 969-1002, 1021-1054, 1072, 1078, 1083, 1123, 1126, 1130
packages/ragbits-agents/src/ragbits/agents/cli.py                                                                  197     179  9.14%    32-70, 83-106, 117-409, 427-437, 450-483
packages/ragbits-agents/src/ragbits/agents/exceptions.py                                                            48      14  70.83%   40-42, 51-52, 86-89, 100-108
packages/ragbits-agents/src/ragbits/agents/tool.py                                                                  79      32  59.49%   15, 100, 125-169
packages/ragbits-agents/src/ragbits/agents/types.py                                                                 15       0  100.00%
packages/ragbits-agents/src/ragbits/agents/mcp/__init__.py                                                           2       0  100.00%
packages/ragbits-agents/src/ragbits/agents/mcp/server.py                                                           143      14  90.21%   174, 183-184, 281, 332-335, 349, 361, 417-420, 434, 447
packages/ragbits-agents/src/ragbits/agents/mcp/utils.py                                                             13       0  100.00%
packages/ragbits-agents/src/ragbits/agents/post_processors/__init__.py                                               3       0  100.00%
packages/ragbits-agents/src/ragbits/agents/post_processors/base.py                                                  58       3  94.83%   12, 119, 124
packages/ragbits-agents/src/ragbits/agents/post_processors/exceptions.py                                            11       0  100.00%
packages/ragbits-agents/src/ragbits/agents/post_processors/supervisor.py                                            89       7  92.13%   15, 32, 179-181, 206-207
packages/ragbits-agents/src/ragbits/agents/tools/__init__.py                                                         4       0  100.00%
packages/ragbits-agents/src/ragbits/agents/tools/memory.py                                                          66       0  100.00%
packages/ragbits-agents/src/ragbits/agents/tools/openai.py                                                          47      10  78.72%   27-29, 46-50, 67-69, 92
packages/ragbits-agents/src/ragbits/agents/tools/todo.py                                                           225     164  27.11%   71-73, 77, 81-92, 101-105, 114-119, 123-140, 150-158, 174, 185-265, 273-293, 298-325, 331-384, 391-428, 433-465, 470-486
packages/ragbits-agents/src/ragbits/agents/tools/types.py                                                            6       0  100.00%
packages/ragbits-agents/tests/unit/test_agent.py                                                                   367       0  100.00%
packages/ragbits-agents/tests/unit/mcp/helpers.py                                                                   36       3  91.67%   21, 26, 61
packages/ragbits-agents/tests/unit/mcp/test_caching.py                                                              21       0  100.00%
packages/ragbits-agents/tests/unit/mcp/test_connect_disconnect.py                                                   28       0  100.00%
packages/ragbits-agents/tests/unit/mcp/test_exceptions.py                                                           25       1  96.00%   20
packages/ragbits-agents/tests/unit/mcp/test_mcp_utils.py                                                            41       0  100.00%
packages/ragbits-agents/tests/unit/post_processors/test_base_post_processors.py                                    101       0  100.00%
packages/ragbits-agents/tests/unit/post_processors/test_supervisor.py                                              131       0  100.00%
packages/ragbits-agents/tests/unit/tools/test_memory.py                                                             94       0  100.00%
packages/ragbits-agents/tests/unit/tools/test_openai.py                                                             65       0  100.00%
packages/ragbits-chat/src/ragbits/chat/__init__.py                                                                   4       0  100.00%
packages/ragbits-chat/src/ragbits/chat/_utils.py                                                                    23       5  78.26%   17, 32, 38-40
packages/ragbits-chat/src/ragbits/chat/api.py                                                                      321     119  62.93%   87-88, 111, 152, 159, 165, 198-212, 223, 235, 294-302, 336, 341-342, 388-407, 413-497, 502, 529-531, 544, 548, 576, 584-587, 620-635, 677-697, 708-761, 766-778
packages/ragbits-chat/src/ragbits/chat/cli.py                                                                        9       2  77.78%   44-52
packages/ragbits-chat/src/ragbits/chat/metrics.py                                                                   44       0  100.00%
packages/ragbits-chat/src/ragbits/chat/auth/__init__.py                                                              4       0  100.00%
packages/ragbits-chat/src/ragbits/chat/auth/backends.py                                                             66       1  98.48%   104
packages/ragbits-chat/src/ragbits/chat/auth/base.py                                                                 31       4  87.10%   47, 60, 74, 87
packages/ragbits-chat/src/ragbits/chat/auth/types.py                                                                36       0  100.00%
packages/ragbits-chat/src/ragbits/chat/client/__init__.py                                                            4       0  100.00%
packages/ragbits-chat/src/ragbits/chat/client/client.py                                                             46      21  54.35%   29-30, 34, 38, 47-48, 57-59, 63, 72, 90-91, 95, 99, 108-109, 118-119, 123, 132
packages/ragbits-chat/src/ragbits/chat/client/conversation.py                                                      136      13  90.44%   65, 67, 69, 83-84, 92, 95, 98-99, 121, 200, 203, 230
packages/ragbits-chat/src/ragbits/chat/client/exceptions.py                                                          4       0  100.00%
packages/ragbits-chat/src/ragbits/chat/history/__init__.py                                                           0       0  100.00%
packages/ragbits-chat/src/ragbits/chat/history/compressors/__init__.py                                               3       0  100.00%
packages/ragbits-chat/src/ragbits/chat/history/compressors/base.py                                                  10       0  100.00%
packages/ragbits-chat/src/ragbits/chat/history/compressors/llm.py                                                   29       1  96.55%   79
packages/ragbits-chat/src/ragbits/chat/interface/__init__.py                                                         2       0  100.00%
packages/ragbits-chat/src/ragbits/chat/interface/_interface.py                                                     138      19  86.23%   111, 163-172, 186, 240-241, 248, 257, 262, 267, 271, 279, 331, 369-370, 387-395, 402
packages/ragbits-chat/src/ragbits/chat/interface/forms.py                                                           50      13  74.00%   59, 64, 79-98, 117
packages/ragbits-chat/src/ragbits/chat/interface/summary.py                                                         50      28  44.00%   18, 27-28, 32-35, 50-52, 56-66, 73-74, 78-82
packages/ragbits-chat/src/ragbits/chat/interface/types.py                                                          230      53  76.96%   122, 132, 146, 169, 212, 221, 230, 237, 246, 425-434, 456-465, 487-496, 508-517, 539-548, 570-579, 591-600, 612-621, 633-642, 654-663, 675-684
packages/ragbits-chat/src/ragbits/chat/interface/ui_customization.py                                                19       0  100.00%
packages/ragbits-chat/src/ragbits/chat/persistence/__init__.py                                                       2       0  100.00%
packages/ragbits-chat/src/ragbits/chat/persistence/base.py                                                           7       1  85.71%   29
packages/ragbits-chat/src/ragbits/chat/persistence/sql.py                                                           92       3  96.74%   295-297
packages/ragbits-chat/tests/unit/test_api.py                                                                       184       1  99.46%   264
packages/ragbits-chat/tests/unit/test_chat_client.py                                                               105       2  98.10%   67, 87
packages/ragbits-chat/tests/unit/test_conversation.py                                                              122       1  99.18%   64
packages/ragbits-chat/tests/unit/test_generic_custom_response.py                                                   192       6  96.88%   132, 165, 184, 199, 227, 252
packages/ragbits-chat/tests/unit/auth/test_list_auth_backend.py                                                    283       0  100.00%
packages/ragbits-chat/tests/unit/history/test_llm_compressor.py                                                     64       0  100.00%
packages/ragbits-chat/tests/unit/persistence/test_sql.py                                                            74       0  100.00%
packages/ragbits-cli/src/ragbits/cli/__init__.py                                                                    35       4  88.57%   80-81, 88-89
packages/ragbits-cli/src/ragbits/cli/_utils.py                                                                      23       4  82.61%   47, 65-67
packages/ragbits-cli/src/ragbits/cli/state.py                                                                       79       3  96.20%   50-51, 61
packages/ragbits-cli/tests/unit/test_state.py                                                                       72       2  97.22%   103-104
packages/ragbits-cli/tests/unit/test_utils.py                                                                       23       0  100.00%
packages/ragbits-core/src/ragbits/core/__init__.py                                                                  16       4  75.00%   20-21, 25-26
packages/ragbits-core/src/ragbits/core/cli.py                                                                        6       0  100.00%
packages/ragbits-core/src/ragbits/core/options.py                                                                   17       0  100.00%
packages/ragbits-core/src/ragbits/core/types.py                                                                      9       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/__init__.py                                                             5       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/metrics/__init__.py                                                    30      14  53.33%   39-56, 64
packages/ragbits-core/src/ragbits/core/audit/metrics/base.py                                                        49       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/traces/__init__.py                                                     80       9  88.75%   49-52, 55-58, 66-69
packages/ragbits-core/src/ragbits/core/audit/traces/base.py                                                        187      60  67.91%   156-165, 178-179, 200, 215-216, 220, 230, 233-234, 249, 256, 258-260, 266-268, 275-278, 338-349, 356-364, 378, 394-413
packages/ragbits-core/src/ragbits/core/audit/traces/cli.py                                                         133      29  78.20%   89-94, 113-140, 157, 164, 173-174, 177-178
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                        4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                           32       5  84.38%   20-21, 24, 77, 92
packages/ragbits-core/src/ragbits/core/embeddings/exceptions.py                                                     17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/embeddings/dense/__init__.py                                                  4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/dense/base.py                                                      9       1  88.89%   44
packages/ragbits-core/src/ragbits/core/embeddings/dense/fastembed.py                                                35       3  91.43%   34, 62-63
packages/ragbits-core/src/ragbits/core/embeddings/dense/litellm.py                                                  58      12  79.31%   19, 134-139, 142, 146-148, 169
packages/ragbits-core/src/ragbits/core/embeddings/dense/local.py                                                    32       5  84.38%   13-14, 52, 68-69
packages/ragbits-core/src/ragbits/core/embeddings/dense/noop.py                                                     32       1  96.88%   99
packages/ragbits-core/src/ragbits/core/embeddings/dense/vertex_multimodal.py                                        60      24  60.00%   13-14, 57, 62, 102-123, 139-148, 175, 194-198
packages/ragbits-core/src/ragbits/core/embeddings/sparse/__init__.py                                                 4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/sparse/bag_of_tokens.py                                           43       1  97.67%   53
packages/ragbits-core/src/ragbits/core/embeddings/sparse/base.py                                                    12       1  91.67%   48
packages/ragbits-core/src/ragbits/core/embeddings/sparse/fastembed.py                                               31       2  93.55%   25, 52
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                              4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                                257      22  91.44%   161-168, 171-179, 186-190, 248-251, 282, 313, 494
packages/ragbits-core/src/ragbits/core/llms/exceptions.py                                                           29       6  79.31%   17, 26-27, 36, 45, 63
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                              12       2  83.33%   30, 51
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                             237     107  54.85%   28, 140, 158-159, 196, 225, 247, 291-406, 460, 464-469, 498-507, 517-565, 575, 604
packages/ragbits-core/src/ragbits/core/llms/local.py                                                               111      37  66.67%   14, 69, 79-80, 94-95, 101, 107, 119-120, 212-279, 294-295
packages/ragbits-core/src/ragbits/core/llms/mock.py                                                                 50       2  96.00%   126, 130
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                            2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/_cli.py                                                               53      22  58.49%   37-45, 59-61, 69-80, 88-90, 102-110
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                               45       1  97.78%   26
packages/ragbits-core/src/ragbits/core/prompt/discovery.py                                                          36       2  94.44%   55-56
packages/ragbits-core/src/ragbits/core/prompt/exceptions.py                                                         13       1  92.31%   17
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                            35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                            189       7  96.30%   105-107, 178, 181, 257, 361
packages/ragbits-core/src/ragbits/core/sources/__init__.py                                                          10       0  100.00%
packages/ragbits-core/src/ragbits/core/sources/azure.py                                                             95      13  86.32%   65-66, 92-102, 189-190
packages/ragbits-core/src/ragbits/core/sources/base.py                                                              74       3  95.95%   46, 185-186
packages/ragbits-core/src/ragbits/core/sources/exceptions.py                                                        16       0  100.00%
packages/ragbits-core/src/ragbits/core/sources/gcs.py                                                               63       0  100.00%
packages/ragbits-core/src/ragbits/core/sources/git.py                                                               94       3  96.81%   188, 195, 211
packages/ragbits-core/src/ragbits/core/sources/google_drive.py                                                     285     104  63.51%   110, 136, 159, 165-177, 187, 198-217, 276-278, 306-321, 364-365, 374-379, 392-404, 434-436, 444-454, 460-473, 490-509, 536, 541, 545-552, 555-558, 575-576, 589-593
packages/ragbits-core/src/ragbits/core/sources/hf.py                                                                75      16  78.67%   14-16, 61, 63-66, 86-87, 124-128, 156, 163-164
packages/ragbits-core/src/ragbits/core/sources/local.py                                                             41       2  95.12%   39, 80
packages/ragbits-core/src/ragbits/core/sources/s3.py                                                               105      17  83.81%   54-57, 75, 88-93, 117, 128-131, 162, 179
packages/ragbits-core/src/ragbits/core/sources/web.py                                                               41       2  95.12%   58, 75
packages/ragbits-core/src/ragbits/core/utils/__init__.py                                                             2       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                          38       1  97.37%   113
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                     79       9  88.61%   17, 55-56, 63-64, 133, 163-165
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                          29       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/dict_transformations.py                                               143      35  75.52%   24, 27, 80, 90, 110-115, 126-133, 147-151, 166-167, 173, 185-191, 195, 254
packages/ragbits-core/src/ragbits/core/utils/function_schema.py                                                     90      19  78.89%   105, 113-127, 134-147, 160, 205, 210, 213-215
packages/ragbits-core/src/ragbits/core/utils/helpers.py                                                             11       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/lazy_litellm.py                                                        30       1  96.67%   38
packages/ragbits-core/src/ragbits/core/utils/pydantic.py                                                            13       2  84.62%   13, 16
packages/ragbits-core/src/ragbits/core/utils/secrets.py                                                             18       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                     3       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/_cli.py                                                        50       4  92.00%   67, 89, 95, 119
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                       103       3  97.09%   53, 214, 286
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                      91       2  97.80%   74, 112
packages/ragbits-core/src/ragbits/core/vector_stores/hybrid.py                                                      34       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/hybrid_strategies.py                                           65       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                                   59       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py                                                   190      15  92.11%   97, 106-109, 125, 168, 312-313, 338-340, 373-375
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                                                      97       5  94.85%   80-95, 160, 181
packages/ragbits-core/src/ragbits/core/vector_stores/weaviate.py                                                   127       5  96.06%   104-132, 271
packages/ragbits-core/tests/conftest.py                                                                             12       0  100.00%
packages/ragbits-core/tests/cli/__init__.py                                                                          0       0  100.00%
packages/ragbits-core/tests/cli/test_cli_trace_handler.py                                                           47       3  93.62%   29, 42, 55
packages/ragbits-core/tests/cli/test_vector_store.py                                                               115       0  100.00%
packages/ragbits-core/tests/integration/sources/test_git.py                                                         68       6  91.18%   147-156
packages/ragbits-core/tests/integration/sources/test_hf.py                                                          19       9  52.63%   16-21, 32-37
packages/ragbits-core/tests/integration/sources/test_s3.py                                                          42       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/__init__.py                                                    0       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/test_keyword_search.py                                        79       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py                                         140       1  99.29%   51
packages/ragbits-core/tests/integration/vector_stores/test_vector_store_sparse.py                                   63       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                         0       0  100.00%
packages/ragbits-core/tests/unit/test_options.py                                                                    21       0  100.00%
packages/ragbits-core/tests/unit/audit/test_cli.py                                                                 107       0  100.00%
packages/ragbits-core/tests/unit/audit/test_metrics.py                                                              35       7  80.00%   14-19, 23
packages/ragbits-core/tests/unit/audit/test_trace.py                                                                98       3  96.94%   17, 20, 23
packages/ragbits-core/tests/unit/embeddings/test_bag_of_tokens.py                                                   52       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_fastembed.py                                                       50       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_from_config.py                                                     39       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_litellm.py                                                         84       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_local.py                                                           42       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_noop.py                                                            26       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_vector_size.py                                                     33       0  100.00%
packages/ragbits-core/tests/unit/embeddings/test_vertex_multimodal.py                                               39       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                    0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_base.py                                                                 186       3  98.39%   77-80
packages/ragbits-core/tests/unit/llms/test_from_config.py                                                           27       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                              216       3  98.61%   170-173
packages/ragbits-core/tests/unit/llms/test_local.py                                                                 74       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                            0       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_preferred_llm.py                                             12       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                                 0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                            65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                            334       1  99.70%   777
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                       0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                      30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                         18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                        2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py                3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py           14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py           14       0  100.00%
packages/ragbits-core/tests/unit/sources/test_aws.py                                                                23       0  100.00%
packages/ragbits-core/tests/unit/sources/test_azure.py                                                              70       0  100.00%
packages/ragbits-core/tests/unit/sources/test_exceptions.py                                                         22       0  100.00%
packages/ragbits-core/tests/unit/sources/test_gcs.py                                                                33       6  81.82%   42-47
packages/ragbits-core/tests/unit/sources/test_git.py                                                               110       0  100.00%
packages/ragbits-core/tests/unit/sources/test_google_drive.py                                                      135      25  81.48%   27-32, 50, 62, 69, 84-95, 184, 197-198, 209-220
packages/ragbits-core/tests/unit/sources/test_hf.py                                                                 12       0  100.00%
packages/ragbits-core/tests/unit/sources/test_local.py                                                              13       0  100.00%
packages/ragbits-core/tests/unit/sources/test_source_discriminator.py                                               36       0  100.00%
packages/ragbits-core/tests/unit/sources/test_web.py                                                                43       0  100.00%
packages/ragbits-core/tests/unit/utils/__init__.py                                                                   0       0  100.00%
packages/ragbits-core/tests/unit/utils/test_config_handling.py                                                      76       2  97.37%   27-28
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                           26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/test_dict_transformations.py                                                 98       0  100.00%
packages/ragbits-core/tests/unit/utils/test_function_schema.py                                                      16       2  87.50%   19, 32
packages/ragbits-core/tests/unit/utils/test_helpers.py                                                               6       0  100.00%
packages/ragbits-core/tests/unit/utils/test_secrets.py                                                              24       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                       13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                                  9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                                37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_base.py                                                          6       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                       81       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_from_config.py                                                  55       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_hybrid.py                                                       74       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_hybrid_strategies.py                                            31       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                   102       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_pgvector.py                                                    262       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                                                      100       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_weaviate.py                                                    142       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                             2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                               91       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/cli.py                                                 40       2  95.00%   86, 105
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                                   0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                                  78       2  97.44%   49, 93
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                                   86      14  83.72%   97, 115, 179-187, 197, 206-208
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                                   0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/enrichers/__init__.py                         4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/enrichers/base.py                            21       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/enrichers/exceptions.py                      14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/enrichers/image.py                           30       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/enrichers/router.py                          25       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/__init__.py                           3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/base.py                              28       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/docling.py                           44       2  95.45%   90, 151
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/exceptions.py                        14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/router.py                            27       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/unstructured.py                      66      24  63.64%   102, 121-123, 135-156, 176-190, 212-213, 233-248
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/__init__.py                      8       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/callbacks.py                    10       1  90.00%   32
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/exceptions.py                   16      10  37.50%   25-33, 49-52
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/hyperlink_callback.py           38      12  68.42%   44-69, 72, 81, 84
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/metadata_callback.py            29       9  68.97%   52-71, 74
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/parser.py                       43       6  86.05%   60-62, 71-73
packages/ragbits-document-search/src/ragbits/document_search/ingestion/parsers/pptx/speaker_notes_callback.py       31      13  58.06%   41-68, 71
packages/ragbits-document-search/src/ragbits/document_search/ingestion/strategies/__init__.py                        5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/strategies/base.py                          102      18  82.35%   156, 212-242, 284
packages/ragbits-document-search/src/ragbits/document_search/ingestion/strategies/batched.py                        69       8  88.41%   172, 200-215, 255-256
packages/ragbits-document-search/src/ragbits/document_search/ingestion/strategies/ray.py                            32       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/strategies/sequential.py                      4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                                   0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                        4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                           14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                            40       5  87.50%   51, 115-118
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                            8       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                         3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/answerai.py                        29       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                            19       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                         27       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/llm.py                             59       1  98.31%   173
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                            10       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/rrf.py                             28       2  92.86%   50, 60
packages/ragbits-document-search/tests/cli/custom_cli_source.py                                                     22       1  95.45%   32
packages/ragbits-document-search/tests/cli/test_ingest.py                                                           56       0  100.00%
packages/ragbits-document-search/tests/cli/test_search.py                                                           71       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                       0       0  100.00%
packages/ragbits-document-search/tests/integration/test_docling.py                                                  10       0  100.00%
packages/ragbits-document-search/tests/integration/test_pptx_parser.py                                              54       9  83.33%   32-34, 52, 71, 74-75, 78-79
packages/ragbits-document-search/tests/integration/test_rerankers.py                                                32       9  71.88%   32-39, 59-64
packages/ragbits-document-search/tests/integration/test_unstructured.py                                             12       4  66.67%   62-67
packages/ragbits-document-search/tests/unit/test_config.py                                                          63       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_parser_router.py                                          24       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_parsers.py                                                47       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                                238       1  99.58%   480
packages/ragbits-document-search/tests/unit/test_document_search_ingest_errors.py                                   38       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                                       13       0  100.00%
packages/ragbits-document-search/tests/unit/test_element_enricher_router.py                                         23       0  100.00%
packages/ragbits-document-search/tests/unit/test_element_enrichers.py                                               56       1  98.21%   25
packages/ragbits-document-search/tests/unit/test_elements.py                                                        21       0  100.00%
packages/ragbits-document-search/tests/unit/test_ingest_strategies.py                                               43       0  100.00%
packages/ragbits-document-search/tests/unit/test_llm_reranker.py                                                    43       0  100.00%
packages/ragbits-document-search/tests/unit/test_rephrasers.py                                                      26       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                       80       1  98.75%   25
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/__init__.py                   0       0  100.00%
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/factories.py                 22       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/__init__.py                                                           0       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/cli.py                                                               46       3  93.48%   133, 135, 137
packages/ragbits-evaluate/src/ragbits/evaluate/evaluator.py                                                         92       1  98.91%   221
packages/ragbits-evaluate/src/ragbits/evaluate/optimizer.py                                                         92      18  80.43%   162-168, 187, 190-191, 194, 198-204, 207-210
packages/ragbits-evaluate/src/ragbits/evaluate/utils.py                                                             58      37  36.21%   31-50, 62-69, 98-101, 117-129, 140-149, 159-160
packages/ragbits-evaluate/src/ragbits/evaluate/dataloaders/__init__.py                                               2       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/dataloaders/base.py                                                  34       4  88.24%   58-60, 79
packages/ragbits-evaluate/src/ragbits/evaluate/dataloaders/document_search.py                                       13       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/dataloaders/exceptions.py                                            10       5  50.00%   10-12, 21-25
packages/ragbits-evaluate/src/ragbits/evaluate/metrics/__init__.py                                                   2       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/metrics/base.py                                                      27       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py                                           23       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/__init__.py                                                14       1  92.86%   37
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/base.py                                                    24       0  100.00%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/document_search.py                                         38       6  84.21%   68-71, 80-84
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/gaia.py                                                   148     102  31.08%   61-66, 73-78, 83-90, 95-107, 113-172, 176-180, 184-207, 211-217, 221-232, 237, 243-249
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/hotpot_qa.py                                              198     150  24.24%   72-82, 89-99, 103-107, 111-141, 145-157, 161-218, 222-226, 239-254, 260-268, 272-278, 282-293, 297-308, 313, 317-342
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/human_eval.py                                             193     148  23.32%   54-119, 138-145, 152-157, 163-180, 185-198, 204-260, 266-272, 276-280, 284-298, 302-308, 312-323
packages/ragbits-evaluate/tests/cli/test_run_evaluation.py                                                          25       0  100.00%
packages/ragbits-evaluate/tests/unit/test_evaluator.py                                                             103       0  100.00%
packages/ragbits-evaluate/tests/unit/test_metrics.py                                                                77       0  100.00%
packages/ragbits-evaluate/tests/unit/test_optimizer.py                                                              68       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/__init__.py                                                       0       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/base.py                                                          15       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/openai_moderation.py                                             19       6  68.42%   27-34
packages/ragbits-guardrails/tests/unit/test_openai_moderation.py                                                    35       0  100.00%
TOTAL                                                                                                            16582    2199  86.74%

Diff against main

Filename                                                                           Stmts    Miss  Cover
-------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-agents/src/ragbits/agents/__init__.py                                +2       0  +100.00%
packages/ragbits-agents/src/ragbits/agents/_main.py                                 +148     +41  -2.57%
packages/ragbits-agents/src/ragbits/agents/cli.py                                    +12     +11  -0.05%
packages/ragbits-agents/src/ragbits/agents/exceptions.py                              +4       0  +2.65%
packages/ragbits-agents/src/ragbits/agents/tool.py                                   +38     +31  -38.07%
packages/ragbits-agents/src/ragbits/agents/post_processors/__init__.py                +3       0  +100.00%
packages/ragbits-agents/src/ragbits/agents/post_processors/base.py                   +58      +3  +94.83%
packages/ragbits-agents/src/ragbits/agents/post_processors/exceptions.py             +11       0  +100.00%
packages/ragbits-agents/src/ragbits/agents/post_processors/supervisor.py             +89      +7  +92.13%
packages/ragbits-agents/src/ragbits/agents/tools/__init__.py                          +2       0  +100.00%
packages/ragbits-agents/src/ragbits/agents/tools/memory.py                           +66       0  +100.00%
packages/ragbits-agents/src/ragbits/agents/tools/todo.py                            +225    +164  +27.11%
packages/ragbits-agents/tests/unit/test_agent.py                                     +28       0  +100.00%
packages/ragbits-agents/tests/unit/post_processors/test_base_post_processors.py     +101       0  +100.00%
packages/ragbits-agents/tests/unit/post_processors/test_supervisor.py               +131       0  +100.00%
packages/ragbits-agents/tests/unit/tools/test_memory.py                              +94       0  +100.00%
packages/ragbits-chat/src/ragbits/chat/api.py                                        +58     +50  -10.83%
packages/ragbits-chat/src/ragbits/chat/interface/_interface.py                       +15      +3  -0.76%
packages/ragbits-chat/src/ragbits/chat/interface/summary.py                          +50     +28  +44.00%
packages/ragbits-chat/src/ragbits/chat/interface/types.py                           +108     +47  -18.12%
packages/ragbits-chat/src/ragbits/chat/persistence/base.py                            +1       0  +2.38%
packages/ragbits-chat/src/ragbits/chat/persistence/sql.py                             +1       0  +0.04%
packages/ragbits-chat/tests/unit/test_api.py                                          +1       0  +0.01%
packages/ragbits-chat/tests/unit/test_generic_custom_response.py                    +192      +6  +96.88%
packages/ragbits-core/src/ragbits/core/llms/litellm.py                               +50     +43  -10.93%
packages/ragbits-core/src/ragbits/core/sources/hf.py                                  +3      -1  +2.28%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/__init__.py                  +3       0  +1.95%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/gaia.py                    +148    +102  +31.08%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/hotpot_qa.py               +198    +150  +24.24%
packages/ragbits-evaluate/src/ragbits/evaluate/pipelines/human_eval.py              +193    +148  +23.32%
TOTAL                                                                              +2033    +833  -3.87%

Results for commit: 1d8c59a

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

…s/evaluate

5piotr · 2025-11-20T10:23:54Z

I will implement parallel evaluation of agent conversation using DeepEval

examples/evaluate/agent-scenarios/personalities.json

examples/evaluate/agent-scenarios/duet_cli.py

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/conversation.py

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/deepeval_evaluator.py

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/scenarios.py

examples/evaluate/agent-scenarios/scenarios.json

…date package and example

…cular imports in future

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/conversation.py

…narios" This reverts commit 3dc123c.

5piotr changed the title ~~857 model2model communication~~ feat: 857 model2model communication Nov 18, 2025

5piotr force-pushed the 857-model2model-communication branch from 76f4f59 to 575cac8 Compare November 18, 2025 10:57

5piotr and others added 10 commits November 18, 2025 12:54

added basic model2model communication with loging

5a60f48

added separate model selection for user and agent

bbb7e84

added goal checker model to evaluate answers

1a9193e

added hotel_api for h2a testing

718ac00

api mock

4748a94

removed conflicting dates from hotel_config

22bcad1

chnged h2a agent from finance to hotel

71a802d

removed goal checking from sim user

416184e

replaced single goals with multitask scenarios

a5286f8

added Scenario 4

7153a5d

5piotr force-pushed the 857-model2model-communication branch from 575cac8 to 7153a5d Compare November 18, 2025 12:54

5piotr added 2 commits November 18, 2025 14:44

restored pyproject.toml and uv.lock

49cc981

fixed linting issues

5001150

5piotr added 7 commits November 19, 2025 10:48

chnged h2a to agent_scenarios and incorporated hotel api into example…

fbc6858

…s/evaluate

fixed linting and updated readme

f0dc987

updated examples readme, deleted integration plan

f0eb605

moved scenarios to evaluate

7f94c3f

duet_cli refactoring

39af935

moved reusable code to packages

6c6d5e9

updated CHANGELOG.md

d6ee6be

5piotr added 6 commits November 20, 2025 14:11

added tool checker for tool usage veryfication

76431b8

changed tool checker from llm based to simple logic

6e7e320

fixed consol log for incomplete tasks

4655096

implemented deepeval metrics

248eb4e

added token tracking for agent

1b59577

added personalities for sim user

8390851

mkosecki reviewed Nov 25, 2025

View reviewed changes

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/scenarios.py Outdated Show resolved Hide resolved

chore: add scenario 5, book room not possible

e7b65b1

mkosecki reviewed Nov 25, 2025

View reviewed changes

examples/evaluate/agent-scenarios/scenarios.json Show resolved Hide resolved

5piotr and others added 9 commits November 25, 2025 12:29

added function for deepeval evaluation

3e2347f

refactoring deepeval_evaluator, removing futures imports

f856757

feat: integrate ChatInterface in evaluation instead of raw agent - up…

472a15f

…date package and example

fix: linter proper formatting

d919ed4

fix: mypy issues

9d947e4

fix: remove HotelChat from init, todo: refactor and fix potential cir…

fa6219a

…cular imports in future

added task status labels to allow for stopping impossible scenarios

3dc123c

added scenario 6 with specified personal data

5d79fb4

chore: refactor imports

a8d5d95

mkosecki reviewed Nov 26, 2025

View reviewed changes

packages/ragbits-evaluate/src/ragbits/evaluate/agent_simulation/conversation.py Outdated Show resolved Hide resolved

5piotr and others added 7 commits November 27, 2025 10:51

Revert "added task status labels to allow for stopping impossible sce…

fb6bee7

…narios" This reverts commit 3dc123c.

added max turns per task parameter

106bdb4

updated docs

95a57f8

updated uv.lock after deepeval

d8586cb

fixed agent_simulation.md linting

d5098e5

whitelisting deepeval dependencies

f1cd05b

Merge branch 'develop' into 857-model2model-communication

1d8c59a

5piotr marked this pull request as ready for review December 1, 2025 11:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: 857 model2model communication #859

feat: 857 model2model communication #859

Uh oh!

5piotr commented Nov 17, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 •

edited

Loading

Uh oh!

5piotr commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: 857 model2model communication #859

Are you sure you want to change the base?

feat: 857 model2model communication #859

Uh oh!

Conversation

5piotr commented Nov 17, 2025

Adding functionality to test agentic models with simulated human interaction

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Summary

Diff against main

Uh oh!

5piotr commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Nov 18, 2025 •

edited

Loading