Add API and CLI command for updating multiple indexes at one

[ludwiktrammer]

No description provided.

[github-actions]

Trivy scanning results.

[github-actions]

Code Coverage Summary

Filename                                                       Stmts    Miss  Cover    Missing
-----------------------------------------------------------  -------  ------  -------  ---------------------------------------------
dbally/_collection.py                                             67       5  92.54%   167, 171, 184-185, 199
dbally/_main.py                                                   23       2  91.30%   39, 73
dbally/assistants/base.py                                         23       0  100.00%
dbally/assistants/openai.py                                       59       2  96.61%   59-76
dbally/audit/event_span.py                                         8       2  75.00%   12, 22
dbally/audit/event_tracker.py                                     34      11  67.65%   35, 48, 59, 69, 83-92
dbally/audit/event_handlers/base.py                               15       0  100.00%
dbally/audit/event_handlers/cli_event_handler.py                  42      27  35.71%   5-8, 24-25, 28-32, 42-44, 55-63, 75-78, 89-92
dbally/audit/event_handlers/langsmith_event_handler.py            25      21  16.00%   6-87
dbally/data_models/audit.py                                       22       0  100.00%
dbally/data_models/execution_result.py                             8       0  100.00%
dbally/data_models/llm_options.py                                 13       0  100.00%
dbally/data_models/prompts/common_validation_utils.py             15       0  100.00%
dbally/data_models/prompts/iql_explainer_prompt_template.py        8       0  100.00%
dbally/data_models/prompts/iql_prompt_template.py                 13       1  92.31%   39
dbally/data_models/prompts/nl_responder_prompt_template.py         8       0  100.00%
dbally/data_models/prompts/prompt_template.py                     28       2  92.86%   27, 35
dbally/data_models/prompts/view_selector_prompt_template.py       12       2  83.33%   33-34
dbally/embedding_client/base.py                                    5       5  0.00%    2-10
dbally/embedding_client/openai.py                                 19      19  0.00%    1-52
dbally/examples/db.py                                             77      77  0.00%    1-163
dbally/examples/recruiting.py                                     46      46  0.00%    1-128
dbally/examples/views.py                                          30      30  0.00%    1-58
dbally/iql/_exceptions.py                                         24       1  95.83%   33
dbally/iql/_processor.py                                          78       7  91.03%   15, 43, 60, 66, 72, 81, 87
dbally/iql/_query.py                                              12       1  91.67%   7
dbally/iql/_type_validators.py                                    39       2  94.87%   24, 28
dbally/iql/syntax.py                                              36       9  75.00%   6-9, 26, 34, 55, 58-61
dbally/iql_generator/iql_generator.py                             32       0  100.00%
dbally/llm_client/base.py                                         20       7  65.00%   50-69
dbally/llm_client/openai_client.py                                21       9  57.14%   19-20, 47-58
dbally/nl_responder/nl_responder.py                               28       3  89.29%   75, 82-88
dbally/nl_responder/token_counters.py                             23      10  56.52%   24-25, 53-63
dbally/prompts/prompt_builder.py                                  20       3  85.00%   6, 27-28
dbally/similarity/detector.py                                     73       6  91.78%   21, 200-204
dbally/similarity/faiss_store.py                                  36      33  8.33%    5-99
dbally/similarity/fetcher.py                                       5       0  100.00%
dbally/similarity/index.py                                        18       6  66.67%   42-43, 49-50, 62-63
dbally/similarity/sqlalchemy_base.py                              40      21  47.50%   17, 35-37, 48-50, 59, 68-71, 81-87, 105-108
dbally/similarity/store.py                                         7       0  100.00%
dbally/utils/errors.py                                             2       0  100.00%
dbally/view_selection/base.py                                      6       0  100.00%
dbally/view_selection/llm_view_selector.py                        20       0  100.00%
dbally/view_selection/random_view_selector.py                      9       9  0.00%    1-27
dbally/views/base.py                                              43       1  97.67%   27
dbally/views/decorators.py                                         6       0  100.00%
dbally/views/methods_base.py                                      33       2  93.94%   68, 76
dbally/views/pandas_base.py                                       35       1  97.14%   62
dbally/views/sqlalchemy_base.py                                   43      10  76.74%   45, 60-62, 75, 82-88
dbally_cli/main.py                                                 5       5  0.00%    1-11
dbally_cli/similarity.py                                          21       0  100.00%
TOTAL                                                           1335     398  70.19%

Diff against main

Filename                         Stmts    Miss  Cover
-----------------------------  -------  ------  --------
dbally/iql/_processor.py            -3      -2  +2.14%
dbally/similarity/detector.py      +73      +6  +91.78%
dbally/views/base.py                +7       0  +0.45%
dbally/views/methods_base.py        +1       0  +0.19%
dbally_cli/main.py                  -2      -2  +100.00%
dbally_cli/similarity.py           +21       0  +100.00%
TOTAL                              +97      +2  +2.67%

Results for commit: 114494f

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

[mhordynski]

Maybe it would be beneficial to add extra method on collection to update all indexes used inside? Something like Collection.update_indexes() which aggregates all SimilarityIndexes and executes update on them

[ludwiktrammer]

Maybe. I viewed the CLI as the primary method of updating collections, and focused less on ways of doing this in code that can't be used by CLI, but you may be right. On the other hand this seems like a separate area, so how about we open a new ticket and do this in a new PR?