Index to search #1276

sampaccoud · 2025-08-07T16:31:17Z

Purpose

We want to add fulltext (and semantic in a second phase) search to Docs.

The goal is to enable efficient and scalable search across document content by pushing relevant data to a dedicated search backend, such as OpenSearch. The backend should be pluggable.

Proposal

Add indexing logic in a search indexer that can be declared as a backend
Implement indexing for the Find backend. See corresponding PR in Find
Implement search views as a proxy
Implement triggers to update search index when a document or its accesses change. Synchronization should be done asyncrhonously as changing a document or its accesses affects all its descendants...

Fixes #322

gitguardian · 2025-09-08T12:39:18Z

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

qbey

First review, I know work is still ongoing and I did not read all the tests... :)

src/backend/core/api/serializers.py

src/backend/core/api/viewsets.py

src/backend/core/services/search_indexers.py

src/backend/core/models.py

src/backend/core/services/search_indexers.py

src/backend/core/tests/commands/test_index.py

src/backend/core/tests/documents/test_api_documents_search.py

src/backend/core/api/viewsets.py

src/backend/impress/settings.py

Search in Docs relies on an external project like "La Suite Find". We need to declare a common external network in order to connect to the search app and index our documents.

We need to content in our demo documents so that we can test indexing.

Add indexer that loops across documents in the database, formats them as json objects and indexes them in the remote "Find" mico-service.

On document content or permission changes, start a celery job that will call the indexation API of the app "Find". Signed-off-by: Fabre Florian <[email protected]>

Signed-off-by: Fabre Florian <[email protected]>

New API view that calls the indexed documents search view (resource server) of app "Find". Signed-off-by: Fabre Florian <[email protected]>

New SEARCH_INDEXER_CLASS setting to define the indexer service class. Raise ImpoperlyConfigured errors instead of RuntimeError in index service. Signed-off-by: Fabre Florian <[email protected]>

Signed-off-by: Fabre Florian <[email protected]>

Filter deleted documents from visited ones. Set default ordering to the Find API search call (-updated_at) BaseDocumentIndexer.search now returns a list of document ids instead of models. Do not call the indexer in signals when SEARCH_INDEXER_CLASS is not defined or properly configured. Signed-off-by: Fabre Florian <[email protected]>

Only documents without title and content are ignored by indexer.

Add SEARCH_INDEXER_COUNTDOWN as configurable setting. Make the search backend creation simplier (only 'get_document_indexer' now). Allow indexation of deleted documents. Signed-off-by: Fabre Florian <[email protected]>

Add bin/fernetkey that generates a key for the OIDC_STORE_REFRESH_TOKEN_KEY setting. Signed-off-by: Fabre Florian <[email protected]>

Add nginx with 'nginx' alias to the 'lasuite-net' network (keycloak calls) Add celery-dev to the 'lasuite-net' network (Find API calls in jobs) Set app-dev alias as 'impress' in the 'lasuite-net' network Add indexer configuration in common settings Signed-off-by: Fabre Florian <[email protected]>

Rename FindDocumentIndexer as SearchIndexer Rename FindDocumentSerializer as SearchDocumentSerializer Rename package core.tasks.find as core.task.search Remove logs on http errors in SearchIndexer Factorise some code in search API view. Signed-off-by: Fabre Florian <[email protected]>

Replace indexer_debounce_lock|release functions by indexer_throttle_acquire() Instead of mutex-like mechanism, simply set a flag in cache for an amount of time that prevents any other task creation. Signed-off-by: Fabre Florian <[email protected]>

Keep ordering by score from Find API on search/ results and fallback search still uses "-update_at" ordering as default Refactor pagination to work with a list instead of a queryset Signed-off-by: Fabre Florian <[email protected]>

Set SEARCH_INDEXER_CLASS=None as default configuration for dev. Rename docker network 'lasuite-net' as 'lasuite' to match with Drive configuration. Signed-off-by: Fabre Florian <[email protected]>

Add documentation for env & Find+Docs configuration in dev mode Signed-off-by: Fabre Florian <[email protected]>

Reduce the number of Find API calls by grouping all the latest changes for indexation : send all the documents updated or deleted since the triggering of the task. Signed-off-by: Fabre Florian <[email protected]>

As we filter the empty documents from the batch during indexing some batches can be empty and cause an error. Now they are ignored. Add --batch-size argument to the index command. Signed-off-by: Fabre Florian <[email protected]>

Use nb_results instead of page/page_size argument for /search API. Signed-off-by: Fabre Florian <[email protected]>

sampaccoud mentioned this pull request Aug 7, 2025

Full-Blown search feature #322

Open

sampaccoud requested a review from joehybird August 7, 2025 16:40

sampaccoud assigned joehybird Aug 7, 2025

sampaccoud added feature add a new feature backend labels Aug 7, 2025

joehybird force-pushed the index-to-search branch from 89fd00e to 526d757 Compare August 13, 2025 14:22

joehybird force-pushed the index-to-search branch 3 times, most recently from 10bfd94 to 5bd6b18 Compare September 8, 2025 12:38

joehybird force-pushed the index-to-search branch from 5bd6b18 to e966594 Compare September 10, 2025 15:18

qbey reviewed Sep 10, 2025

View reviewed changes

joehybird force-pushed the index-to-search branch 16 times, most recently from 7cfa907 to 7255ec2 Compare September 15, 2025 13:01

qbey reviewed Sep 15, 2025

View reviewed changes

joehybird force-pushed the index-to-search branch from 7255ec2 to ee1105f Compare September 17, 2025 06:29

joehybird force-pushed the index-to-search branch 2 times, most recently from 652c868 to e9fdc43 Compare November 13, 2025 09:06

joehybird requested a review from qbey November 13, 2025 09:47

joehybird force-pushed the index-to-search branch 2 times, most recently from 553332f to bca3a22 Compare November 14, 2025 09:26

sampaccoud and others added 21 commits November 14, 2025 13:37

🔧(compose) configure external network for communication with search

fd49f28

Search in Docs relies on an external project like "La Suite Find". We need to declare a common external network in order to connect to the search app and index our documents.

✨(backend) add dummy content to demo documents

254a541

We need to content in our demo documents so that we can test indexing.

✨(backend) add document search indexer

cb77f68

Add indexer that loops across documents in the database, formats them as json objects and indexes them in the remote "Find" mico-service.

✨(backend) add async triggers to enable document indexation with find

1a9011f

On document content or permission changes, start a celery job that will call the indexation API of the app "Find". Signed-off-by: Fabre Florian <[email protected]>

🔧(compose) Add some ignore for docker-compose local overrides

0145a05

Signed-off-by: Fabre Florian <[email protected]>

✨(backend) add unit test for the 'index' command

47fd1df

Signed-off-by: Fabre Florian <[email protected]>

✨(backend) add document search view

1b36f6b

New API view that calls the indexed documents search view (resource server) of app "Find". Signed-off-by: Fabre Florian <[email protected]>

✨(backend) improve search indexer service configuration

ff08b3e

New SEARCH_INDEXER_CLASS setting to define the indexer service class. Raise ImpoperlyConfigured errors instead of RuntimeError in index service. Signed-off-by: Fabre Florian <[email protected]>

✨(backend) refactor indexation signals and fix circular import issues

f49902a

Signed-off-by: Fabre Florian <[email protected]>

✨(backend) Index partially empty documents

57a6f73

Only documents without title and content are ignored by indexer.

✨(backend) Index deleted documents

ad92bf6

Add SEARCH_INDEXER_COUNTDOWN as configurable setting. Make the search backend creation simplier (only 'get_document_indexer' now). Allow indexation of deleted documents. Signed-off-by: Fabre Florian <[email protected]>

🔧(backend) tool for valid fernet key used in OIDC token storage

f9c0a2b

Add bin/fernetkey that generates a key for the OIDC_STORE_REFRESH_TOKEN_KEY setting. Signed-off-by: Fabre Florian <[email protected]>

🔧(compose) disable indexer in default configuration

78e2bc6

Set SEARCH_INDEXER_CLASS=None as default configuration for dev. Rename docker network 'lasuite-net' as 'lasuite' to match with Drive configuration. Signed-off-by: Fabre Florian <[email protected]>

📝(backend) add fulltext search documentation

a83f140

Add documentation for env & Find+Docs configuration in dev mode Signed-off-by: Fabre Florian <[email protected]>

✨(backend) use batches in indexing task

5427f18

Reduce the number of Find API calls by grouping all the latest changes for indexation : send all the documents updated or deleted since the triggering of the task. Signed-off-by: Fabre Florian <[email protected]>

🩹(backend) fix empty indexation batch

3652732

As we filter the empty documents from the batch during indexing some batches can be empty and cause an error. Now they are ignored. Add --batch-size argument to the index command. Signed-off-by: Fabre Florian <[email protected]>

joehybird force-pushed the index-to-search branch 2 times, most recently from 81070d0 to 94c792f Compare November 14, 2025 14:17

✨(backend) adapt to Find new search pagination

f2106dd

Use nb_results instead of page/page_size argument for /search API. Signed-off-by: Fabre Florian <[email protected]>

joehybird force-pushed the index-to-search branch from 94c792f to f2106dd Compare November 14, 2025 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Index to search #1276

Index to search #1276

Uh oh!

sampaccoud commented Aug 7, 2025 •

edited by joehybird

Loading

Uh oh!

gitguardian bot commented Sep 8, 2025 •

edited

Loading

Uh oh!

qbey left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Index to search #1276

Are you sure you want to change the base?

Index to search #1276

Uh oh!

Conversation

sampaccoud commented Aug 7, 2025 • edited by joehybird Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Proposal

Uh oh!

gitguardian bot commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

️✅ There are no secrets present in this pull request anymore.

Uh oh!

qbey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sampaccoud commented Aug 7, 2025 •

edited by joehybird

Loading

gitguardian bot commented Sep 8, 2025 •

edited

Loading