Improve index cleanup #5709
Open
Improve index cleanup #5709
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nickva
force-pushed
the
fix-index-cleanup
branch
from
5b5ca2b to
b1751b9
Compare
rnewson
reviewed
Oct 23, 2025
| ok. | ||
|
|
||
| cleanup(DbName, ActiveSigs) -> | ||
| cleanup(DbName, #{} = SigMap) -> |
There was a problem hiding this comment.
a _search_cleanup during a cluster upgrade will crash then? I think that's fine as they are rare but might be worth noting somewhere.
There was a problem hiding this comment.
We can add an upgrade clause as well. It's easy enough in this case
There was a problem hiding this comment.
Added an upgrade clause
nickva
force-pushed
the
fix-index-cleanup
branch
2 times, most recently
from
e79027c to
1ccf4c6
Compare
Fix these issues:
* Index cleanup triggered by smoosh only cleaned view indexes and purge
checkpoints, make sure to also clean nouveau and search indexes and
checkpoints.
* `/_search_cleanup` endpoint only cleaned indexes on the coordinator node,
despite being a clustered fabric endpoint. Fix it so it cleans indexes on
other nodes as well as most users would expect.
* Nouveau cleanup didn't clean purge checkpoints, so make it do so. Use the
mrview purge checkpoint strategy for all indexes. This improves dreyfus logic
to avoid traversing internal disk paths from clouseau.
* Make `_view_cleanup` clean all the index types. This is a bit dirty, but it
may be better than adding another cleanup `_index_cleanup` API. Left
`_search_cleanup` and `_nouveau_cleanup` APIs as is for now.
Some optimizations:
* For each index clean request fetch ddocs once. Calculate signatures and
then call the remote node cleanup logic with them. This avoids fetching
design documents multiple times or sending all of them to the worker nodes.
This is what Nouveau is doing so stick with that nice pattern.
* Use erpc for remote calls. Our Erlang version is high enough (25+) to use
the multple requests pattern from erpc. This is more compact than rexi. The
absolute timeout pattern makes it simpler to have a global timeout for the
whole request.
Cleanups:
* Make purge checkpoint fetching and cleanup more uniform. Use common utility
logic in `couch_index_util` for all indexes.
* Make index cleanup similar between all three indexes. Use the same erpc
pattern for all of them.
* Move index cleanup functions from fabric.erl to its own module --
fabric_index_cleanup.erl
* Rename fabric function names more uniform and clearly indicate if it cleans
indexes on all nodes or just the current node.
* Add the db name to dreyfus `#index{}` record when it initializes so it
matches Nouveau.
nickva
force-pushed
the
fix-index-cleanup
branch
from
1ccf4c6 to
912b00c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix these issues:
Index cleanup triggered by smoosh only cleaned view indexes and purge checkpoints, make sure to also clean nouveau and search indexes and checkpoints.
/_search_cleanupendpoint only cleaned indexes on the coordinator node, despite being a clustered fabric endpoint. Fix it so it cleans indexes on other nodes as well as most users would expect.Nouveau cleanup didn't clean purge checkpoints, so make it do so. Use the mrview purge checkpoint strategy for all indexes. This improves dreyfus logic to avoid traversing internal disk paths from clouseau.
Make
_view_cleanupclean all the index types. This is a bit dirty, but it may be better than adding another cleanup_index_cleanupAPI. Left_search_cleanupand_nouveau_cleanupAPIs as is for now.Some optimizations:
For each index clean request fetch ddocs once. Calculate signatures and then call the remote node cleanup logic with them. This avoids fetching design documents multiple times or sending all of them to the worker nodes. This is what Nouveau is doing so stick with that nice pattern.
Use erpc for remote calls. Our Erlang version is high enough (25+) to use the multple requests pattern from erpc. This is more compact than rexi. The absolute timeout pattern makes it simpler to have a global timeout for the whole request.
Cleanups:
Make purge checkpoint fetching and cleanup more uniform. Use common utility logic in
couch_index_utilfor all indexes.Make index cleanup similar between all three indexes. Use the same erpc pattern for all of them.
Move index cleanup functions from fabric.erl to its own module -- fabric_index_cleanup.erl
Rename fabric function names more uniform and clearly indicate if it cleans indexes on all nodes or just the current node.
Add the db name to dreyfus
#index{}record when it initializes so it matches Nouveau.