Don't normalize fields of type text when the index mode is LogsDB or TSDB #131317
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…TSDB. In the context of logsdb and tsdb, not many fields are configured to be of type text, and given the trade offs that logsdb and tsdb provide that is geared towards storage reduction, it makes sense not to store a normalized version of text fields. Closes elastic#129183
Kubik42
added
>enhancement
>breaking
Team:StorageEngine
:StorageEngine/Mapping
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
Hi @Kubik42, I've created a changelog YAML for you. Note that since this PR is labelled |
|
Hi @Kubik42, I've updated the changelog YAML for you. Note that since this PR is labelled |
|
Hi @Kubik42, I've updated the changelog YAML for you. Note that since this PR is labelled |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes the default behavior for norms on
textfields in logsdb and tsdb indices. Prior to this change, norms were enabled by default, with the option to disable them via manual configurations. After this change, norms will be disabled by default. Note, because we don't support enabling norms from a disabled state (code), unless manually enabled from the get-go, users will not be able to enable norms ontextfields in logsdb and tsdb indices.In the context of logsdb and tsdb, not many fields are configured to be of type
text, and given the trade offs that logsdb and tsdb provide that is geared towards storage reduction, it makes sense not to store a normalized version oftextfields.Closes #129183
During benchmarking, there was a minor improvement in disk usage:
More benchmarks and details available here.
With this change, we're also replacing the existing
TextParms.normswithParameter.normsParamto be more in line with the existing functions in that class. This change is covered by the existing tests.