Add template_id to patterned-text type #131401
Conversation
| @@ -474,6 +496,10 @@ private FieldType resolveFieldType( | |||
| final IndexMode indexMode, | |||
| final String fullFieldName | |||
| ) { | |||
| if (requireDocValueSkippers) { | |||
There was a problem hiding this comment.
It seems weird to have both useDocValuesSkipper and requireDocValuesSkipper fields. But I don't think it's a good idea to use the name-based way to deciding whether or not to use docValueSkippers. It seems like we should just enforce that they are always used for KeywordFields that are created for the purpose of being a templateId.
There was a problem hiding this comment.
This reads more like enableDocValuesSkipper and useDocValuesSkipper. Can we clean up the logic to apply this?
There was a problem hiding this comment.
We talked offline about changing useDocValuesSkipper to enableDocValuesSkipper and requireDocValuesSkipper to forceDocValuesSkipper. It's still a bit weird to have two such parameters, but one is needed to generally enable the use of doc values skippers, and another it needed for types (such as templateId) which are certain they want to use skippers, and don't require and name-based logic to decide (like host.name). With future changes to skippers, we can probably clean this up some.
| Integer.MAX_VALUE, | ||
| indexCreatedVersion, | ||
| IndexMode.LOGSDB, | ||
| null, |
There was a problem hiding this comment.
Should we be wiring a IndexSortConfig through here? I'm still a bit confused about how we want to control sorting.
There was a problem hiding this comment.
No index sorting will be explicitly configured for now, see:
There was a problem hiding this comment.
I also find it weird that it's propagated through this class.. Looking at the uses, I see shouldUseDocValuesSkipper that's just inappropriate - this logic belongs to the LogsdbIndexModeSettingsProvider that can inject a setting to enable skiplists when appropriate.
elasticsearchmachine
added
the
needs:triage
server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java
Outdated
Show resolved
Hide resolved
| @@ -222,6 +223,7 @@ public Builder(final String name, final MappingParserContext mappingParserContex | |||
| mappingParserContext.getIndexSettings().getMode(), | |||
| mappingParserContext.getIndexSettings().getIndexSortConfig(), | |||
| USE_DOC_VALUES_SKIPPER.get(mappingParserContext.getSettings()), | |||
There was a problem hiding this comment.
Should we remove this, or use regular indexing when this is not set? Maybe something to discuss with @martijnvg when he's back.
.../src/main/java/org/elasticsearch/xpack/logsdb/patternedtext/PatternedTextValueProcessor.java
Outdated
Show resolved
Hide resolved
...test/java/org/elasticsearch/xpack/logsdb/patternedtext/PatternedTextValueProcessorTests.java
Outdated
Show resolved
Hide resolved
| search: | ||
| index: test | ||
| body: | ||
| docvalue_fields: [ "foo.template_id" ] |
There was a problem hiding this comment.
We should also test that it can be used in index sort config.
|
Fixes #128937 |
parkertimmins
added
>non-issue
:StorageEngine/Mapping
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
For patterned-text mapper
foo, add a sub-field calledfoo.template_id. This is the an 8-byte hash of the template doc_value column. Unlike the template,template_idis accessible and can be used for querying, aggregations, etc. Thetemplate_idis stored as a KeywordField and can use any features of the KeyworkFieldType.template_idhas doc_values, and is not stored or indexed. It uses doc value skippers, which should be quite fast given that the index will be sorted on template.