Add ES93BloomFilterStoredFieldsFormat for efficient field existence checks #137331

fcofdez · 2025-10-29T11:55:24Z

Introduces a new stored fields format that builds a Bloom filter for a specific field to enable fast existence checks without storing the field itself. This delegates storage of all other fields to another StoredFieldsFormat while maintaining the ability to quickly determine if a document might contain the target field.

Missing parts

Merge support for the bloom filter
Dynamic bloom filter sizing
More extensive testing

…hecks Introduces a new stored fields format that builds a Bloom filter for a specific field to enable fast existence checks without storing the field itself. This delegates storage of all other fields to another StoredFieldsFormat while maintaining the ability to quickly determine if a document might contain the target field.

tlrx

Looks great! I left many comments but most of them are nits.

I think this is ready for review, we'll improve the format step by step.

server/src/main/java/org/elasticsearch/index/codec/bloomfilter/BloomFilterHashFunctions.java

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

tlrx · 2025-10-30T09:47:33Z

...t/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormatTests.java

+                    "",
+                    TestUtil.getDefaultCodec().storedFieldsFormat(),
+                    ByteSizeValue.ofKb(2),
+                    IdFieldMapper.NAME


I wonder if we could have a better coverage if we were randomly using some field names that are generated in BaseStoredFieldsFormatTestCase?

I'm afraid that we'll broke some of the integration tests that rely on these fields actually being stored.

tlrx · 2025-10-30T10:11:27Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+
+        private void finishBloomFilterStoredFormat() throws IOException {
+            BloomFilterMetadata bloomFilterMetadata = null;
+            if (bloomFilterFieldInfo != null) {


If this is null, then we haven't seen an _id stored field. Does it still make sense to write an empty file then?

Oh, for merges maybe?

Yes, it's for merges. Once we get merges working, bloomFilterFieldInfo should be always not null

Makes sense, thanks!

tlrx · 2025-10-30T10:14:46Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+ */
+public class ES93BloomFilterStoredFieldsFormat extends StoredFieldsFormat {
+    public static final String STORED_FIELDS_BLOOM_FILTER_FORMAT_NAME = "ES93BloomFilterStoredFieldsFormat";
+    public static final String STORED_FIELDS_BLOOM_FILTER_EXTENSION = "sfbf";


sfbf will be Special Francisco's Bloom Filter in my mind forever once this is merged.

tlrx · 2025-10-30T10:16:43Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+            var success = false;
+            try {
+                bloomFilterFieldReader = BloomFilterFieldReader.open(directory, si, fn, context, segmentSuffix);
+                success = true;


Do we need to return the delegateReader even if bloomFilterFieldReader is null?

Not necessarily, but I would expect that the bloom filter field reader will be always != null

Skip skipping the bloom filter field for all data types Compute the bloom filter size in a simpler way

elasticsearchmachine · 2025-10-30T14:09:40Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2025-10-30T14:16:33Z

Hi @fcofdez, I've created a changelog YAML for you.

tlrx

LGTM

tlrx · 2025-10-30T15:32:43Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+                    + " or less (rounded to nearest power of two)"
+            );
+        }
+        this.bloomFilterSizeInBits = (int) Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE);


nit:

Suggested change

this.bloomFilterSizeInBits = (int) Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE);

this.bloomFilterSizeInBits = Math.toIntExact(Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE));

Nit isn't addressed yet

Addressed in 1e4ea4c

tlrx · 2025-10-30T15:38:35Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+            if (info.getName().equals(bloomFilterFieldName)) {
+                addToBloomFilter(info, value);
+            } else {
+                delegateWriter.writeField(info, value);


I think it depends on how this bloom stored field format will be registered. If it's registered only for the _id field through (an adjusted version of) PerFieldFormatSupplier, what I think we'll do, then I prefer to have all field behave the same as we know other types of fields won't be called.

tlrx · 2025-10-30T15:40:07Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+        }
+
+        private void addToBloomFilter(FieldInfo info, BytesRef value) {
+            bloomFilterFieldInfo = info;


nit: I think there was an assertion on info.name being equal to the bloomFilterFieldName which I liked

Addressed in 4293207

tlrx · 2025-10-30T15:40:53Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+
+        private void finishBloomFilterStoredFormat() throws IOException {
+            BloomFilterMetadata bloomFilterMetadata = null;
+            if (bloomFilterFieldInfo != null) {


Makes sense, thanks!

…z/elasticsearch into bloom-filter-stored-fields-format

…ields-format

kkrik-es

@martijnvg knows this part better for sure.

server/src/main/java/org/elasticsearch/index/codec/bloomfilter/BloomFilterHashFunctions.java

martijnvg

LGTM 👍

martijnvg · 2025-10-31T13:31:59Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+                    + " or less (rounded to nearest power of two)"
+            );
+        }
+        this.bloomFilterSizeInBits = (int) Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE);


Nit isn't addressed yet

martijnvg · 2025-10-31T14:47:49Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+    public static final String STORED_FIELDS_BLOOM_FILTER_FORMAT_NAME = "ES93BloomFilterStoredFieldsFormat";
+    public static final String STORED_FIELDS_BLOOM_FILTER_EXTENSION = "sfbf";
+    public static final String STORED_FIELDS_METADATA_BLOOM_FILTER_EXTENSION = "sfbfm";
+    private static final int VERSION_START = 1;


nit: I think typically codec versions start at zero?

Addressed in 605c93d

martijnvg · 2025-10-31T15:03:36Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+
+        @Override
+        public int merge(MergeState mergeState) throws IOException {
+            // Skip merging the bloom filter for now


Out of curiosity what is the plan to support merging? Iirc ES87BloomFilterPostingsFormat supports this by rebuilding the bloom filter from the merged terms.

Ideally we would just OR the bitsets, but that depends a bit on the sizing strategy. Otherwise we could just rebuild the bloom filter as you mentioned.

martijnvg · 2025-10-31T15:09:25Z

...c/main/java/org/elasticsearch/index/codec/bloomfilter/ES93BloomFilterStoredFieldsFormat.java

+            this.bloomFilterData = bloomFilterData;
+        }
+
+        public boolean mayContainTerm(String field, BytesRef term) throws IOException {


Nit: instead of makeing the BloomFilterFieldReader public, maybe keep it package protected / private like other classes in here and add an public interface for this specific method?

Addressed in 6550601

…ields-format

fcofdez · 2025-11-03T11:28:03Z

Thanks all for the reviews!

fcofdez requested a review from tlrx October 29, 2025 11:55

fcofdez added WIP :StorageEngine/TSDB You know, for Metrics v9.3.0 labels Oct 29, 2025

tlrx reviewed Oct 30, 2025

View reviewed changes

fcofdez added 2 commits October 30, 2025 15:00

Review comments

a1af7b9

Skip skipping the bloom filter field for all data types Compute the bloom filter size in a simpler way

Add more randomness

78eafbf

fcofdez requested a review from tlrx October 30, 2025 14:08

fcofdez marked this pull request as ready for review October 30, 2025 14:09

fcofdez removed the WIP label Oct 30, 2025

fcofdez requested review from kkrik-es and martijnvg October 30, 2025 14:09

elasticsearchmachine added the Team:StorageEngine label Oct 30, 2025

fcofdez added the >enhancement label Oct 30, 2025

Update docs/changelog/137331.yaml

d26a6e1

tlrx approved these changes Oct 30, 2025

View reviewed changes

fcofdez added 3 commits October 30, 2025 23:08

Improve test a bit

d68c2ad

Merge branch 'bloom-filter-stored-fields-format' of github.com:fcofde…

a8336c1

…z/elasticsearch into bloom-filter-stored-fields-format

Merge remote-tracking branch 'origin/main' into bloom-filter-stored-f…

416f1da

…ields-format

kkrik-es approved these changes Oct 31, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/codec/bloomfilter/BloomFilterHashFunctions.java Show resolved Hide resolved

server/src/main/java/org/elasticsearch/index/codec/bloomfilter/BloomFilterHashFunctions.java Show resolved Hide resolved

martijnvg approved these changes Oct 31, 2025

View reviewed changes

fcofdez requested a review from burqen November 3, 2025 08:28

fcofdez added 5 commits November 3, 2025 09:51

Add more assertions

49fcc66

Start version from 0

605c93d

Use toIntExact

1e4ea4c

Add assertion for bloom filter name

4293207

Extract interface

6550601

Merge remote-tracking branch 'origin/main' into bloom-filter-stored-f…

6b07204

…ields-format

fcofdez merged commit 2f6ea7a into elastic:main Nov 3, 2025
34 checks passed

	this.bloomFilterSizeInBits = (int) Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE);
	this.bloomFilterSizeInBits = Math.toIntExact(Math.multiplyExact(closestPowerOfTwoBloomFilterSizeInBytes, Byte.SIZE));

Add ES93BloomFilterStoredFieldsFormat for efficient field existence checks #137331

Add ES93BloomFilterStoredFieldsFormat for efficient field existence checks #137331

Uh oh!

Conversation

fcofdez commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Missing parts

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 30, 2025

Uh oh!

elasticsearchmachine commented Oct 30, 2025

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkrik-es left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fcofdez commented Nov 3, 2025

fcofdez commented Oct 29, 2025 •

edited

Loading