search optimization

[GrzegorzPustulka]

I conducted performance tests in 3 scenarios on the sentinel-2-l2a collection, which has around 42 million products and 480GB of data.
The performance test consisted of maintaining 200 requests per second, querying the sentinel-2-l2a collection with random date ranges and random geometry intersections, limit 100.

Scenario I - 100GB indexes, 8 shards:
Average response time: 5.57s
Average transfer 400MB/s
200 responses - 18967
500 responses - 396
CPU usage on opensearch server 98-100%

Scenario II - 1 index, 1 shard:
Average response time: 1.14s
Average transfer 767MB/s
200 responses - 43599
500 responses - 10040
CPU usage on opensearch server 98-100%

Scenario III - quarterly indexes, based on product datetime and collection. Example items_sentinel-2-l2a-Q1-2015, we specify in the API which opensearch indexes to search in:
Average response time: 732.72ms
Average transfer 1.2 GB/s
200 responses - 53614
500 responses - 0
CPU usage on opensearch server around 60%
Notes: data distribution was uneven e.g. Q1 2015 had X times less data than Q1 2022.

Conclusions:
The best performance solution is specifying in the API which opensearch/elasticsearch indexes should search - users usually search by collection and datetime.

Solution:
Sentinel-l2a has much larger products than other collections. So to eliminate that some indexes could be 2GB while others 30GB:

1. We index around 30GB per index - +/- because we fill up to full day, so new index starts from full day, so index looks like X-2025-04-01:2025-07-03

2. In API we get indexes based on alias and cache them, so we don't execute this query on every request

3. In API we specify which indexes we'll search in

Are you open to this change? If yes then I'd be happy to prepare MR with these changes - might take some time since it's a big change.

[jonhealy1]

@GrzegorzPustulka This sounds really good.

[GrzegorzPustulka]

@jonhealy1 how do we look at backward compatibility? if users have indexes of this type: items_sentinel-2-l1c_73656e74696e656c2d322d6c3163-000001?

[jonhealy1]

We should be able to provide backward compatibility for existing index naming conventions. The current codebase resolves indexes for each collection via the indices() utility function which maps collection IDs to index aliases or names, and could potentially be updated to support both the legacy and new index formats during a transition period.

• Support both index patterns in the API for now, so existing users aren’t disrupted.
• Add a deprecation warning (e.g., via logging) when the legacy format is used in indices() or related resolution logic.
• Announce a deprecation timeline and provide migration guidance for users to switch to the new indexing scheme.

[jonhealy1]

I am going to tag some other people who may have an opinion on these potential changes - @jamesfisher-geo @rhysrevans3 @Zaczero @StijnCaerts

[GrzegorzPustulka]

Okay, so I'll wait for everyone else's opinions then

[jonhealy1]

Personally I feel like it's a good idea and worth starting on. People may have stronger opinions on specific implementation details.

[rhysrevans3]

The performance benefits look great. I have a couple of questions just to clarify some points:

1. In Scenario I does "100GB indexes, 8 shards" mean you had 5 indexes each with 8 shards?
2. In Scenario III how many indexes and shards did you use?
3. Is the planned solution effectively moving some of the datetime filtering to the API? By filtering the indexes that are being search on.
4. Was this kind of filtering of which indexes are being searched on being done in Scenario III?

[GrzegorzPustulka]

@rhysrevans3

1. In scenario 1, there were 5 indexes: 4x100GB and 1x80GB, each with 8 shards.
2. The data was from 2015-2025, so there were 4 indexes per year, totaling 42 indexes (excluding Q3 and Q4 of 2025), each with 1 shard
3. Yes, directly in the API we specify which indexes OpenSearch should search
4. Yes, the test was designed so that date ranges were randomly selected between 2015-2025, e.g., 2015-01-01 to 2017-04-04.The API would choose which indexes matched that range and those were passed to the query

[GrzegorzPustulka]

According to this article, I might even reduce the size from 30GB to 20GB

https://opensearch.org/blog/optimize-opensearch-index-shard-size/

[jonhealy1]

Is this something that we should implement as a stac-api-extension

@GrzegorzPustulka Are there some use-cases where an organization may not want this functionality? If we could implement it as an extension and then have a flag where we can turn it on/off like how we do with the Transactions Extension now.

[Zaczero]

Scenario I - 100GB indexes, 8 shards

Was this scenario checked with 8 servers (1 shard/server) or 1 server (8 shards/server)?

Also, what are the general specs of the server(s) (CPUs, RAM)?

What OpenSearch version?

Can I assume you were using mostly the default OpenSearch configuration?

[GrzegorzPustulka]

@Zaczero

1 server(1 index, 8 shards)
CPU - AMD EPYC 7502 32-Core Processor,
RAM - 512GB
OpenSearch version - 2.19.1
mostly default OpenSearch configuration, with minor changes

[GrzegorzPustulka]

@jonhealy1
If someone has very small collection datasets, they basically don't need this, because it won't optimize anything for them but it also won't slow anything down. The flag idea is okay with me

[Zaczero]

This is unrelated to the search index optimization, but rather to the deployment itself.

Scenario I ... 8 shards:
Average response time: 5.57s
CPU usage on opensearch server 98-100%

Scenario II ... 1 shard:
Average response time: 1.14s
CPU usage on opensearch server 98-100%

It seems like you are trying to optimize for maximum throughput, sacrificing responsiveness under light/medium load.

In such a case, indeed, starting fewer threads per request (1 shard on v2.19 == 1 thread) will use less overall system resources and increase total throughput. Having more shards in this max-load scenario will only be beneficial if the shards are distributed over several servers.

I can think of a few potential optimizations for this deployment scenario (that is, data warehouse that is mostly read-only).

• Increase maximum segment size from default 5GB to something around 100GB (index.merge.policy.max_merged_segment). This will reduce the total amount of work per thread and should further increase total throughput (ref). Generally, I believe large segments should be preferred for STAC deployments.
• Further reduce the number of segments (that is, search work per shard) by making merging more aggressive by decreasing index.merge.policy.segments_per_tier and index.merge.policy.max_merge_at_once from default 10 to around 3.
• Increasing index.refresh_interval will decrease initial segment fragmentation and should only help in the STAC deployment scenario. Changing from the default 1s to 60s sounds like a good option.

I would personally aim for a more balanced approach and expand the benchmark to also cover the medium-load scenario. I would then focus on having 1 shard per server, with concurrent segment search configured (ref). I would probably also switch the index codecs from default (LZ4) to zstd as it has a much better compression ratio with very small downsides (ref).

[Zaczero]

Do you mind sharing your benchmark script?

If I understand this correctly, you create smaller indexes that are grouped quarterly, and because of these smaller indexes performance improves significantly. That's somewhat surprising!

My theory is that you see this behavior because with the default configuration and large dataset, you have many segments that are scanned linearly. Splitting a large index into several smaller ones decreases the number of segments per index, which makes them easier and less intensive to search. But I believe this could also be achieved by simply tuning the OpenSearch config (check my previous comment, related).