[TSDB] Reconsider the start time of the first backing index.

[gmarouli]

Every time series index has a start_time and an end_time and only accepts data within these boundaries, they are both required settings.

A time series data stream consists of a series of backing time series indices with no overlapping boundaries. The start_time of a backing index is calculated to be the same as the end_time of the previous backing index.

In this issue, we would like to rethink how the start_time of the first index is determined. When a time series data stream is created, it creates the first backing index and it uses the index.look_back_time property to determine how far in time the start_time of the first index will be.

This has been confusing the user who consider that the index.look_back_time means that this how long back in time they can index data.

We would like to rethink the way the start_time of the first backing index of a data stream is determined. For example, can we leave it empty or should we set it Unix Epoch?

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

We would like to rethink the way the start_time of the first backing index of a data stream is determined. For example, can we leave it empty or should we set it Unix Epoch?

This would allow indexing older metrics easier. However if much more older metrics are indexed then current ones, this can lead to the first backing index holding much more data compared to subsequent backing indices.

[gmarouli]

However if much more older metrics are indexed then current ones, this can lead to the first backing index holding much more data compared to subsequent backing indices.

That is true, it might be hard to evaluate the trade-offs without any user data.

[kkrik-es]

We always have this problem, even for the first backing index.. I think it's too restrictive to disallow loading older data - just in case we load too much. More so, lifecycle should be handling data aging, so for most real-world applications with sizeable datasets, older data will be aggressively downsampled and/or recycled.