Distributor ingestion limits

[aleks-p]

Adds a way for distributors to stop allowing requests through, controlled via a tenant override. The tenant override is defined as:

ingestion_limit:
  period_type: hour
  period_limit_mb: 1024
  limit_reached: true
  next_limit_reset: 1738274400
  sampling:
    num_requests: 1
    period: 2m

For now, an external system will manage the tenant overrides, in the future we might create a special component for this. Once the limit is reached (ingestion_limit.limit_reached == true):

• distributors will only let a few requests through, controlled via the sampling section in the config
• for rejections, distributors will respond with an error containing more information:
  resource_exhausted: limit of 1 GiB/hour reached, next reset at 2025-01-30T22:00:00Z

[simonswine]

LGTM

I think sorting out the docs would be good, otherwise I think this is ready to go

[simonswine]

You can set an advanced or experimental struct tag, to not have it appear here.

Alternatively we should fill this with more description.

Mimir does that: https://github.com/grafana/mimir/blob/ae924bf9a2f373a600dce34c078a9602929dd480/pkg/util/validation/limits.go#L107

[aleks-p]

Thanks!

I just tried, the advanced and experimental category don't seem to make a difference (it still gets generated). Setting doc:"hidden" removes it which I think makes sense considering we will likely move this config to object storage.