feat: Heartbeat shard statistics

[Theis-Mathiassen]

What changed?
We added functionality to record the load as a moving average for each shard, where the weight of a new data point depends on how recently the average was last updated.

Why?
This is done to smooth the load input for the shard distributor, this is desirable as the load can change sporadically.
It is also necessary to save the load of each shard in ETCD, as to persist it (In case the handler crashes) and make it available to each instance of shard distributors.

How did you test it?
We have created some unit tests, and tried to run it with the canary service:
TestRecordHeartbeatUpdatesShardStatistics:
This test verifies that when an executor sends a heartbeat with ShardLoad information for a shard, the ShardStatistics for that shard are correctly updated in the store, specifically the SmoothedLoad and LastUpdateTime. It also ensures that LastMoveTime remains unchanged if not explicitly updated.

TestRecordHeartbeatSkipsShardStatisticsWithNilReport:
This test ensures that if an executor's heartbeat includes a nil ShardStatusReport for a particular shard, the existing ShardStatistics for that shard are not updated or created. It also verifies that valid shard reports are processed correctly.

Potential risks
None, since it is for the shard distributor, which is not utilized in production yet.

Release notes
It is not, since it is for the shard distributor, which is not utilized in production yet.

Documentation Changes
No, but maybe some documentation should be created, later.

[eleonoradgr]

I am worried that we are generating too much writes load to etcd,do you think we can avoid writing if the the load is stable?

[Theis-Mathiassen]

Yeah I think that is possible, so as an example if load from heartbeat is within +/- 5% of the currently stored value, we do not update it?

Some problems with this approach might be:

Because of how ewmaSmoothedLoad calculates the new load, if there have been a longer time since last load change, the new load will have greater impact.

So I fear if we have a very stable shard load with some spikes in the load, the smoothing effect is lost, excuse my drawing, but i hope it illustrates the idea:

It might still be possible, we will probably just have to rethink the update function, how we can avoid this.
We will look into how to solve this, and comment again when we have something.

[AndreasHolt]

Have implemented something that can help throttle the writes in 158e030.
We now only write

1. If load has fluctuated enough
2. If enough time has passed since last write (we don't want to cause a stats cleanup just because load hasn't fluctuated)

Edit:
e0779ec decouples the stats cleanup, which before this change was coupled to heartbeat TTL when determining whether a shard stat is stale.