Make storage request more peers if no checkpoints can be found

[kostko]

In case the storage worker is unable to initialize because there are no usable checkpoints available, make it able to request additional peers from the P2P subsystem (they could be requested through discovery etc.).

Similar should probably be done for other protocols, e.g. in case key manager client is unable to do queries it should try to discover more peers etc.

[martintomazic]

Context:
Target number of protocol peers is maintained by peer manager service that ensures both the minimum number of consensus registered peers and total number of peers (registered or non-registered). The former are being discovered via consensus service, while the latter are discovered via seed nodes discovery (see).

Problem:
If host node has sufficient number of peers connected, but none of them has usable checkpoints, host might never trigger new discovery** even though with current peerset it may never have checkpoints available (or at least in reasonable time).

** Assuming peers that never record neither success nor failure for a given protocol are not pruned and replaced fast.

Naive solution:

1. Add new method to p2p.Service or rpc.P2P interface:

// ForcePeerDiscovery discovers up to max new peers for the specified protocol.
ForcePeerDiscovery(p core.ProtocolID, max int)

2. If no checkpoints received (or only genesis), call method above.
3. Repeat similar to 2. for other protocols.

Alternative/Complementary solution:
RegisterProtocol in addition to triggering tracking and peer management for the given protocol (fetch part), is also responsible for triggering background advertisement (serve part). Ideally, advertisement would be RegisterProtocolServer responsibility.

We could keep it simple and start advertising only after sufficient number of checkpoints. Likely we could generalize this and propagate generic readiness closure up to advertisement's retry.

Proposed solution:
Probably both solutions should be implemented, but we could start with naive solution first.

The last approach would require a bit of refactor, but it would likely already solve the root cause one of the root causes for this issue (peers registering too early/without checkpoints). Still we cannot assume peers behave honest, which should be mitigated with fallback discovery (naive approach).

Opinions?

[martintomazic]

Also we should definitely limit fallback discovery, else e.g. if you set your trust root too much back in time, you will keep adding new peers indefinitely. They will be pruned, but still...

There is also minor overlap with #5823 (comment), especially given that often time probably this was result of poor trust root?

Update:
Prior to all that I would like to get in #5751, given that is relatively simple and was likely the major root cause for host having sufficient number of connections, yet none of them supports checkpoints.

[kostko]

Prior to all that I would like to get in #5751, given that is relatively simple and was likely the major root cause for host having sufficient number of connections, yet none of them supports checkpoints.

Yes, that would be a good first step.

[martintomazic]

With #6277 merged, the root cause for this problem should be omitted. I suggest to postpone this task for now, until this issue is reported again.