State sync worker should be only responsible for initializing and syncing the state

[martintomazic]

WIP

Follow-up of #6306, inspired by #6299 (comment), part of the #6307.

After that we will refactor state sync worker into smaller workers (#6307) = #6299.

Finally, we will have simple PR that optimizes things.

TODO:

• Move out pruner
• Move out p2p server registration
• Add tests
• Checkpoint sync is independent worker and once it finishes we should close the checkpointsync p2p client, else we are leaking peermgr goroutine...

End goal:
Runtime storage worker should orchestrate workers, implement hooks and be responsible for the status:

1. Init state, by filling the possible missing rounds in nodedb and reindexed history.
2. Depending on the result from 1. start checkpoint sync worker, that also handles lifetime of the underlying checkpoint sync clients(!).
3. State is now fully initialized (update the status)
4. Start remaining workers:
   • Checkpointer, also responsible for genesis checkpoint, even if not enabled or could be separate task.
   • Availability nudger
   • Diff sync worker.
5. Register p2p protocol servers
6. Register prune handler.

This way all workers that actually do work can be easily tested in isolation.

[netlify]

✅ Deploy Preview for oasisprotocol-oasis-core canceled.

Name	Link
🔨 Latest commit	8bc3c32
🔍 Latest deploy log	https://app.netlify.com/projects/oasisprotocol-oasis-core/deploys/68b093799a64b00008a3049e

[martintomazic]

On one hand I like granular packages, so that I can reuse simple words [package_name].New, and conveniently return Worker type. I can use [package_name].Config etc. If package has a good name this becomes very useful... Generally, the namespace is very clean.

On the other hand having workers nested under single package, I can make them private, and thus they are not seen in a documentation, which in this context is probably a good thing. update: Not true, as documentation should then be moved to the worker orchestrating... Finally, all this packages should be under internal, to make it clear sub-packaging is only for namespacing and API is not stable (e.g. storage/committee even storage). I think pkgsite documentation correctly hides everything under internal and by default shows only what the client (someone importing oasis protocol) should use and know about. Still not convinced entirely with neither of the approaches :).

If we decide to go with the latter approach, then runtime_worker.go from this PR should probably be moved inside committee package. In such case renaming committee to statesync was premature, given we will have many workers not directly related to the state sync inside this package. Regardless, we should definitely find a better name, but maybe later. update: Renaming to statesync is definitely premature given that likely we would have diffsync worker. I would however keep refactor that internally renames Node to Worker as this is not node...

The good point it is relatively easy to refactor between any of the approaches. Finally, this PR reinforced that State sync worker should be only responsible for initializing and syncing the state so factoring out checkpointer and availability nudger after #6306 is the right step to do.