BlockExchange Performance and reliability

[benbierens]

Tests show that performance of the block exchange varies significantly in controlled networks.
In some cases, downloads fail when datasets are available both in the network, and in peers to which existing connections exist. Reliability must be improved.

Work done:

• Reliablity test write-up: https://hackmd.io/rwPtPJ7KTw6cGjhN0zNYig
• Block retry: Add basic retry functionality #1119
• Block protocol write up: https://hackmd.io/ZAblhSiPTRahEz9w0mulcA
  • simple simulator for looking at graphs/sampling: https://github.com/gmega/block.protocol

[benbierens]

Reliability Test write-up: https://hackmd.io/rwPtPJ7KTw6cGjhN0zNYig

[gmega]

This is an important step: #1119

[gmega]

Write-up on me trying to build a mental model of out how I think this should work: https://hackmd.io/ZAblhSiPTRahEz9w0mulcA

[benbierens]

It's been a while since I've done anything for this topic. So I dusted off some of performance tracking tests. (They never made it into the official release tests because they don't really give a clear pass or fail.)

Block-retransmit was an issue where a node would receive the same block multiple times from multiple peers, wasting bandwidth and slowing down total transfer. It has gone down from 2% retransmit in the "normal" swarm test for 0.2.4 to 0% for latest master for the same test. I'll see about getting some reliability numbers.