Design and implement a way to benchmark the TPS throughput of a Miden network.
This includes designing the framework as well as the infrastructure orchestration required to perform such testing easily on-demand.
imo it makes sense to decouple the benchmark inputs from the network at the logical RPC boundary. That is, the benchmark should just be pointed at the network's RPC url, and 🚀.
Bottleneck
Generating transaction proofs will be the main bottleneck we face. On the surface of it, there are two possible solutions:
- Configure the network to ignore verification failures
- Pre-generate all proofs in advance
Both have pros and cons, ideal from a benchmark perspective would be (2).
Integrity
We can have very fast processing if the network simply drop all transactions. This can happen if transactions are reverted (expected). The bencher will need to detect and handle this.
A final summary should include a scan of the network to determine #transactions which didn't make it.
Scenarios
We should create common situations and scenarios which can be added to the benchmark list. Most benchmark runners include this sort of thing, we should see if what we want is supported.
Network setup
We can of course benchmark directly on public networks, but we will want to test different configurations. To me this implies some sort of docker-based cluster setup which can be deployed easily.
We'll also want to run this locally perhaps, on a single machine to iron out kinks.
This also complicates data generation. Running on a live network implies that data cannot be replayed. Which is okay I guess, no way around that.
An open question is whether we want to store raw data semi-permanently so we can easily run benchmarks and iterate locally. e.g. store data on S3 which we can fetch and run easily. And regenerate once the data format breaks. This would assume a given network state e.g. a fresh network.
Additional problems are validator keys and signing if we're testing official network snapshots locally, since we probably don't want to access those (?).
Design and implement a way to benchmark the TPS throughput of a Miden network.
This includes designing the framework as well as the infrastructure orchestration required to perform such testing easily on-demand.
imo it makes sense to decouple the benchmark inputs from the network at the logical RPC boundary. That is, the benchmark should just be pointed at the network's RPC url, and 🚀.
Bottleneck
Generating transaction proofs will be the main bottleneck we face. On the surface of it, there are two possible solutions:
Both have pros and cons, ideal from a benchmark perspective would be (2).
Integrity
We can have very fast processing if the network simply drop all transactions. This can happen if transactions are reverted (expected). The bencher will need to detect and handle this.
A final summary should include a scan of the network to determine #transactions which didn't make it.
Scenarios
We should create common situations and scenarios which can be added to the benchmark list. Most benchmark runners include this sort of thing, we should see if what we want is supported.
Network setup
We can of course benchmark directly on public networks, but we will want to test different configurations. To me this implies some sort of docker-based cluster setup which can be deployed easily.
We'll also want to run this locally perhaps, on a single machine to iron out kinks.
This also complicates data generation. Running on a live network implies that data cannot be replayed. Which is okay I guess, no way around that.
An open question is whether we want to store raw data semi-permanently so we can easily run benchmarks and iterate locally. e.g. store data on S3 which we can fetch and run easily. And regenerate once the data format breaks. This would assume a given network state e.g. a fresh network.
Additional problems are validator keys and signing if we're testing official network snapshots locally, since we probably don't want to access those (?).