REDLINE: A MagicBlock Validator Benchmarking Tool

Meet the REDLINE, a high-performance benchmarking tool specifically designed for load-testing MagicBlock validators. REDLINE was developed to analyze and ensure the performance of our validator infrastructure under heavy, real-world conditions. We are sharing it with the community in the hope that it will be a valuable resource for other developers and validator operators.

With REDLINE, you can simulate high-throughput scenarios, identify performance bottlenecks, and gain a comprehensive understanding of your validator's operational limits. It offers a suite of flexible configuration options and a variety of benchmark modes to rigorously test every aspect of your validator's performance.

Note on Compatibility: While REDLINE is optimized for MagicBlock validators, its RPC-based benchmark modes can be used to test any validator that conforms to the official Solana RPC-API specification.

---

Features

Here are some of the key features of REDLINE:

• Unified Benchmark Runner: Run both TPS (Transactions Per Second) and RPS (Requests Per Second) benchmarks, and even mix them together in the same run.
• Multi-threaded Execution: Simulate realistic, high-load conditions by running multiple benchmark instances in parallel.
• Per-Request Statistics: Get detailed statistics on the performance of each benchmark mode, including latency and throughput metrics.
• Flexible Benchmark Modes: Test different aspects of your validator's performance with a variety of built-in benchmark modes.
• Customizable Configuration: Use a simple TOML file to configure every aspect of the benchmark, from connection settings to workload mix.
• Comprehensive Reporting: Generate detailed, human-readable reports from your benchmark results, and compare performance between runs.
• Minimal Resource Footprint: REDLINE is designed to run locally alongside your validator without significantly skewing the results.
• Accurate Measurements: REDLINE is engineered to provide precise measurements of latencies and throughput for in-depth performance analysis.

---

Getting Started

Here’s how to get up and running with REDLINE:

1. Clone the Repository

git clone https://github.com/magicblock-labs/redline.git
cd redline

2. Build the Binaries

make build

This will build both the redline and redline-assist binaries in release mode and place them in the target/release directory.

3. Prepare the Benchmark

Before running a benchmark, you'll need to create and fund the necessary accounts.

make prepare CONFIG=config.example.toml

This command uses the redline-assist tool to get all the on-chain accounts ready for the benchmark based on your configuration file.

4. Run the Benchmark

Now you're ready to run the benchmark.

make bench CONFIG=config.example.toml

This will start the benchmark with the parameters specified in your configuration file.

---

Usage

REDLINE includes two binaries: redline for running the benchmark and redline-assist for helper utilities. The makefile provides a convenient interface for common operations.

Command	Description
make build	Compiles the redline and redline-assist binaries in release mode.
make prepare	Prepares the environment for a benchmark run by creating and funding the necessary accounts, using the specified CONFIG file.
make bench	Runs the benchmark with the configuration from the specified CONFIG file. Results are saved as a timestamped JSON file in the runs/ directory.
make report	Generates a detailed, human-readable report from the latest benchmark results file.
make bench-report	A convenience command that first runs the benchmark and then immediately generates a report.
make compare	Compares the results of the two most recent benchmark runs and highlights performance regressions or improvements. You can customize the SENSITIVITY of the regression detection (default is 15%).
make bench-compare	Runs a new benchmark and then compares its results with the previous run.
make clean	Deletes the latest benchmark result file from the runs/ directory.
make clean-all	Deletes the entire runs/ directory, removing all benchmark result files.

---

Configuration

REDLINE uses a TOML file for configuration. Here's an overview of the available options:

# The number of parallel threads to run the benchmark on.
parallelism = 1

# Number of payers/transaction signers
payers = 8

# Path to keypairs
keypairs = "keypairs/"

# The pubkey of the validator (ER) identity, which
# is used as a target for account delegation
authority = "mAGicPQYBMvcYveUZA5F5UNNwyHvfYh5xkLS2Fr1mev"

[connection]
# The URL of the main chain node.
chain-url = "http://api.devnet.solana.com"
# The URL of the ephemeral node.
ephem-url = "http://127.0.0.1:8899"
# The type of HTTP connection to use.
# Options: "http1" or "http2"
http-connection-type = "http2"
# The maximum number of HTTP connections.
http-connections-count = 16
# The maximum number of WebSocket connections.
ws-connections-count = 16

[benchmark]
# The total number of iterations.
iterations = 100000
# The target rate of requests or transactions per second.
rate = 3000
# The number of concurrent tasks.
concurrency = 64
# The frequency, in milliseconds, at which account cloning should be triggered.
clone-frequency-ms = 1000
# Whether to perform a preflight check for transactions.
preflight-check = false
# The number of accounts to use for the benchmark.
accounts-count = 8
# The benchmark mode to run.
mode = { mixed = [
    { mode = { high-cu-cost = { iters = 23 } }, weight = 50 },
    { mode = { simple-byte-set = {} }, weight = 29 },
    { mode = { read-write = {} }, weight = 20 },
    { mode = { commit = { accounts-per-transaction = 2 } }, weight = 1 },
] }

[confirmations]
# Whether to subscribe to account notifications.
subscribe-to-accounts = true
# Whether to subscribe to signature notifications.
subscribe-to-signatures = true
# Whether to use `getSignatureStatuses` for confirmations.
get-signature-status = false
# Whether to enforce total synchronization for confirmations.
enforce-total-sync = true

[data]
# The encoding for account data.
# Options: "base58", "base64", "base64+zstd"
account-encoding = "base64+zstd"
# The size of the accounts.
# Options: "bytes128", "bytes512", "bytes2048", "bytes8192"
account-size = "bytes128"

---

Conclusion

We hope REDLINE proves to be a useful tool for testing your validator infrastructure. If you have any questions or feedback, please feel free to open an issue or pull request on our GitHub repository.