Payment benchmark #664

joostjager · 2025-10-15T08:47:30Z

This PR adds a new async payment benchmark test to integration_tests_rust.rs. The test sets up two nodes with a channel and measures performance while sending 1,000 concurrent payments. It uses Tokio’s multi-threaded runtime to spawn payments as separate async tasks and tracks completion through LDK events. The benchmark provides a simple way to gauge throughput and identify performance bottlenecks in payment handling.

Related: lightningdevkit/rust-lightning#4161

ldk-reviews-bot · 2025-10-15T08:47:33Z

👋 I see @tnull was un-assigned.
If you'd like another reviewer assignment, please click here.

joostjager · 2025-10-21T13:04:00Z

Converted test to criterion benchmark and added it to CI. Also removed the payment throttling, now that it has been established that there is a bug in lightning-net-tokio that causes this. For the benchmark, the 1000 payments can now simply be launched at the same time.

Criterion reports outliers, so this should be able to detect to occassional 30 sec hangs in the payments test.

Cargo.toml

tests/benchmarks.rs

ldk-reviews-bot · 2025-10-23T13:04:43Z

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-25T13:05:28Z

🔔 2nd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-27T13:06:14Z

🔔 3rd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

To enable more realistic testing with sqlite as a backend.

Introduces a criterion-based benchmark that sends 1000 concurrent payments between two LDK nodes to measure total duration. Also adds a CI job to automatically run the benchmark.

joostjager · 2025-10-28T10:37:23Z

Rebased onto async test framework changes.

tnull · 2025-10-28T13:07:22Z

Mind rebasing once more to fix CI?

tnull

Thanks! The test store changes don't make sense to me, I believe you intended something else there. Otherwise some comments.

tnull · 2025-10-28T13:10:00Z

tests/common/mod.rs

-	let node = builder.build_with_store(test_sync_store).unwrap();
+	let kv_store: Arc<DynStore> = match config.store_type {
+		TestStoreType::InMemory => {
+			Arc::new(TestSyncStore::new(config.node_config.storage_dir_path.into()))


This doesn't do what you think it does. This is TestSyncStore, a store that writes to multiple backends to ensure they are kept in-sync, not the in-memory TestStore.

Ah, I missed that indeed. Renamed InMemory to TestSyncStore. The intention of the commit is to allow a test/benchmark to run with sqlite (only).

.github/workflows/rust.yml

Cargo.toml

tests/benchmarks.rs

tnull · 2025-10-28T13:21:06Z

tests/benchmarks.rs

+
+	// Spawn each payment as a separate async task
+	task::spawn(async move {
+		println!("{}: Starting payment", payment_hash.0.as_hex());


Are we positive all the printing IO doesn't impact the performance? Should we omit this for 'production' benchmarking?

I did wonder the same. Maybe I should indeed remove it to be sure. Would that be a cfg flag?

Could be, but do we really need them in the first place?

tests/benchmarks.rs

tnull · 2025-10-28T13:24:15Z

tests/benchmarks.rs

+			// Pre-check the HTLC slots to try to avoid the performance impact of a failed payment.
+			while node_a.list_channels()[0].next_outbound_htlc_limit_msat == 0 {
+				println!("{}: Waiting for HTLC slots to free up", payment_hash.0.as_hex());
+				tokio::time::sleep(std::time::Duration::from_millis(100)).await;


How did you determine 100ms here? Do we have an intuition if this could starve any of the waiting payers?

It isn't great indeed. Basically something that looked 'reasonable' to me. I think it is a weakness in the API that a user even needs to do this.

tnull · 2025-10-29T09:48:19Z

tests/common/mod.rs

+			Arc::new(TestSyncStore::new(config.node_config.storage_dir_path.into()))
+		},
+		TestStoreType::Sqlite => Arc::new(
+			SqliteStore::new(


Let's just use the default build() here, which defaults to SQLite.

If you prefer to have it explicit, feel free to rename build to build_with_sqlite_store and have a new build method on Builder call it. Then you can explicitly set build_with_sqlite_store here.

tnull · 2025-10-29T09:50:31Z

tests/benchmarks.rs

+			// Pre-check the HTLC slots to try to avoid the performance impact of a failed payment.
+			while node_a.list_channels()[0].next_outbound_htlc_limit_msat == 0 {
+				println!("{}: Waiting for HTLC slots to free up", payment_hash.0.as_hex());
+				tokio::time::sleep(std::time::Duration::from_millis(100)).await;


tnull · 2025-10-29T09:51:00Z

Cargo.toml


 vss-client = "0.3"
 prost = { version = "0.11.6", default-features = false}
+criterion = { version = "0.7.0", features = ["async_tokio"] }


Why is this not a dev-dependency?

joostjager force-pushed the payment-benchmark branch from ee6d232 to 53a16a2 Compare October 15, 2025 08:48

joostjager mentioned this pull request Oct 15, 2025

Retry payment for local failures lightningdevkit/rust-lightning#4161

Open

joostjager added this to Weekly Goals Oct 16, 2025

joostjager self-assigned this Oct 16, 2025

joostjager force-pushed the payment-benchmark branch 4 times, most recently from fffca97 to 058ede1 Compare October 21, 2025 13:01

joostjager marked this pull request as ready for review October 21, 2025 13:04

joostjager requested a review from tnull October 21, 2025 13:04

joostjager force-pushed the payment-benchmark branch from 058ede1 to 6bdb4d0 Compare October 21, 2025 13:07

joostjager commented Oct 21, 2025

View reviewed changes

Cargo.toml Show resolved Hide resolved

tests/benchmarks.rs Show resolved Hide resolved

joostjager removed the request for review from tnull October 28, 2025 09:33

joostjager added 2 commits October 28, 2025 11:07

Configurable test store

07d359b

To enable more realistic testing with sqlite as a backend.

Add async payment throughput benchmark

7a808a5

Introduces a criterion-based benchmark that sends 1000 concurrent payments between two LDK nodes to measure total duration. Also adds a CI job to automatically run the benchmark.

joostjager force-pushed the payment-benchmark branch from 6bdb4d0 to 7a808a5 Compare October 28, 2025 10:36

tnull reviewed Oct 28, 2025

View reviewed changes

joostjager added 2 commits October 28, 2025 15:56

f: rename to TestSyncStore

672dd46

f: separate benchmarks CI job

7771cab

tnull reviewed Oct 29, 2025

View reviewed changes

Uh oh!

Payment benchmark #664

Are you sure you want to change the base?

Payment benchmark #664

Conversation

joostjager commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joostjager commented Oct 21, 2025

Uh oh!

Uh oh!

Uh oh!

ldk-reviews-bot commented Oct 23, 2025

Uh oh!

ldk-reviews-bot commented Oct 25, 2025

Uh oh!

ldk-reviews-bot commented Oct 27, 2025

Uh oh!

joostjager commented Oct 28, 2025

Uh oh!

tnull commented Oct 28, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joostjager commented Oct 15, 2025 •

edited

Loading

ldk-reviews-bot commented Oct 15, 2025 •

edited

Loading