Add "silentpayments" module implementing BIP352 (take 4, limited to full-node scanning)

[theStack]

Description

This PR implements BIP352 with scanning limited to full-nodes. Light-client scanning is planned to be added in a separate PR in the future. The following 5 API functions are currently introduced:

Sender side [BIP description]:

• secp256k1_silentpayments_sender_create_outputs: given a list of $n$ secret keys $a_1 ... a_n$, a serialized outpoint, and a list of recipients (each consisting of silent payments scan pubkey and spend pubkey), create the corresponding transaction outputs (x-only public keys) for the sending transaction

Receiver side, label creation [BIP description]:

• secp256k1_recipient_create_label: given a scan secret key and label integer, calculate the corresponding label_tweak and label public key
• secp256k1_recipient_create_labeled_spend_pubkey: given a spend public key and a label public key, create the corresponding labeled spend public key

Receiver side, scanning [BIP description]:

• secp256k1_recipient_prevouts_summary_create: given a list of $n$ public keys $A_1 ... A_n$ and a serialized outpoint, create a prevouts_summary object needed for scanning
• secp256k1_recipient_scan_outputs: given a prevouts_summary object, a recipients scan secret key and spend public key, and the relevant transaction outputs (x-only public keys), scan for outputs belonging to the recipients and and return the tweak(s) needed for spending the output(s). Optionally, a label_lookup callback function can be provided to also scan for labels.

For a higher-level overview on what these functions exactly do, it's suggested to look at a corresponding Python implementation that was created based on the secp256k1lab project (it passes the test vectors, so this "executable pseudo-code" should be correct).

Changes to the previous take

Based on the latest state of the previous PR #1698 (take 3), the following changes have been made:

• rebased on master (adapting to the new unit test framework added in Introduce (mini) unit test framework #1734; thanks to furszy for his mini-guide)
• removed functionality that is only relevant for light clients (_prevout_summary_{parse,serialize}, __recipient_create_output_pubkeys), adapted tests and benchmark accordingly
• adapted examples to have both Bob and Carol do full-node scanning, with a TODO to switch the latter to light-client scanning once available in the future
• addressed various review comments, if still relevant (Add BIP352 module (take 3) #1698 (comment), Add BIP352 module (take 3) #1698 (comment), Add BIP352 module (take 3) #1698 (comment), Add BIP352 module (take 3) #1698 (comment))

The scope reduction isn't immediately visible in commit count (only one commit was only introducing light-client relevant functionality and could be completely removed), but the review burden compared #1698 is still significantly lower in terms of LOC, especially in the receiving commit.

Open questions / TODOs

• Recent proposals of reducing the worst-case scanning time (see posts by w0xlt and jonasnick, Add BIP352 module (take 3) #1698 (comment) ff.) are not taken into account yet.
• Not providing prevouts_summary (de)serialization functionality yet in the API poses the risk that users try to do it anyway by treating the opaque object as "serialized". How to cope with that? Is adding a "don't do this" comment in API header sufficient?

[w0xlt]

Added the optimized version on top of this PR:
w0xlt@8d16914

For more context:
#1698 (comment)

[theStack]

Small supplementary update: I've created a corresponding Python implementation of the provided API functions based on secp256k1lab (https://github.com/theStack/secp256k1lab/blob/add_bip352_module_review_helper/src/secp256k1lab/bip352.py) (also linked in the PR description). The hope is that this makes reviewing this PR a bit easier by having a less noisy, "executable pseudo-code"-like description on what happens under the hood. The code passes the BIP352 test vectors and hence should be correct.

Added the optimized version on top of this PR: w0xlt@8d16914

For more context: #1698 (comment)

Thanks for rebasing on top of this PR, much appreciated! I will take a closer look within the next days.

[w0xlt]

Nit: Not related to optimization, but the diff below removes some redundant public-key serialization code:

diff --git a/src/modules/silentpayments/main_impl.h b/src/modules/silentpayments/main_impl.h
index 106da20..922433d 100644
--- a/src/modules/silentpayments/main_impl.h
+++ b/src/modules/silentpayments/main_impl.h
@@ -21,6 +21,19 @@
 /** magic bytes for ensuring prevouts_summary objects were initialized correctly. */
 static const unsigned char secp256k1_silentpayments_prevouts_summary_magic[4] = { 0xa7, 0x1c, 0xd3, 0x5e };
 
+/* Serialize a ge to compressed 33 bytes. Keeps eckey_pubkey_serialize usage uniform
+ * (expects non-const ge*), and centralizes the VERIFY_CHECK. */
+static SECP256K1_INLINE void secp256k1_sp_ge_serialize33(const secp256k1_ge* in, unsigned char out33[33]) {
+    size_t len = 33;
+    secp256k1_ge tmp = *in;
+    int ok = secp256k1_eckey_pubkey_serialize(&tmp, out33, &len, 1);
+#ifdef VERIFY
+    VERIFY_CHECK(ok && len == 33);
+#else
+    (void)ok;
+#endif
+}
+
 /** Sort an array of silent payment recipients. This is used to group recipients by scan pubkey to
  *  ensure the correct values of k are used when creating multiple outputs for a recipient.
  *
@@ -68,13 +81,11 @@ static int secp256k1_silentpayments_calculate_input_hash_scalar(secp256k1_scalar
     secp256k1_sha256 hash;
     unsigned char pubkey_sum_ser[33];
     unsigned char input_hash[32];
-    size_t len;
     int ret, overflow;
 
     secp256k1_silentpayments_sha256_init_inputs(&hash);
     secp256k1_sha256_write(&hash, outpoint_smallest36, 36);
-    ret = secp256k1_eckey_pubkey_serialize(pubkey_sum, pubkey_sum_ser, &len, 1);
-    VERIFY_CHECK(ret && len == sizeof(pubkey_sum_ser));
+    secp256k1_sp_ge_serialize33(pubkey_sum, pubkey_sum_ser);
     secp256k1_sha256_write(&hash, pubkey_sum_ser, sizeof(pubkey_sum_ser));
     secp256k1_sha256_finalize(&hash, input_hash);
     /* Convert input_hash to a scalar.
@@ -85,15 +96,13 @@ static int secp256k1_silentpayments_calculate_input_hash_scalar(secp256k1_scalar
      * an error to ensure strict compliance with BIP0352.
      */
     secp256k1_scalar_set_b32(input_hash_scalar, input_hash, &overflow);
-    ret &= !secp256k1_scalar_is_zero(input_hash_scalar);
+    ret = !secp256k1_scalar_is_zero(input_hash_scalar);
     return ret & !overflow;
 }
 
 static void secp256k1_silentpayments_create_shared_secret(const secp256k1_context *ctx, unsigned char *shared_secret33, const secp256k1_ge *public_component, const secp256k1_scalar *secret_component) {
     secp256k1_gej ss_j;
     secp256k1_ge ss;
-    size_t len;
-    int ret;
 
     secp256k1_ecmult_const(&ss_j, public_component, secret_component);
     secp256k1_ge_set_gej(&ss, &ss_j);
@@ -103,12 +112,7 @@ static void secp256k1_silentpayments_create_shared_secret(const secp256k1_contex
      * impossible at this point considering we have already validated the public key and
      * the secret key.
      */
-    ret = secp256k1_eckey_pubkey_serialize(&ss, shared_secret33, &len, 1);
-#ifdef VERIFY
-    VERIFY_CHECK(ret && len == 33);
-#else
-    (void)ret;
-#endif
+    secp256k1_sp_ge_serialize33(&ss, shared_secret33);
 
     /* Leaking these values would break indistinguishability of the transaction, so clear them. */
     secp256k1_ge_clear(&ss);
@@ -585,7 +589,6 @@ int secp256k1_silentpayments_recipient_scan_outputs(
                 secp256k1_ge output_negated_ge, tx_output_ge;
                 secp256k1_gej tx_output_gej, label_gej;
                 unsigned char label33[33];
-                size_t len;
 
                 secp256k1_xonly_pubkey_load(ctx, &tx_output_ge, tx_outputs[j]);
                 secp256k1_gej_set_ge(&tx_output_gej, &tx_output_ge);
@@ -595,7 +598,6 @@ int secp256k1_silentpayments_recipient_scan_outputs(
                 secp256k1_ge_neg(&output_negated_ge, &output_ge);
                 secp256k1_gej_add_ge_var(&label_gej, &tx_output_gej, &output_negated_ge, NULL);
                 secp256k1_ge_set_gej_var(&label_ge, &label_gej);
-                ret = secp256k1_eckey_pubkey_serialize(&label_ge, label33, &len, 1);
                 /* Serialize must succeed because the point was just loaded.
                  *
                  * Note: serialize will also fail if label_ge is the point at infinity, but we know
@@ -603,7 +605,7 @@ int secp256k1_silentpayments_recipient_scan_outputs(
                  * Thus, we know that label_ge = tx_output_gej + output_negated_ge cannot be the
                  * point at infinity.
                  */
-                VERIFY_CHECK(ret && len == 33);
+                secp256k1_sp_ge_serialize33(&label_ge, label33);
                 label_tweak = label_lookup(label33, label_context);
                 if (label_tweak != NULL) {
                     found = 1;
@@ -617,7 +619,6 @@ int secp256k1_silentpayments_recipient_scan_outputs(
                 secp256k1_gej_neg(&label_gej, &tx_output_gej);
                 secp256k1_gej_add_ge_var(&label_gej, &label_gej, &output_negated_ge, NULL);
                 secp256k1_ge_set_gej_var(&label_ge, &label_gej);
-                ret = secp256k1_eckey_pubkey_serialize(&label_ge, label33, &len, 1);
                 /* Serialize must succeed because the point was just loaded.
                  *
                  * Note: serialize will also fail if label_ge is the point at infinity, but we know
@@ -625,7 +626,7 @@ int secp256k1_silentpayments_recipient_scan_outputs(
                  * Thus, we know that label_ge = tx_output_gej + output_negated_ge cannot be the
                  * point at infinity.
                  */
-                VERIFY_CHECK(ret && len == 33);
+                secp256k1_sp_ge_serialize33(&label_ge, label33);
                 label_tweak = label_lookup(label33, label_context);
                 if (label_tweak != NULL) {
                     found = 1;

[w0xlt]

nit: The following diff removes the implicit cast and clarifies that k is 4 bytes

diff --git a/src/modules/silentpayments/main_impl.h b/src/modules/silentpayments/main_impl.h
index 922433d..d94aed6 100644
--- a/src/modules/silentpayments/main_impl.h
+++ b/src/modules/silentpayments/main_impl.h
@@ -512,7 +512,8 @@ int secp256k1_silentpayments_recipient_scan_outputs(
     secp256k1_xonly_pubkey output_xonly;
     unsigned char shared_secret[33];
     const unsigned char *label_tweak = NULL;
-    size_t j, k, found_idx;
+    size_t j, found_idx;
+    uint32_t k;
     int found, combined, valid_scan_key, ret;
 
     /* Sanity check inputs */

[jonasnick]

Thanks @theStack for the new PR. I can confirm that this PR is a rebased version of #1698, with the light client functionality removed and comments addressed, except for:

• #1698 (comment)
• #1698 (comment)
• #1698 (review) (only the last one, "elemement")

[jonasnick]

Not providing prevouts_summary (de)serialization functionality yet in the API poses the risk that users try to do it anyway by treating the opaque object as "serialized". How to cope with that? Is adding a "don't do this" comment in API header sufficient?

Is there a reason for serializing prevouts_summary without light client functionality? If not, I think the don't do this comment is sufficient. Right now, in contrast to the docs of all other opaque objects, this is missing, however:

The exact representation of data inside the opaque data structures is implementation defined and not guaranteed to be portable between different platforms or versions.

[theStack]

@w0xlt, @jonasnick: Thanks for the reviews! I've addressed the suggested changes:

• in _recpient_scan_outputs: changed the type of k to uint32_t (comment above)
• in _recipient_create_label: added a scan key validity check (+added a test for that) (#1698 - comment)
• unified all mentions of "Silent Payments" to title case in the header API and example (#1698 - comment)
• fixed typo s/elemement/element/ (#1698 - review)
• in _recipient_scan_outputs: fixed comment in second label candidate (review above)
• extended the API header comment for the _prevouts_summary opaque data structure, to point out that the data structure is implementation defined (like docs of all other opaque structs) (comment above)

Nit: Not related to optimization, but the diff below removes some redundant public-key serialization code:

Given that this compressed-pubkey-serialization pattern shows up repeatedly also in other modules (ellswift, musig), I think it would make the most sense to add a general helper (e.g. in eckey{,_impl}.h), which could be done in an independent PR. I've opened issue #1773 to see if there is conceptual support for doing this.

Not providing prevouts_summary (de)serialization functionality yet in the API poses the risk that users try to do it anyway by treating the opaque object as "serialized". How to cope with that? Is adding a "don't do this" comment in API header sufficient?

Is there a reason for serializing prevouts_summary without light client functionality? If not, I think the don't do this comment is sufficient.

Good point, I can't think of a good reason for full nodes wanting to serialize prevouts_summary.

[nymius]

To address the open questions, I’ve reviewed the proposed changes by @w0xlt on 8d16914.

I'm going to focus more on the key aspects I extracted from the review and the merits of each change, rather on the big O improvement claims, because I didn't get that far.

These are multiple different changes rather than a single one, so to make the review easier I suggest to brake it in multiple commits. I would state on each of them the purpose and the real case scenario where the change would be relevant.

Also, I would use clearer names for the variables or at least document their purpose.

The changes I've identified so far are the following:

• Improve label lookup using hash table: I think this is implementation dependent and should be improved by the user rather than by the library itself.
  Examples, as part of the documentation, are a usage demonstration, although can point the user the best practices, I prefer clarity rather than performance on them. For example, on the rust-secp256k1 bindings, I used a HashMap for the example because it is a familiar standard structure for Rust users. If they would like to gain more performance there, they have other tools available to replace that structure by themselves.
• On secp256k1_silentpayments_recipient_sort_cmp, I understood the change: (r1->index < r2->index) ? -1 : (r1->index > r2->index) ? 1 : 0 is to make the unstable secp256k1_hsort implementation stable. Considering it only affects secp256k1_silentpayments_sender_create_outputs, which didn't receive more changes than a variable removal, what are the performance improvements there?
• SECP256K1_SP_SCAN_BATCH: I just learned about it (ge_set_gej_all) during this review, but seems that affine to jacobian conversion is needed for the comparison against labels, and is faster to do it in batches. I think the impact of this improvement will only affect the small subset of transaction with large amount of outputs. A good benchmark for this would be coinjoin transactions, although is not supported by BIP 352, a test case is doable.
• Double head: I'm not sure of the target of this change, I guess is to skip already found or scanned inputs, but couldn't figure out what is tracking each head.
• Binary tree search for x-only lookups: I think it explain itself, faster lookups on the not labeled case. This may have its merits, but I need to remove the other changes to have a clear answer.

In general I agree with @jonasnick that we should define a clear target to benchmark and improve. As I've said before, the base case should be a wallet with a single label for change.
For other improvements, I would try to match up plausible real world scenarios before making complex changes in the code base of the PR.
Finally, by looking at bench_silentpayments_full_tx_scan, the use_labels case is very simple. If we want to test performance improvements on the label lookup, I would start there.

In conclusion, from the proposed commit and the discussion around it, the only changes I've found clear enough to consider are:

• Tracking of found outputs: simple enough, small performance improvement for the usual case, better for larger transactions.

[w0xlt]

Thanks @nymius for reviewing the changes, addressing the main points, and proposing a simplification.

I’m currently splitting the optimization commit into smaller pieces to make it easier to review.
I’ll also take a closer look at your commit and run it against the benchmark files.

The only part of the discussion that still feels a bit ambiguous is the “base” or “usual” case.
From my understanding of #1698 (review), the concern is not about typical usage, but rather about an attacker crafting malicious transactions with many outputs, causing the scanning process to take hours.

So the goal of this optimization would be to mitigate that scenario, not the collaborative one.

[w0xlt]

I ran the examples/silentpayments_mixed_1.c file with simplified the version suggested by @nymius jonasnick@311b4eb . It shows slightly worse performance (see below), but the simpler approach may still be worth it

Without the secp256k1_silentpayments_recipient_sort_cmp stabilization, I got 82s for the complex version vs. 114s for the simpler one. Whether the 30s difference justifies the additional complexity is up for discussion — I don’t have a strong opinion.

Answering the questions: secp256k1_silentpayments_recipient_sort_cmp stabilization + heads speed up the non-adversarial case. The stable sort ensures that the transaction outputs are ordered sequentially by the index $k$.

The optimized receiver implementation relies on a heuristic (the head pointers) that assumes the next output it is looking for ($k+1$) is located immediately after the previous one ($k$).

• With Stable Sort: The scanner complexity is roughly O(N) (Linear).
• Without Stable Sort: The scanner complexity degrades to O(N²) (Quadratic).