Eliminate a bunch more recursion; expand the `TreeLike` iterator trait a bit #722

apoelstra · 2024-08-18T23:01:56Z

This PR replaces the recursive derived implementations of PartialEq, PartialOrd, Hash, fmt::Debug and fmt::Display. Along the way it expands the iter::TreeLike trait to make it a bit more useful (adding a Ternary variant and making the existing Nary variant generic as long as you can get a length out of it and index into it).

The new fmt::Debug method works using a new DisplayNode wrapper around Terminal which allows iterating over a script in the same way as it's displayed, treating things like or_i(0, X) as l:X and implementing the c:pk_k aliases and so on. This seems generally useful but this PR does not expose it in the public API because I'd like to make some breaking changes to Terminal down the line and don't want to expand the API.

The new Ord impl orders things alphabetically (assuming it is bug-free at least) by using the same iterator as the display logic. This is both nonrecursive and a more useful ordering for anybody who cares about the exact ordering. On the other hand, if anybody is depending on the existing ordering this will break their code. I would assume not, since the existing ordering is a weird ad-hoc thing based on the order that derive(PartialOrd) happens to use.

Similarly, it adds a Terminal::fragment_name accessor which returns a &'static str representing the fragment name as displayed. This is also not put into the public API.

The block of type parameters used in the debug output of Miniscripts is moved into a fmt::Display impl on Type itself. This is part of the public API.

This PR does not replace the Clone impl, which is currently derived. This impl actually isn't recursive, since it just clones the Arcs in the first layer of the script. Maybe we want to implement this manually and do a "deep copy"? This would be useful for users who have keys with interior mutability or something. I have a followup which does this, but didn't include it because I felt it might be controversial and need its own discussion.

This PR also does not replace the implicit Drop impl, because that's hard to do and I haven't done it. But we're making progress.

It's lazy and takes some extra allocations to use Nary for the andor combinator; introduce a dedicated Ternary variant for this. This also means that Nary will be exclusively used for thresholds, which should give us some more freedom to mess with it.

This somewhat complicates implementation of the trait, but not by much, and eliminates a ton of refcount accesses and increments, and a ton of Arc::clones. As we will see, having an Arc<[T]> is somewhat restrictive in addition to requiring allocations. Later we will want to replace this with an enum that covers multiple kinds of arrays. I was unsure whether to name the new associated type NAry or Nary. It seems more natural to use NAry but the existing Tree::Nary variant uses the lowercase spelling so I stuck with that. (And similarly used nary_ for the method names rather than n_ary.)

This uses the new iterator extensions to implement a `DisplayNode` wrapper around a `Terminal` which understands that pubkeys, threshold k values, etc., count as "children" for display purposes. We can then embed all the alias/wrapper logic in the `as_node` method and the new `fragment_name` method. The latter is generally useful and ought to be public; given a fragment, it outputs its name, without recursing but with consideration of the aliasing rules. Previously this logic was embedded in the Terminal::condition_fmt method. The resulting display algorithm is much easier to follow, although it uses more lines of code (primarily in the form of large repetitive match statements). cargo bloat shows there is a slight reduction in code size attributed to the miniscript crate, though a slight increase in the total code size for an example binary which basically implements the `string_rtt` test. I think with some effort we should be able to reduce the code size of this algorithm. The important thing though is that it's not recursive. The exact results are as follows (though they have limited use without my committing the actual example program, which is a little ugly and not generally useful). Before: File .text Size Crate Name 0.0% 0.9% 7.3KiB miniscript miniscript::miniscript::astelem::<impl miniscript::expression::FromTree for miniscript::miniscript::decode::Terminal<Pk,Ctx>>::from_tree 0.0% 0.7% 5.8KiB miniscript miniscript::miniscript::types::Type::type_check 0.0% 0.6% 4.4KiB miniscript miniscript::miniscript::types::extra_props::ExtData::threshold 0.0% 0.5% 4.0KiB miniscript <miniscript::Error as core::fmt::Display>::fmt 0.0% 0.5% 4.0KiB miniscript miniscript::miniscript::astelem::<impl miniscript::miniscript::decode::Terminal<Pk,Ctx>>::conditional_fmt 0.0% 0.5% 3.7KiB miniscript miniscript::miniscript::wrap_into_miniscript 0.0% 0.4% 3.0KiB miniscript miniscript::miniscript::types::extra_props::ExtData::or_i 0.0% 0.4% 3.0KiB miniscript <miniscript::miniscript::decode::Terminal<Pk,Ctx> as core::cmp::PartialEq>::eq 0.0% 0.3% 2.8KiB miniscript <miniscript::miniscript::types::Error as core::fmt::Display>::fmt 0.0% 0.3% 2.5KiB miniscript miniscript::expression::Tree::from_slice_delim 0.0% 0.3% 2.4KiB miniscript miniscript::miniscript::types::extra_props::ExtData::type_check 0.0% 0.3% 2.4KiB miniscript alloc::collections::btree::node::BalancingContext<K,V>::bulk_steal_left 0.0% 0.3% 2.2KiB miniscript miniscript::miniscript::split_expression_name 0.0% 0.3% 2.2KiB miniscript <miniscript::Error as core::fmt::Debug>::fmt 0.0% 0.3% 2.1KiB miniscript miniscript::miniscript::types::extra_props::ExtData::and_or 0.0% 0.2% 1.9KiB miniscript miniscript::miniscript::types::extra_props::ExtData::or_b 0.0% 0.2% 1.9KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.7% 33.0% 262.1KiB And 1454 smaller methods. Use -n N to show more. 0.9% 40.7% 323.2KiB filtered data size, the file size is 34.8MiB After: File .text Size Crate Name 0.0% 0.9% 7.3KiB miniscript miniscript::miniscript::astelem::<impl miniscript::expression::FromTree for miniscript::miniscript::decode::Terminal<Pk,Ctx>>::from_tree 0.0% 0.7% 5.8KiB miniscript miniscript::miniscript::types::Type::type_check 0.0% 0.7% 5.7KiB miniscript miniscript::miniscript::display::<impl miniscript::miniscript::decode::Terminal<Pk,Ctx>>::conditional_fmt 0.0% 0.6% 4.4KiB miniscript miniscript::miniscript::types::extra_props::ExtData::threshold 0.0% 0.5% 4.3KiB miniscript <miniscript::miniscript::display::DisplayNode<Pk,Ctx> as miniscript::iter::tree::TreeLike>::as_node 0.0% 0.5% 4.0KiB miniscript <miniscript::Error as core::fmt::Display>::fmt 0.0% 0.5% 3.7KiB miniscript miniscript::miniscript::wrap_into_miniscript 0.0% 0.4% 3.0KiB miniscript miniscript::miniscript::types::extra_props::ExtData::or_i 0.0% 0.3% 2.8KiB miniscript <miniscript::miniscript::types::Error as core::fmt::Display>::fmt 0.0% 0.3% 2.5KiB miniscript miniscript::expression::Tree::from_slice_delim 0.0% 0.3% 2.4KiB miniscript miniscript::miniscript::types::extra_props::ExtData::type_check 0.0% 0.3% 2.4KiB miniscript alloc::collections::btree::node::BalancingContext<K,V>::bulk_steal_left 0.0% 0.3% 2.2KiB miniscript miniscript::miniscript::split_expression_name 0.0% 0.3% 2.2KiB miniscript <miniscript::Error as core::fmt::Debug>::fmt 0.0% 0.3% 2.1KiB miniscript miniscript::miniscript::types::extra_props::ExtData::and_or 0.0% 0.2% 1.9KiB miniscript miniscript::miniscript::types::extra_props::ExtData::or_b 0.0% 0.2% 1.9KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.0% 0.2% 1.8KiB miniscript core::slice::sort::merge 0.7% 32.2% 256.3KiB And 1423 smaller methods. Use -n N to show more. 0.9% 40.2% 320.4KiB filtered data size, the file size is 35.2MiB

This implements an ordering based on the display ordering, though a more efficiently than just serializing to strings and comparing those. Empirically this adds a couple kb to the total code size, but not as much as the nonrecursive fmt shrunk it. We expect somewhat larger code to implement such a nontrivial algorithm nonrecursively (and in fact the old algorithm was "wrong" in that it did not implement any coherent ordering). This is a breaking change because it changes the order of Miniscripts.

apoelstra

Successfully ran local tests on d0c0c14.

sanket1729 · 2024-08-18T23:45:31Z

Maybe we want to implement this manually and do a "deep copy"? This would be useful for users who have keys with interior mutability or something. I have a followup which does this, but didn't include it because I felt it might be controversial and need its own discussion.

I don't see any harm in having a "deep copy". Or we can wait for someone to complain or explicitly request that feature. I can forsee users using keys with interior mutability when scanning funds from descriptors keys containing wildcards.

sanket1729

ACK d0c0c14.

sanket1729 · 2024-08-19T00:21:25Z

src/miniscript/display.rs

+    /// Display no types.
+    None,
+    /// Display all types, including the initial type.
+    All(Type),


nit:

In cccaaec, why do we need Type argument here? We are always calling this with self.ty?

@sanket1729 yes, but the self of self.ty is a Miniscript while the self of conditional_fmt is a Terminal. So we do need to pass in self.ty explicitly like this. It's not available from conditional_fmt otherwise.

Ah, makes sense

apoelstra · 2024-08-19T02:53:09Z

I don't see any harm in having a "deep copy". Or we can wait for someone to complain or explicitly request that feature. I can forsee users using keys with interior mutability when scanning funds from descriptors keys containing wildcards.

Nice. I'll PR it since I've already got the code written.

We are incorrectly serializing and_n as "and_b". In master this is fixed by rust-bitcoin#722 which rewrites the Display impl completely.

When fuzzing my new non-recursive tree parsing logic, I noticed that we were deviating from the released 12.0 in the way that we display l:0. This is an ambiguous fragment which can be displayed either as l:0 or u:0. In our released code we use u:0 so stick with that. This was unintentially changed as part of rust-bitcoin#722. Change it back. While we are at it add a regression test for rust-bitcoin#735

1259375 miniscript: make display prefer 'u' over 'l' in the fragment l:0 (Andrew Poelstra) 67fdc50 descriptor: reject strings of the form "tr(<key>,)" (Andrew Poelstra) 00cac40 descriptor: add unit test demonstrating sanity-checking behavior in <= 12.x (Andrew Poelstra) Pull request description: This PR has three changes which are mostly unrelated except that they were all found when fuzzing my "rewrite expression parser to be nonrecursive" branch against 12.x. * First is a unit test demonstrating #734. It doesn't fix anything, just tests the current behavior. * Second is a fix for #736 (backported in #735). * Third tweaks the new `Display` code from #722 to change how the ambiguous `l:0`/`u:0` is serialized, to match 12.x. ACKs for top commit: sanket1729: ACK 1259375 Tree-SHA512: 921d65a1efd49bda0f9db488a2854b004e14518f584d832497a9dbc13a845ceec99544375385570c6ac42d4985277e8dcbb3aa8654de93235cf9067ba601f91d

9cedc49 bump version to 12.3.0 (Andrew Poelstra) 9420012 fix: zero sequence check on relative time (Chris Hyunhum Cho) 5147c6f descriptor: don't accept strings of the form tr(<key>,) (Andrew Poelstra) 6401c22 miniscript: fix string serialization of and_n (Andrew Poelstra) Pull request description: We are incorrectly serializing and_n as "and_b", and incorrectly parsing `tr(<key>,)`. In master the `and_b` thing is fixed by #722 which rewrites the Display impl completely and the `tr(<key>,)` thing will be fixed as part of a coming series of PRs to clean up expression parsing. Also includes #740 since it showed up in time. ACKs for top commit: sanket1729: ACK 9cedc49 Tree-SHA512: edd45781db5b712e6013a396f6840ce74a6b32861b58a4df25e457c46c845c225dcde95f6c33db7fa314a1c4d2857515f40608d9c66e9432994cadf3d2f10e78

…ion; expand the `TreeLike` iterator trait a bit d0c0c14ae26e353c950319182fd853ee90b8eb54 miniscript: nonrecursive implementation of PartialEq/Eq/Hash (Andrew Poelstra) 117003f3d1c89129734288153a870abb7e940e45 miniscript: implement PartialOrd/Ord nonrecursively (Andrew Poelstra) cccaaec8bc189f5dcf8212c40d0f752ce90d9063 miniscript: non-recursive Display implementation (Andrew Poelstra) 47bed0c281aaf67cba19eb4b7feda86e464b29cb iter: get rid of Arc::clones and allocations for n-ary nodes (Andrew Poelstra) 67d6ff78ad044ab0048e796e376529b6d2be9191 iter: introduce Ternary variant (Andrew Poelstra) Pull request description: This PR replaces the recursive derived implementations of `PartialEq`, `PartialOrd`, `Hash`, `fmt::Debug` and `fmt::Display`. Along the way it expands the `iter::TreeLike` trait to make it a bit more useful (adding a `Ternary` variant and making the existing `Nary` variant generic as long as you can get a length out of it and index into it). The new `fmt::Debug` method works using a new `DisplayNode` wrapper around `Terminal` which allows iterating over a script in the same way as it's displayed, treating things like `or_i(0, X)` as `l:X` and implementing the `c:pk_k` aliases and so on. This seems generally useful but this PR does not expose it in the public API because I'd like to make some breaking changes to `Terminal` down the line and don't want to expand the API. The new `Ord` impl orders things alphabetically (assuming it is bug-free at least) by using the same iterator as the display logic. This is both nonrecursive and a more useful ordering for anybody who cares about the exact ordering. On the other hand, if anybody is depending on the *existing* ordering this will break their code. I would assume not, since the existing ordering is a weird ad-hoc thing based on the order that `derive(PartialOrd)` happens to use. Similarly, it adds a `Terminal::fragment_name` accessor which returns a `&'static str` representing the fragment name as displayed. This is also not put into the public API. The block of type parameters used in the debug output of Miniscripts is moved into a `fmt::Display` impl on `Type` itself. This is part of the public API. This PR does **not** replace the `Clone` impl, which is currently derived. This impl actually isn't recursive, since it just clones the `Arc`s in the first layer of the script. Maybe we want to implement this manually and do a "deep copy"? This would be useful for users who have keys with interior mutability or something. I have a followup which does this, but didn't include it because I felt it might be controversial and need its own discussion. This PR also does **not** replace the implicit `Drop` impl, because that's hard to do and I haven't done it. But we're making progress. ACKs for top commit: sanket1729: ACK d0c0c14ae26e353c950319182fd853ee90b8eb54. Tree-SHA512: edc42b5ed7d2b562e93d3c061328c2020f710534a5cad3878b849d900d697be4919232e1f1c178fdc5338bd2e6ef71ebcfed90b2b8b8e577d8f781d887771bc1

apoelstra added 5 commits August 18, 2024 16:18

iter: introduce Ternary variant

67d6ff7

It's lazy and takes some extra allocations to use Nary for the andor combinator; introduce a dedicated Ternary variant for this. This also means that Nary will be exclusively used for thresholds, which should give us some more freedom to mess with it.

miniscript: nonrecursive implementation of PartialEq/Eq/Hash

d0c0c14

apoelstra force-pushed the 2024-08--recursion branch from 1b53dec to d0c0c14 Compare August 18, 2024 23:04

apoelstra commented Aug 18, 2024

View reviewed changes

sanket1729 approved these changes Aug 19, 2024

View reviewed changes

apoelstra merged commit d67fb00 into rust-bitcoin:master Aug 19, 2024

apoelstra deleted the 2024-08--recursion branch August 19, 2024 03:07

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Aug 31, 2024

miniscript: fix string serialization of and_n

98ea9df

We are incorrectly serializing and_n as "and_b". In master this is fixed by rust-bitcoin#722 which rewrites the Display impl completely.

apoelstra mentioned this pull request Aug 31, 2024

[backport] Several small fixes for 12.x #735

Merged

apoelstra added a commit to apoelstra/rust-miniscript that referenced this pull request Sep 1, 2024

miniscript: fix string serialization of and_n

6401c22

We are incorrectly serializing and_n as "and_b". In master this is fixed by rust-bitcoin#722 which rewrites the Display impl completely.

apoelstra mentioned this pull request Sep 1, 2024

Misc compatibility-with-12.x things #737

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Eliminate a bunch more recursion; expand the `TreeLike` iterator trait a bit #722

Eliminate a bunch more recursion; expand the `TreeLike` iterator trait a bit #722

Uh oh!

apoelstra commented Aug 18, 2024

Uh oh!

apoelstra left a comment

Uh oh!

sanket1729 commented Aug 18, 2024 •

edited

Loading

Uh oh!

sanket1729 left a comment •

edited

Loading

Uh oh!

sanket1729 Aug 19, 2024

Uh oh!

apoelstra Aug 19, 2024

Uh oh!

sanket1729 Aug 19, 2024

Uh oh!

apoelstra commented Aug 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Eliminate a bunch more recursion; expand the TreeLike iterator trait a bit #722

Eliminate a bunch more recursion; expand the TreeLike iterator trait a bit #722

Uh oh!

Conversation

apoelstra commented Aug 18, 2024

Uh oh!

apoelstra left a comment

Choose a reason for hiding this comment

Uh oh!

sanket1729 commented Aug 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanket1729 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanket1729 Aug 19, 2024

Choose a reason for hiding this comment

Uh oh!

apoelstra Aug 19, 2024

Choose a reason for hiding this comment

Uh oh!

sanket1729 Aug 19, 2024

Choose a reason for hiding this comment

Uh oh!

apoelstra commented Aug 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Eliminate a bunch more recursion; expand the `TreeLike` iterator trait a bit #722

Eliminate a bunch more recursion; expand the `TreeLike` iterator trait a bit #722

sanket1729 commented Aug 18, 2024 •

edited

Loading

sanket1729 left a comment •

edited

Loading