Skip to content

Commit 955593c

Browse files
committed
Merge #1670: Introduce O(n) canonicalization algorithm
956d0a9 test(chain): Update test docs to stop referencing `get_chain_position` (志宇) 1508ae3 chore: Fix typos (志宇) af92199 refactor(wallet): Reuse chain position instead of obtaining new one (Jiri Jakes) caa0f13 docs(wallet): Explain `.take` usage (志宇) 1196405 refactor(chain): Reorganize `TxGraph::insert_anchor` logic for clarity (志宇) 4706315 chore(chain): Address `CanonicalIter` nitpicks (志宇) 68f7b77 test(chain): Add canonicalization test (志宇) da0c43e refactor(chain)!: Rename `LastSeenIn` to `ObservedIn` (志宇) d4102b4 perf(chain): add benchmarks for canonicalization logic (志宇) e34024c feat(chain): Derive `Clone` on `IndexedTxGraph` (志宇) e985445 docs: Add ADR for `O(n)` canonicalization algorithm (志宇) 4325e2c test(chain): Add transitive anchor tests (志宇) 8fbee12 feat(chain)!: rm `get_chain_position` and associated methods (志宇) 582d6b5 feat(chain)!: `O(n)` canonicalization algorithm (志宇) f6192a6 feat(chain)!: Add `run_until_finished` methods (志宇) 0aa39f9 feat(chain)!: `TxGraph` contain anchors in one field (志宇) Pull request description: Fixes #1665 Replaces #1659 ### Description Previously, getting the canonical history of transactions/UTXOs required calling `TxGraph::get_chain_position` on each transaction. This was highly inefficient and resulted in an `O(n^2)` algorithm. The situation is especially problematic when we have many unconfirmed conflicts. This PR introduces an `O(n)` algorithm to determine the canonical set of transactions in `TxGraph`. The algorithm's premise is as follows: 1. If transaction `A` is determined to be canonical, all of `A`'s ancestors must also be canonical. 2. If transaction `B` is determined to be NOT canonical, all of `B`'s descendants must also be NOT canonical. 3. If a transaction is anchored in the best chain, it is canonical. 4. If a transaction conflicts with a canonical transaction, it is NOT canonical. 5. A transaction with a higher last-seen has precedence. 6. Last-seen values are transitive. A transaction's collective last-seen value is the max of it's last-seen value and all of it's descendants. We maintain two mutually-exclusive `txid` sets: `canoncial` and `not_canonical`. Imagine a method `mark_canonical(A)` that is based on premise 1 and 2. This method will mark transaction `A` and all of it's ancestors as canonical. For each transaction that is marked canonical, we can iterate all of it's conflicts and mark those as `non_canonical`. If a transaction already exists in `canoncial` or `not_canonical`, we can break early, avoiding duplicate work. This algorithm iterates transactions in 3 runs. 1. Iterate over all transactions with anchors in descending anchor-height order. For any transaction that has an anchor pointing to the best chain, we call `mark_canonical` on it. We iterate in descending-height order to reduce the number of anchors we need to check against the `ChainOracle` (premise 1). The purpose of this run is to populate `non_canonical` with all transactions that directly conflict with anchored transactions and populate `canonical` with all anchored transactions and ancestors of anchors transactions (transitive anchors). 2. Iterate over all transactions with last-seen values, in descending last-seen order. We can call `mark_canonical` on all of these that do not already exist in `canonical` or `not_canonical`. 3. Iterate over remaining transactions that contains anchors (but not in the best chain) and have no last-seen value. We treat these transactions in the same way as we do in run 2. #### Benchmarks Thank you to @ValuedMammal for working on this. ```sh $ cargo bench -p bdk_chain --bench canonicalization ``` Benchmark results (this PR): ``` many_conflicting_unconfirmed::list_canonical_txs time: [709.46 us 710.36 us 711.35 us] many_conflicting_unconfirmed::filter_chain_txouts time: [712.59 us 713.23 us 713.90 us] many_conflicting_unconfirmed::filter_chain_unspents time: [709.95 us 711.16 us 712.45 us] many_chained_unconfirmed::list_canonical_txs time: [2.2604 ms 2.2641 ms 2.2680 ms] many_chained_unconfirmed::filter_chain_txouts time: [3.5763 ms 3.5869 ms 3.5979 ms] many_chained_unconfirmed::filter_chain_unspents time: [3.5540 ms 3.5596 ms 3.5652 ms] nested_conflicts_unconfirmed::list_canonical_txs time: [660.06 us 661.75 us 663.60 us] nested_conflicts_unconfirmed::filter_chain_txouts time: [650.15 us 651.36 us 652.71 us] nested_conflicts_unconfirmed::filter_chain_unspents time: [658.37 us 661.54 us 664.81 us] ``` Benchmark results (master): https://github.com/evanlinjin/bdk/tree/fix/1665-master-bench ``` many_conflicting_unconfirmed::list_canonical_txs time: [94.618 ms 94.966 ms 95.338 ms] many_conflicting_unconfirmed::filter_chain_txouts time: [159.31 ms 159.76 ms 160.22 ms] many_conflicting_unconfirmed::filter_chain_unspents time: [163.29 ms 163.61 ms 163.96 ms] # I gave up running the rest of the benchmarks since they were taking too long. ``` ### Notes to the reviewers * ***PLEASE MERGE #1733 BEFORE THIS PR!*** We had to change the signature of `ChainPosition` to account for transitive anchors and unconfirmed transactions with no `last-seen` value. * The canonicalization algorithm is contained in `/crates/chain/src/canonical_iter.rs`. * Since the algorithm requires traversing transactions ordered by anchor height, and then last-seen values, we introduce two index fields in `TxGraph`; `txs_by_anchor` and `txs_by_last_seen`. Methods `insert_anchor` and `insert_seen_at` are changed to populate these index fields. * An ADR is added: `docs/adr/0003_canonicalization_algorithm.md`. This is based on the work in #1592. ### Changelog notice * Added: Introduce an `O(n)` canonicalization algorithm. This logic is contained in `/crates/chain/src/canonical_iter.rs`. * Added: Indexing fields in `TxGraph`; `txs_by_anchor_height` and `txs_by_last_seen`. Pre-indexing allows us to construct the canonical history more efficiently. * Removed: `TxGraph` methods: `try_get_chain_position` and `get_chain_position`. This is superseded by the new canonicalization algorithm. ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing #### New Features: * [x] I've added tests for the new feature * [x] I've added docs for the new feature #### Bugfixes: * [x] This pull request breaks the existing API * [x] I've added tests to reproduce the issue which are now passing * [x] I'm linking the issue being fixed by this PR ACKs for top commit: ValuedMammal: ACK 956d0a9 nymius: ACK 956d0a9 oleonardolima: utACK 956d0a9 jirijakes: ACK 956d0a9 Tree-SHA512: 44963224abf1aefb3510c59d0eb27e3a572cd16f46106fd92e8da2e6e12f0671dcc1cd5ffdc4cc80683bc9e89fa990eba044d9c64d9ce02abc29a08f4859b69e
2 parents ab08b8c + 956d0a9 commit 955593c

File tree

18 files changed

+1242
-606
lines changed

18 files changed

+1242
-606
lines changed

.github/workflows/cont_integration.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,8 @@ jobs:
5151
cargo update -p tokio-util --precise "0.7.11"
5252
cargo update -p indexmap --precise "2.5.0"
5353
cargo update -p security-framework-sys --precise "2.11.1"
54+
cargo update -p csv --precise "1.3.0"
55+
cargo update -p unicode-width --precise "0.1.13"
5456
- name: Build
5557
run: cargo build --workspace --exclude 'example_*' ${{ matrix.features }}
5658
- name: Test

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ cargo update -p tokio --precise "1.38.1"
7878
cargo update -p tokio-util --precise "0.7.11"
7979
cargo update -p indexmap --precise "2.5.0"
8080
cargo update -p security-framework-sys --precise "2.11.1"
81+
cargo update -p csv --precise "1.3.0"
82+
cargo update -p unicode-width --precise "0.1.13"
8183
```
8284

8385
## License

crates/bitcoind_rpc/tests/test_emitter.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -389,6 +389,7 @@ fn tx_can_become_unconfirmed_after_reorg() -> anyhow::Result<()> {
389389
assert_eq!(
390390
get_balance(&recv_chain, &recv_graph)?,
391391
Balance {
392+
trusted_pending: SEND_AMOUNT * reorg_count as u64,
392393
confirmed: SEND_AMOUNT * (ADDITIONAL_COUNT - reorg_count) as u64,
393394
..Balance::default()
394395
},

crates/chain/Cargo.toml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,15 @@ rusqlite = { version = "0.31.0", features = ["bundled"], optional = true }
2828
rand = "0.8"
2929
proptest = "1.2.0"
3030
bdk_testenv = { path = "../testenv", default-features = false }
31-
31+
criterion = { version = "0.2" }
3232

3333
[features]
3434
default = ["std", "miniscript"]
3535
std = ["bitcoin/std", "miniscript?/std", "bdk_core/std"]
3636
serde = ["dep:serde", "bitcoin/serde", "miniscript?/serde", "bdk_core/serde"]
3737
hashbrown = ["bdk_core/hashbrown"]
3838
rusqlite = ["std", "dep:rusqlite", "serde"]
39+
40+
[[bench]]
41+
name = "canonicalization"
42+
harness = false
Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
use bdk_chain::{keychain_txout::KeychainTxOutIndex, local_chain::LocalChain, IndexedTxGraph};
2+
use bdk_core::{BlockId, CheckPoint};
3+
use bdk_core::{ConfirmationBlockTime, TxUpdate};
4+
use bdk_testenv::hash;
5+
use bitcoin::{
6+
absolute, constants, hashes::Hash, key::Secp256k1, transaction, Amount, BlockHash, Network,
7+
OutPoint, ScriptBuf, Transaction, TxIn, TxOut,
8+
};
9+
use criterion::{black_box, criterion_group, criterion_main, Criterion};
10+
use miniscript::{Descriptor, DescriptorPublicKey};
11+
use std::sync::Arc;
12+
13+
type Keychain = ();
14+
type KeychainTxGraph = IndexedTxGraph<ConfirmationBlockTime, KeychainTxOutIndex<Keychain>>;
15+
16+
/// New tx guaranteed to have at least one output
17+
fn new_tx(lt: u32) -> Transaction {
18+
Transaction {
19+
version: transaction::Version::TWO,
20+
lock_time: absolute::LockTime::from_consensus(lt),
21+
input: vec![],
22+
output: vec![TxOut::NULL],
23+
}
24+
}
25+
26+
fn spk_at_index(txout_index: &KeychainTxOutIndex<Keychain>, index: u32) -> ScriptBuf {
27+
txout_index
28+
.get_descriptor(())
29+
.unwrap()
30+
.at_derivation_index(index)
31+
.unwrap()
32+
.script_pubkey()
33+
}
34+
35+
fn genesis_block_id() -> BlockId {
36+
BlockId {
37+
height: 0,
38+
hash: constants::genesis_block(Network::Regtest).block_hash(),
39+
}
40+
}
41+
42+
fn tip_block_id() -> BlockId {
43+
BlockId {
44+
height: 100,
45+
hash: BlockHash::all_zeros(),
46+
}
47+
}
48+
49+
/// Add ancestor tx confirmed at `block_id` with `locktime` (used for uniqueness).
50+
/// The transaction always pays 1 BTC to SPK 0.
51+
fn add_ancestor_tx(graph: &mut KeychainTxGraph, block_id: BlockId, locktime: u32) -> OutPoint {
52+
let spk_0 = spk_at_index(&graph.index, 0);
53+
let tx = Transaction {
54+
input: vec![TxIn {
55+
previous_output: OutPoint::new(hash!("bogus"), locktime),
56+
..Default::default()
57+
}],
58+
output: vec![TxOut {
59+
value: Amount::ONE_BTC,
60+
script_pubkey: spk_0,
61+
}],
62+
..new_tx(locktime)
63+
};
64+
let txid = tx.compute_txid();
65+
let _ = graph.insert_tx(tx);
66+
let _ = graph.insert_anchor(
67+
txid,
68+
ConfirmationBlockTime {
69+
block_id,
70+
confirmation_time: 100,
71+
},
72+
);
73+
OutPoint { txid, vout: 0 }
74+
}
75+
76+
fn setup<F: Fn(&mut KeychainTxGraph, &LocalChain)>(f: F) -> (KeychainTxGraph, LocalChain) {
77+
const DESC: &str = "tr([ab28dc00/86h/1h/0h]tpubDCdDtzAMZZrkwKBxwNcGCqe4FRydeD9rfMisoi7qLdraG79YohRfPW4YgdKQhpgASdvh612xXNY5xYzoqnyCgPbkpK4LSVcH5Xv4cK7johH/0/*)";
78+
let cp = CheckPoint::from_block_ids([genesis_block_id(), tip_block_id()])
79+
.expect("blocks must be chronological");
80+
let chain = LocalChain::from_tip(cp).unwrap();
81+
82+
let (desc, _) =
83+
<Descriptor<DescriptorPublicKey>>::parse_descriptor(&Secp256k1::new(), DESC).unwrap();
84+
let mut index = KeychainTxOutIndex::new(10);
85+
index.insert_descriptor((), desc).unwrap();
86+
let mut tx_graph = KeychainTxGraph::new(index);
87+
88+
f(&mut tx_graph, &chain);
89+
(tx_graph, chain)
90+
}
91+
92+
fn run_list_canonical_txs(tx_graph: &KeychainTxGraph, chain: &LocalChain, exp_txs: usize) {
93+
let txs = tx_graph
94+
.graph()
95+
.list_canonical_txs(chain, chain.tip().block_id());
96+
assert_eq!(txs.count(), exp_txs);
97+
}
98+
99+
fn run_filter_chain_txouts(tx_graph: &KeychainTxGraph, chain: &LocalChain, exp_txos: usize) {
100+
let utxos = tx_graph.graph().filter_chain_txouts(
101+
chain,
102+
chain.tip().block_id(),
103+
tx_graph.index.outpoints().clone(),
104+
);
105+
assert_eq!(utxos.count(), exp_txos);
106+
}
107+
108+
fn run_filter_chain_unspents(tx_graph: &KeychainTxGraph, chain: &LocalChain, exp_utxos: usize) {
109+
let utxos = tx_graph.graph().filter_chain_unspents(
110+
chain,
111+
chain.tip().block_id(),
112+
tx_graph.index.outpoints().clone(),
113+
);
114+
assert_eq!(utxos.count(), exp_utxos);
115+
}
116+
117+
pub fn many_conflicting_unconfirmed(c: &mut Criterion) {
118+
const CONFLICTING_TX_COUNT: u32 = 2100;
119+
let (tx_graph, chain) = black_box(setup(|tx_graph, _chain| {
120+
let previous_output = add_ancestor_tx(tx_graph, tip_block_id(), 0);
121+
// Create conflicting txs that spend from `previous_output`.
122+
let spk_1 = spk_at_index(&tx_graph.index, 1);
123+
for i in 1..=CONFLICTING_TX_COUNT {
124+
let tx = Transaction {
125+
input: vec![TxIn {
126+
previous_output,
127+
..Default::default()
128+
}],
129+
output: vec![TxOut {
130+
value: Amount::ONE_BTC - Amount::from_sat(i as u64 * 10),
131+
script_pubkey: spk_1.clone(),
132+
}],
133+
..new_tx(i)
134+
};
135+
let update = TxUpdate {
136+
txs: vec![Arc::new(tx)],
137+
..Default::default()
138+
};
139+
let _ = tx_graph.apply_update_at(update, Some(i as u64));
140+
}
141+
}));
142+
c.bench_function("many_conflicting_unconfirmed::list_canonical_txs", {
143+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
144+
move |b| b.iter(|| run_list_canonical_txs(&tx_graph, &chain, 2))
145+
});
146+
c.bench_function("many_conflicting_unconfirmed::filter_chain_txouts", {
147+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
148+
move |b| b.iter(|| run_filter_chain_txouts(&tx_graph, &chain, 2))
149+
});
150+
c.bench_function("many_conflicting_unconfirmed::filter_chain_unspents", {
151+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
152+
move |b| b.iter(|| run_filter_chain_unspents(&tx_graph, &chain, 1))
153+
});
154+
}
155+
156+
pub fn many_chained_unconfirmed(c: &mut Criterion) {
157+
const TX_CHAIN_COUNT: u32 = 2100;
158+
let (tx_graph, chain) = black_box(setup(|tx_graph, _chain| {
159+
let mut previous_output = add_ancestor_tx(tx_graph, tip_block_id(), 0);
160+
// Create a chain of unconfirmed txs where each subsequent tx spends the output of the
161+
// previous one.
162+
for i in 0..TX_CHAIN_COUNT {
163+
// Create tx.
164+
let tx = Transaction {
165+
input: vec![TxIn {
166+
previous_output,
167+
..Default::default()
168+
}],
169+
..new_tx(i)
170+
};
171+
let txid = tx.compute_txid();
172+
let update = TxUpdate {
173+
txs: vec![Arc::new(tx)],
174+
..Default::default()
175+
};
176+
let _ = tx_graph.apply_update_at(update, Some(i as u64));
177+
// Store the next prevout.
178+
previous_output = OutPoint::new(txid, 0);
179+
}
180+
}));
181+
c.bench_function("many_chained_unconfirmed::list_canonical_txs", {
182+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
183+
move |b| b.iter(|| run_list_canonical_txs(&tx_graph, &chain, 2101))
184+
});
185+
c.bench_function("many_chained_unconfirmed::filter_chain_txouts", {
186+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
187+
move |b| b.iter(|| run_filter_chain_txouts(&tx_graph, &chain, 1))
188+
});
189+
c.bench_function("many_chained_unconfirmed::filter_chain_unspents", {
190+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
191+
move |b| b.iter(|| run_filter_chain_unspents(&tx_graph, &chain, 0))
192+
});
193+
}
194+
195+
pub fn nested_conflicts(c: &mut Criterion) {
196+
const CONFLICTS_PER_OUTPUT: usize = 3;
197+
const GRAPH_DEPTH: usize = 7;
198+
let (tx_graph, chain) = black_box(setup(|tx_graph, _chain| {
199+
let mut prev_ops = core::iter::once(add_ancestor_tx(tx_graph, tip_block_id(), 0))
200+
.collect::<Vec<OutPoint>>();
201+
for depth in 1..GRAPH_DEPTH {
202+
for previous_output in core::mem::take(&mut prev_ops) {
203+
for conflict_i in 1..=CONFLICTS_PER_OUTPUT {
204+
let mut last_seen = depth * conflict_i;
205+
if last_seen % 2 == 0 {
206+
last_seen /= 2;
207+
}
208+
let ((_, script_pubkey), _) = tx_graph.index.next_unused_spk(()).unwrap();
209+
let value =
210+
Amount::ONE_BTC - Amount::from_sat(depth as u64 * 200 - conflict_i as u64);
211+
let tx = Transaction {
212+
input: vec![TxIn {
213+
previous_output,
214+
..Default::default()
215+
}],
216+
output: vec![TxOut {
217+
value,
218+
script_pubkey,
219+
}],
220+
..new_tx(conflict_i as _)
221+
};
222+
let txid = tx.compute_txid();
223+
prev_ops.push(OutPoint::new(txid, 0));
224+
let _ = tx_graph.insert_seen_at(txid, last_seen as _);
225+
let _ = tx_graph.insert_tx(tx);
226+
}
227+
}
228+
}
229+
}));
230+
c.bench_function("nested_conflicts_unconfirmed::list_canonical_txs", {
231+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
232+
move |b| b.iter(|| run_list_canonical_txs(&tx_graph, &chain, GRAPH_DEPTH))
233+
});
234+
c.bench_function("nested_conflicts_unconfirmed::filter_chain_txouts", {
235+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
236+
move |b| b.iter(|| run_filter_chain_txouts(&tx_graph, &chain, GRAPH_DEPTH))
237+
});
238+
c.bench_function("nested_conflicts_unconfirmed::filter_chain_unspents", {
239+
let (tx_graph, chain) = (tx_graph.clone(), chain.clone());
240+
move |b| b.iter(|| run_filter_chain_unspents(&tx_graph, &chain, 1))
241+
});
242+
}
243+
244+
criterion_group!(
245+
benches,
246+
many_conflicting_unconfirmed,
247+
many_chained_unconfirmed,
248+
nested_conflicts,
249+
);
250+
criterion_main!(benches);

0 commit comments

Comments
 (0)