perf: reduce contention on trie cache #4953
Description
We're currently using a RwLock to restrict access to the TrieLayerCache:
pub struct TrieLayerCache {
last_id: usize,
layers: HashMap<H256, TrieLayer>,
}
pub struct Store {
db: Arc<DBWithThreadMode<MultiThreaded>>,
trie_cache: Arc<RwLock<TrieLayerCache>>,
flatkeyvalue_control_tx: std::sync::mpsc::SyncSender<FKVGeneratorControlMessage>,
}This seems to be causing congestion at critical times, as taking the write lock for apply_updates needs to wait for all the read locks to be freed.
I suggest the following approach to keep this minimal, trading off memory and a little CPU for minimal locking:
pub struct TrieLayerCache {
last_id: usize,
layers: HashMap<H256, Arc<TrieLayer>>,
}
pub struct Store {
db: Arc<DBWithThreadMode<MultiThreaded>>,
trie_cache: Arc<Mutex<Arc<TrieLayerCache>>>,
flatkeyvalue_control_tx: std::sync::mpsc::SyncSender<FKVGeneratorControlMessage>,
}As can be seen, it's a little loaded on Arc. The reason is this allows us to:
- Have an immutable clone of
TrieLayerCacheon each openTrie; - Easily swap the main version with a new one on
apply_updates.
So, apply_updates would do something like:
let old_cache = self.trie_cache.lock().clone(); // Clones the Arc
let new_cache = TrieLayerCache::clone_from(&*old_cache); // Locklessly copy the contents, which are themselves just Arcs
new_cache.insert(new_layer);
*self.trie_cache.lock() = new_cache; // Replace the old cache atomically, all clones are still validAnd open_storage_trie et all:
let cache = self.cache.clone(); // Clone the Arc, it's immutable and reference counted so it remains valid
Trie::new(cache, etc...);Locking would now be restricted to pointer swapping, rather than things like trie iteration.
The inner Arc allows for cheap cloning of live layers.
Metadata
Metadata
Assignees
Type
Projects
Status