Improve `GetAccount` performance

[Mirko-von-Leipzig]

GetAccount performance has been shown to be slow during the recent testnet.

There are probably some low hanging fruit and optimisations we can make to improve matters.

One idea is to move most of the data loading away from sqlite and instead fetch it from the account tree itself. If possible, this may also let us remove the historical tracking of account storage within sqlite which is a pain.

Some starting points:

• The account tree doesn't use a persistent backend, but we should use rocksdb. This would also fix our slow start up times.
• Check what data we can take from ^^ instead of sqlite.
• We don't have sqlite migration support, so we may need to punt on the removal of data, but maybe its also a good time to start on this.

[Mirko-von-Leipzig]

This also relates to #1969

[bobbinth]

One idea is to move most of the data loading away from sqlite and instead fetch it from the account tree itself. If possible, this may also let us remove the historical tracking of account storage within sqlite which is a pain.

I think as a general rule we should try to follow the following:

• The "historical data" (i.e., the data that changed in the last ~50 blocks) should live only in memory.
• Most authentication data (Merkle paths) lives in auxiliary databases.

The current AccountStateForest covers most of the above pretty well (and the rest is covered by the AccountTree). I think the bulk of this task would be to clean up AccountStateForest a bit, make it work with the RocksDB backend, and update the GetAccount endpoint to pull most of the data from the AccountStateForest (rather than from SQLite as it does currently).

For the last point, I would love to see if we can get rid of things like select_account_vault_at_block() and reconstruct_storage_map_from_db() as the required data should be in the AccountStateForest - though, maybe not surfaced in the most convenient way.

The account tree doesn't use a persistent backend

Did you mean AccountStateForest here?

[kkovaacs]

For the last point, I would love to see if we can get rid of things like select_account_vault_at_block() and reconstruct_storage_map_from_db() as the required data should be in the AccountStateForest - though, maybe not surfaced in the most convenient way.

Unfortunately that's only partially possible with the current AccountStateForest:

• Vault assets can be reconstructed from AccountStateForest (see e0c33e2)
• Storage map details however cannot be reconstructed from the underlying LargeSmtForest. StorageMapKeys are hashed before being inserted:

  node/crates/store/src/account_state_forest/mod.rs

  Line 477 in f8dcdfe

  raw_map_entries.iter().map(|(raw_key, value)| (raw_key.hash().into(), *value)),

  This means that we cannot recover the original StorageMapKey that we'd need to return in AccountResponse.

[kkovaacs]

Storage map details however cannot be reconstructed from the underlying LargeSmtForest. StorageMapKeys are hashed before being inserted:

For storage maps, we'd need to add some additional bookkeeping (something like this, but I'm not sure this should be put into AccountStateForest).

[bobbinth]

Ah yes, good point! I haven't thought this through - but could we keep track of original "raw" keys only for the historical data?

Another option is to change the API to be based on hashed keys - but I'm not sure that's a good idea.

[kkovaacs]

Ah yes, good point! I haven't thought this through - but could we keep track of original "raw" keys only for the historical data?

Not sure I follow you here. If by historical data you mean the map keys that have been changed in the last N blocks -- sure, that's possible, but to serve the request we'd still need to query the database if the in-memory data doesn't cover all keys that have been requested (the case with all_entries = true).

In case the request contains a list of map keys, we can just hash those keys and perform the lookup through the AccountStateForest IIUC.