Lock-in RPC api

[Mirko-von-Leipzig]

Our RPC api is mostly complete. This issue serves to discuss any remaining changes or outstanding functionality that we may want.

We're only looking to finalize the RPC public facing method definitions. This excludes the actual input/output message definitions as we still want to separately decide on the schema/codec.

In other words, this issue is only about finalizing the method signatures and intents.

---

Account syncing

SyncAccountVault and SyncAccountStorageMaps currently paginate by returning all changes within a block range, even ones which overwrite each other. This lets us delimit pages at block boundaries, and since blocks are limited in capacity, this means we have a manageable response size.

The downside of this is that the client throws most of this away, retaining only the aggregated information i.e. the latest value for each slot/key. It would therefore be more efficient for both server and client if we only send the final value.

The original intention was to let the client sync to any block desired, but that roles has been filled by SyncMmr iiuc. So a client will

1. use SyncMmr to determine which exact block to sync to
2. use the account methods to end at that block
3. drop all intermediate values

Instead, we could only do (1) + send final values. The open questions are:

• Are there any use cases which are served by the current API?
• How do we paginate, if not at block boundaries?
  • We could simply paginate by key i.e. index
  • If clients don't trust server, we could also send the merkle proof for the page

---

Block and proof streams

#1960 and #1961 touch on this, but it might be nicer to have the store push new blocks and proofs, instead of having clients poll for them.

#1960 might also introduce a dedicated GetProof endpoint.

The benefit would be lower latency between store instances, and no unnecessary polls (though the latter is probably a minor issue). Downside is slightly more complexity, and potentially a DoS vector if many open stream connections. This may be better as an internal method only - though it would be nice to have this be public so anyone can create a personal RPC+store using these (instead we would have to have either both polling and streaming, or only polling).

The idea would be to have separate subscriptions for blocks and proofs, starting at some block height forwards. This enables syncing from any block height onward, maybe in combination with SyncMmr for security.

---

Mempool transaction status

We may want an endpoint for clients to know the status of a transaction within the mempool.

This is useful so that a client can know if the transaction(s) are still pending, or have been reverted for some reason. While the client can detect an expired transaction, it cannot do this if the transaction builds on some other expired state, or if the node resets, or just decides to evict the transaction for whatever reason.

The current client implementation marks a transaction as failed after some configurable number of blocks have passed. Which is of course not a perfect solution.

We could have a dedicated block-producer endpoint for this (re-exposed via RPC ofc). Where the mempool would retain information about transactions still in the mempool, as well as recently reverted transactions (kept for some configurable amount of time/blocks).

ito I think we can make this efficient by having a dedicated task connected to the mempool via a channel. The mempool sends transaction events to this task, which aggregates and manages them. The RPC endpoint then interacts with the state from this task. Given sufficient channel capacity, I don't think this endpoint would be an overload.

Open question: what states would we be interested in?

• submitted aka in-mempool aka in-queue aka pending
• reverted (kept for e.g. 5minutes)
• committed(BlockNumber) (kept for e.g. 5minutes)
• in batch? in pending block? probably not needed..

[igamigo]

The downside of this is that the client throws most of this away, retaining only the aggregated information i.e. the latest value for each slot/key. It would therefore be more efficient for both server and client if we only send the final value.

The original intention was to let the client sync to any block desired, but that roles has been filled by SyncMmr iiuc. So a client will

1. use SyncMmr to determine which exact block to sync to
2. use the account methods to end at that block
3. drop all intermediate values
   Instead, we could only do (1) + send final values.

Pasting this mostly for reference: some of this is touched upon here

[bobbinth]

A few other thoughts:

• It is not clear to me if the CheckNullifiers endpoint (at least as it currently is) is all that useful. @igamigo - do we use it anywhere on the client? One of the reason for this is that it always returns a response against the chain tip - but the user may not be synced to the chain tip - in which case, the returned paths are not all that useful. If we do want to keep it, we should probably add something like a block_num parameter to it.
• We may want to normalize the naming between SubmitProvenTransaction and SubmitProvenBatch. We could just say SubmitProvenTransactionBatch or maybe SubmitProvenTxBatch and then rename SubmitProvenTransaction into SubmitProvenTx.
• We could probably simplify SyncNotes, SyncTransactions, and SyncNullifiers etc. requests to use BlockRange where block_to is not optional. The thinking is that the client usually does SyncChainMmr first, and then makes requests to these endpoints to get specific block ranges. So, the whole logic of trying to guess what the block_to should be could go away. This could also lead to simplifying the PaginationInfo in the responses for these endpoints.

[igamigo]

It is not clear to me if the CheckNullifiers endpoint (at least as it currently is) is all that useful. @igamigo - do we use it anywhere on the client? One of the reason for this is that it always returns a response against the chain tip - but the user may not be synced to the chain tip - in which case, the returned paths are not all that useful.

Nothing on the client logic ever invokes it. I think the only other endpoint that we don't use is GetBlockByNumber, but I guess an explorer would want to use that one. Maybe also worth thinking if there any of the existing ones should be expanded to support more explorer-related usecases (I recall there being a request from pioneers to show specific data on the explorer that was not trivial to do, but I forget the details).

We could probably simplify SyncNotes, SyncTransactions, and SyncNullifiers etc. requests to use BlockRange where block_to is not optional. The thinking is that the client usually does SyncChainMmr first, and then makes requests to these endpoints to get specific block ranges.

On the client we've already made a couple of these instances non-optional: 0xMiden/miden-client#1960 (comment), so I'd agree.