Network transactions tracking issue

[Mirko-von-Leipzig]

This likely still needs feedback; so I'll avoid adding (most) subissues just yet.

---

This serves as the tracking issue for the initial implementation of network transactions.

Prior discussion held in #606 with both major threads containing valuable information. I'll summarize the results below but still recommend reading through the discussion.

What are network transactions

Network transactions are transactions that are executed and proven by the node itself. Users can trigger these by creating notes targeting network accounts. The node will detect these and create network transactions to execute/consume them.

Design overview

Execution and proving of these network notes will be handled by a new node component tx-builder (real name pending).

This component will receive network notes from the mempool which are then queued until they can be successfully executed against the target account, and submitted as a normal transaction via RPC.

The tx-builder can fetch the required account state from the store via RPC. Ideally we would interact with the RPC component so that we don't have write access to the store and get tx proof verification via rpc in-line. Using the store and block-producer clients is also an option though.

flowchart LR
    A[tx-builder]
    B[mempool]
    C[rpc]
    
    A --> |submit transaction| C
    B --> |network note| A
    B --> |"network tx reverted|committed"| A
    C --> |account state| A

Loading

Initial simplifications

Restrictions we impose to simplify implementations which may be relaxed in the future once things stabilize and we come up with better designs.

Network accounts

Network accounts will be fully managed by the network. This means the tx-builder can assume that only itself will update network accounts. Otherwise we would have to account for external transactions changing the account state under our feet. With this simplification we can fetch the latest account state from the store and know that it cannot change without our intervention.

This means we only need the account state for in progress/queue network notes.

Note status

Only consider network notes that have been committed. This comes at the expense of extra latency and removing the possibility of note erasure (both user tx and network tx consuming it in the same block).

This simplification means we don't need to worry about the user tx reverting. The mempool can send network notes to the builder as they are committed.

Execution stacking

The builder could consume multiple notes within the same transaction for efficiency, but it is currently not possible to build a transaction by consuming notes incrementally. aka we have to apply all notes at once, atomically. This can be revisited once we can execute notes incrementally.

Note types

Notes will be restricted to some subset of known notes initially. This makes it simpler to validate that a note can be executed against an account.

Open questions

Validation

• Should transactions which create invalid network notes be rejected? e.g. if they target a private account, or non-existing account.
• Who performs this validation? mempool or tx-builder?

Multi-account txs

If txs can affect multiple accounts one day, is this something we would want to add to network txs? Seems unlikely at first blush.

Fault tolerance

What happens if some subset of the components fail? Currently we run all components in the same process, so one failure brings down all of them. However in the future this is not ideal and each component would ideally survive and somehow reset on its own. This isn't only an issue for this feature, but rather a broader concern. I think for now we can ignore piecemeal failure.

We do need to handle the node restarting. With the current design of only allowing committed network notes, we can fetch the set of initial notes to process from the store.

This does mean we could get different results though. Such a query could also return network notes that we previously discarded as invalid (e.g. targeting a non-existent account), but are valid now (account now exists). We would need to decide on the policy for this.

RPC design

How complex can network account state become? Will it require a streaming endpoint, or can we assume that a single request will retrieve the full state at once.

This will inform our RPC endpoint design (if any), since it may diverge from the normal RPC account state retrieval - which needs to handle streaming of some kind.

If the builder gets direct access to the store and block-producer (instead of via RPC), then it may make sense to begin splitting the store gRPC definitions into multiple services:

store.rpc             # Endpoints intended for RPC component
store.block-producer  # get_{tx|batch|block}_inputs & apply_block
store.tx-builder      # get_network_account_state

Relaxing simplifications aka future expansions

Execution stacking can be added whenever its possible to incrementally build a tx. We should consider whether we need more information from the mempool wrt ordering of the origin txs. e.g. the network tx will be dependent on all origin txs in the stacked set.

Send network notes as soon as they are accepted which means we have to account for user tx reverts. This implies we must track account deltas and tx/note dependencies in the builder. Though this might already be required to handle network tx reverts.

Expand note types. Fairly self-explanatory, probably just handled on a case-by-case basis.

Allowing external mutation of network accounts. This would likely require closer coupling between the mempool and the builder. The mempool could inform the builder whenever a network account's state changes which in turn could use that to make decisions on its internal set of pending network txs.

[bobbinth]

Great write up! I think this covers basically everything. A few comments:

The tx-builder can fetch the required account state from the store via RPC. Ideally we would interact with the RPC component so that we don't have write access to the store and get tx proof verification via rpc in-line. Using the store and block-producer clients is also an option though.

I was thinking of the tx-builder as a "trusted" component - i.e., it would be run at the same level as the store or block producer. So, we can trust that it'll generate valid proofs and that it won't try to write anything to the store. And then, I think it is fine if it "talks" to the store and block producer directly.

• Should transactions which create invalid network notes be rejected? e.g. if they target a private account, or non-existing account.
• Who performs this validation? mempool or tx-builder?

I'm inclined not to do this right now. If we do want to do this, it'll probably have to be done at the mempool level (or maybe even RPC level?). That is, we'd want to discard the transaction before accepting it to the mempool - but this may have non-negligible performance implications, and could be done separately in the future anyway.

If txs can affect multiple accounts one day, is this something we would want to add to network txs? Seems unlikely at first blush.

Yeah - I wouldn't worry about this for now.

We do need to handle the node restarting. With the current design of only allowing committed network notes, we can fetch the set of initial notes to process from the store.

I think this is a separate issue and ideally we may want to run the note in different processes in a single machine anyways because the cost of restarting different components is different. For example, restarting RPC or block producer is current pretty cheap, but restarting the store is very expensive. So, if the RPC or block producer crush, ideally, we'd restart only them.

This does mean we could get different results though. Such a query could also return network notes that we previously discarded as invalid (e.g. targeting a non-existent account), but are valid now (account now exists). We would need to decide on the policy for this.

This could be solved by making the state of the tx-builder persistent. We have a similar issue for the block producer (#23).

This will inform our RPC endpoint design (if any), since it may diverge from the normal RPC account state retrieval - which needs to handle streaming of some kind.

The way I'm thinking the tx-builder will build transactions is by using TransactionExecutor from miden-base. The main thing that the executor needs is a DataStore implementation. For tx-builder this implementation could keep things in memory and could also have access to the store to load relevant things. For example, on first request to get_transaction_inputs() for a specific account_id, it could load the account from the store and then cache it in memory so that for subsequent requests account state would be available without the need for the round trip to the store.

The are some discussions in 0xPolygonMiden/miden-base#938 and 0xPolygonMiden/miden-base#831 (comment) about potentially modifying this interface.

[Mirko-von-Leipzig]

This does mean we could get different results though. Such a query could also return network notes that we previously discarded as invalid (e.g. targeting a non-existent account), but are valid now (account now exists). We would need to decide on the policy for this.

This could be solved by making the state of the tx-builder persistent. We have a similar issue for the block producer (#23).

I think this gets complicated quickly because part of the state also exists in the mempool and in gRPC comms. Ideally there would be one single source of truth (store) that everyone can reset from.

As an example: part of the overall state are the gRPC comms between mempool, block-producer and the tx builder. e.g. if the tx builder dies while it is receiving a new note, or status of network tx then that information is lost forever because the mempool won't resend it. The issue is that these comms aren't atomic so there are many possible race failures.

A similar example for #23 is a block-producer crash after the store has accepted the block, but before this is registered with the solution for #23.

Some options:

• Some sort of atomic communication, e.g. apply updates one both sides, and both send ack once complete, otherwise rollback.
• Make local reset state fallible, e.g. mempool could verify local data against store

The issue is that the more components store their view locally, the more complex recovery becomes (I think). But we can punt on that for now I think.

[bobbinth]

I think this gets complicated quickly because part of the state also exists in the mempool and in gRPC comms. Ideally there would be one single source of truth (store) that everyone can reset from.

Agree that this could get complicated, but also not sure if we can push everything to the store. I think of the store as containing the data that is already "committed" - while mempool contains data which is mostly in-flight. So, technically, if the block producer crashes, we only loose the mempool and people can re-submit transaction, but the integrity of the data is not affected.

So, overall, I think our persistence (at least for the mempool) could be "best effort" - i.e., if on restart we are able to re-build most of the mempool that's fine. We would need to make sure that w/e the mempool gets re-initialized with does not conflict with the store, but I think that may not be too complicated (and if it does conflict, we can drop more data from the mempool).

[Mirko-von-Leipzig]

Agree that this could get complicated, but also not sure if we can push everything to the store. I think of the store as containing the data that is already "committed" - while mempool contains data which is mostly in-flight. So, technically, if the block producer crashes, we only loose the mempool and people can re-submit transaction, but the integrity of the data is not affected.

So, overall, I think our persistence (at least for the mempool) could be "best effort" - i.e., if on restart we are able to re-build most of the mempool that's fine. We would need to make sure that w/e the mempool gets re-initialized with does not conflict with the store, but I think that may not be too complicated (and if it does conflict, we can drop more data from the mempool).

I think so long as we assign correct ownership to the data then we can make it work. store for committed data and block-producer/mempool for inflight data makes good sense.

This implies that tx-builder shouldn't have its own persisted source of truth though - it and the mempool are coupled, so if either dies both should "reset". The mempool then re-aligns with the store on startup, and begins rebuilding and thereby reseeds the tx-builder. Problem for another day, but I think that sounds robust to me.

[bobbinth]

This implies that tx-builder shouldn't have its own persisted source of truth though - it and the mempool are coupled, so if either dies both should "reset". The mempool then re-aligns with the store on startup, and begins rebuilding and thereby reseeds the tx-builder. Problem for another day, but I think that sounds robust to me.

I would say that if mempool dies that tx-builder dies too, but if tx-builder dies, does mempool need to die as well?

Though, maybe the solution is that tx-builder does not need persistent storage but just gets info from mempool/store.