feat: committor service

[thlorenz]

Summary

Adding a full fledged committor service to support the following kind of commits:

• commits up to 10Kb as long as the size increase per commit is <=1Kb
• max accounts per bundle is 3 without lookup tables and 12 with lookup tables
• commit results are persisted into a SQlite database in the ledger directory which makes it
  easy to manually retry them or do this automatically (next committor version will support this)

Details

The following crates were developed
here and integrated
into the validator:

magicblock-rpc-client

• thin wrapper around the solana rpc client with added features like improved transaction
  signature status checking (including sane retries)
• lookup table convenience methods
• multi account fetch with known max limit
• waiting for slots

magicblock-table-mania

• a lookup table manager that allows to create, extend and deactivate + close lookup tables
• the API supports reserving pubkeys which will be added to a table and when they need to be
  used in a transaction the needed table addresses are provided
• once a pubkey is released this is noted and a table is deactivated and closed on chain once
  all its pubkeys are released

magicblock-committor-program

• allows the committor service to upload account data to be committed in chunks
• supports creation/resizing of buffers and filling them with transactions running in parallel
  (out of order is ok)

magicblock-committor-service

• commits a changeset as fast and efficiently as possible
• first prefers transactions that don't require lookup tables since those take time to become
  available
• then prefers args only commits to the use of buffers (which require preparation)

Sample Flow

1. Validator starts up and reserves common pubkeys (used in all commits) with the committor
   service
2. Account are cloned and for each account the pubkeys for commits are reserved to prepare the
   lookup tables which we might need as fast as possible
3. Account(s) commit is scheduled and registered as we did before
4. The commits are processed via the committor service and the on chain transaction signature
   is provided to the user via logs in a transaction (as we did before, just now we have to wait
   for the commit to complete)

For those commits the committor service ensures that accounts with the same commit (bundle) id
are always processed atomically.

The committor service also picks the best possible strategy to commit each changeset,
preferring speed and then cost.

It also inserts configurable (via code) compute budget instructions to each transaction.

The outcome of each commit is persisted to a database which allows manual (and in the next
version) automated retries of failed commits.
Succeessful commits are also persisted to the database and can be used for diagnostics. In the
future they should be removed since no retry is necessary.

On chain signatures for process-commit/finaliz/undelegate transactions are also persisted to
the database in a separate table.

This table is queried by the validator to log those signatures as part of a transaction that
the user waits for.

Greptile Summary

This PR introduces a comprehensive committor service to the MagicBlock validator for managing account changes and transactions. The implementation includes a new Solana program for handling commits, a service layer for managing commit operations, and integration with lookup tables for transaction optimization.

• Added magicblock-committor-program that handles chunked account data commits with proper buffer management and security checks
• Added magicblock-committor-service that orchestrates commit operations with support for bundling, lookup tables, and persistent status tracking via SQLite
• Added magicblock-table-mania for managing address lookup tables to optimize transaction sizes
• Added magicblock-rpc-client with enhanced transaction handling and batched account fetching
• Integrated committor service with existing account management systems while maintaining backward compatibility

[greptile-apps]

_{113 file(s) reviewed, 128 comment(s)
Edit PR Review Bot Settings | Greptile}