Feat: multi threaded scheduler #589
Conversation
WalkthroughReplaces the single-file scheduler with a new multi-executor TransactionScheduler, adds an ExecutionCoordinator and per-account bitmask Read/Write locks, renames WorkerId→ExecutorId, updates test harness and tests for multi-executor configs, adjusts workspace dependencies, and changes validator runtime bootstrap to configure executor count dynamically. (50 words) Changes
Sequence Diagram(s)sequenceDiagram
participant Main as Validator Main
participant Scheduler as TransactionScheduler
participant Coord as ExecutionCoordinator
participant Exec as TransactionExecutor
participant Locks as LocksCache/AccountLock
Main->>Scheduler: spawn()
activate Scheduler
rect rgb(245,250,240)
Note over Scheduler: Event loop waits on (executor ready | new txn | new block)
end
loop Runtime events
alt Executor ready
Exec-->>Scheduler: ready(executor_id)
Scheduler->>Coord: unlock_accounts(executor_id)
Coord->>Locks: clear executor locks
Scheduler->>Coord: next_blocked_transaction(executor_id)?
Coord-->>Scheduler: txn or none
alt txn available
Scheduler->>Exec: dispatch txn
end
else New transaction arrives
Scheduler->>Coord: queue_transaction(...)/try_acquire_locks(...)
Coord->>Locks: attempt locks
alt locks acquired
Scheduler->>Exec: dispatch txn
else contention
Coord-->>Scheduler: enqueue blocked txn (with blocker id)
end
else New block
Scheduler->>Scheduler: transition_to_new_slot()
end
end
deactivate Scheduler
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (4)📓 Common learnings📚 Learning: 2025-11-04T10:53:50.922ZApplied to files:
📚 Learning: 2025-10-14T09:56:14.047ZApplied to files:
📚 Learning: 2025-10-21T14:00:54.642ZApplied to files:
🧬 Code graph analysis (1)magicblock-processor/tests/scheduling.rs (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 50
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
magicblock-accounts-db/src/index.rs (1)
376-409: Minor inconsistency: Raw u32 in packing on line 401.Lines 392 properly unpacks using
bytes!(#unpack, val, Offset, Blocks), but line 401 uses rawu32types:let index_value = bytes!(#pack, new_offset, u32, remainder, u32);For consistency with the broader refactor, this should probably be:
let index_value = bytes!(#pack, new_offset, Offset, remainder, Blocks);This doesn't affect correctness (since both are
u32aliases) but would improve type consistency.magicblock-accounts/src/errors.rs (3)
22-26: Rename typo:CommittorSerivceError→CommittorServiceError(breaking API if left as-is)Spelling error in public enum variants; fix in both
AccountsErrorandScheduledCommitsProcessorError.- #[error("CommittorSerivceError: {0}")] - CommittorSerivceError(#[from] CommittorServiceError), + #[error("CommittorServiceError: {0}")] + CommittorServiceError(#[from] CommittorServiceError), @@ - #[error("CommittorSerivceError")] - CommittorSerivceError(#[from] CommittorServiceError), + #[error("CommittorServiceError")] + CommittorServiceError(#[from] CommittorServiceError),Consider adding a temporary deprecated alias if semver stability is needed.
Also applies to: 63-65
28-33: Avoid unnecessary boxing and align error types
TokioOneshotRecvErroris boxed here but not inScheduledCommitsProcessorError. Drop boxing for consistency. Similarly, consider unboxing other small error types unless there’s a strong reason.- #[error("TokioOneshotRecvError")] - TokioOneshotRecvError(#[from] Box<tokio::sync::oneshot::error::RecvError>), + #[error("TokioOneshotRecvError")] + TokioOneshotRecvError(#[from] tokio::sync::oneshot::error::RecvError),Optionally unbox
url::ParseErrorandsolana_sdk::transaction::TransactionErroras well for uniformity.Also applies to: 59-65
49-51: Include all fields in error display
FailedToSendCommitTransactioncurrently renders only{0}but carries twoHashSet<Pubkey>s that are dropped from the message. Include them for diagnostics.- #[error("FailedToSendCommitTransaction '{0}'")] - FailedToSendCommitTransaction(String, HashSet<Pubkey>, HashSet<Pubkey>), + #[error("FailedToSendCommitTransaction '{0}', write_set={1:?}, read_set={2:?}")] + FailedToSendCommitTransaction(String, HashSet<Pubkey>, HashSet<Pubkey>),
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (4)
Cargo.lockis excluded by!**/*.lockmagicblock-bank/tests/utils/elfs/noop.sois excluded by!**/*.somagicblock-bank/tests/utils/elfs/solanax.sois excluded by!**/*.somagicblock-bank/tests/utils/elfs/sysvars.sois excluded by!**/*.so
📒 Files selected for processing (107)
.github/actions/setup-solana/action.yml(1 hunks)Cargo.toml(7 hunks)Makefile(1 hunks)docs/rpc.md(1 hunks)geyser-grpc-proto/Cargo.toml(0 hunks)geyser-grpc-proto/LICENSE_APACHE2(0 hunks)geyser-grpc-proto/build.rs(0 hunks)geyser-grpc-proto/proto/geyser.proto(0 hunks)geyser-grpc-proto/proto/solana-storage.proto(0 hunks)geyser-grpc-proto/src/lib.rs(0 hunks)magicblock-account-cloner/Cargo.toml(1 hunks)magicblock-account-cloner/src/account_cloner.rs(2 hunks)magicblock-account-cloner/src/account_cloner_stub.rs(0 hunks)magicblock-account-cloner/src/bpf_loader_v1.rs(1 hunks)magicblock-account-cloner/src/lib.rs(1 hunks)magicblock-account-cloner/src/remote_account_cloner_client.rs(0 hunks)magicblock-account-cloner/src/remote_account_cloner_worker.rs(0 hunks)magicblock-account-cloner/src/util.rs(1 hunks)magicblock-account-cloner/tests/remote_account_cloner.rs(0 hunks)magicblock-account-dumper/Cargo.toml(0 hunks)magicblock-account-dumper/src/account_dumper.rs(0 hunks)magicblock-account-dumper/src/account_dumper_bank.rs(0 hunks)magicblock-account-dumper/src/account_dumper_stub.rs(0 hunks)magicblock-account-dumper/src/lib.rs(0 hunks)magicblock-account-fetcher/Cargo.toml(0 hunks)magicblock-account-fetcher/src/account_fetcher.rs(0 hunks)magicblock-account-fetcher/src/account_fetcher_stub.rs(0 hunks)magicblock-account-fetcher/src/lib.rs(0 hunks)magicblock-account-fetcher/src/remote_account_fetcher_client.rs(0 hunks)magicblock-account-fetcher/src/remote_account_fetcher_worker.rs(0 hunks)magicblock-account-fetcher/tests/remote_account_fetcher.rs(0 hunks)magicblock-account-updates/Cargo.toml(0 hunks)magicblock-account-updates/src/account_updates.rs(0 hunks)magicblock-account-updates/src/account_updates_stub.rs(0 hunks)magicblock-account-updates/src/lib.rs(0 hunks)magicblock-account-updates/src/remote_account_updates_client.rs(0 hunks)magicblock-account-updates/src/remote_account_updates_shard.rs(0 hunks)magicblock-account-updates/src/remote_account_updates_worker.rs(0 hunks)magicblock-account-updates/tests/remote_account_updates.rs(0 hunks)magicblock-accounts-api/src/bank_account_provider.rs(0 hunks)magicblock-accounts-api/src/internal_account_provider.rs(0 hunks)magicblock-accounts-api/src/internal_account_provider_stub.rs(0 hunks)magicblock-accounts-api/src/lib.rs(0 hunks)magicblock-accounts-db/Cargo.toml(1 hunks)magicblock-accounts-db/src/index.rs(17 hunks)magicblock-accounts-db/src/index/iterator.rs(3 hunks)magicblock-accounts-db/src/index/utils.rs(2 hunks)magicblock-accounts-db/src/lib.rs(9 hunks)magicblock-accounts-db/src/snapshot.rs(4 hunks)magicblock-accounts-db/src/storage.rs(11 hunks)magicblock-accounts-db/src/tests.rs(6 hunks)magicblock-accounts/Cargo.toml(2 hunks)magicblock-accounts/README.md(0 hunks)magicblock-accounts/src/accounts_manager.rs(0 hunks)magicblock-accounts/src/config.rs(1 hunks)magicblock-accounts/src/errors.rs(1 hunks)magicblock-accounts/src/external_accounts_manager.rs(0 hunks)magicblock-accounts/src/lib.rs(0 hunks)magicblock-accounts/src/scheduled_commits_processor.rs(8 hunks)magicblock-accounts/src/traits.rs(0 hunks)magicblock-accounts/src/utils/mod.rs(0 hunks)magicblock-accounts/tests/commit_delegated.rs(0 hunks)magicblock-accounts/tests/ensure_accounts.rs(0 hunks)magicblock-accounts/tests/stubs/mod.rs(0 hunks)magicblock-accounts/tests/stubs/scheduled_commits_processor_stub.rs(0 hunks)magicblock-aperture/Cargo.toml(1 hunks)magicblock-aperture/README.md(1 hunks)magicblock-aperture/src/encoder.rs(1 hunks)magicblock-aperture/src/error.rs(1 hunks)magicblock-aperture/src/lib.rs(1 hunks)magicblock-aperture/src/processor.rs(1 hunks)magicblock-aperture/src/requests/http/get_account_info.rs(1 hunks)magicblock-aperture/src/requests/http/get_balance.rs(1 hunks)magicblock-aperture/src/requests/http/get_block.rs(1 hunks)magicblock-aperture/src/requests/http/get_block_height.rs(1 hunks)magicblock-aperture/src/requests/http/get_block_time.rs(1 hunks)magicblock-aperture/src/requests/http/get_blocks.rs(1 hunks)magicblock-aperture/src/requests/http/get_blocks_with_limit.rs(1 hunks)magicblock-aperture/src/requests/http/get_fee_for_message.rs(1 hunks)magicblock-aperture/src/requests/http/get_identity.rs(1 hunks)magicblock-aperture/src/requests/http/get_latest_blockhash.rs(1 hunks)magicblock-aperture/src/requests/http/get_multiple_accounts.rs(1 hunks)magicblock-aperture/src/requests/http/get_program_accounts.rs(1 hunks)magicblock-aperture/src/requests/http/get_signature_statuses.rs(1 hunks)magicblock-aperture/src/requests/http/get_signatures_for_address.rs(1 hunks)magicblock-aperture/src/requests/http/get_slot.rs(1 hunks)magicblock-aperture/src/requests/http/get_token_account_balance.rs(1 hunks)magicblock-aperture/src/requests/http/get_token_accounts_by_delegate.rs(1 hunks)magicblock-aperture/src/requests/http/get_token_accounts_by_owner.rs(1 hunks)magicblock-aperture/src/requests/http/get_transaction.rs(1 hunks)magicblock-aperture/src/requests/http/get_version.rs(1 hunks)magicblock-aperture/src/requests/http/is_blockhash_valid.rs(1 hunks)magicblock-aperture/src/requests/http/mocked.rs(1 hunks)magicblock-aperture/src/requests/http/mod.rs(1 hunks)magicblock-aperture/src/requests/http/request_airdrop.rs(1 hunks)magicblock-aperture/src/requests/http/send_transaction.rs(1 hunks)magicblock-aperture/src/requests/http/simulate_transaction.rs(1 hunks)magicblock-aperture/src/requests/mod.rs(1 hunks)magicblock-aperture/src/requests/params.rs(1 hunks)magicblock-aperture/src/requests/payload.rs(1 hunks)magicblock-aperture/src/requests/websocket/account_subscribe.rs(1 hunks)magicblock-aperture/src/requests/websocket/log_subscribe.rs(1 hunks)magicblock-aperture/src/requests/websocket/mod.rs(1 hunks)magicblock-aperture/src/requests/websocket/program_subscribe.rs(1 hunks)magicblock-aperture/src/requests/websocket/signature_subscribe.rs(1 hunks)magicblock-aperture/src/requests/websocket/slot_subscribe.rs(1 hunks)magicblock-aperture/src/server/http/dispatch.rs(1 hunks)
⛔ Files not processed due to max files limit (34)
- magicblock-aperture/src/server/http/mod.rs
- magicblock-aperture/src/server/mod.rs
- magicblock-aperture/src/server/websocket/connection.rs
- magicblock-aperture/src/server/websocket/dispatch.rs
- magicblock-aperture/src/server/websocket/mod.rs
- magicblock-aperture/src/state/blocks.rs
- magicblock-aperture/src/state/cache.rs
- magicblock-aperture/src/state/mod.rs
- magicblock-aperture/src/state/signatures.rs
- magicblock-aperture/src/state/subscriptions.rs
- magicblock-aperture/src/state/transactions.rs
- magicblock-aperture/src/tests.rs
- magicblock-aperture/src/utils.rs
- magicblock-aperture/tests/accounts.rs
- magicblock-aperture/tests/blocks.rs
- magicblock-aperture/tests/mocked.rs
- magicblock-aperture/tests/node.rs
- magicblock-aperture/tests/setup.rs
- magicblock-aperture/tests/transactions.rs
- magicblock-aperture/tests/websocket.rs
- magicblock-api/Cargo.toml
- magicblock-api/src/errors.rs
- magicblock-api/src/external_config.rs
- magicblock-api/src/fund_account.rs
- magicblock-api/src/genesis_utils.rs
- magicblock-api/src/geyser_transaction_notify_listener.rs
- magicblock-api/src/init_geyser_service.rs
- magicblock-api/src/lib.rs
- magicblock-api/src/magic_validator.rs
- magicblock-api/src/slot.rs
- magicblock-api/src/tickers.rs
- magicblock-bank/Cargo.toml
- magicblock-bank/README.md
- magicblock-bank/src/address_lookup_table.rs
💤 Files with no reviewable changes (44)
- magicblock-accounts/README.md
- magicblock-account-updates/src/lib.rs
- magicblock-accounts/tests/stubs/mod.rs
- magicblock-accounts-api/src/lib.rs
- geyser-grpc-proto/LICENSE_APACHE2
- magicblock-accounts/src/utils/mod.rs
- magicblock-account-fetcher/Cargo.toml
- magicblock-accounts-api/src/internal_account_provider_stub.rs
- magicblock-account-fetcher/src/lib.rs
- magicblock-account-dumper/src/lib.rs
- magicblock-account-cloner/src/account_cloner_stub.rs
- magicblock-account-fetcher/src/remote_account_fetcher_worker.rs
- magicblock-account-updates/src/remote_account_updates_shard.rs
- magicblock-accounts-api/src/internal_account_provider.rs
- magicblock-account-dumper/src/account_dumper.rs
- magicblock-account-updates/src/remote_account_updates_worker.rs
- magicblock-accounts/src/accounts_manager.rs
- magicblock-account-updates/src/account_updates_stub.rs
- magicblock-accounts/tests/commit_delegated.rs
- magicblock-accounts/src/traits.rs
- geyser-grpc-proto/Cargo.toml
- magicblock-account-fetcher/src/remote_account_fetcher_client.rs
- magicblock-account-updates/src/remote_account_updates_client.rs
- magicblock-accounts/tests/stubs/scheduled_commits_processor_stub.rs
- magicblock-account-fetcher/src/account_fetcher.rs
- magicblock-account-updates/src/account_updates.rs
- magicblock-account-updates/Cargo.toml
- magicblock-account-cloner/src/remote_account_cloner_client.rs
- magicblock-accounts/src/lib.rs
- geyser-grpc-proto/proto/geyser.proto
- geyser-grpc-proto/build.rs
- magicblock-account-dumper/src/account_dumper_bank.rs
- magicblock-accounts/tests/ensure_accounts.rs
- magicblock-account-dumper/src/account_dumper_stub.rs
- magicblock-account-updates/tests/remote_account_updates.rs
- magicblock-account-cloner/tests/remote_account_cloner.rs
- magicblock-accounts/src/external_accounts_manager.rs
- geyser-grpc-proto/proto/solana-storage.proto
- magicblock-account-fetcher/src/account_fetcher_stub.rs
- geyser-grpc-proto/src/lib.rs
- magicblock-account-dumper/Cargo.toml
- magicblock-accounts-api/src/bank_account_provider.rs
- magicblock-account-cloner/src/remote_account_cloner_worker.rs
- magicblock-account-fetcher/tests/remote_account_fetcher.rs
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-10-21T14:00:54.642Z
Learnt from: bmuddha
PR: magicblock-labs/magicblock-validator#578
File: magicblock-aperture/src/requests/websocket/account_subscribe.rs:18-27
Timestamp: 2025-10-21T14:00:54.642Z
Learning: In magicblock-aperture account_subscribe handler (src/requests/websocket/account_subscribe.rs), the RpcAccountInfoConfig fields data_slice, commitment, and min_context_slot are currently ignored—only encoding is applied. This is tracked as technical debt in issue #579: https://github.com/magicblock-labs/magicblock-validator/issues/579
Applied to files:
magicblock-aperture/src/requests/websocket/account_subscribe.rs
📚 Learning: 2025-10-21T10:34:59.140Z
Learnt from: bmuddha
PR: magicblock-labs/magicblock-validator#578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Applied to files:
magicblock-accounts-db/src/lib.rs
🧬 Code graph analysis (50)
magicblock-aperture/src/requests/http/get_version.rs (1)
magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_account_info.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-core/src/link/accounts.rs (1)
new(39-49)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_fee_for_message.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (3)
parse_error(106-111)invalid_params(78-83)transaction_verification(92-97)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_token_account_balance.rs (4)
magicblock-aperture/src/requests/http/mod.rs (1)
read_account_with_ensure(100-115)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
invalid_params(78-83)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_latest_blockhash.rs (2)
magicblock-aperture/src/state/blocks.rs (1)
from(96-101)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_slot.rs (1)
magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_multiple_accounts.rs (4)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-core/src/link/accounts.rs (1)
new(39-49)magicblock-aperture/src/utils.rs (1)
new(130-140)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/send_transaction.rs (3)
test-integration/test-tools/src/integration_test_context.rs (2)
send_transaction(850-869)signature(63-65)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/mod.rs (1)
magicblock-aperture/src/error.rs (1)
invalid_request(99-104)
magicblock-aperture/src/requests/http/get_blocks.rs (4)
magicblock-aperture/src/requests/http/get_blocks_with_limit.rs (2)
get_blocks_with_limit(13-30)start_slot(27-27)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
invalid_params(78-83)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/server/http/dispatch.rs (5)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-aperture/src/requests/http/mod.rs (2)
extract_bytes(59-92)parse_body(47-56)magicblock-aperture/src/server/http/mod.rs (1)
new(40-55)magicblock-aperture/src/state/mod.rs (1)
new(76-94)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_transaction.rs (2)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_token_accounts_by_delegate.rs (6)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
parse_error(106-111)magicblock-aperture/src/requests/http/get_program_accounts.rs (1)
accounts(40-47)magicblock-aperture/src/requests/http/get_token_accounts_by_owner.rs (1)
accounts(66-72)magicblock-aperture/src/utils.rs (1)
new(130-140)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_token_accounts_by_owner.rs (5)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
parse_error(106-111)magicblock-aperture/src/requests/http/get_program_accounts.rs (1)
accounts(40-47)magicblock-aperture/src/utils.rs (1)
new(130-140)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/is_blockhash_valid.rs (2)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_program_accounts.rs (4)
magicblock-accounts-db/src/lib.rs (2)
get_program_accounts(180-200)slot(229-231)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/utils.rs (3)
from(22-26)from(93-117)new(130-140)magicblock-aperture/src/requests/payload.rs (5)
encode(61-79)encode(105-115)encode(120-133)encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_block_height.rs (1)
magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_block.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (5)
from(33-35)from(39-41)from(45-47)from(51-53)from(57-59)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/get_balance.rs (2)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/http/get_block_time.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
custom(120-125)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-accounts-db/src/index/utils.rs (2)
magicblock-accounts-db/src/index/table.rs (1)
new(14-21)magicblock-accounts-db/src/index/iterator.rs (1)
new(16-45)
magicblock-aperture/src/requests/http/mocked.rs (3)
magicblock-aperture/src/requests/payload.rs (5)
encode_no_context(85-100)encode_no_context(138-148)encode(61-79)encode(105-115)encode(120-133)magicblock-aperture/src/state/mod.rs (1)
new(76-94)magicblock-aperture/src/server/http/dispatch.rs (1)
new(54-67)
magicblock-aperture/src/requests/http/request_airdrop.rs (3)
magicblock-aperture/src/error.rs (1)
invalid_request(99-104)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-account-cloner/src/util.rs (1)
magicblock-rpc-client/src/lib.rs (2)
get_cus_from_transaction(593-602)get_logs_from_transaction(578-582)
magicblock-aperture/src/requests/http/get_identity.rs (1)
magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/websocket/account_subscribe.rs (1)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)
magicblock-aperture/src/requests/http/get_signature_statuses.rs (2)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-aperture/src/requests/websocket/signature_subscribe.rs (2)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/state/subscriptions.rs (1)
next_subid(255-257)
magicblock-accounts-db/src/index/iterator.rs (1)
magicblock-accounts-db/src/lib.rs (1)
next(386-394)
magicblock-aperture/src/requests/http/simulate_transaction.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/error.rs (1)
transaction_simulation(85-90)magicblock-aperture/src/requests/payload.rs (3)
encode(61-79)encode(105-115)encode(120-133)
magicblock-accounts-db/src/tests.rs (2)
magicblock-accounts-db/src/lib.rs (3)
next(386-394)get_account(335-338)new(42-72)magicblock-core/src/traits.rs (1)
get_account(12-12)
magicblock-aperture/src/requests/params.rs (1)
magicblock-aperture/src/encoder.rs (6)
encode(25-30)encode(76-87)encode(93-105)encode(115-129)encode(142-171)encode(181-200)
magicblock-accounts-db/src/storage.rs (2)
magicblock-accounts-db/src/lib.rs (1)
flush(320-323)magicblock-accounts-db/src/index.rs (2)
flush(411-419)reload(424-430)
magicblock-aperture/src/lib.rs (5)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-aperture/src/processor.rs (3)
new(52-61)start(73-83)run(90-139)magicblock-aperture/src/server/http/mod.rs (2)
new(40-55)run(67-84)magicblock-aperture/src/server/websocket/mod.rs (2)
new(64-81)run(89-104)magicblock-aperture/tests/setup.rs (1)
new(83-138)
magicblock-aperture/src/requests/websocket/program_subscribe.rs (4)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/encoder.rs (2)
from(43-50)from(54-63)magicblock-aperture/src/utils.rs (2)
from(22-26)from(93-117)magicblock-aperture/src/requests/params.rs (5)
from(27-29)from(33-35)from(39-41)from(45-47)from(51-53)
magicblock-account-cloner/src/account_cloner.rs (2)
magicblock-committor-service/src/stubs/changeset_committor_stub.rs (1)
oneshot(57-57)magicblock-account-cloner/src/util.rs (1)
get_tx_diagnostics(7-18)
magicblock-accounts/src/scheduled_commits_processor.rs (3)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-account-cloner/src/lib.rs (1)
new(57-71)test-kit/src/lib.rs (1)
new(80-82)
magicblock-aperture/src/requests/http/get_blocks_with_limit.rs (3)
magicblock-aperture/src/requests/http/get_blocks.rs (1)
start_slot(35-35)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-accounts-db/src/index.rs (5)
magicblock-accounts-db/src/index/utils.rs (1)
new(49-59)magicblock-accounts-db/src/storage.rs (5)
new(81-127)new(319-382)size(267-270)offset(184-190)reload(224-253)magicblock-accounts-db/src/index/table.rs (1)
new(14-21)magicblock-accounts-db/src/index/iterator.rs (1)
new(16-45)magicblock-accounts-db/src/index/tests.rs (1)
allocation(428-442)
magicblock-aperture/src/error.rs (1)
magicblock-rpc-client/src/lib.rs (1)
error(191-193)
magicblock-aperture/src/requests/websocket/log_subscribe.rs (1)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)
magicblock-accounts-db/src/snapshot.rs (1)
magicblock-accounts-db/src/lib.rs (1)
new(42-72)
magicblock-aperture/src/requests/payload.rs (3)
magicblock-aperture/src/encoder.rs (8)
encode(25-30)encode(76-87)encode(93-105)encode(115-129)encode(142-171)encode(181-200)from(43-50)from(54-63)magicblock-aperture/src/error.rs (5)
from(33-35)from(39-41)from(45-47)from(51-53)from(57-59)magicblock-aperture/src/utils.rs (2)
from(22-26)from(93-117)
magicblock-aperture/src/requests/websocket/mod.rs (5)
magicblock-aperture/src/server/websocket/dispatch.rs (1)
dispatch(63-84)magicblock-aperture/src/requests/websocket/account_subscribe.rs (1)
account_subscribe(14-38)magicblock-aperture/src/requests/websocket/program_subscribe.rs (1)
program_subscribe(18-48)magicblock-aperture/src/requests/websocket/signature_subscribe.rs (1)
signature_subscribe(17-47)magicblock-aperture/src/requests/websocket/slot_subscribe.rs (1)
slot_subscribe(8-12)
magicblock-account-cloner/src/lib.rs (4)
magicblock-chainlink/src/remote_account_provider/program_account.rs (2)
try_from(92-103)lamports(126-129)programs/magicblock/src/utils/instruction_utils.rs (4)
modify_accounts(154-160)modify_accounts_instruction(162-195)disable_executable_check_instruction(292-302)enable_executable_check_instruction(304-314)magicblock-account-cloner/src/bpf_loader_v1.rs (1)
try_from(31-79)magicblock-account-cloner/src/util.rs (1)
get_tx_diagnostics(7-18)
magicblock-aperture/src/processor.rs (3)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-aperture/src/state/blocks.rs (2)
new(47-61)set_latest(64-67)magicblock-aperture/src/state/subscriptions.rs (6)
new(330-343)send_account_update(125-131)send_program_update(162-169)send_signature_update(190-203)send_logs_update(223-229)send_slot(250-252)
magicblock-aperture/src/requests/http/get_signatures_for_address.rs (3)
magicblock-aperture/src/requests/mod.rs (1)
params(22-26)magicblock-aperture/src/requests/params.rs (5)
from(27-29)from(33-35)from(39-41)from(45-47)from(51-53)magicblock-aperture/src/requests/payload.rs (2)
encode_no_context(85-100)encode_no_context(138-148)
magicblock-aperture/src/requests/http/mod.rs (9)
magicblock-aperture/src/requests/http/get_token_account_balance.rs (2)
size_of(56-56)get_token_account_balance(17-71)magicblock-aperture/src/server/http/dispatch.rs (1)
dispatch(81-109)magicblock-aperture/src/error.rs (4)
invalid_request(99-104)parse_error(106-111)invalid_params(78-83)transaction_verification(92-97)magicblock-aperture/src/requests/http/get_multiple_accounts.rs (2)
pubkeys(31-39)get_multiple_accounts(13-43)magicblock-aperture/src/requests/params.rs (2)
deserialize(78-110)deserialize(135-168)magicblock-aperture/src/requests/http/get_account_info.rs (1)
get_account_info(12-40)magicblock-aperture/src/requests/http/get_balance.rs (1)
get_balance(9-24)magicblock-aperture/src/requests/http/send_transaction.rs (1)
send_transaction(14-51)magicblock-aperture/src/requests/http/simulate_transaction.rs (1)
simulate_transaction(22-92)
magicblock-aperture/src/encoder.rs (3)
magicblock-aperture/src/requests/payload.rs (5)
encode(61-79)encode(105-115)encode(120-133)encode_no_context(85-100)encode_no_context(138-148)magicblock-aperture/src/utils.rs (3)
from(22-26)from(93-117)new(130-140)magicblock-core/src/link/accounts.rs (1)
new(39-49)
magicblock-accounts-db/src/lib.rs (8)
magicblock-accounts-db/src/index/utils.rs (1)
new(49-59)magicblock-accounts/src/scheduled_commits_processor.rs (2)
new(66-90)new(415-430)magicblock-accounts-db/src/storage.rs (4)
new(81-127)new(319-382)offset(184-190)flush(214-219)magicblock-accounts-db/src/index.rs (3)
new(96-126)flush(411-419)remove_account(229-261)magicblock-accounts-db/src/snapshot.rs (2)
new(35-49)database_path(143-145)magicblock-accounts-db/src/index/iterator.rs (2)
new(16-45)next(50-53)magicblock-aperture/src/requests/http/get_program_accounts.rs (1)
get_program_accounts(13-55)magicblock-core/src/traits.rs (3)
get_account(12-12)remove_account(13-13)remove_where(14-17)
Manual Deploy AvailableYou can trigger a manual deploy of this PR branch to testnet: Alternative: Comment
Comment updated automatically when the PR is synchronized. |
There was a problem hiding this comment.
Actionable comments posted: 7
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
Cargo.toml(1 hunks)magicblock-api/src/magic_validator.rs(1 hunks)magicblock-processor/Cargo.toml(1 hunks)magicblock-processor/src/executor/mod.rs(4 hunks)magicblock-processor/src/lib.rs(0 hunks)magicblock-processor/src/scheduler.rs(0 hunks)magicblock-processor/src/scheduler/coordinator.rs(1 hunks)magicblock-processor/src/scheduler/locks.rs(1 hunks)magicblock-processor/src/scheduler/mod.rs(1 hunks)magicblock-processor/src/scheduler/tests.rs(1 hunks)magicblock-validator/Cargo.toml(1 hunks)magicblock-validator/src/main.rs(3 hunks)
💤 Files with no reviewable changes (2)
- magicblock-processor/src/lib.rs
- magicblock-processor/src/scheduler.rs
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-21T10:34:59.140Z
Learnt from: bmuddha
PR: magicblock-labs/magicblock-validator#578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Applied to files:
magicblock-processor/src/scheduler/locks.rs
🧬 Code graph analysis (4)
magicblock-processor/src/scheduler/tests.rs (1)
magicblock-processor/src/scheduler/coordinator.rs (5)
new(31-36)new(56-64)try_acquire_locks(133-172)get_ready_executor(102-104)get_blocked_transaction(121-126)
magicblock-api/src/magic_validator.rs (2)
magicblock-processor/src/scheduler/coordinator.rs (2)
new(31-36)new(56-64)magicblock-processor/src/scheduler/mod.rs (1)
new(56-93)
magicblock-processor/src/scheduler/mod.rs (2)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-processor/src/scheduler/coordinator.rs (3)
new(31-36)new(56-64)is_ready(97-99)
magicblock-processor/src/scheduler/coordinator.rs (2)
magicblock-processor/src/scheduler/locks.rs (1)
next_transaction_id(111-122)magicblock-processor/src/scheduler/mod.rs (1)
new(56-93)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: run_make_ci_test
- GitHub Check: run_make_ci_test
- GitHub Check: run_make_ci_format
- GitHub Check: run_make_ci_lint
🔇 Additional comments (24)
Cargo.toml (1)
148-148: LGTM!The addition of
rustc-hash = "2.1"as a workspace dependency is appropriate for the new scheduler's hash-based locking and contention structures.magicblock-validator/Cargo.toml (1)
19-19: LGTM!The
num_cpusdependency addition supports the dynamic executor sizing based on available CPU cores, which aligns with the multi-threaded scheduler implementation.magicblock-processor/Cargo.toml (1)
41-45: LGTM!The dependency additions are appropriate:
rustc-hashsupports the new locking and contention structures in the schedulersolana-keypair(dev-dependency) is used in the new test suite for creating mock transactionsmagicblock-validator/src/main.rs (3)
53-65: Verify thread allocation consistency across the codebase.The runtime is configured with
(num_cpus::get() / 2).max(1)worker threads for the async runtime. Inmagicblock-api/src/magic_validator.rs(lines 305-308), the transaction executors are calculated as(num_cpus::get() / 2).saturating_sub(1).max(1). This creates a potential imbalance:
- Async runtime threads:
(num_cpus / 2).max(1)- Execution threads:
(num_cpus / 2 - 1).max(1)(scheduler) + executorsFor example, on an 8-core system:
- Async runtime: 4 threads
- Scheduler overhead: 1 thread
- Executors: 3 threads
- Total execution: 4 threads
This seems intentional, but please confirm that the asymmetry is expected and document why the scheduler thread is accounted for in one calculation but not the other.
67-137: LGTM!The refactored
run()function preserves all the previous initialization logic correctly, including logger setup, config loading, validator startup, and the informational startup messages.
139-153: LGTM!The
print_infohelper function is correctly restored with appropriate conditional logging based on theRUST_LOGenvironment variable.magicblock-api/src/magic_validator.rs (1)
305-312: LGTM!The dynamic executor sizing based on CPU cores is well-implemented:
- Uses
saturating_sub(1)to account for the scheduler thread overhead- Ensures at least 1 executor with
.max(1)- Clear comment explaining the allocation strategy
The thread allocation strategy has been flagged for verification in
magicblock-validator/src/main.rsto ensure consistency across the codebase.magicblock-processor/src/executor/mod.rs (2)
24-24: LGTM!The import update correctly reflects the rename from
WorkerIdtoExecutorIdand the new module structure (scheduler::locks).
34-34: LGTM!The type changes from
WorkerIdtoExecutorIdare consistently applied across all field declarations and constructor parameters.Also applies to: 54-54, 69-69, 72-72
magicblock-processor/src/scheduler/tests.rs (5)
17-46: LGTM!The
create_mock_transactionhelper is well-designed:
- Clean API with
(Pubkey, bool)tuples for account specification- Properly constructs transactions with appropriate writable/readonly metadata
- Wraps in
TransactionWithIdfor coordinator testing
50-94: LGTM!Basic concurrency tests are comprehensive and well-structured:
test_non_conflicting_transactions: Validates parallel execution without conflictstest_read_read_no_contention: Confirms multiple read locks can coexistClear assertions with descriptive failure messages.
98-168: LGTM!Contention tests thoroughly cover all locking conflict scenarios:
- Write-write blocking
- Write-read blocking
- Read-write blocking
Each test properly validates that the blocked transaction reports the correct blocker executor ID.
172-285: LGTM!Advanced scenario tests provide excellent coverage:
- Multiple mixed locks with complex contention patterns
- Dependency chains (txn3 → txn2 → txn1)
- Full executor pool saturation and rescheduling
These tests validate the core scheduling logic under realistic conditions.
289-467: LGTM!Edge case tests comprehensively cover boundary conditions:
- Empty transaction (no accounts)
- Multiple concurrent read locks
- Rapid lock/unlock cycles
- Multiple transactions blocked on same executor
- Multi-account contention
- Pool exhaustion
- Transaction blocked by queued transaction (ID-based blocking)
This test suite provides strong confidence in the coordinator's correctness.
magicblock-processor/src/scheduler/coordinator.rs (8)
1-6: LGTM!Clear module documentation explaining the ExecutionCoordinator's role in the scheduling process.
24-37: LGTM!The
TransactionWithIdwrapper correctly associates transactions with unique IDs for tracking purposes. The constructor usesnext_transaction_id()from the locks module.
56-64: LGTM!The constructor properly initializes all coordinator state:
- Per-executor blocked transaction queues
- Per-executor acquired lock lists
- Ready executor pool with all executors initially available
- Empty contention map and lock cache
97-109: LGTM!The executor pool management methods are straightforward and correct:
is_ready()checks for available executorsget_ready_executor()pops an executor from the ready poolrelease_executor()returns an executor to the ready pool
112-118: LGTM!The
unlock_accountsmethod correctly iterates through all acquired locks for an executor and releases them. Usingpop()in a while loop efficiently drains the vector.
121-126: LGTM!The
get_blocked_transactionmethod correctly retrieves the next blocked transaction from an executor's queue usingpop_front()for FIFO ordering.
133-173: Verify the atomicity of lock acquisition rollback.The
try_acquire_locksmethod attempts to acquire all necessary locks and rolls back on partial failure (lines 155-163). However, there's a potential race condition:
- Executor A acquires locks for accounts [X, Y]
- Executor B tries to acquire locks for [X, Z] and fails on X
- Executor B's rollback (lines 156-160) calls
contend()on all previously acquired locks- The
contend()call marks contention but doesn't verify the locks are still held by the same transactionThe code on line 158 calls
lock.contend(transaction.id)on locks that were just acquired and are being rolled back. This seems intended to mark that this transaction is contending for these locks, but the semantic meaning should be clarified.Please verify:
- Is
contend(transaction.id)on line 158 setting contention correctly for locks that were just released?- Should contention be set before or after
unlock()?- Does the order matter for the coordinator's subsequent scheduling decisions?
Consider adding inline comments to clarify the rollback semantics.
169-171: LGTM!On successful lock acquisition, the transaction is removed from the contention map, which correctly reflects that it's no longer waiting.
magicblock-processor/src/scheduler/mod.rs (2)
166-173: Slot transition ordering: confirm no concurrent index updates during reset.index.store(0, Relaxed) during a new slot could race with executors still fetch_add-ing within the previous slot, producing non-monotonic ordering. Ensure executors have released the per-slot AccountsDb read lock before this reset, or use a slot-epoch barrier.
Based on learnings: executors hold a slot-scoped AccountsDb read lock and release at slot boundaries. Please confirm transition_to_new_slot is only invoked after all executors drop that read lock, or add coordination to enforce it. Also consider Release on store and Acquire where read to publish the epoch switch.
176-188: Reschedule loop logic: confirm blocked queue keying and release semantics.Loop always drains get_blocked_transaction(blocker) while scheduling onto exec that may change. This seems intentional (drain transactions unblocked by the original blocker), but please confirm:
- get_blocked_transaction(blocker) is indeed keyed by the “previous blocker” executor id.
- release_executor(exec) correctly returns the last unused ready executor to the pool when no blocked txn exists.
| # bundled sqlite 3.44 | ||
| rusqlite = { version = "0.37.0", features = ["bundled"] } | ||
| rustc_version = "0.4" | ||
| rustc-hash = "2.1" |
There was a problem hiding this comment.
This is never used?
On a side note, this weekend, I discovered hashbrown based on foldhash, which is faster on longer inputs than fxhash.
There was a problem hiding this comment.
it is used in the Lock contention tracking, since the keys are fixed (32 bytes) it's massively faster than siphash, and for a such a small values it's on par with foldhash, but that's an interesting algo to keep in mind, thanks.
|
|
||
| /// The maximum number of concurrent executors supported by the bitmask. | ||
| /// One bit is reserved for the write flag. | ||
| pub(super) const MAX_SVM_EXECUTORS: u32 = ReadWriteLock::BITS - 1; |
There was a problem hiding this comment.
Wouldn't it be worth casting that down to u8? Ditto for ExecutorId.
It might not improve perfs and would require lots of casts, but it is slightly confusing to have such large values for tiny ID.
There was a problem hiding this comment.
u32 is mostly used for convenience with (as you mentioned) casting operations and compatibility with TransactionID. The performance is not affected, as modern registers are 64 bit anyway, and they sign extend u8/u16, for u32 there's even a hardware optimization (absence of the REX prefix on x86)
| self.handle_new_transaction(txn).await; | ||
| } | ||
| // A new block has been produced. | ||
| _ = block_produced.recv() => { |
There was a problem hiding this comment.
Wouldn't that be the most priorotized thing to update?
There was a problem hiding this comment.
well, yeah I guess it wouldn't hurt to prioritize this, fixed in 4e218fa
| for id in 0..count { | ||
| // Each executor has a channel capacity of 1, as it | ||
| // can only process one transaction at a time. | ||
| let (transactions_tx, transactions_rx) = channel(1); |
There was a problem hiding this comment.
Could we extract this into separate const within function? It is highly important that no one ever sets anything rather than 1 here or the whole thing breaks.
There was a problem hiding this comment.
Put into a const. Technically nothing would break, since the executor readiness doesn't depend on free channel capacity, but rather is tracked via the Coordinator itself. 4e218fa
| tokio::select! { | ||
| biased; | ||
| // A worker has finished its task and is ready for more. | ||
| Some(executor) = self.ready_rx.recv() => { |
There was a problem hiding this comment.
Assume there's a stream of TXs that use same accounts as writable [a, b]. Obviously only 1 Executor can work, while all others will be getting TXs queued up in blocked_transactions. So we get
E1 executing [a, b]. blocked_transaction[E1] = vec![[a, b], [a, b], [a, b], [a, b]]
E2 stale - blocked_transaction[E1] = vec![[a, b], [a, b], [a, b], [a, b]]
E3 same as E2
... and so on
vec![[a, b], [a, b], [a, b], [a, b]] - TXs with [a, b] as writable
Now assume that this spam of txs is over. E1 gets cleaned up and is ready. Since E2, E3, ... do not execute anything ready_rx is empty so no one can wake them up. Those txs are effectively stuck until a new tx arrives and wakes up some Executor.
It seems to clean this heap of tx we will require around same amount of other txs
There was a problem hiding this comment.
Here again the assumption is that transactions with those accounts can end up in other executor queues, that is impossible since the executor can have transaction in its queue, if and only if it is holding the lock for one of the accounts in the transaction. If I have messed up this part of scheduling guarantee, please point me to the code, I'll fix it.
There was a problem hiding this comment.
Wait, in that case the flow is as follows, right?
- E1 gets TX1 with writable
[a,b] - We get another TX2 with writable
[a,b] - E2 is ready, we hit
handle_ready_executor - Account is locked and blocker is E1, so we populate blocked_transactions[E1]
- Again we get new TX3
[a,b], nowblocked_transaction[E1] = vec![[a,b], [a,b]]
With above we will
6. Now E1 exits, we get into reschedule_blocked_transactions, schedules from blocked_transactions [a,b], now blocked_transactions[E1] = vec![[a,b]]
7. executor = self.coordinator.get_ready_executor(); is E2, we go at the begginning of the loop
8. let txn = self.coordinator.get_blocked_transaction(blocker); is blocked_transaction[E1], [a, b]
9. We attempt to schedule and fail, free executor again E2 and we repeat and repeat
For test sufficient to have
E1 executing [a, b]
blocked_transactions[E1] = vec![[a,b], [a, b]]
E2 is ready
E1 finishes execution
There was a problem hiding this comment.
Ok, I see the error, thanks, great catch 👍🏽
There was a problem hiding this comment.
Fixed in 4e218fa. So now we just bail from the loop upon first lock failure. Which prevents indefinite release/retry lock cycle. Since we don't have frontrunning for now, potentially wasted throughput is acceptable. We can review the algo together once the priority based rescheduling/reordering will be implemented. Please take another look at the fix.
There was a problem hiding this comment.
Are there integration tests for the whole Scheduler execution flow? Would be good to see the same transaction set being executed end-end and seeing that the Scheduler is empty at the end and everything works as expected.
This file rather seems to be unit tests for Coordinator
There was a problem hiding this comment.
Good point, I'll try to add fully fledged scheduler tests to cover as many cases as possible. I'll push them in the upcoming commits.
There was a problem hiding this comment.
Added a separate file for these tests, which tests the the whole scheduler/executor bundle as a black box, magicblock-processor/tests/scheduling.rs
There was a problem hiding this comment.
Let me know if you have any other test scenarios which we need to cover
| self.schedule_transaction(exec, txn).await; | ||
| executor = self.coordinator.get_ready_executor(); | ||
| } else { | ||
| self.coordinator.release_executor(exec); |
There was a problem hiding this comment.
I'd propose here stealing mechanism rather than making Executor stale, it would help resolve this issue
There was a problem hiding this comment.
We will implement some form of stealing in the near future, but for now we have settled on more or less first in - first served policy. Once we have formalized the ruleset for front-running and priorities, we can work on stealing algorithm as well. As of now stealing would imply getting transactions, which are blocked on the queues of other executors, and even if their lock conditions have been satisfied by now, it would cause front running and out of order execution.
There was a problem hiding this comment.
I agree, but we sort of already have that issue where since some transaction is not blocked by anyone it could skip through all of earlier blocked transactions.
Also with the structure of next_blocked_transaction & queue_transaction we have a shuffle of transactions, where next_blocked_transaction takes from front and if not scheduled pushed at the end via queue_transaction
There was a problem hiding this comment.
Yes, you are right, that's why I said more or less policy:). Well, right now implementing work stealing would introduce a bit of complexity involved in rescan of all the available queues, and I'd like to take a step by step approach, and just make this PR as small as possible. We can introduce work stealing in the subsequent one, once we make sure that the base cases work without any issues.
As for the reshuffle, you are right, that's what I caught yesterday as well, and sort of fixed. Now there's an explicit flag, which forces the transaction to be pushed to the front, if it was taken from front (and back is only used for new transactions) it's not elegant, but we can improve upon the design during the work on a work stealing.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
Cargo.toml(1 hunks)magicblock-processor/Cargo.toml(1 hunks)magicblock-processor/src/executor/mod.rs(4 hunks)magicblock-processor/src/scheduler/coordinator.rs(1 hunks)magicblock-processor/src/scheduler/locks.rs(1 hunks)magicblock-processor/src/scheduler/mod.rs(1 hunks)magicblock-processor/src/scheduler/tests.rs(1 hunks)
🧰 Additional context used
🧠 Learnings (7)
📓 Common learnings
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.896Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
📚 Learning: 2025-11-04T10:53:50.896Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.896Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Applied to files:
magicblock-processor/src/scheduler/mod.rsmagicblock-processor/src/scheduler/tests.rsmagicblock-processor/src/executor/mod.rsmagicblock-processor/src/scheduler/locks.rsmagicblock-processor/src/scheduler/coordinator.rs
📚 Learning: 2025-11-04T10:48:00.047Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/mod.rs:217-219
Timestamp: 2025-11-04T10:48:00.047Z
Learning: In magicblock-validator, the codebase uses a pattern where types containing non-Send/non-Sync fields (like Rc<RefCell<...>>) are marked with unsafe impl Send when they are guaranteed to be confined to a single thread through careful API design and thread spawning patterns.
Applied to files:
magicblock-processor/src/scheduler/mod.rsmagicblock-processor/src/scheduler/locks.rs
📚 Learning: 2025-10-21T10:34:59.140Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Applied to files:
magicblock-processor/src/scheduler/mod.rsmagicblock-processor/src/executor/mod.rsmagicblock-processor/src/scheduler/locks.rsmagicblock-processor/src/scheduler/coordinator.rs
📚 Learning: 2025-10-28T13:15:42.706Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
Applied to files:
magicblock-processor/src/scheduler/mod.rsmagicblock-processor/src/scheduler/locks.rsmagicblock-processor/src/scheduler/coordinator.rs
📚 Learning: 2025-10-21T14:00:54.642Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-aperture/src/requests/websocket/account_subscribe.rs:18-27
Timestamp: 2025-10-21T14:00:54.642Z
Learning: In magicblock-aperture account_subscribe handler (src/requests/websocket/account_subscribe.rs), the RpcAccountInfoConfig fields data_slice, commitment, and min_context_slot are currently ignored—only encoding is applied. This is tracked as technical debt in issue #579: https://github.com/magicblock-labs/magicblock-validator/issues/579
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-14T09:56:14.047Z
Learnt from: taco-paco
Repo: magicblock-labs/magicblock-validator PR: 564
File: test-integration/programs/flexi-counter/src/processor/call_handler.rs:122-125
Timestamp: 2025-10-14T09:56:14.047Z
Learning: The file test-integration/programs/flexi-counter/src/processor/call_handler.rs contains a test smart contract used for integration testing, not production code.
Applied to files:
magicblock-processor/src/scheduler/tests.rs
🧬 Code graph analysis (3)
magicblock-processor/src/scheduler/mod.rs (4)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-processor/src/executor/mod.rs (4)
new(66-107)spawn(131-142)run(156-195)transition_to_new_slot(199-204)magicblock-processor/src/scheduler/coordinator.rs (3)
new(32-37)new(57-65)is_ready(106-108)magicblock-processor/src/scheduler.rs (2)
TransactionScheduler(40-148)TransactionScheduler(26-38)
magicblock-processor/src/scheduler/tests.rs (1)
magicblock-processor/src/scheduler/coordinator.rs (5)
new(32-37)new(57-65)try_acquire_locks(142-181)get_ready_executor(111-113)next_blocked_transaction(130-135)
magicblock-processor/src/scheduler/coordinator.rs (2)
magicblock-processor/src/scheduler/locks.rs (1)
next_transaction_id(111-122)magicblock-processor/src/scheduler/mod.rs (1)
new(58-91)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: run_make_ci_lint
🔇 Additional comments (2)
magicblock-processor/Cargo.toml (1)
42-42: Workspace dependencies appropriately added for multi-threaded scheduler.The additions of
rustc-hashandsolana-keypairare consistent with the new lock-based scheduler design.rustc-hashis a fast hasher for fixed-size account keys used in contention tracking, andsolana-keypairprovides test utilities for the executor suite.Also applies to: 46-46
Cargo.toml (1)
148-148:rustc-hashversion pinned appropriately for workspace.Version 2.1 is a stable release of the Rust compiler's hash implementation, ideal for the fixed-size account key hashing in the new lock contention tracking subsystem introduced by the multi-threaded scheduler.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
magicblock-aperture/src/requests/http/mocked.rs (1)
197-201: Type mismatch remains unresolved:commitmentshould be Vec, not array.The previously flagged issue still exists.
RpcBlockCommitment { commitment: Option<Vec<_>> }expects a Vec, but an array[0; 32]is provided.Apply this diff to fix:
- let response = RpcBlockCommitment { - commitment: Some([0; 32]), - total_stake: 0, - }; + let response = RpcBlockCommitment { + commitment: Some(vec![0; 32]), + total_stake: 0, + };magicblock-aperture/src/requests/http/get_blocks_with_limit.rs (1)
22-24: Address overflow risks in end_slot calculation.The addition of
+ 1toblock_height()on line 24 partially addresses the past review comment's off-by-one concern, but critical overflow issues remain unresolved:
- Line 22:
start_slot + limitcan overflow when both values are large (e.g., high slot number + 500,000 limit).- Line 24:
block_height() + 1can overflow ifblock_height()returnsu64::MAX.Both operations should use
saturating_addto prevent panic in release builds or wrap-around behavior.Apply this diff to fix the overflow issues:
- let end_slot = start_slot + limit; + let end_slot = start_slot.saturating_add(limit); // Calculate the end slot, ensuring it does not exceed the latest block height. - let end_slot = (end_slot).min(self.blocks.block_height() + 1); + let end_slot = end_slot.min(self.blocks.block_height().saturating_add(1));magicblock-processor/src/scheduler/mod.rs (1)
172-190: Restore reserved executors when a reschedule attempt fails.
reschedule_blocked_transactionspulls the next ready executor before callingschedule_transaction. If lock acquisition still fails, we break out of the loop without putting that executor back intoready_executors, so one contention cycle silently shrinks the pool. After a few such misses every worker gets leaked,is_ready()staysfalse, and the scheduler stops dispatching even though threads are idle. Please defer taking the next executor until after a successful schedule (or explicitly release it on failure).- while let Some(exec) = executor { + while let Some(exec) = executor { let txn = self.coordinator.next_blocked_transaction(blocker); let scheduled = if let Some(txn) = txn { - executor = self.coordinator.get_ready_executor(); - self.schedule_transaction(exec, txn, true).await + let scheduled = + self.schedule_transaction(exec, txn, true).await; + if scheduled { + executor = self.coordinator.get_ready_executor(); + } else { + executor = None; + } + scheduled } else { self.coordinator.release_executor(exec); break; }; if !scheduled { break; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (8)
Cargo.toml(2 hunks)magicblock-aperture/Cargo.toml(1 hunks)magicblock-aperture/src/requests/http/get_blocks_with_limit.rs(1 hunks)magicblock-aperture/src/requests/http/get_token_account_balance.rs(2 hunks)magicblock-aperture/src/requests/http/mocked.rs(3 hunks)magicblock-aperture/src/requests/http/request_airdrop.rs(1 hunks)magicblock-processor/src/scheduler/mod.rs(1 hunks)magicblock-table-mania/src/manager.rs(1 hunks)
🧰 Additional context used
🧠 Learnings (7)
📓 Common learnings
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.896Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
📚 Learning: 2025-10-21T14:00:54.642Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-aperture/src/requests/websocket/account_subscribe.rs:18-27
Timestamp: 2025-10-21T14:00:54.642Z
Learning: In magicblock-aperture account_subscribe handler (src/requests/websocket/account_subscribe.rs), the RpcAccountInfoConfig fields data_slice, commitment, and min_context_slot are currently ignored—only encoding is applied. This is tracked as technical debt in issue #579: https://github.com/magicblock-labs/magicblock-validator/issues/579
Applied to files:
magicblock-aperture/src/requests/http/request_airdrop.rsmagicblock-aperture/src/requests/http/get_token_account_balance.rsmagicblock-aperture/src/requests/http/mocked.rsmagicblock-processor/src/scheduler/mod.rsmagicblock-aperture/src/requests/http/get_blocks_with_limit.rs
📚 Learning: 2025-10-21T11:00:18.396Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-aperture/src/encoder.rs:176-187
Timestamp: 2025-10-21T11:00:18.396Z
Learning: In the magicblock validator, the current slot is always the root slot. The SlotEncoder in magicblock-aperture/src/encoder.rs correctly sets `root: slot` because there is no lag between current and root slots in this architecture.
Applied to files:
magicblock-aperture/src/requests/http/get_token_account_balance.rsmagicblock-aperture/src/requests/http/mocked.rsmagicblock-aperture/src/requests/http/get_blocks_with_limit.rs
📚 Learning: 2025-11-04T10:53:50.896Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.896Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-11-04T10:48:00.047Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/mod.rs:217-219
Timestamp: 2025-11-04T10:48:00.047Z
Learning: In magicblock-validator, the codebase uses a pattern where types containing non-Send/non-Sync fields (like Rc<RefCell<...>>) are marked with unsafe impl Send when they are guaranteed to be confined to a single thread through careful API design and thread spawning patterns.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-21T10:34:59.140Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-28T13:15:42.706Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
🧬 Code graph analysis (3)
magicblock-aperture/src/requests/http/request_airdrop.rs (2)
magicblock-aperture/src/error.rs (2)
invalid_request(99-104)invalid_params(78-83)magicblock-aperture/src/requests/mod.rs (1)
params(22-26)
magicblock-table-mania/src/manager.rs (1)
magicblock-committor-service/src/intent_execution_manager/intent_execution_engine.rs (1)
rand(587-587)
magicblock-processor/src/scheduler/mod.rs (4)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-processor/src/scheduler/coordinator.rs (3)
new(32-37)new(57-65)is_ready(106-108)magicblock-processor/src/executor/mod.rs (4)
new(66-107)spawn(131-142)run(156-195)transition_to_new_slot(199-204)magicblock-processor/src/scheduler.rs (2)
TransactionScheduler(40-148)TransactionScheduler(26-38)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: run_make_ci_lint
- GitHub Check: run_make_ci_test
- GitHub Check: Build Project
🔇 Additional comments (12)
magicblock-aperture/src/requests/http/mocked.rs (3)
84-89: LGTM! Slot context standardized.This change addresses the previous review comment by standardizing on
block_height()for consistency with other handlers.
103-108: LGTM! Slot context standardized.Consistent with the standardization applied to other handlers.
120-125: LGTM! Slot context standardized.This completes the standardization across all three handlers flagged in the previous review.
magicblock-aperture/src/requests/http/get_token_account_balance.rs (2)
4-7: LGTM: Necessary imports for canonical token amount formatter.The added imports enable the use of Solana's
token_amount_to_ui_amount_v3function, which provides standardized UI amount formatting for SPL tokens.
64-71: Excellent refactor: adopts Solana's canonical UI amount formatter.This change addresses the previous concern about
ui_amount_stringformatting by delegating totoken_amount_to_ui_amount_v3, which handles edge cases like scientific notation and precision trimming correctly.The
Nonevalues forinterest_bearing_configandscaled_ui_amount_configare appropriate for standard SPL tokens. If Token-2022 extended tokens (interest-bearing, scaled UI amounts) are in scope, additional parsing of mint extensions would be required—acceptable technical debt for now.magicblock-aperture/src/requests/http/request_airdrop.rs (3)
18-20: Improved error message for disabled faucet.The error message has been updated to clearly indicate that the faucet is disabled, which is more informative than the previous generic "method is not supported" message.
27-29: Good addition of lamports validation.The zero-amount validation correctly prevents no-op airdrop requests and provides a clear error message.
25-26: Verify: Previous review suggested explicit Pubkey type cast.The earlier review comment on these lines recommended adding an explicit type annotation to cast
pubkeytoPubkeytype (i.e.,let pubkey: Pubkey = some_or_err!(pubkey);), stating this matched the project's existing pattern. While the lamports validation has been added, the type cast was not implemented.Please confirm whether the explicit cast is necessary for type consistency across the codebase or if the current implicit conversion is sufficient.
magicblock-aperture/Cargo.toml (1)
69-69: LGTM! Workspace reference applied.This change addresses the previous review comment by using a workspace reference for
rand, ensuring centralized version management and consistency with other dependencies.Cargo.toml (2)
94-94: Verify the intentional removal of log level filtering.The
release_max_level_infofeature has been removed, which means log statements will no longer be filtered at compile time in release builds. This could impact performance and binary size.Is this intentional for improved production debugging, or should the feature be retained for performance?
148-148: LGTM! rustc-hash addition is appropriate.The addition of
rustc-hashfor lock contention tracking with fixed 32-byte keys is well-justified and faster than siphash for this use case, as discussed in previous comments.magicblock-table-mania/src/manager.rs (1)
380-381: API migration is correct.The code properly uses the rand 0.9 API:
rand::rng()replacesrand::thread_rng(), andRng::random_range(a..b)replacesgen_range(a, b)with support for inclusive ranges. The range0..=u64::MAXis valid syntax for the new API.
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test-kit/src/lib.rs (1)
86-115: Update the constructor docs to match the new behavior.Bullet 3 still claims we “spawn a TransactionScheduler with one worker thread,” but
new_with_confignow accepts an executor count and an option to defer startup. Please revise the doc comment so it explains the configurable worker pool and optional deferred spawn.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (5)
magicblock-core/src/link/transactions.rs(2 hunks)magicblock-processor/src/scheduler/mod.rs(1 hunks)magicblock-processor/tests/fees.rs(1 hunks)magicblock-processor/tests/scheduling.rs(1 hunks)test-kit/src/lib.rs(6 hunks)
🧰 Additional context used
🧠 Learnings (7)
📓 Common learnings
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.922Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
📚 Learning: 2025-11-04T10:53:50.922Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/locks.rs:110-122
Timestamp: 2025-11-04T10:53:50.922Z
Learning: In magicblock-processor, the TransactionScheduler runs in a single, dedicated thread and will always remain single-threaded. The `next_transaction_id()` function in scheduler/locks.rs uses `unsafe static mut` which is safe given this architectural guarantee.
Applied to files:
magicblock-processor/src/scheduler/mod.rstest-kit/src/lib.rsmagicblock-processor/tests/scheduling.rs
📚 Learning: 2025-11-04T10:48:00.070Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: magicblock-processor/src/scheduler/mod.rs:217-219
Timestamp: 2025-11-04T10:48:00.070Z
Learning: In magicblock-validator, the codebase uses a pattern where types containing non-Send/non-Sync fields (like Rc<RefCell<...>>) are marked with unsafe impl Send when they are guaranteed to be confined to a single thread through careful API design and thread spawning patterns.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-21T10:34:59.140Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-accounts-db/src/lib.rs:63-72
Timestamp: 2025-10-21T10:34:59.140Z
Learning: In magicblock-validator, the AccountsDb "stop-the-world" synchronizer is managed at the processor/executor level, not at the AccountsDb API level. Transaction executors in magicblock-processor hold a read lock (sync.read()) for the duration of each slot and release it only at slot boundaries, ensuring all account writes happen under the read lock. Snapshot operations acquire a write lock, blocking until all executors release their read locks. This pattern ensures mutual exclusion between writes and snapshots without requiring read guards in AccountsDb write APIs.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-28T13:15:42.706Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 596
File: magicblock-processor/src/scheduler.rs:1-1
Timestamp: 2025-10-28T13:15:42.706Z
Learning: In magicblock-processor, transaction indexes were always set to 0 even before the changes in PR #596. The proper transaction indexing within slots will be addressed during the planned ledger rewrite.
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-21T14:00:54.642Z
Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-aperture/src/requests/websocket/account_subscribe.rs:18-27
Timestamp: 2025-10-21T14:00:54.642Z
Learning: In magicblock-aperture account_subscribe handler (src/requests/websocket/account_subscribe.rs), the RpcAccountInfoConfig fields data_slice, commitment, and min_context_slot are currently ignored—only encoding is applied. This is tracked as technical debt in issue #579: https://github.com/magicblock-labs/magicblock-validator/issues/579
Applied to files:
magicblock-processor/src/scheduler/mod.rs
📚 Learning: 2025-10-14T09:56:14.047Z
Learnt from: taco-paco
Repo: magicblock-labs/magicblock-validator PR: 564
File: test-integration/programs/flexi-counter/src/processor/call_handler.rs:122-125
Timestamp: 2025-10-14T09:56:14.047Z
Learning: The file test-integration/programs/flexi-counter/src/processor/call_handler.rs contains a test smart contract used for integration testing, not production code.
Applied to files:
magicblock-processor/tests/scheduling.rs
🧬 Code graph analysis (4)
magicblock-processor/src/scheduler/mod.rs (4)
magicblock-core/src/link.rs (1)
link(56-82)magicblock-processor/src/scheduler/coordinator.rs (3)
new(32-37)new(57-65)is_ready(106-108)magicblock-processor/src/executor/mod.rs (4)
new(66-107)spawn(131-142)run(156-195)transition_to_new_slot(199-204)magicblock-processor/src/scheduler.rs (2)
TransactionScheduler(40-148)TransactionScheduler(26-38)
magicblock-processor/tests/fees.rs (1)
test-kit/src/lib.rs (1)
new_with_config(94-152)
test-kit/src/lib.rs (2)
magicblock-accounts/src/scheduled_commits_processor.rs (2)
new(66-90)new(418-433)magicblock-aperture/tests/setup.rs (1)
new(83-138)
magicblock-processor/tests/scheduling.rs (2)
test-kit/src/lib.rs (4)
new_with_config(94-152)new(82-84)get_transaction(200-208)get_account(297-307)magicblock-processor/src/scheduler/mod.rs (1)
new(58-91)
🪛 GitHub Actions: Run CI - Format
magicblock-processor/tests/scheduling.rs
[error] 1-7: cargo fmt --check failed. Detected unstaged formatting changes in tests/scheduling.rs. Run 'cargo fmt' (or update formatting) to fix code style issues.
🪛 GitHub Actions: Run CI - Lint
magicblock-processor/tests/scheduling.rs
[error] 562-562: the loop variable i is only used to index accounts (clippy: needless_range_loop). The lint suggests using an iterator, e.g., for in accounts.iter().take(num_txs).skip(1) { ... }
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Build Project
- GitHub Check: run_make_ci_test
🔇 Additional comments (3)
magicblock-core/src/link/transactions.rs (2)
54-54: LGTM: Debug derives improve observability.The Debug trait additions on
TransactionStatusandTransactionExecutionResultare beneficial for debugging and logging, especially valuable in the new multi-threaded scheduler context.Also applies to: 80-80
84-84: All construction sites have been properly updated with thelogsfield.Verification confirms that the single construction site in
magicblock-processor/src/executor/processing.rs:202-210correctly initializes all three struct fields, including the newlogsfield (logs: meta.log_messages.clone(),). No other construction sites or alternative instantiation patterns exist in the codebase.magicblock-processor/tests/fees.rs (1)
270-308: Zero-fee setup still behaves as before.Using
new_with_config(0, 1, false)keeps the slot in gasless mode while spawning the scheduler eagerly, so the test’s intent is preserved.
This introduces a new multi-threaded transaction scheduler to the
magicblock-processor, replacing the previous single-threaded implementation. This new design significantly improves performance by processing non-conflicting transactions in parallel.The core of this feature is a central scheduler that dispatches transactions to a pool of worker threads. A lock-free coordination mechanism based on bitmasks is used to manage account access, ensuring that only transactions with non-overlapping write sets are executed concurrently.
Changes in this PR include:
Summary by CodeRabbit
Performance Improvements
Behavior Changes
Tests