Problem
When the requester-side transfer timeout fires in fetch_blocks_via_rdma (pegaflow-core/src/backing/rdma_fetch.rs), the completion receivers and all staged SegmentAllocs are dropped, returning the pinned staging chunks to the pool — while the RDMA READ work requests already posted to the NIC may still be in flight. A late completion then DMA-writes into memory the pool may have handed to someone else: silent data corruption, strictly worse than the lock leak this path was reviewed under.
The pool's memory stays registered with the NIC (whole-pool MR), so the HCA will not fault the write — nothing stops the stale READ from landing.
Same family: after a completion error, execute_batch (pegaflow-transfer/src/rc_backend/session.rs) returns with inflight non-empty, and the subsequent session drop destroys a QP with outstanding WQEs. Verbs allows destroy-with-outstanding (driver quiesces), but any WQE that completed into freed local memory before the destroy has already corrupted it. Before #399's worker-leak fix, leaked QPs incidentally masked this; now that sessions actually die, the window is real.
Suggested fix directions
- On timeout/error, transition the QP to
ERR (flushes outstanding WQEs) and drain the CQ before dropping the staged allocations; only then release the lock and return.
- Alternatively, keep the staging chunks alive (hold the
Arc<PinnedAllocation>s) until the session confirms quiescence.
Context
Flagged during the #399 review (pre-existing; not a regression of #399).
🤖 Generated with Claude Code
Problem
When the requester-side transfer timeout fires in
fetch_blocks_via_rdma(pegaflow-core/src/backing/rdma_fetch.rs), the completion receivers and all stagedSegmentAllocs are dropped, returning the pinned staging chunks to the pool — while the RDMA READ work requests already posted to the NIC may still be in flight. A late completion then DMA-writes into memory the pool may have handed to someone else: silent data corruption, strictly worse than the lock leak this path was reviewed under.The pool's memory stays registered with the NIC (whole-pool MR), so the HCA will not fault the write — nothing stops the stale READ from landing.
Same family: after a completion error,
execute_batch(pegaflow-transfer/src/rc_backend/session.rs) returns withinflightnon-empty, and the subsequent session drop destroys a QP with outstanding WQEs. Verbs allows destroy-with-outstanding (driver quiesces), but any WQE that completed into freed local memory before the destroy has already corrupted it. Before #399's worker-leak fix, leaked QPs incidentally masked this; now that sessions actually die, the window is real.Suggested fix directions
ERR(flushes outstanding WQEs) and drain the CQ before dropping the staged allocations; only then release the lock and return.Arc<PinnedAllocation>s) until the session confirms quiescence.Context
Flagged during the #399 review (pre-existing; not a regression of #399).
🤖 Generated with Claude Code