Kernel transfer backend causes decode TPOT tail spikes under concurrent KV transfers

[xiaguan]

Summary

The kernel transfer backend solves an important throughput problem for highly
fragmented KV transfers: it replaces many directional CUDA memcpy submissions
with one descriptor upload and one copy-kernel launch.

However, when the transfer runs concurrently with model decode on the same GPU,
the copy kernel competes for GPU execution and memory resources. In a
serving-oriented A/B test on an RTX 5090, bidirectional kernel transfers
sustained the target transfer rate but increased decode TPOT p99 by 46.6%
and max TPOT by 80.6%.

This suggests that TransferMode::Kernel should remain available, but backend
selection at registration should account for latency policy and transfer shape
rather than only the model type.

Relevant implementation

At the current repository head:

• TransferMode is selected per instance at registration and is shared by both
  load and save worker pools.
• KernelBackend launches up to one 256-thread block per descriptor, capped at
  65,535 blocks.
• MemcpyBackend coalesces adjacent ranges, then submits one directional CUDA
  memcpy per merged range.
• The connector currently selects kernel mode for MLA models and direct mode
  otherwise.

Test setup

The test used a release build and a real Qwen3-4B pure-decode workload, with a
standalone CUDA transfer stressor matching the current PegaFlow backend
semantics.

Decode workload:

• GPU: RTX 5090, PCIe 5.0 x16
• CUDA: 13.1
• Context length: 2,048
• Batch size: 1
• Decode steps: 512
• Warmup steps: 32
• Iterations per run: 3
• ITL samples per run: 1,437
• Three runs per main mode, with rotated execution order
• Transfer stress started only after model initialization and CUDA Graph capture

Transfer workload:

• 16,384 shuffled descriptors per direction
• 4 KiB per descriptor
• 64 MiB H2D + 64 MiB D2H per burst
• One burst every 50 ms
• Target aggregate rate: 2.5 GiB/s
• Host payload allocated as mapped pinned memory
• H2D and D2H submitted concurrently on separate streams

The hybrid result below is an experimental policy using kernel for H2D and
direct memcpy for D2H; it is not a current PegaFlow mode.

Results

Values are means of the per-run metrics across three runs. Max is the maximum
observed across the three runs.

Mode	Avg TPOT	P50	P99	Max	Effective transfer rate
No transfer	6.221 ms	6.220 ms	6.255 ms	6.335 ms	—
Direct	6.327 ms (+1.7%)	6.335 ms (+1.9%)	6.542 ms (+4.6%)	6.654 ms	2.436 GiB/s
Kernel	6.414 ms (+3.1%)	6.034 ms (-3.0%)	9.172 ms (+46.6%)	11.441 ms	2.497 GiB/s
Hybrid	6.234 ms (+0.2%)	6.074 ms (-2.3%)	7.683 ms (+22.8%)	8.807 ms	2.497 GiB/s

The lower kernel p50 should not be interpreted as a speedup. The periodic 50 ms
transfers affect only a subset of decode steps: unaffected steps remain fast
while transfer-overlapping steps form a separate tail.

Direction isolation produced similar kernel interference in both directions:

Backend	Direction	Decode TPOT p99	Effective transfer rate
Kernel	H2D	7.632 ms	1.249 GiB/s
Kernel	D2H	7.587 ms	1.249 GiB/s
Direct	H2D	6.219 ms	1.249 GiB/s
Direct	D2H	6.235 ms	1.249 GiB/s

On this GPU, H2D and D2H kernel interference is approximately symmetric. The
larger bidirectional tail comes from concurrent kernel transfers rather than one
direction being universally worse.

As a PCIe sanity check, contiguous pinned-memory DMA reached 52.54 GiB/s H2D
and 53.20 GiB/s D2H. The lower direct result above is therefore caused by
per-fragment submission overhead, not PCIe bandwidth.

Discussion

I think the kernel backend is still valuable, especially when a direct transfer
expands into thousands of merged ranges. The open question is how registration
should express the tradeoff between transfer throughput and serving tail
latency.

Possible directions:

1. Keep Kernel as an explicit backend, but avoid selecting it solely from
   model type for latency-sensitive serving.
2. Add an Auto policy based on observable transfer shape, such as merged range
   count, total bytes, and average range size.
3. Allow load and save policies to differ, while avoiding a hard-coded
   directional choice because the direction-specific result is GPU-dependent.
4. Evaluate lower-priority transfer streams, a bounded grid size, or
   chunked/yielding kernel launches to reduce decode interference.
5. Add a serving-interference benchmark alongside backend-only throughput
   benchmarks, since transfer throughput alone misses this regression.

The desired outcome is to preserve the fragmented-transfer throughput advantage
without making TPOT p99 unpredictable for colocated decode.

[xiaguan]

Follow-up: grid, priority, chunk/yield, and Green Context sweep

I ran a second sweep to look for a useful operating point between the Direct backend's stable decode tail and the Kernel backend's transfer throughput.

Setup

• RTX 5090, 170 SMs, PCIe 5.0 x16, CUDA 13.1
• Qwen3-4B pure decode: context length 2048, batch size 1, 512 decode steps, 32 warmup steps
• 3 iterations per run, 1,437 ITL samples per run
• Each transfer burst contains 16,384 shuffled 4 KiB descriptors per direction: 64 MiB H2D + 64 MiB D2H
• One H2D stream and one D2H stream, mapped pinned host memory
• A burst every 50 ms, targeting 2.5 GiB/s aggregate transfer bandwidth

The harness is standalone, but its Kernel path follows the current backend semantics: descriptor-based gather/scatter kernels submitted asynchronously on two streams.

Serving results

The baseline contains six runs. The other rows contain three runs except chunk=1024, which is one run. Deltas are relative to the baseline.

Mode	Mean ITL	p50	p99	Max	Effective transfer
No transfer	6.222 ms	6.224 ms	6.253 ms	6.347 ms	—
Direct	6.201 ms	6.230 ms	6.541 ms (+4.6%)	6.703 ms	2.404 GiB/s
Kernel, current full grid	6.424 ms (+3.2%)	6.041 ms	9.204 ms (+47.2%)	11.463 ms	2.497 GiB/s
Kernel, grid=256	6.227 ms (+0.1%)	5.978 ms	8.061 ms (+28.9%)	11.873 ms	2.497 GiB/s
Kernel, grid=384	6.280 ms (+0.9%)	—	8.128 ms (+30.0%)	12.220 ms	2.497 GiB/s
Kernel, chunk=4096 with completion backpressure	6.455 ms (+3.7%)	6.047 ms	8.484 ms (+35.7%)	8.586 ms (+35.3%)	2.497 GiB/s
Kernel, chunk=2048 with completion backpressure	6.503 ms (+4.5%)	6.028 ms	7.485 ms (+19.7%)	7.605 ms (+19.8%)	2.497 GiB/s
Kernel, chunk=1024 with completion backpressure	6.785 ms (+9.0%)	7.063 ms	7.113 ms (+13.8%)	7.188 ms (+13.2%)	2.497 GiB/s

The lower p50 values in some periodic-transfer modes are not a speedup. They are a distribution effect caused by periodic interference.

1. Bounding the grid helps, but does not fully bound the tail

A transfer-only sweep, with continuous transfers and no decode workload, produced:

Mode	Raw transfer throughput
Direct, fragmented copies	2.920 GiB/s
Kernel, current full grid	47.730 GiB/s
Kernel, grid=8	48.600 GiB/s
Kernel, grid=16	57.100 GiB/s
Kernel, grid=32	63.565 GiB/s
Kernel, grid=64	56.998 GiB/s
Kernel, grid=128	70.242 GiB/s
Kernel, grid=256	69.798 GiB/s
Kernel, grid=512	67.410 GiB/s
Kernel, grid=1024	50.264 GiB/s

Therefore, using the current very large grid is worse for both raw transfer throughput and decode tail on this device. A bounded grid in the 128–512 range is a better operating region here. grid=256 gave the best formal serving mean, but rare maxima were still around 10.5–11.9 ms across runs. I would expose a device/policy-specific grid limit rather than hard-code 256 for every GPU.

2. Chunking only helps when it adds real completion backpressure

Enqueueing smaller chunks without waiting did not help: chunk=2048 enqueue-only had a 9.006 ms p99, essentially the same as the full kernel launch.

The useful sequence was:

1. Submit one paired H2D/D2H chunk.
2. Wait or poll for that chunk's completion.
3. Yield the transfer worker.
4. Submit the next chunk.

Adding a fixed sleep after completion was not useful:

Post-completion sleep	p99 ITL
0 us	7.420 ms
25 us	7.417 ms
50 us	7.468 ms
100 us	7.563 ms

The completion boundary is what matters, not the sleep duration. Transfer completion latency under concurrent decode increased from about 5.9 ms for a full submission to 10.2 ms, 18.7 ms, and 36.6 ms at chunk sizes 4096, 2048, and 1024 respectively. All of them still sustained the paced 2.497 GiB/s target because the period was 50 ms.

The current TransferBackend contract is enqueue-only and explicitly avoids synchronization. I therefore would not block inside submit_transfer. Chunk progress should live in the transfer worker as an event-driven state machine: submit a chunk, record/query an event, yield while incomplete, then advance to the next chunk.

3. Stream priority did not provide cross-context QoS

The device reports a stream-priority range of [0, -5], where a lower numeric value has higher priority. The default transfer stream is already at the lowest priority available. Raising the transfer stream to -5 did not change the full-kernel result materially (p99 remained about 9.10 ms).

I also temporarily raised the decode main stream to -5; this did not help either:

• Full kernel: 9.114 ms p99
• Grid=32: 8.410 ms p99
• Green Context with 8 SMs: 9.061 ms p99

The transfer and decode workloads are in separate processes/CUDA contexts, so stream priority did not act as cross-context QoS in this experiment. A same-context or MPS setup may behave differently, but changing only the transfer stream priority is not an effective control here.

4. Green Context SM limiting did not improve decode tail

Green Context creation worked on the RTX 5090, and exact 8/16/32/64/128-SM partitions were created. Raw transfer throughput remained about 47.6 GiB/s even with only 8 SMs, but the serving p99 did not improve:

Transfer SM allocation	p99 ITL
8 SMs	9.059 ms
16 SMs	9.086 ms
32 SMs	9.058 ms
64 SMs	9.187 ms
128 SMs	9.192 ms

This only restricts the transfer context's SMs; the decode context is not isolated onto the complementary partition, while PCIe, L2, and memory-system contention remain shared. The added complexity does not appear justified for this use case.

Proposed direction

I think the Kernel backend is still worth retaining, with two explicit policies:

• Throughput-balanced: bounded grid, with grid=256 as the measured RTX 5090 starting point. It kept mean ITL near baseline and improved p99 from +47.2% to +28.9%, although rare maxima remain.
• Tail-bounded: chunk_descriptors=2048 with event-based completion backpressure and no fixed sleep. It kept the 2.497 GiB/s paced target while limiting both p99 and max regression to about 20%, at a 4.5% mean ITL cost.

For workloads with strict latency requirements, Direct remains the safest option: +4.6% p99 in this test, but only 2.404 GiB/s effective bandwidth, about 3.8% below the target. chunk=1024 smooths the tail further, but its 9% mean ITL regression makes it an unattractive default.

An API along these lines should be sufficient:

KernelOptions {
    max_grid_blocks: ...,
    chunk_descriptors: Option<usize>,
}

The values should be selected by a device-aware policy/profile rather than treated as universal constants.