Correctly support video-modality RL in Miles

[JingwenGu0829]

Goal

Miles need to fix the current buggy pipeline in supportting video inputs RL.

Problem

The rollout and training sides currently require different representations of the same video.

SGLang's HTTP /generate interface is source-oriented: video_data carries a URL, path, or data URI, and SGLang loads the video, samples frames, runs its multimodal processor, and expands the video placeholder. It cannot consume Miles's model-specific processed video tensors through the existing JSON API. Miles currently has an equivalent image transport path in compute_request_payload, but no corresponding video path.

The training side is tensor-oriented. Miles runs process_vision_info and the model processor during dataset/rollout processing, then stores the expanded prompt tokens and processor outputs in Sample.multimodal_train_inputs (Dataset, compute_prompt_ids_from_sample). Training conversion forwards only these tensors (convert_samples_to_train_data); Megatron moves them to GPU and passes them as model kwargs (training data preparation, Megatron model call). Megatron does not load a video URL during model forward.

Therefore, a source-based rollout currently requires maintaining two sets of prompts:

training:  processor-expanded prompt IDs + processed video tensors
rollout:   tokenizer-only prompt IDs + video source

The two processors must use identical frame-sampling and resizing options, otherwise training and rollout operate on different inputs or produce different placeholder lengths.

Possible approaches

1. Maintain two different prompts: video tensor on megatron side, URL on rollout side

1.1 Integrate both representations into Miles core

Miles can retain the raw source alongside the processed tensors and propagate a second prompt-ID representation through Dataset, Sample, single-turn, partial rollout, multi-turn, and prefill logprob recomputation.

This can support the complete rollout feature matrix, but it is invasive: generic rollout and scoring code must inherit the video-specific dual-prompt contract, so in partial rollout and multiturn, for each \generate sent the prompt needs to be re-concatenated again.

This fundementally misaligns with the basic logic of miles&slime family of RL framework (see the attached image below)

Here is what it looks like if we choose this way: #1689

1.2 Implement source-based video rollout as an experimental example

A custom generate function can own video prompt construction and the SGLang payload while delegating non-video samples to the existing generators. Single-turn, partial continuation, and multi-turn suffix reconstruction can then be implemented without changing the core single-turn or multi-turn semantics.

The limitation is that unrelated core consumers do not automatically understand this custom representation. In particular, --recompute-logprobs-via-prefill cannot correctly replay the video request without either a custom rollout/scoring implementation or a new generic prefill-payload hook. Other custom generate functions also do not gain video support automatically. In this way video understanding can't serve as a generic feature of Miles, but otherwise as an independent example feature.

1.3 Other Distributed source constraints on either solution

Passing a source instead of video data also assumes every SGLang worker can resolve the same value. Local paths require identical shared mounts; private or expiring URLs require consistent credentials; remote loading adds latency and failure modes; mutable URLs can return different content across retries. SGLang also decodes the video again on every request, including resumed or multi-turn requests.

These constraints become significant in multi-node RL and are independent of prompt propagation correctness.

2. Unifying on processed video tensors

Using the locally processed video representation for both training and rollout would remove the shared-URL requirement and make the Miles processor the single source of truth. However, the current SGLang HTTP contract cannot accept or reuse the processor tensor bundle and its associated grid/frame metadata. Encoding these tensors into JSON would also be large and inefficient, while treating video frames as ordinary images can change model-specific video semantics.

A proper tensor-based design would require coordinated Miles and SGLang changes:

1. define an engine-neutral processed-multimodal contract containing tensors, processor metadata, and canonical prompt IDs;
2. add a binary/shared-memory transport instead of embedding large tensors in JSON;
3. extend SGLang to accept preprocessed video inputs and bypass its media loading and processor expansion;
4. validate that generation, partial continuation, multi-turn requests, and prefill scoring all use the same canonical token sequence;
5. preserve the existing source-based API as an optional lightweight path.

[coding-famer]

I previously implemented something similar based on Approach 1(for slime and single-turn only) https://github.com/coding-famer/slime/tree/feat/video.

One thing worth noting is that SGLang applies additional handling to multimodal token IDs, so the processor-expanded prompt token IDs can be sent directly in the request. SGLang will internally convert them back to text before multimodal processing, with the trade-off that this is no longer a true TITO path. Therefore, maintaining a single set of prompt token IDs should be sufficient.

It may also be worth implementing lazy loading for multimodal inputs alongside video support and enabling it by default when video inputs are present(slime/miles will load all images to the dataset when initializing now). I also experimented with this approach here: https://github.com/coding-famer/slime/tree/feat/lazy-multimodal.

If there’s anything I can help with going forward, I’d be very happy to contribute and collaborate on this.

[JingwenGu0829]

@coding-famer Thanks for the comment. However, I think SGLang’s /generate does not support processor-expanded video prompt token IDs, so a single set of prompt token IDs is not sufficient for video inputs

[coding-famer]

@JingwenGu0829 I see. I checked again and confirmed that SGLang does not currently support this. In my previous implementation, I modified SGLang’s source code and submitted a PR, but it looks like those changes were ultimately not merged. I believe this should ideally be handled on the SGLang side, since expanded token inputs are already supported for images. I previously verified the feasibility of using expanded tokens for videos with Slime. Although my fork now appears to send text to the /generate endpoint, that is because Slime’s multimodal handling was changed later.