feat(mindone/diffusers): add sana_video to diffusers #1461
Conversation
Summary of ChangesHello @Cui-yshoho, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands the video generation capabilities of the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the Sana-Video model and its associated pipelines (SanaVideoPipeline and SanaImageToVideoPipeline) into the diffusers library. The implementation is comprehensive, covering the model, pipelines, documentation, and necessary __init__ updates. While the core logic appears sound, I've identified a few areas for improvement, primarily in the documentation for clarity and in the code for better organization and maintainability. My feedback focuses on correcting some inaccuracies in the documentation and suggesting structural improvements to the code.
| <!-- Copyright 2025 The SANA-Video Authors and HuggingFace Team. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. --> | ||
|
|
||
| # SanaVideoPipeline | ||
|
|
||
| <div class="flex flex-wrap space-x-1"> | ||
| <img alt="LoRA" src="https://img.shields.io/badge/LoRA-d8b4fe?style=flat"/> | ||
| </div> | ||
|
|
||
| [SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer](https://huggingface.co/papers/2509.24695) from NVIDIA and MIT HAN Lab, by Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, Haozhe Liu, Hongwei Yi, Hao Zhang, Muyang Li, Yukang Chen, Han Cai, Sanja Fidler, Ping Luo, Song Han, Enze Xie. | ||
|
|
||
| The abstract from the paper is: | ||
|
|
||
| *We introduce SANA-Video, a small diffusion model that can efficiently generate videos up to 720x1280 resolution and minute-length duration. SANA-Video synthesizes high-resolution, high-quality and long videos with strong text-video alignment at a remarkably fast speed, deployable on RTX 5090 GPU. Two core designs ensure our efficient, effective and long video generation: (1) Linear DiT: We leverage linear attention as the core operation, which is more efficient than vanilla attention given the large number of tokens processed in video generation. (2) Constant-Memory KV cache for Block Linear Attention: we design block-wise autoregressive approach for long video generation by employing a constant-memory state, derived from the cumulative properties of linear attention. This KV cache provides the Linear DiT with global context at a fixed memory cost, eliminating the need for a traditional KV cache and enabling efficient, minute-long video generation. In addition, we explore effective data filters and model training strategies, narrowing the training cost to 12 days on 64 H100 GPUs, which is only 1% of the cost of MovieGen. Given its low cost, SANA-Video achieves competitive performance compared to modern state-of-the-art small diffusion models (e.g., Wan 2.1-1.3B and SkyReel-V2-1.3B) while being 16x faster in measured latency. Moreover, SANA-Video can be deployed on RTX 5090 GPUs with NVFP4 precision, accelerating the inference speed of generating a 5-second 720p video from 71s to 29s (2.4x speedup). In summary, SANA-Video enables low-cost, high-quality video generation. [this https URL](https://github.com/NVlabs/SANA).* | ||
|
|
||
| This pipeline was contributed by SANA Team. The original codebase can be found [here](https://github.com/NVlabs/Sana). The original weights can be found under [hf.co/Efficient-Large-Model](https://hf.co/collections/Efficient-Large-Model/sana-video). | ||
|
|
||
| Available models: | ||
|
|
||
| | Model | Recommended dtype | | ||
| |:-----:|:-----------------:| | ||
| | [`Efficient-Large-Model/SANA-Video_2B_480p_diffusers`](https://huggingface.co/Efficient-Large-Model/ANA-Video_2B_480p_diffusers) | `mindspore.bfloat16` | | ||
|
|
||
| Refer to [this](https://huggingface.co/collections/Efficient-Large-Model/sana-video) collection for more information. | ||
|
|
||
| Note: The recommended dtype mentioned is for the transformer weights. The text encoder and VAE weights must stay in `mindspore.bfloat16` or `mindspore.float32` for the model to work correctly. Please refer to the inference example below to see how to load the model with the recommended dtype. | ||
|
|
||
| ::: mindone.diffusers.SanaVideoPipeline | ||
|
|
||
| ::: mindone.diffusers.pipelines.sana.pipeline_sana_video.SanaVideoPipelineOutput |
There was a problem hiding this comment.
I've found a few small issues in this documentation file:
- The copyright header from lines 2-13 uses a mix of HTML and markdown comments. It would be cleaner to use a consistent style, preferably HTML comments (
<!-- ... -->). - In the abstract on line 25, the link to the original codebase is malformed:
[this https URL](...). It should be[this URL](...). - In the 'Available models' table on line 33, the link to the Hugging Face model has a typo:
ANA-Video_2B_480p_diffusersinstead ofSANA-Video_2B_480p_diffusers. - On line 41, the path to the output class seems incorrect. It should likely be
mindone.diffusers.pipelines.sana_video.SanaVideoPipelineOutput.
| def apply_rotary_emb( | ||
| hidden_states: ms.Tensor, | ||
| freqs_cos: ms.Tensor, | ||
| freqs_sin: ms.Tensor, | ||
| ): | ||
| x1, x2 = hidden_states.unflatten(-1, (-1, 2)).unbind(-1) | ||
| cos = freqs_cos[..., 0::2] | ||
| sin = freqs_sin[..., 1::2] | ||
| out = mint.empty_like(hidden_states) | ||
| out[..., 0::2] = x1 * cos - x2 * sin | ||
| out[..., 1::2] = x1 * sin + x2 * cos | ||
| return out.type_as(hidden_states) |
There was a problem hiding this comment.
The apply_rotary_emb function is defined inside the __call__ method. For better code organization, readability, and potential reuse, it would be better to define it as a static method of the SanaLinearAttnProcessor3_0 class.
| if not return_dict: | ||
| return (output,) | ||
|
|
||
| return Transformer2DModelOutput(sample=output) |
There was a problem hiding this comment.
The SanaVideoTransformer3DModel returns a Transformer2DModelOutput. This is misleading, as this is a 3D model that outputs a 5D video tensor. The name Transformer2DModelOutput and its docstring (which describes a 4D tensor) can cause confusion. Consider defining and using a Transformer3DModelOutput for 3D models to improve clarity and API consistency.
Cui-yshoho
force-pushed
the
sana_video_1208
branch
4 times, most recently
from
5c9a41a to
071654f
Compare
Cui-yshoho
force-pushed
the
sana_video_1208
branch
from
071654f to
571a92e
Compare
1 participant
What does this PR do?
Fixes # (issue)
Adds # (feature)
Before submitting
What's New. Here are thedocumentation guidelines
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@xxx