Upgrade amd-integration branch to v0.5.3 tag from upstream. - #173
Merged
diptorupd merged 297 commits intoFeb 23, 2026
Merged
Conversation
<!-- .github/pull_request_template.md --> ## 📌 Description this pr adds gemm overlapped with two-shot allreduce (with multimem instructions) <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…fer-ai#1711) <!-- .github/pull_request_template.md --> ## 📌 Description This PR removes some dependencies on `torch.utils.cpp_extension": 1. After flashinfer-ai#1608 , we should no longer rely on `_get_cuda_arch_flags`, this PR fixes the issue. 2. Remove dependency on `_get_pybind11_abi_build_flags` as mentioned in flashinfer-ai#1591. After flashinfer-ai#1641 we will remove all dependencies on `torch.utils.cpp_extension`. ## 🔍 Related Issues flashinfer-ai#1591 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
) <!-- .github/pull_request_template.md --> ## 📌 Description skip unsupported (non-SM90) test cases for xqa. Added in cuda device SM check for xqa module. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: jimmzhou <jimmzhou@nvidia.com>
In our tests DeepSeek-V3 model quality was severely degraded when using `MaxNumTopGroups = 16`. This PR makes it so we use `MaxNumTopGroups = 16` for Kimi-K2 and `MaxNumTopGroups = 4` for DeepSeek.
…i#1685) <!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
<!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
The macro DISPATCH_COMPUTE_CAP_NUM_THREADS is used for compatibility with kernels less than sm80, but some kernel launches are not included in it, which breaks compatibility with sm75. flashinfer-ai#1697 <!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> Please help check the impact on kernel correctness. Signed-off-by: IriKa Qiu <qiujie.jq@gmail.com>
## 📌 Description
Summary: fix `flashinfer download-cubin` and double download speed
The previous version was attempting to use `tqdm` module as callable.
The new version replaces the custom logging handler with
`tqdm.contrib.logging` helper. Fixes `'module' object is not callable.
Did you mean: 'tqdm.tqdm(...)'?`
The `ThreadPoolExecutor` is now correctly wrapped in a context manager.
The progress bar is updated by a future done callback.
`requests.get` creates a new session object for each GET request. This
is pretty inefficient, because each request has to perform DNS lookup,
TCP handshake, and HTTPS handshake including certificate validation.
A `requests.Session` can be shared between requests and across threads
to make use of HTTP keep-alive. This change more than doubles the
download speed and reduces the load on the server.
## 🔍 Related Issues
Fixes flashinfer-ai#1728
## 🚀 Pull Request Checklist
Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.
### ✅ Pre-commit Checks
- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.
> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).
## 🧪 Tests
- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).
## Reviewer Notes
<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->
---------
Signed-off-by: Christian Heimes <cheimes@redhat.com>
<!-- .github/pull_request_template.md --> ## 📌 Description User prefers a standalone wheel for cubin files in flashinfer, this PR implements this feature. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…de (flashinfer-ai#1719) <!-- .github/pull_request_template.md --> ## 📌 Description Add device debug information and disable cuda kernel optimization in verbose mode. ## 🔍 Related Issues N/A ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…flashinfer-ai#1737) <!-- .github/pull_request_template.md --> ## 📌 Description Follow up of flashinfer-ai#1718 , this PR adds the github workload to build flashinfer-cubin wheel and publish it to pypi. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…lashinfer-ai#1739) <!-- .github/pull_request_template.md --> ## 📌 Description This PR bumps the base image of [flashinfer-ci-cu130](https://hub.docker.com/r/flashinfer/flashinfer-ci-cu130) from [nvidia/cuda:13.0.0-devel-ubuntu24.04](https://hub.docker.com/layers/nvidia/cuda/13.0.0-devel-ubuntu24.04/images/sha256-435220c0fef35cbf712e11999f8670a83835ef3cdd18564e5e8122f83078c88c) to [nvidia/cuda:13.0.1-devel-ubuntu24.04](https://hub.docker.com/layers/nvidia/cuda/13.0.1-devel-ubuntu24.04/images/sha256-84e5f33efdccadc21599978ea7e6ed80cbe19bb432f9c339b15a66c4025ac983) The updated base image container contains the CUDA Toolkit 13.0 Update 1 <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…sed on CUDA Version (flashinfer-ai#1742) <!-- .github/pull_request_template.md --> ## 📌 Description Current `flashinfer-ci-cu[12x, 130]` containers all install [nvidia-cudnn-cu12](https://pypi.org/project/nvidia-cudnn-cu12/). However, in CUDA 13 environments, [nvidia-cudnn-cu13](https://pypi.org/project/nvidia-cudnn-cu13/) should be installed. The PR modifies `install_python_packages.sh` such that if CUDA 13 is used, `nvidia-cudnn-cu13>=9.12.0.46` is installed (note that [9.12.0.46](https://pypi.org/project/nvidia-cudnn-cu13/#history) is the earliest version) <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
<!-- .github/pull_request_template.md --> ## 📌 Description Unit test fixes: * Refactored test_mla_decode_kernel to run from pytest * Added skip to test_mnnvl_custom_comm when world size is too large * Added asserts to the cascade API when not using Hopper ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [V] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [V] Tests have been added or updated as needed. - [V] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
<!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> The PR add support for GroupedGemm tvm_binding from FlashInfer side. - ```flashinfer/tvm_binding/grouped_gemm_fp8.cu``` contains implementation of dispatching templates to ```group_gemm::CutlassFP8GroupwiseScaledGroupGEMMSM100```, supporting JIT compilation - ```flashinfer/tvm_binding/grouped_gemm_fp8_jit_tvm_binding.cu``` contains declaration of above function - ```flashinfer/flashinfer/jit/gemm``` contains interface exposed to tvm ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: Anrui Liu <anruil@catalyst-fleet1.cs.cmu.edu>
<!-- .github/pull_request_template.md --> ## 📌 Description enable the test as mentioned in flashinfer-ai#1697 ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes cc: @yzh119 Co-authored-by: Zihao Ye <expye@outlook.com>
<!-- .github/pull_request_template.md --> ## 📌 Description There was an issue in parsing the torch major version, so this PR fixes the problem. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
<!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues flashinfer-ai#1720 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: Avery Yingyi Huang <yingyihuang2000@outlook.com>
…` on B40 (flashinfer-ai#1750) <!-- .github/pull_request_template.md --> ## 📌 Description Slightly increase `atol` from `2e-3` to `3e-3` to pass the unit test. Also, explicitly specify the backend to make reference choice deterministic. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…nfer-ai#1754) <!-- .github/pull_request_template.md --> ## 📌 Description Temporarily marking test_trtllm_gen_fused_moe mxfp8_bf16 cases as xfail until we converge on fix without causing regression on B200. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: jimmzhou <jimmzhou@nvidia.com>
…er-ai#1756) <!-- .github/pull_request_template.md --> ## 📌 Description Current PR: * Adds a compute capability check function (`filter_backends_by_compute_capability()`) to `flashinfer_benchmark.py`. Backends are now skipped during benchmarks if not supported on the current compute capability. - Previous behavior of unsupported cases was undefined. - Dictionary of supported routine, compute capability, and backend combinations are placed in `flashinfer_benchmark_utils.py` * Updates the `benchmarks/README.md` file to document a support matrix on the bottom. * Updates the sample testlist and outputs with the latest `flashinfer_benchmark.py` usage and sample outputs. * Updates `README.md` to make it clear that SM architecture 75 is supported. <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…hinfer-ai#1760) <!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues comments in flashinfer-ai#1453 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
… list (flashinfer-ai#1757) <!-- .github/pull_request_template.md --> ## 📌 Description fix flashinfer-ai#1745 ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Co-authored-by: Zihao Ye <yezihhhao@gmail.com> Co-authored-by: Zihao Ye <98052487+zihaoye@users.noreply.github.com>
…-ai#1759) <!-- .github/pull_request_template.md --> ## 📌 Description Remove trailing slash from FLASHINFER_CUBINS_REPOSITORY before concatenating the filename, to avoid redundant double slashes in the URL. Even though servers may handle double slashes, this makes URL construction consistent and cleaner. <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> --------- Signed-off-by: Tarun Kumar <takumar@redhat.com>
…test_trtllm_batch_decode mismatch error (flashinfer-ai#1755)
<!-- .github/pull_request_template.md --> ## 📌 Description Fix the CI of flashinfer-ai#1641 ## 🔍 Related Issues flashinfer-ai#1641 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…tion (flashinfer-ai#2084) <!-- .github/pull_request_template.md --> ## 📌 Description - change `bmm1_scale` and `bmm2_scale` to `Union[float, torch.Tensor]`. notice that when using tensor, it must be applied by log2e - **remove the `bmm1_scale_log2_tensor` and `bmm2_scale_tensor` in the `xqa_batch_decode_with_kv_cache_mla`** - update trtllm-gen FMHA kernels TODO: do the same refactor for xqa kernels. The support for the device side scales was removed in flashinfer-ai#2033 ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Attention scale parameters now accept either floats or 1-element tensors across prefill, decode and runtime; tensor scales are validated and applied on-device and pointer-backed scale paths are supported. * **Chores** * Updated FMHA artifact path and checksum constants; added a public utility import and removed an obsolete inline comment. * **Tests** * Updated tests to exercise device/tensor-or-scalar scale flows, removed legacy per-tensor call-site args, and added device-scale parametrization for several test variants. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
<!-- .github/pull_request_template.md --> ## 📌 Description Add shuffling and blockmajorK layout in dpskv3 fused_moe fp8_blockscaled tests. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded MoE test suite with per-expert weight shuffling, optional block-layout conversion, selectable weight-processing modes, and dynamic kernel flags. * Added a reference FP8 block-scale validation path and centralized accuracy checks for clearer correctness verification. * **Refactor** * Centralized test utilities: quantization mode and test-skip logic moved into shared helpers for consistent gating across MoE tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Zihao Ye <expye@outlook.com>
<!-- .github/pull_request_template.md --> ## 📌 Description Added DSR1 MLA test, and split up the trtllm_batch_decode_mla function. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved test suite for batch decoding by making maximum sequence length configurable, adding parameterized runs across short and long lengths, and introducing a compatibility wrapper to preserve legacy behavior. This enhances coverage and validation across varied sequence-length scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Zihao Ye <expye@outlook.com>
<!-- .github/pull_request_template.md --> ## 📌 Description In flashinfer-ai#1898, it was raised that trtllm-gen's attention kernels fail for batch size 1. The prefill kernel was fixed in flashinfer-ai#1912 and prefill tests have been enabled. Further updates to trtllm-gen kernels have also fixed the decode batch size 1 issue. Current PR re-enables testing. <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded batch_decode test scenarios to cover additional small-batch and page-size combinations. * Increased coverage for max_in_kv_len by testing multiple length options instead of a single value. * Restored previously marked-as-expected-failure case to run normally, improving overall test pass coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Zihao Ye <expye@outlook.com>
…ai#2113 (flashinfer-ai#2114) <!-- .github/pull_request_template.md --> ## 📌 Description This pr adds a parameter `return_lse_base_on_e` to control the base of LSE returned by MLA. Default to `False`, which keeps the same with current implementation. If `return_lse_base_on_e` is `True`, multiply the final LSE by `loge2` to maintain consistency with the standard softmax and FA3. ## 🔍 Related Issues flashinfer-ai#2113 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a run-time option to control whether returned log‑sum‑exp (LSE) baselines are scaled by ln(2) (default: disabled). * **Bug Fixes** * Conditional scaling ensures returned LSE values are consistent when the option is enabled, improving numerical consistency. * **Chores** * The new option is exposed in public APIs and bindings and is propagated through the execution path. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: augusto.yjh <augusto.yjh@antgroup.com>
<!-- .github/pull_request_template.md --> ## 📌 Description Update xqa license based on NVIDIA/TensorRT-LLM#8807 <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues flashinfer-ai#1977 <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated project licensing to Apache License 2.0 with extended copyright years through 2025. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Qidi Sang <200703406+qsang-nv@users.noreply.github.com>
<!-- .github/pull_request_template.md --> ## 📌 Description <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [ ] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [ ] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Attention ops now accept tensor-based per-head scaling (q/kv) in C++ and Python paths, enabling dynamic or per-tensor quantization scales. * Python APIs and docs updated to accept float or tensor scales. * **Tests** * Batch-decode tests adjusted to use per-sequence cache/block sizing for more accurate memory dimensioning. * **Documentation** * Docstrings updated to describe tensor-or-scalar scale inputs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Qidi Sang <200703406+qsang-nv@users.noreply.github.com>
<!-- .github/pull_request_template.md --> ## 📌 Description 9.0a was removed from installation documentation by accident, in some recent PRs. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. -->
…er-ai#2086)" This reverts commit 9a79b78.
…cubin (flashinfer-ai#2123) <!-- .github/pull_request_template.md --> ## 📌 Description flashinfer-cubin package building failed because we flashinfer/utils.py relies on nvidia-ml-py which is not specified as part of build system requirements of the package. ## 🔍 Related Issues <!-- Link any related issues here --> ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [x] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added a new build system dependency to support enhanced system functionality. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
- IS_CUDA blocks take upstream v0.5.3 verbatim for all shared files - IS_HIP blocks restored from amd-integration HEAD - jit/attention/__init__.py: IS_CUDA uses .modules, IS_HIP uses .modules_hip - jit/activation.py: IS_CUDA uses TVM FFI API, IS_HIP keeps pytorch_extension_utils - jit/__init__.py: IS_CUDA adds JitSpecRegistry/JitSpecStatus/MissingJITCacheError, removes load_cuda_ops from IS_HIP (deprecated upstream) - __init__.py: IS_CUDA adds new v0.5.3 symbols (nvfp4, xqa_mla, layernorm, etc.) - Rename pytorch_hip.py -> modules_hip.py for consistency with upstream rename - jit/env.py and jit/core.py: take --theirs (IS_HIP restored in follow-up commits) Co-authored-by: Cursor <cursoragent@cursor.com>
- Wrap IS_CUDA block with if IS_CUDA guard (keeps all v0.5.3 CUDA logic)
- Add elif IS_HIP block with:
- has_amd_flashinfer_jit_cache(): detect amd_flashinfer_jit_cache package
- _get_aot_dir_hip(): priority amd_flashinfer_jit_cache.get_jit_cache_dir()
with version check, fallback to _package_root / "data" / "aot"
- _get_workspace_dir_name(): GCN arch from torch device props, now versioned
as FLASHINFER_CACHE_DIR / flashinfer_version / arch (matches CUDA pattern)
- FLASHINFER_INCLUDE_DIR / FLASHINFER_CSRC_DIR from get_include_paths
- Remove FIXME comment that referenced v0.4.1 upgrade
Co-authored-by: Cursor <cursoragent@cursor.com>
- Conditional imports: IS_CUDA uses tvm_ffi/CompilationContext/cpp_ext, IS_HIP uses CompilationContext_hip/cpp_ext_hip - check_rocm_arch() restored from amd-integration HEAD using hip_utils - JitSpecStatus, JitSpecRegistry, jit_spec_registry: no CUDA dependency, ported to HIP as-is (identical implementation) - JitSpec.load(): IS_HIP uses torch.ops/torch.classes.load_library(), IS_CUDA uses tvm_ffi.load_module() - gen_jit_spec(): IS_CUDA uses v0.5.3 debug/verbose flags and check_cuda_arch(), IS_HIP uses HIP cflags from amd-integration HEAD and check_rocm_arch(); both paths adopt new jit_spec_registry.register(spec) pattern - load_cuda_ops() not restored: removed upstream and unused in HIP path Co-authored-by: Cursor <cursoragent@cursor.com>
- get_jit_cache_dir() already present (no change needed) - Update fallback_version from 0.0.0 to 0.5.3 to align with upgrade - build_backend.py: aot_hip.compile_and_package_modules() API unchanged Co-authored-by: Cursor <cursoragent@cursor.com>
utils.py: - Make pynvml a lazy import inside get_gpu_memory_bandwidth(); pynvml is NVIDIA-only and was never reachable on ROCm (function raises ValueError for non-CUDA devices before the import), but the top-level import caused ModuleNotFoundError on every flashinfer import on HIP systems activation.py: - Guard import of get_fp4_quantization_module with IS_CUDA; fp4_quantization module unconditionally imports CUDA-only jit symbols (sm121a_nvcc_flags) that do not exist on HIP, causing ImportError on module load jit/attention/modules_hip.py: - Replace shared generate_additional_params (from .utils) with a HIP-specific variant that emits std::optional<at::Tensor> / at::Tensor PyTorch types; the shared CUDA version emits Optional<ffi::Tensor> (TVM FFI types) which caused a C++ compile error in the HIP pybind templates .pre-commit-config.yaml: - Add --ignore-missing-imports to the mypy pre-commit hook args; torch, jinja2, pynvml, and other runtime deps are absent from the isolation env (ROCm torch ships from AMD's repo, not PyPI) but were causing pre-existing mypy failures for every flashinfer file that imports torch Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR upgrades the amd-integration branch to v0.5.3 tag from upstream, migrating from PyTorch-based tensor bindings to TVM FFI (Foreign Function Interface) for better framework interoperability. The changes replace PyTorch-specific TORCH_LIBRARY_FRAGMENT macros with TVM_FFI_DLL_EXPORT_TYPED_FUNC, update tensor type definitions from at::Tensor to TensorView, and modernize error handling from TORCH_CHECK to TVM_FFI_ICHECK.
Changes:
- Migrated from PyTorch tensor bindings to TVM FFI throughout the codebase
- Added new utility functions and type aliases (ConstBool, computeEffectiveRows, etc.)
- Updated build infrastructure including Docker configuration, CI workflows, and version management
- Added new benchmarking utilities and test coverage improvements
Reviewed changes
Copilot reviewed 139 out of 602 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| csrc/nv_internal/include/tensorrt_llm/common/cudaUtils.h | Added ConstBool type alias |
| csrc/nv_internal/cpp/kernels/quantization.cu | Added computeEffectiveRows helper and new quantization function |
| csrc/logging.cc | Migrated from TORCH_LIBRARY_FRAGMENT to TVM_FFI_DLL_EXPORT_TYPED_FUNC |
| csrc/group_gemm_sm90.cu | Updated from PyTorch tensors to TensorView with TVM FFI |
| csrc/group_gemm_sm120_binding.cu | Migrated binding from PyTorch to TVM FFI |
| csrc/group_gemm_sm100_binding.cu | Migrated binding with multiple function exports |
| csrc/group_gemm_mxfp4_groupwise_sm100.cu | Updated dispatch macros and error handling |
| csrc/group_gemm_fp8_groupwise_sm120.cu | Migrated to TVM FFI with updated contiguity checks |
| csrc/group_gemm_fp8_groupwise_sm100.cu | Updated to TensorView and TVM error checking |
| csrc/group_gemm.cu | Migrated segment GEMM to TVM FFI |
| csrc/gemm_sm120_binding.cu | Updated binding to TVM FFI |
| csrc/gemm_sm100_binding.cu | Updated binding to TVM FFI |
| csrc/gemm_groupwise_sm120.cu | Migrated with updated dimension access methods |
| csrc/gemm_groupwise_sm100.cu | Updated to TensorView with improved error messages |
| csrc/fused_moe/cutlass_backend/cutlass_fused_moe_instantiation.cu | Added new template instantiations |
| csrc/fmha_cutlass_sm100_pybind.cu | Removed PyTorch pybind file |
| csrc/fmha_cutlass_sm100_binding.cu | Added TVM FFI binding file |
| csrc/fmha_cutlass_sm100.cu | Migrated to TVM FFI with Optional support |
| csrc/flashinfer_xqa_ops.cu | Removed PyTorch ops file |
| csrc/flashinfer_xqa_binding.cu | Added TVM FFI binding file |
| csrc/flashinfer_sampling_ops.cu | Removed PyTorch ops file |
| csrc/flashinfer_sampling_binding.cu | Added TVM FFI binding file |
| csrc/flashinfer_rope_ops.cu | Removed PyTorch ops file |
| csrc/flashinfer_rope_binding.cu | Added TVM FFI binding file |
| csrc/flashinfer_quantization_binding.cu | Updated binding exports |
| csrc/flashinfer_page_binding.cu | Added page operations binding |
| csrc/flashinfer_norm_binding.cu | Added normalization operations binding |
| csrc/flashinfer_mla_binding.cu | Updated MLA binding exports |
| csrc/flashinfer_gemm_sm90_binding.cu | Updated GEMM SM90 binding |
| csrc/flashinfer_gemm_binding.cu | Updated GEMM binding exports |
| csrc/flashinfer_cascade_ops.cu | Removed PyTorch ops file |
| csrc/flashinfer_cascade_binding.cu | Added TVM FFI binding file |
| csrc/dsv3_router_gemm.cu | Added new router GEMM implementation |
| csrc/cutlass_mla.cu | Migrated to TVM FFI |
| csrc/cascade.cu | Migrated cascade operations to TVM FFI |
| csrc/bmm_fp8.cu | Migrated batched matrix multiply to TVM FFI |
| csrc/blackwell_fmha_plan.cu | Updated planning function to TVM FFI |
| csrc/batch_prefill_sm90_jit_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_prefill_sm90_jit_binding.cu | Added TVM FFI binding file |
| csrc/batch_prefill_sm90_customize_config.jinja | Removed pytorch_extension_utils include |
| csrc/batch_prefill_sm90_config.inc | Removed config file |
| csrc/batch_prefill_jit_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_prefill_jit_binding.cu | Added TVM FFI binding file with additional parameters |
| csrc/batch_prefill_fp8_sm90.cu | Migrated to TVM FFI with Array support |
| csrc/batch_prefill_config.inc | Removed config file |
| csrc/batch_pod_kernel_inst.jinja | Added batch POD kernel instantiation template |
| csrc/batch_pod_jit_binding.cu | Added batch POD JIT binding |
| csrc/batch_pod_customize_config.jinja | Added batch POD config template |
| csrc/batch_mla_sm90_run.cu | Migrated to TVM FFI with new parameter |
| csrc/batch_mla_sm90_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_mla_sm90_plan.cu | Updated planning to return Array |
| csrc/batch_mla_sm90_binding.cu | Added TVM FFI binding |
| csrc/batch_mla_run.cu | Migrated to TVM FFI |
| csrc/batch_mla_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_mla_plan.cu | Updated to return Array |
| csrc/batch_mla_config.jinja | Updated parameter type |
| csrc/batch_mla_binding.cu | Added TVM FFI binding |
| csrc/batch_decode_mla_run.cu | Migrated to TVM FFI |
| csrc/batch_decode_mla_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_decode_mla_plan.cu | Updated to return Array |
| csrc/batch_decode_mla_cute_sm80.cu | Migrated to TVM FFI |
| csrc/batch_decode_mla_binding.cu | Added TVM FFI binding |
| csrc/batch_decode_jit_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_decode_jit_binding.cu | Added TVM FFI binding |
| csrc/batch_decode_config.inc | Removed config file |
| csrc/batch_decode.cu | Migrated to TVM FFI |
| csrc/batch_attention_jit_pybind.cu | Removed PyTorch pybind file |
| csrc/batch_attention_jit_binding.cu | Added TVM FFI binding |
| csrc/batch_attention_customize_config.jinja | Added v_scale parameter |
| csrc/batch_attention.cu | Migrated to TVM FFI with v_scale support |
| csrc/aot_extension_utils.h | Removed file |
| csrc/activation.cu | Removed file |
| ci/docker-tags.yml | Added Docker image tags |
| ci/bash.sh | Added docker pull command |
| build_utils.py | Added shared build utilities |
| build_backend.py | Added custom build backend |
| benchmarks/routines/moe.py | Updated benchmark utilities |
| benchmarks/routines/flashinfer_benchmark_utils.py | Added backend filtering by compute capability |
| benchmarks/flashinfer_benchmark.py | Added use_cupti argument |
| benchmarks/bench_trtllm_gen_mla.py | Fixed FLOPS calculation |
| benchmarks/bench_sampling.py | Added new sampling benchmarks |
| benchmarks/bench_mm_fp8.py | Added FP8 matrix multiplication benchmark |
| benchmarks/bench_hopper_fp8_attention.py | Updated FLOPS calculation |
| benchmarks/bench_hopper_attention.py | Updated FLOPS calculations |
| benchmarks/bench_groupwise_gemm_fp8_blackwell.py | Added scale_major_mode parameter |
| benchmarks/bench_block_sparse_attention.py | Updated FLOPS calculation |
| benchmarks/bench_blackwell_attention.py | Updated FLOPS calculation and added head_dim=64 tests |
| amd-flashinfer-jit-cache/pyproject.toml | Updated fallback version |
| CONTRIBUTING.md | Updated versioning scheme documentation |
| 3rdparty/cutlass | Updated submodule commit |
| .pre-commit-config.yaml | Added mypy configuration |
| .gitmodules | Added spdlog submodule |
| .github/workflows/update-codeowners.yml | Added CODEOWNERS update workflow |
| .github/workflows/release_wheel_x86_64.yml | Removed workflow |
| .github/workflows/release_wheel_sglang_x86_64.yml | Removed workflow |
| .github/workflows/release_wheel_aarch64.yml | Removed workflow |
| .github/workflows/release_pypi_sdist.yml | Removed workflow |
| .github/workflows/release-ci-docker.yml | Enhanced with tagging and PR automation |
| .github/workflows/new-issue.yml | Added issue triage workflow |
| .github/workflows/build-doc.yml | Added PR support |
| .github/CODEOWNERS | Added CODEOWNERS file |
| .devcontainer/cuda/install/install_python_packages.sh | Removed file |
| .devcontainer/cuda/install/install_python.sh | Removed file |
| .devcontainer/cu130/devcontainer.json | Added CUDA 13.0 devcontainer |
| .devcontainer/cu129/devcontainer.json | Added CUDA 12.9 devcontainer |
| .devcontainer/cu128/devcontainer.json | Added CUDA 12.8 devcontainer |
| .devcontainer/cu126/devcontainer.json | Updated devcontainer configuration |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -1181,6 +1181,9 @@ using Int = ConstExprWrapper<int, VALUE>; | |||
| template <bool VALUE> | |||
| using Bool = ConstExprWrapper<bool, VALUE>; | |||
|
|
|||
| template <bool VALUE> | |||
| using ConstBool = ConstExprWrapper<bool, VALUE>; | |||
There was a problem hiding this comment.
ConstBool is a duplicate alias of the existing Bool type alias defined at line 1182. Both are ConstExprWrapper<bool, VALUE>. Consider removing the duplicate or renaming if they serve different purposes.
Suggested change
| using ConstBool = ConstExprWrapper<bool, VALUE>; | |
| using ConstBool = Bool<VALUE>; |
| TensorView page_locked_int_workspace_buffer, TensorView qo_indptr, TensorView kv_indptr, | ||
| TensorView kv_len_arr, int64_t total_num_rows, int64_t batch_size, int64_t num_qo_heads, | ||
| int64_t num_kv_heads, int64_t page_size, bool enable_cuda_graph, int64_t head_dim_qk, | ||
| int64_t head_dim_vo, bool causal, int64_t window_left, int64_t fixed_split_size, |
There was a problem hiding this comment.
The parameter 'disable_split_kv' is declared as int64_t but appears to be a boolean flag based on its name. Consider changing the type to bool for clarity.
diptorupd
changed the title
This was referenced Feb 21, 2026
diptorupd
merged commit Feb 23, 2026
782c68a
into
AMD-Ecosystem:amd-integration
0 of 9 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates amd-flashinfer to v0.5.3 tag from upstream