Yongye Zhu and Claude Opus 4.7
964c6eb485
[Attention][TokenSpeed MLA] Fix decode FP8 numerics: pass output_scale and assert FP8 Q
...
The decode kernel needs both bmm scales to recover correct outputs from an
FP8 KV cache: bmm1 (softmax_scale = scale * q_scale * k_scale) and bmm2
(output_scale = k_scale, since V is stored as V_real / k_scale). We were
only passing bmm1, which left bmm2 = 1.0 and produced silently wrong output.
Also assert query dtype is float8_e4m3fn on entry to forward_mqa.
supports_quant_query_input=True (inherited from MLACommonImpl) tells the
upstream pipeline to FP8-quantize Q via _decode_concat_quant_fp8_op; the
kernel is shape-specialized for FP8 Q + FP8 KV, so any other dtype here
means the upstream quant path didn't run and the kernel will produce
garbage. Failing loud beats failing silent.
Verified: gsm8k matches reference with TOKENSPEED_MLA decode +
FLASH_ATTN prefill on Kimi-K2.5-NVFP4 / TP=4 / B200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-05-06 02:49:09 +00:00
Yongye Zhu and Claude Opus 4.7
d0e6514bf8
[Attention] Add TOKENSPEED_MLA backend for DeepSeek R1 prefill + decode on Blackwell
...
Wires the tokenspeed_mla CuTe DSL kernels into vLLM as a new MLA backend,
covering both prefill (tokenspeed_mla_prefill) and decode
(tokenspeed_mla_decode). Targets Blackwell (SM100) with FP8 KV cache and
DeepSeek R1 MLA dimensions; users opt in via -ac
'{"backend":"TOKENSPEED_MLA","mla_prefill_backend":"TOKENSPEED_MLA"}'.
Includes numeric parity tests against the trtllm reference kernels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-05-06 02:49:09 +00:00
Matthew Bonanni and GitHub
be5983b874
[Docs] Add non-causal support to attention backend docs ( #41643 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-05-04 20:35:15 +00:00
Luka Govedič and GitHub
d58c42e19c
[vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload ( #36823 )
...
Signed-off-by: Luka Govedič <lgovedic@redhat.com >
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com >
2026-05-01 23:41:15 -04:00
John Calderon and GitHub
964a4bc2a5
[MM][CG] Support ViT CG for Qwen2.5-VL ( #40830 )
...
Signed-off-by: John Calderon <jcalderon@nvidia.com >
2026-05-02 11:10:14 +08:00
f3fef12350
[Attention] Abstract the MLA prefill backends and eliminate cuDNN ( #32623 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 13:36:20 -04:00
bc635fad23
[ROCm][Deepseek] dsv3.2 further optimization ( #41217 )
...
Signed-off-by: ganyi <ygan@amd.com >
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com >
2026-05-01 23:06:00 +09:00
sychen52 and GitHub
947138b6c2
Add nvfp4 kv cache support ( #40177 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-05-01 04:55:16 +00:00
4d5c89295b
(bugfix): block_size check for flex attn ( #41363 )
...
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-04-30 18:59:26 -07:00
b4806c8ee1
[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided ( #40960 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-30 15:33:12 -07:00
22524f7a92
[Feat] CPU fp8 attn for AMX/AVX-512 ( #39445 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-04-29 20:43:21 +08:00
+8
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Jiangyun Zhu and GitHub
e8ee2a78db
[Attention] use diff kv backend for mimo v2 flash ( #40045 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-04-24 11:25:55 +00:00
079a4cf399
[MoE] Move cutlass moe to fused_moe/experts/ ( #40574 )
...
Signed-off-by: Jackmin801 <ongjackm@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-04-24 06:05:49 +00:00
4506319a28
[compile] mla + group fp8 fusion ( #38877 )
...
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-21 23:16:58 -04:00
Shanshan Shen and GitHub
936e0b79aa
[MM][CG] Optimize default max_frames_per_batch auto-infer for ViT CUDA graph video inference ( #40445 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-04-21 14:47:53 +00:00
Shanshan Shen and GitHub
8097591286
[Doc] Update ViT CUDA graph doc for mixed (image+video) inputs ( #40355 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-04-21 02:31:09 +00:00
bai and GitHub
191e3fdaa1
Update flashinfer to 0.6.8 ( #39959 )
...
Signed-off-by: bai <v@gor.io >
2026-04-20 10:37:23 -07:00
Rita Brugarolas and GitHub
fb5635d3f9
[ROCm] Add MLA dual RMS norm fusion (Q, KV) pass for DeepSeek/Kimi-K2 ( #39242 )
...
Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com >
2026-04-20 14:56:27 +00:00
7243e02aa1
[ROCm][Feature] Enable AITER MLA attention backend to work with Eagle3 speculative decoding on ROCm ( #39616 )
...
Signed-off-by: larryli2-amd <larryli2@amd.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-04-20 09:44:43 -05:00
Jee Jee Li and GitHub
bfde49e287
[DOC] Add fuse_minimax_qk_norm ( #39782 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-04-18 00:41:37 -07:00
Michael Goin and GitHub
1174723eba
Fix TURBOQUANT backend selection in cuda.py ( #40060 )
...
Signed-off-by: Michael Goin <mgoin64@gmail.com >
2026-04-17 07:31:41 -07:00
wang.yuqi and GitHub
8d2cff8140
[Examples] Resettle Observability examples. ( #40123 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-17 03:13:31 -07:00
f4b42df048
[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity ( #38479 )
...
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-04-14 19:57:13 -07:00
1a9353bb02
[MoE] Move GPT OSS Triton kernel experts into fused_moe/experts/ ( #39007 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Jackmin801 <ongjackm@gmail.com >
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com >
2026-04-14 19:27:39 +00:00
80118853f4
[MM][Perf][CG] Support ViT full CUDA graph for Qwen3-VL video inference ( #38061 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-04-14 16:49:32 +08:00
Yongye Zhu and GitHub
739e5945dc
[Quantization] [Refactor] Create special "GptOssMxfp4MoeMethod" ( #39604 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-04-13 12:53:58 -04:00
a776a48b1c
[MoE] Move DEEP_GEMM into experts/ subdirectory ( #39005 )
...
Signed-off-by: Jackmin801 <ongjackm@gmail.com >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-04-08 19:23:08 +00:00
bnellnm and GitHub
b2b2c5239e
[MoE Refactor] Split up compressed_tensors_moe.py ( #38960 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-06 20:07:54 -04:00
1f5ec2889c
[mla] Support fused FP8/NVFP4 output quantization in MLA attention ( #35792 ) ( #36205 )
...
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com >
Signed-off-by: Carl Y <4531192+carlyou@users.noreply.github.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
2026-04-02 21:16:11 -04:00
2ce3d0ce36
[Feature] KV cache per-token-head INT8/FP8 quantization ( #38378 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: yangyang4991 <yangyang4991@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-04-02 08:13:26 -04:00
Stefano Castagnetta and GitHub
6183cae1bd
[Bugfix] Restrict TRTLLM attention to SM100, fixing GB300 (SM103) hang ( #38730 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-04-01 12:08:40 -07:00
Monishver and GitHub
c09ad767cd
Feature/silu block quant fusion v1 ( #32996 )
...
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com >
2026-04-01 18:50:43 +00:00
Li, Jiang and GitHub
36d7f19897
[CPU] Support head_size 512 in cpu_attn ( #38676 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-04-01 05:42:27 +00:00
077a9a8e37
[torch.compile] Refactor Attention Quant Fusion Pass and Remove Boilerplate ( #37373 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
2026-03-31 14:15:50 -04:00
Andreas Karatzas and GitHub
43cc5138e5
[ROCm][CI] Fix cross-attention dispatch for encoder-decoder models ( #38450 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-03-28 22:08:03 -07:00
Cyrus Leung and GitHub
a9213c0ffe
[Doc] Fix outdated reference to CUDAGraphManager ( #38209 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-03-26 01:52:38 -07:00
Rohan Potdar and GitHub
a0e8c74005
[ROCm]: Update rope+kvcache fusion conditions and disable custom op by default ( #36716 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
2026-03-25 20:58:44 +00:00
R0CKSTAR and GitHub
242c93f744
[Docs] Adds vllm-musa to custom_op.md ( #37840 )
...
Signed-off-by: Xiaodong Ye <yeahdongcn@gmail.com >
2026-03-25 11:54:36 +00:00
189ddefbfd
[ROCm] Attention selector reordering ( #36702 )
...
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com >
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
Co-authored-by: Micah Williamson <micah.williamson@amd.com >
2026-03-25 17:42:56 +08:00
Baorun (Lauren) Mu and GitHub
9d0351c91d
[Docs] Add Encoder (ViT) CUDA Graphs section to CUDA Graphs design doc ( #37914 )
...
Signed-off-by: Baorun Mu <bmu@nvidia.com >
2026-03-24 19:53:24 -07:00
82580b10ac
[Perf] Disable inductor runtime asserts by default for serving perfor… ( #37485 )
...
Signed-off-by: tianrengao <terrygao87@gmail.com >
Co-authored-by: Tianren Gao <tianren@fb.com >
2026-03-24 19:37:51 -04:00
Andrew Xia and GitHub
9ace378a63
[Frontend][Responses API] Fix arrival_time recording for TTFT on initial request ( #37498 )
...
Signed-off-by: Andrew Xia <axia@meta.com >
2026-03-23 09:58:08 +00:00
Robert Shaw and GitHub
4383f1532e
[MoE] Move PF Methods to Folder ( #35927 )
2026-03-22 02:42:59 -06:00
87bd91892f
[MoE Refactor] Mxfp4 oracle rebased ( #37128 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-03-21 03:37:04 +00:00
0674d1fee7
[PluggableLayer][MM] Add PluggableLayer for CustomQwen2Decoder ( #37293 )
...
Signed-off-by: Wangbei25 <wangbei41@huawie.com >
Signed-off-by: Wangbei25 <wangbei41@huawei.com >
Co-authored-by: Wangbei25 <wangbei41@huawie.com >
2026-03-20 06:24:07 +00:00
bnellnm and GitHub
91be5f9be3
[MoE Refactor] Rename "naive" all2all backend ( #36294 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-03-19 15:50:34 -04:00
Lucas Kabela and GitHub
7769b58307
[torch.compile][BE][Multimodal] Remove requirement to set_model_tag to avoid cache conflict ( #37345 )
...
Signed-off-by: Lucas Kabela <lucaskabela@meta.com >
2026-03-19 17:26:12 +00:00
Wei Zhao and GitHub
b36adfa349
[Perf] Set Flashinfer sparse MLA as default backend for FP8 kv cache ( #37252 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-03-17 20:09:20 +00:00
Matthew Bonanni and GitHub
93f3c8e531
[Misc] Add float16 to CacheDType ( #37199 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-03-16 13:24:48 -07:00
