karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,719 Commits 354 Branches 175 Tags
d2fbaf73c19e2bf519e4e4fc954ec8c14ba1e1fc
Commit Graph
877 Commits
Author SHA1 Message Date
Yongye ZhuandClaude Opus 4.8 bb4844dba3 [Model] Add fmha_sm100 MSA indexer backend + fp8 index cache for MiniMax M3 
Add an SM100/Blackwell lightning-indexer impl that computes the per-128-block
QK max-scores with fmha_sm100's score-only (OnlyScore) path and selects the
top-k blocks with the existing Triton minimax_m3_index_topk kernel, mirroring
how the main MSA attention pairs the SM100 attend with Triton. Decode and
prefill requests are split manually (decode-first batch) and each side gets its
own _fmha_sm100_plan / _fmha_sm100 call. Auto-selected on SM100 when
topk_blocks in (4, 8, 16, 32) for both bf16 and fp8 index caches; falls back to
the Triton indexer otherwise. The builder declares AttentionCGSupport.NEVER
(eager; the attention is broken out of the graph by _run_attention).

Extend the fused qknorm+rope+kv-insert kernel to optionally emit fp8 (e4m3) for
the index-K cache and index-Q via a direct cast with no scale tensors (RMSNorm
outputs are O(1) and scalar scales do not change top-k ordering). Only the index
outputs go fp8; q/k/v and q_out stay bf16 and bit-identical to the existing
path. MiniMaxM3IndexerCache now accepts fp8 caches and the model allocates
index_q in the cache dtype.

Tests: test_fmha_sm100_indexer_matches_reference (bf16/fp8 x prefill/decode) and
test_msa_indexer_impl_matches_triton (full impl parity vs the Triton indexer
through the real metadata builders); fp8 fused-kernel parity is covered in
test_fused_minimax_m3_qknorm_rope_kv_insert.

AI assistance (Claude Code) was used for this change.

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:35 +00:00
Thien TranGitHubYongye Zhu
 		4c62663315 [M3] Enable FP8 sparse GQA (#45744) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-16 21:38:03 -07:00
Li, JiangandGitHub a7fdfeef72 [CPU] Support Gemma Diffusion (#45690) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-16 14:39:56 +08:00
+1
youkaichaoGitHubOpenAI CodexIsotr0pyThien TranBugen ZhaoJee Jee LiRoger WangfunctionstackxYongye ZhuJee Jee Li
 		0a1c5034f5 [Model] Add MiniMax M3 support (#45381) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-06-16 01:01:25 +08:00
Xin HeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kunshang Ji
 		5ed15f42b9 Fix the E8M0 scale computation in the MXFP4 (W4A4) MOE CUTLASS kernel (#43557) 
Signed-off-by: Xin He <xin3.he@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-15 06:04:54 -07:00
Li, JiangandGitHub 8760f972ca [CPU] Refine CPU attention frontend (#45391) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-14 19:26:54 -07:00
Shengqi ChenGitHubXin LiShawRongChange72
 		78e7293bb1 [Build] Fix CUDA arch build coverage gaps (#45277) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Xin Li <xinli-sw@users.noreply.github.com>
Co-authored-by: ShawRong <ShawRong@users.noreply.github.com>
Co-authored-by: Change72 <Change72@users.noreply.github.com>
 2026-06-13 22:09:20 -07:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		71b961dd35 [Perf] SM90 cutlass fp8 mm supports odd M by swap_ab, 180~290% kernel performance improvement (#44572) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-13 12:05:45 -07:00
Isotr0pyandGitHub 6635279d8a [Migration] Migrate GGUF quantization support to plugin (#39612) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-12 12:02:21 -07:00
Chris LeonardGitHubShengqi Chen
 		a014dddbaa [11b/n] Migrate Machete kernels to torch stable ABI (#45304) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-12 10:36:49 +00:00
Chris LeonardandGitHub 7021be66e8 [11a/n] Migrate Marlin kernels to torch stable ABI (#45176) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
 2026-06-11 21:22:37 -07:00
wcyGitHublyd1992
 		f06aefb4e3 [CPU] Add missing scalar fallback for CPU W4A8 INT4 GEMM (#44523) 
Signed-off-by: wcy <233313160abc@gmail.com>
Co-authored-by: lyd1992 <liuyudong@iscas.ac.cn>
 2026-06-11 08:52:01 +00:00
Juan Pérez de AlgabaandGitHub f219788f91 [Security] Fix info disclosure via int32 truncation in GGUF dequantize kernels (#44971) 
Signed-off-by: jperezde <jperezde@redhat.com>
 2026-06-11 08:05:14 +00:00
Chris LeonardGitHubShengqi Chen
 		6e64c1bab1 [10c/n] Migrate MoE kernels to torch stable ABI (#44565) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-10 23:02:26 -07:00
Yuanyuan ChenGitHubLi, Jiang <jiang1.li@intel.com>
 		0b995f8609 Use std::bit_cast for type punning in CPU kernels (#45089) 
Signed-off-by: Yuanyuan Chen <cyyever@outlook.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-06-10 22:07:44 -07:00
velonica0andGitHub f31bc2ea60 [CPU][RISC-V] Enable oneDNN W8A8 INT8 to run on RISC-V (#44478) 
Signed-off-by: velonica0 <like@mail.nankai.edu.cn>
 2026-06-11 04:09:05 +00:00
Micah WilliamsonandGitHub e1ed89dbee Revert "[Kernel] Speed up silu_and_mul_per_block_quant with warp-shuf… (#45066) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-06-09 14:12:06 -07:00
SII-yangdianGitHubSII-yangdian
 		a4b14b98c6 [Kernel] Speed up silu_and_mul_per_block_quant with warp-shuffle reduction + vectorized I/O (#44173) 
Signed-off-by: SII-yangdian <yangdian@sii.edu.cn>
Co-authored-by: SII-yangdian <yangdian@sii.edu.cn>
 2026-06-09 10:41:26 -07:00
Andreas KaratzasandGitHub 2385e140d6 [ROCm][CI] Stabilize sleep-mode memory release (#43022) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-09 16:51:12 +08:00
Charlie FuandGitHub 228bcc436b [ROCm][Kernel] Enable permute_cols for ROCm (#44674) 
Signed-off-by: charlifu <charlifu@amd.com>
 2026-06-07 09:50:03 +00:00
Markus HartikainenGitHubCursorFrida AnderssonDouglas LehrTJian
 		bc5745a00f [ROCm][MLA] Replace torch.cat in sparse-MLA forward_mqa with fused concat_mla_q (#42838) 
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
Signed-off-by: Frida Andersson <fanderss@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Frida Andersson <fanderss@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-06 18:20:50 -05:00
JartXGitHubTJian
 		062b05ff3a [ROCm][Perf] Fused MoE W4A16 HIP kernel for AMD RDNA3 (gfx1100) (#44075) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-06 15:30:39 -05:00
TJianandGitHub aa6fb8a329 [Bugfix] [ROCm] [Critical] fallback to regular abi for ROCm (#44648) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-06-05 15:51:17 +00:00
Chris LeonardandGitHub 56aff0dd15 [10/n] Migrate cuda_view and silu_and_mul_per_block_quant kernels to torch stale ABI. (#44334) 2026-06-04 20:14:43 -07:00
Yongye ZhuandGitHub b5235fca2e [DSv4] Adding TRTLLM gen attention kernel (#43827) 2026-06-04 07:35:09 -07:00
Chris LeonardGitHubShengqi Chen
 		59d0236193 [10b/n] Migrate custom all-reduce, DeepSeek V4 fused MLA, MiniMax reduce-RMS, and MXFP8 MoE to libtorch stable ABI (#44365) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-04 00:29:46 +08:00
Charlie FuandGitHub 71df063c49 Enable perf_token_group_quant/_C_stable_libtorch for ROCm (#42758) 
Signed-off-by: charlifu <charlifu@amd.com>
 2026-06-02 23:23:28 -07:00
SeongJun LeeGitHublesj0610
 		3099de3617 [Kernel][MoE] Add GELU_TANH to CPU, CUTLASS, and WNA16 MoE backends (#42027) 
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
 2026-06-02 17:12:08 -04:00
Chris LeonardandGitHub 4d93bc35c9 Migrate header files to torch stable abi (#44013) 2026-06-02 08:09:52 -07:00
Rukhaiya2004GitHubRukhaiyaAkash kaothalkarLi, Jiang <jiang1.li@intel.com>
 		689b0eeb9e [HARDWARE][POWER] Enable SHM communicator support for PowerPC (#43754) 
Signed-off-by: Rukhaiya <rukhaiya@c643n08aix1-lp1.pok.stglabs.ibm.com>
Signed-off-by: Rukhaiya <bibirukhaiya123@gmail.com>
Co-authored-by: Rukhaiya <rukhaiya@c643n08aix1-lp1.pok.stglabs.ibm.com>
Co-authored-by: Akash kaothalkar <61960177+Akashcodes732@users.noreply.github.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-06-02 18:06:32 +08:00
Fadi ArafehGitHubLi, Jiang <jiang1.li@intel.com>
 		0b25cf4419 [CPU][Perf] Enable fused kernels for GDN's gated delta rules (#43534) 
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-06-02 08:00:48 +00:00
wcyGitHubLi, Jiang <jiang1.li@intel.com>
 		98f1279815 [CPU][RISC-V] Add missing RVV cpu_types helpers for WNA16 (#42730) 
Signed-off-by: wcy <233313160abc@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-06-01 14:56:41 +08:00
Lanze LiuGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		124fac10cb [Bugfix] Fix RMSNorm kernels to multiply in weight's native dtype (#42379) 
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-29 23:16:53 -07:00
Jee Jee LiGitHubYiliu Dong
 		559d6710bf [PERF]MiniMax-M2 gate kernel (#38445) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: qianlihuang <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
 2026-05-29 18:28:34 -07:00
JartXandGitHub 0cff0741ff [Kernel][ROCm] Native W4A16 kernel for AMD RDNA3 (gfx1100) — fp16 + bf16 (#41394) 
Signed-off-by: JartX <sagformas@epdcenter.es>
 2026-05-29 11:04:40 +00:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		22a58640b4 [9/n] Migrate attention and cache kernels to torch stable ABI (continued) (#43717) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-29 04:44:45 +00:00
Matthias GehreandGitHub a9ec46d4b7 [ROCm][Perf] Support N=5 in wvSplitK skinny GEMM kernels for speculative decoding (#40687) 
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
 2026-05-28 16:28:21 +00:00
Wentao YeandGitHub 64e1218673 [Perf] Optimize moe permute by pre-allocate buffer, 9~14% kernel performance improvement (#43014) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-28 06:18:26 -07:00
tonyliu312GitHubTony Liumergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Shengqi Chen
 		6cc8577421 [Kernel] Marlin MoE: include SM 12.x in default arch list (#40923) 
Signed-off-by: Tony Liu <tonyliu0512@gmail.com>
Co-authored-by: Tony Liu <tonyliu0512@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-28 15:30:26 +08:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		284e6f543d [8/n] Migrate merge_attn_states, mamba, sampler to torch stable ABI (continued) (#43361) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-27 09:35:24 -07:00
Yongye ZhuandGitHub 6ab6ffb428 [Feat][DSV4] Fuse q pad into deepseek v4 fused kernel (#43162) 2026-05-26 05:12:54 -10:00
zhao, zhenhuiandGitHub 771e1e48b1 [CPU] Enable non-divisible GQA for decode workitems in mixed batches (#43032) 
Signed-off-by: zhejiangxiaomai <zhenhui.zhao@intel.com>
 2026-05-26 14:15:47 +08:00
Jee Jee LiandGitHub d4004455d2 [Kernel] Remove NormGateLinear (#43554) 
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-05-25 09:49:19 +00:00
Jakub ZakrzewskiGitHubWentao YeYongye Zhu
 		5bb8d2767a [Kernel] Batch invariant NVFP4 linear using cutlass (#39912) 
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-05-23 09:41:12 -04:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		a7be0f342d [7/n] Migrate pos_encoding and norm kernels to libtorch stable ABI (continued) (#43209) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-23 13:20:00 +08:00
gnovackandGitHub f743254143 DSv4 fused Q-norm kernel grid refactor (#42353) 2026-05-22 15:21:33 -07:00
velonica0GitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Li, Jiang <jiang1.li@intel.com>
 		c68c55d43e [CPU][RISC-V] Add VLEN=256 support to RVV attention kernels (#42943) 
Signed-off-by: velonica0 <like@mail.nankai.edu.cn>
Signed-off-by: velonica0 <47554626+velonica0@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-05-21 04:50:49 -07:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		07aeaf9d4d [6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued) (#42663) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-20 00:18:12 -07:00
Wentao YeandGitHub 37ece593c1 [Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement (#42774) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-18 16:38:12 -07:00
Wentao YeandGitHub 00e20e76f7 [Refactor] Remove dead cuda kernels (#42767) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-18 11:14:21 -07:00