karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,719 Commits 354 Branches 175 Tags
d2fbaf73c19e2bf519e4e4fc954ec8c14ba1e1fc
Commit Graph
984 Commits
Author SHA1 Message Date
Yongye ZhuandClaude Opus 4.8 d2fbaf73c1 [Model] M3 MSA indexer: unify top-k buffer, drop numpy, GPU-only decode plan 
Follow-ups on the cudagraph-capturable MSA indexer:

- Top-k: both decode and prefill now write into the single shared, persistent
  topk_indices_buffer (decode at [:, :nd], prefill at [:, nd:]) and return views
  into it -- no fresh per-step top-k allocations.
- Build the decode plan + flat page table entirely with torch on-GPU: drop numpy
  and CpuGpuBuffer; segment offsets/lengths are computed via torch.cumsum into the
  persistent int32 buffers, and the request-major page table is scattered into the
  buffer via the on-GPU page indptr (the run bounds reads by indptr, so the full
  buffer is passed and no host page count is needed).
- No GPU->CPU sync on the decode path: scalars come from host ints
  (num_decode_tokens // num_decodes), and seq_lens.cpu() is confined to the eager
  prefill branch. The impl forward (fmha OnlyScore + Triton top-k) was already
  sync-free.

test_msa_indexer_impl_matches_triton now also asserts both outputs are views into
the persistent buffer. 42/42 in test_minimax_m3.py pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Yongye Zhu <yongye@inferact.ai>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:36 +00:00
Yongye ZhuandClaude Opus 4.8 bb4844dba3 [Model] Add fmha_sm100 MSA indexer backend + fp8 index cache for MiniMax M3 
Add an SM100/Blackwell lightning-indexer impl that computes the per-128-block
QK max-scores with fmha_sm100's score-only (OnlyScore) path and selects the
top-k blocks with the existing Triton minimax_m3_index_topk kernel, mirroring
how the main MSA attention pairs the SM100 attend with Triton. Decode and
prefill requests are split manually (decode-first batch) and each side gets its
own _fmha_sm100_plan / _fmha_sm100 call. Auto-selected on SM100 when
topk_blocks in (4, 8, 16, 32) for both bf16 and fp8 index caches; falls back to
the Triton indexer otherwise. The builder declares AttentionCGSupport.NEVER
(eager; the attention is broken out of the graph by _run_attention).

Extend the fused qknorm+rope+kv-insert kernel to optionally emit fp8 (e4m3) for
the index-K cache and index-Q via a direct cast with no scale tensors (RMSNorm
outputs are O(1) and scalar scales do not change top-k ordering). Only the index
outputs go fp8; q/k/v and q_out stay bf16 and bit-identical to the existing
path. MiniMaxM3IndexerCache now accepts fp8 caches and the model allocates
index_q in the cache dtype.

Tests: test_fmha_sm100_indexer_matches_reference (bf16/fp8 x prefill/decode) and
test_msa_indexer_impl_matches_triton (full impl parity vs the Triton indexer
through the real metadata builders); fp8 fused-kernel parity is covered in
test_fused_minimax_m3_qknorm_rope_kv_insert.

AI assistance (Claude Code) was used for this change.

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:35 +00:00
Thien TranGitHubYongye Zhu
 		4c62663315 [M3] Enable FP8 sparse GQA (#45744) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-16 21:38:03 -07:00
Thien TranGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		5bdc01bcc3 [M3] Tune Triton indexer score decode for spec-decode (#45743) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 21:07:34 -07:00
liangel-02andGitHub 20a5f8b43b [FlexAttention] make custom mask mods fully cudagraphable (#45232) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-06-17 11:53:12 +08:00
Cameron QuiliciGitHubHongxia YangTJian
 		efd15e192a [Bugfix][ROCm] Fix MiniMax-M3 FP8 KV cache dtype (#45720) 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-17 03:14:45 +00:00
Xiaohong (Sean) ChenGitHubYanan Cao
 		ce3ef17bec [Kernel][Helion][1/N] Add Helion kernel for rms_norm_per_block_quant (#36895) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
 2026-06-16 22:09:52 +08:00
Li, JiangandGitHub a7fdfeef72 [CPU] Support Gemma Diffusion (#45690) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-16 14:39:56 +08:00
Wentao YeandGitHub e18fe932ca [Perf] Optimize DSv4 prefill chunk planning, 4.0% E2E Throughput Improvement (#45061) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-15 19:50:21 +00:00
+1
youkaichaoGitHubOpenAI CodexIsotr0pyThien TranBugen ZhaoJee Jee LiRoger WangfunctionstackxYongye ZhuJee Jee Li
 		0a1c5034f5 [Model] Add MiniMax M3 support (#45381) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-06-16 01:01:25 +08:00
Xin HeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kunshang Ji
 		5ed15f42b9 Fix the E8M0 scale computation in the MXFP4 (W4A4) MOE CUTLASS kernel (#43557) 
Signed-off-by: Xin He <xin3.he@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-15 06:04:54 -07:00
Li, JiangandGitHub 8760f972ca [CPU] Refine CPU attention frontend (#45391) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-14 19:26:54 -07:00
Amanzhol SalykovandGitHub 725c3bc808 [ROCm][Perf] Enable W4A16 FlyDSL MoE (#44400) 
Signed-off-by: amd-asalykov <asalykov@amd.com>
Signed-off-by: Amanzhol Salykov <asalykov@amd.com>
 2026-06-14 00:14:39 -07:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		71b961dd35 [Perf] SM90 cutlass fp8 mm supports odd M by swap_ab, 180~290% kernel performance improvement (#44572) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-13 12:05:45 -07:00
Fangzhou AiGitHubClaude Fable 5
 		badddd254f [ROCm][DSV4][Perf] Fuse inverse-RoPE and cache bf16 wo_a in o-projection (#45103) 
Signed-off-by: Fangzhou Ai <fangzhouai@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 15:57:09 -05:00
Michael GoinandGitHub 9eaacb23ec [Kernel] Consolidate Marlin thread-tile padding across all dense Marlin paths (#45295) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-06-12 13:46:21 -07:00
Isotr0pyandGitHub 6635279d8a [Migration] Migrate GGUF quantization support to plugin (#39612) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-12 12:02:21 -07:00
Xiaohong (Sean) ChenGitHubYanan Cao
 		272c16953e [Kernel][Helion][1/N] Add Helion kernel for dynamic_per_token_scaled_fp8_quant (#33790) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
 2026-06-12 12:50:06 -04:00
Yuwen ZhouandGitHub 0cd9b7af25 [CPU] Support CPU W4A16 INT4 MoE (#43409) 
Signed-off-by: yuwenzho <yuwen.zhou@intel.com>
 2026-06-12 07:12:37 +00:00
+1
Lucas WilkinsonGitHubMartin KuklaMatthew BonanniDipika SikkaNickLucchejiahancAlec KohlhoffPorras HuangClaudescoootscooob
 		eb28452b10 [Model] Add DiffusionGemma Support (#45163) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
 2026-06-11 22:17:35 -07:00
Divakar VermaandGitHub 1ce3cdc5c1 [ROCm][CI] fix fp8 support for test_deepep_moe (#45302) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2026-06-12 00:16:14 -05:00
Fangzhou AiGitHubvLLM Contributor
 		fcf5115c45 [ROCm][DSv4][Perf] Flash-decode split-K decode attention kernel (#44899) 
Co-authored-by: vLLM Contributor <contributor@vllm.ai>
 2026-06-12 03:17:52 +00:00
Wentao YeandGitHub 5a6c7b7ab5 [Bug] Fix test flashmla for DSv4 (#45052) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-11 16:22:26 -04:00
Xiaohong (Sean) ChenGitHubYanan Caomergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		2ec6594db9 [Kernel][Helion][1/N] Add Helion kernel for per_token_group_fp8_quant (#36902) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-11 08:59:08 -07:00
Jiangyun ZhuandGitHub f81daf8880 [Attention] add triton diff-kv backend for mimo (#41797) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-06-11 11:36:31 -04:00
Harry MellorandGitHub 03878d1c22 Deprecations for v0.23 and v0.24 (#44992) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-11 14:35:38 +00:00
x41lakazamandGitHub 3508cb78d4 [Bugfix] Fix broken profile_modular_kernel.py (#43300) 2026-06-11 12:17:23 +01:00
Dan BlanaruGitHubWentao Yemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		82d6b59f04 [CI/Build] Skip test_use_trtllm_attention on non-CUDA platforms (#44687) 
Signed-off-by: Dan Blanaru <48605845+DanBlanaru@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-10 18:18:42 -04:00
Ilya MarkovandGitHub 6471ec75bd [EPLB] Reject NCCL-based EPLB communicators with async EPLB (#44978) 
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
 2026-06-10 19:51:27 +00:00
JartXGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		32daf56b42 [Refactor] Rename rocm_moe.py to rocm_moe_rdna.py (#45011) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-10 17:02:09 +08:00
Jiangyun ZhuandGitHub 7a89b72564 [Perf] fuse qk rmsnorm rope gate for qwen3.5 (#44176) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-06-09 22:12:17 +08:00
Tyler Michael SmithGitHubClaude Opus 4.6
 		e2f993dc41 [WideEP] Integrate DeepEP v2 (#41183) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-06-08 18:07:29 -07:00
bnellnmandGitHub dc68bd8c41 [MoE Refactor] FusedMoE/MoERunner inversion refactor (#41184) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-06-08 10:42:58 -04:00
Agata DobrzyniewiczandGitHub 2ed0a9627b [Kernel][Test] Make kernel tests for mamba dual-HW (CUDA + XPU) (#42736) 
Signed-off-by: Dobrzyniewicz, Agata <agata.dobrzyniewicz@intel.com>
 2026-06-08 08:22:47 +08:00
JartXGitHubTJian
 		062b05ff3a [ROCm][Perf] Fused MoE W4A16 HIP kernel for AMD RDNA3 (gfx1100) (#44075) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-06 15:30:39 -05:00
Vadim GimpelsonGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		fa27d4e9cf [PERF] [Qwen3.5] Split mixed prefill+decode batches: route decodes to the recurrent kernel (#44700) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-06 22:13:50 +08:00
Tushar JainGitHubTushar Jain
 		38fd2405f3 use split_group for pytorch process group creation (#41980) 
Signed-off-by: Tushar Jain <tushar00jain@users.noreply.github.com>
Co-authored-by: Tushar Jain <tushar00jain@users.noreply.github.com>
 2026-06-04 14:36:07 -04:00
Agata DobrzyniewiczandGitHub a947f7a420 [Kernel][Test] Extend lightning_attn and awq_triton kernel tests to XPU (#43307) 
Signed-off-by: Dobrzyniewicz, Agata <agata.dobrzyniewicz@intel.com>
 2026-06-04 14:25:59 -04:00
bnellnmandGitHub 439203d32c [Bugfix] Fix test_cutlass_moe.py (#44380) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-06-04 14:18:52 -04:00
Divakar VermaGitHubAndreas Karatzas
 		06f94633e7 [ROCm][CI] Add test for Aiter unified attn kernel (#44436) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-04 16:15:05 +00:00
Michael GoinandGitHub 06ee2d8433 [Quant] Support compressed-tensors WNA8O8Int linears and WNInt embeddings (#44340) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-06-04 07:40:33 -07:00
Yongye ZhuandGitHub b5235fca2e [DSv4] Adding TRTLLM gen attention kernel (#43827) 2026-06-04 07:35:09 -07:00
Ilya MarkovGitHubMarkov IlyaNicolò LucchesiTyler Michael SmithTyler Michael Smith
 		4f423bd5bc [EPLB] Nixl communicator optimization. Zero-copy transfers (#41633) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-04 03:40:34 +00:00
Li, JiangandGitHub 823d271c0d [Attention][CPU] Standardize kv layout to blocks first (#44393) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-03 19:03:09 +08:00
Charlie FuandGitHub 71df063c49 Enable perf_token_group_quant/_C_stable_libtorch for ROCm (#42758) 
Signed-off-by: charlifu <charlifu@amd.com>
 2026-06-02 23:23:28 -07:00
SeongJun LeeGitHublesj0610
 		3099de3617 [Kernel][MoE] Add GELU_TANH to CPU, CUTLASS, and WNA16 MoE backends (#42027) 
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
 2026-06-02 17:12:08 -04:00
Fadi ArafehGitHubLi, Jiang <jiang1.li@intel.com>
 		0b25cf4419 [CPU][Perf] Enable fused kernels for GDN's gated delta rules (#43534) 
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-06-02 08:00:48 +00:00
Jee Jee LiGitHubYiliu Dong
 		559d6710bf [PERF]MiniMax-M2 gate kernel (#38445) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: qianlihuang <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
 2026-05-29 18:28:34 -07:00
bnellnmandGitHub 7b98f498cd [MoE Refactor] Remove supports_expert_map (#43108) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-05-29 17:26:56 -04:00
JartXandGitHub 0cff0741ff [Kernel][ROCm] Native W4A16 kernel for AMD RDNA3 (gfx1100) — fp16 + bf16 (#41394) 
Signed-off-by: JartX <sagformas@epdcenter.es>
 2026-05-29 11:04:40 +00:00