karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,717 Commits 354 Branches 175 Tags
bb4844dba3c620490f190207f00a597ab7d08507
Commit Graph
17717 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yongye ZhuandClaude Opus 4.8 bb4844dba3 [Model] Add fmha_sm100 MSA indexer backend + fp8 index cache for MiniMax M3 
Add an SM100/Blackwell lightning-indexer impl that computes the per-128-block
QK max-scores with fmha_sm100's score-only (OnlyScore) path and selects the
top-k blocks with the existing Triton minimax_m3_index_topk kernel, mirroring
how the main MSA attention pairs the SM100 attend with Triton. Decode and
prefill requests are split manually (decode-first batch) and each side gets its
own _fmha_sm100_plan / _fmha_sm100 call. Auto-selected on SM100 when
topk_blocks in (4, 8, 16, 32) for both bf16 and fp8 index caches; falls back to
the Triton indexer otherwise. The builder declares AttentionCGSupport.NEVER
(eager; the attention is broken out of the graph by _run_attention).

Extend the fused qknorm+rope+kv-insert kernel to optionally emit fp8 (e4m3) for
the index-K cache and index-Q via a direct cast with no scale tensors (RMSNorm
outputs are O(1) and scalar scales do not change top-k ordering). Only the index
outputs go fp8; q/k/v and q_out stay bf16 and bit-identical to the existing
path. MiniMaxM3IndexerCache now accepts fp8 caches and the model allocates
index_q in the cache dtype.

Tests: test_fmha_sm100_indexer_matches_reference (bf16/fp8 x prefill/decode) and
test_msa_indexer_impl_matches_triton (full impl parity vs the Triton indexer
through the real metadata builders); fp8 fused-kernel parity is covered in
test_fused_minimax_m3_qknorm_rope_kv_insert.

AI assistance (Claude Code) was used for this change.

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:35 +00:00
Thien TranGitHubYongye Zhu
 		4c62663315 [M3] Enable FP8 sparse GQA (#45744) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-16 21:38:03 -07:00
Itay AlroyGitHubovidiusm
 		d78650cf97 [CI][NIXL] Pin NIXL to 1.2.0 (#45843) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Signed-off-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com>
Co-authored-by: ovidiusm <ovidium@nvidia.com>
 2026-06-16 21:29:34 -07:00
Thien TranGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		5bdc01bcc3 [M3] Tune Triton indexer score decode for spec-decode (#45743) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 21:07:34 -07:00
liangel-02andGitHub 20a5f8b43b [FlexAttention] make custom mask mods fully cudagraphable (#45232) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-06-17 11:53:12 +08:00
Ting SUNGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7b5d60cc37 [Bugfix][V1] Clean up compiled-model bytecode hooks on VllmRunner exit (#45195) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 20:31:17 -07:00
Nick HillandGitHub 14b438a98b [ModelRunnerV2] Various model/config compatibility fixes (#45868) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-17 03:23:01 +00:00
nehmathe2GitHubTJianmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Divakar VermaAndreas Karatzas
 		2785a5e0e6 [Bugfix][ROCm] Fix FP8 per-tensor scale rank mismatch causing Inductor assertion failure (#44912) 
Signed-off-by: nehmathe2 <nehmathe2@gmail.com>
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Signed-off-by: nehmathe <nehmathe@amd.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Divakar Verma <divakar.verma@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-16 20:17:42 -07:00
Cameron QuiliciGitHubHongxia YangTJian
 		efd15e192a [Bugfix][ROCm] Fix MiniMax-M3 FP8 KV cache dtype (#45720) 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-17 03:14:45 +00:00
Chaojun ZhangGitHubKunshang Ji
 		556b063e45 [XPU] Fix test_spec_decode_logprobs: use FLASH_ATTN for XPU in GPU_DETERMINISM_KWARGS (#44468) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-17 11:07:04 +08:00
Kevin H. LuuGitHubClaude Opus 4.8
 		aa0ac8a661 [CI] Run pre-commit on self-hosted vllm-runners (#45865) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 19:49:22 -07:00
FedericoandGitHub b831374cf1 [Bugfix][Gemma4] Fix parsing when thinking is disabled (#45832) 
Signed-off-by: Federico Iezzi <fiezzi@google.com>
 2026-06-17 02:41:36 +00:00
Dakai AnGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		71bc19dbdd [Bugfix] Fix MoE model load OOM in FlashInfer_TRTLLM backend with sleep mode (#45589) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 19:36:51 -07:00
Kunshang JiandGitHub ef2c40dc00 [XPU][CI] fix server test file path (#45870) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-17 09:06:25 +08:00
sungsoo haGitHubThomas Parnell
 		4bf699d310 [Kernel] Support DS Mamba tail copy for MTP align mode (#45473) 
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Co-authored-by: Thomas Parnell <tom.parnell@gmail.com>
 2026-06-16 22:50:30 +00:00
Stan WozniakandGitHub 520828789c Apply LRU policy only to proper cache entries (#42656) 
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com>
 2026-06-16 21:49:15 +00:00
Mohammad Miadh AngkadGitHubMatthew Bonanni
 		9d4dc4ca2f [Kernel] Support GLM-5 dimensions for TRT-LLM ragged MLA prefill (#43525) 
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-06-16 20:49:47 +00:00
FedericoandGitHub b9684d99e9 [Bugfix] Gemma4: skip forced JSON for required/named tool choice (#45795) 
Signed-off-by: Federico Iezzi <fiezzi@google.com>
 2026-06-16 20:38:29 +00:00
Divakar VermaandGitHub 4fadf9c92c [ROCm][CI] fix multimodel run cmds (#45858) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2026-06-16 15:31:52 -05:00
Nick HillandGitHub d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
Flora FengandGitHub 475a6ad18a [Misc] Update Mergify tool-calling label (#45853) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-06-16 19:08:00 +00:00
Hongxia YangandGitHub f2beaa80c8 [ROCm][Quant] mxfp8 moe/linear gfx950 tuning for MiniMax-M3 (#45725) 
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
 2026-06-16 18:50:40 +00:00
Song ZhixinGitHubNick Hill
 		8e27a9c215 [PERF] Fuse multi-group block table staged writes (#44944) 
Signed-off-by: jesse <szxfml@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 10:53:27 -07:00
alexbi29GitHubAlex Bilichenko
 		7d567172fc [Bugfix] Fix Qwen3 prompt tool-call reasoning false positive (#45763) 
Signed-off-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
Co-authored-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
 2026-06-16 17:48:01 +00:00
ChaunceyandGitHub f00e163f35 [Frontend] Add Streaming Parser Engine and new MinimaxM2 Parser (#45701) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-06-16 13:38:17 -04:00
Dao007foreverGitHubClaude Opus 4.8
 		44b2512767 [KV Connector][Mooncake] Add cache_prefix to namespace store keys (#45767) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 10:24:20 -07:00
kourosh hakhamaneshiGitHubClaude Opus 4.8
 		188c68798e [KVConnector][MoRIIO] Allow overriding the advertised host IP (#45488) 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 17:18:37 +00:00
Ting SUNGitHubShengqi Chen
 		c45f681932 [Bugfix][Core] Fall back when numactl --membind is blocked in constrained containers (#45438) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-16 09:49:56 -07:00
Tiezhen WANGGitHubClaude
 		89e8645a9e [Model] Remove Dots1ForCausalLM (#45637) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-17 00:32:18 +08:00
Wentao YeandGitHub 88a9cdd439 [Model Runner V2] Enable GraniteMOE for MRv2 by default (#45461) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-16 09:31:32 -07:00
Micah WilliamsonandGitHub 6f612fbedf [ROCm][CI] Patch conftest to resolve occasional OOMs (#45722) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-06-16 10:00:15 -05:00
Sting LinandGitHub 506ec6d656 Upgrade tpu-inference to v0.22.1 (#45793) 2026-06-16 07:54:57 -07:00
yifei wuGitHubHarry Mellor
 		a52205bccf [Model] Add HrmTextForCausalLM (Hierarchical Reasoning Model — Text) (#43098) 
Signed-off-by: Wuyifei <wuyifei@me.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-16 22:41:41 +08:00
Tuukka SarviGitHubCyrus Leung
 		3d34f8cbdc [ROCm][Cleanup] Remove stale AITER FA hybrid KV-cache TODO (#44178) 
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-06-16 07:28:06 -07:00
Carl YandGitHub eb04c769d3 feat: MLA prefill enable FA4 fp8 output (#43050) 
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com>
 2026-06-16 07:10:59 -07:00
Xiaohong (Sean) ChenGitHubYanan Cao
 		ce3ef17bec [Kernel][Helion][1/N] Add Helion kernel for rms_norm_per_block_quant (#36895) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
 2026-06-16 22:09:52 +08:00
Ajay AnuboluGitHubMatthew Bonanni
 		bf5149b516 [Bugfix] Fix FlashMLA sparse accuracy with topk_length and zero-init padding (#36616) 
Signed-off-by: AjAnubolu <anuboluajay@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-06-16 07:09:00 -07:00
Tahsin TunanandGitHub cca3365b73 [Rust Frontend] Add CORS support (#45753) 
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
 2026-06-16 13:47:11 +00:00
Matthew BonanniGitHubClaude
 		040df8f2ea [CI] Fix attention benchmark smoke test (#45728) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-16 13:43:37 +00:00
Francesco FuscoGitHubClaude
 		ced32bb474 [Perf] Add VLLM_TRITON_FORCE_FIRST_CONFIG to skip Triton autotuning (#42425) 
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-16 15:16:45 +02:00
Netanel HaberGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		c5e5c33fcd [Bugfix][MoE] Restore routed output unpadding before shared expert add (#45707) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 16:06:28 +03:00
Mike GandGitHub a8c86eeb16 [Quant] Support modelopt_mixed on Ampere (SM80/SM86) (#45306) 
Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com>
 2026-06-16 08:43:44 -04:00
Andreas KaratzasandGitHub 7e179e4bc0 [ROCm][CI] Gate incompatible HF references on Transformers v5 (#41532) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-16 20:34:11 +08:00
Lalithnarayan CGitHubTyler Michael SmithClaudeTyler Michael Smith
 		405c7cf283 [ZenCPU] Add zencpu Platform Runtime Logging and Docs (#42726) 
Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-06-16 08:23:12 -04:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		3f53e2138f [Refactor] Remove Fp8OnlineLinearMethod as scheduled (#45463) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 04:35:58 -07:00
Hank HanandGitHub d53f4593ce [KV Connector][Mooncake] Pipeline-parallel support for PD-disaggregated serving with Mooncake connector (#44528) 
Signed-off-by: hanhan.hank <hanhan.hank@bytedance.com>
Signed-off-by: Hank Han <hanhan7630@outlook.com>
 2026-06-16 04:35:38 -07:00
Shanshan ShenGitHubRoger WangIsotr0py
 		ad32608e24 [MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR (#43586) 
Signed-off-by: shen-shanshan <467638484@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-16 04:35:20 -07:00
Thien TranandGitHub b2cfae777d Add Triton recompile detection (#45631) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-06-16 18:25:28 +08:00
wangxiyuanandGitHub 3f1ff1ff14 [Misc]Clean up useless test (#45792) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-06-16 09:53:08 +00:00
wenjun liuGitHubzengxianKunshang Ji
 		c69c73418a [XPU][CI] add intel xpu cases for nightly CI (#44372) 
Signed-off-by: wenjun.liu <wenjun.liu@intel.com>
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Co-authored-by: zengxian <xiangdong.zeng@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-16 16:35:08 +08:00