karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,719 Commits 354 Branches 175 Tags
d2fbaf73c19e2bf519e4e4fc954ec8c14ba1e1fc
Commit Graph
5981 Commits
Author SHA1 Message Date
Yongye ZhuandClaude Opus 4.8 d2fbaf73c1 [Model] M3 MSA indexer: unify top-k buffer, drop numpy, GPU-only decode plan 
Follow-ups on the cudagraph-capturable MSA indexer:

- Top-k: both decode and prefill now write into the single shared, persistent
  topk_indices_buffer (decode at [:, :nd], prefill at [:, nd:]) and return views
  into it -- no fresh per-step top-k allocations.
- Build the decode plan + flat page table entirely with torch on-GPU: drop numpy
  and CpuGpuBuffer; segment offsets/lengths are computed via torch.cumsum into the
  persistent int32 buffers, and the request-major page table is scattered into the
  buffer via the on-GPU page indptr (the run bounds reads by indptr, so the full
  buffer is passed and no host page count is needed).
- No GPU->CPU sync on the decode path: scalars come from host ints
  (num_decode_tokens // num_decodes), and seq_lens.cpu() is confined to the eager
  prefill branch. The impl forward (fmha OnlyScore + Triton top-k) was already
  sync-free.

test_msa_indexer_impl_matches_triton now also asserts both outputs are views into
the persistent buffer. 42/42 in test_minimax_m3.py pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Yongye Zhu <yongye@inferact.ai>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:36 +00:00
Yongye ZhuandClaude Opus 4.8 bb4844dba3 [Model] Add fmha_sm100 MSA indexer backend + fp8 index cache for MiniMax M3 
Add an SM100/Blackwell lightning-indexer impl that computes the per-128-block
QK max-scores with fmha_sm100's score-only (OnlyScore) path and selects the
top-k blocks with the existing Triton minimax_m3_index_topk kernel, mirroring
how the main MSA attention pairs the SM100 attend with Triton. Decode and
prefill requests are split manually (decode-first batch) and each side gets its
own _fmha_sm100_plan / _fmha_sm100 call. Auto-selected on SM100 when
topk_blocks in (4, 8, 16, 32) for both bf16 and fp8 index caches; falls back to
the Triton indexer otherwise. The builder declares AttentionCGSupport.NEVER
(eager; the attention is broken out of the graph by _run_attention).

Extend the fused qknorm+rope+kv-insert kernel to optionally emit fp8 (e4m3) for
the index-K cache and index-Q via a direct cast with no scale tensors (RMSNorm
outputs are O(1) and scalar scales do not change top-k ordering). Only the index
outputs go fp8; q/k/v and q_out stay bf16 and bit-identical to the existing
path. MiniMaxM3IndexerCache now accepts fp8 caches and the model allocates
index_q in the cache dtype.

Tests: test_fmha_sm100_indexer_matches_reference (bf16/fp8 x prefill/decode) and
test_msa_indexer_impl_matches_triton (full impl parity vs the Triton indexer
through the real metadata builders); fp8 fused-kernel parity is covered in
test_fused_minimax_m3_qknorm_rope_kv_insert.

AI assistance (Claude Code) was used for this change.

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-17 05:32:35 +00:00
Thien TranGitHubYongye Zhu
 		4c62663315 [M3] Enable FP8 sparse GQA (#45744) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-16 21:38:03 -07:00
Thien TranGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		5bdc01bcc3 [M3] Tune Triton indexer score decode for spec-decode (#45743) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 21:07:34 -07:00
liangel-02andGitHub 20a5f8b43b [FlexAttention] make custom mask mods fully cudagraphable (#45232) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-06-17 11:53:12 +08:00
Ting SUNGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7b5d60cc37 [Bugfix][V1] Clean up compiled-model bytecode hooks on VllmRunner exit (#45195) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 20:31:17 -07:00
Cameron QuiliciGitHubHongxia YangTJian
 		efd15e192a [Bugfix][ROCm] Fix MiniMax-M3 FP8 KV cache dtype (#45720) 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cameron Quilici <cjquilici@gmail.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-06-17 03:14:45 +00:00
Chaojun ZhangGitHubKunshang Ji
 		556b063e45 [XPU] Fix test_spec_decode_logprobs: use FLASH_ATTN for XPU in GPU_DETERMINISM_KWARGS (#44468) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-17 11:07:04 +08:00
FedericoandGitHub b831374cf1 [Bugfix][Gemma4] Fix parsing when thinking is disabled (#45832) 
Signed-off-by: Federico Iezzi <fiezzi@google.com>
 2026-06-17 02:41:36 +00:00
sungsoo haGitHubThomas Parnell
 		4bf699d310 [Kernel] Support DS Mamba tail copy for MTP align mode (#45473) 
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Co-authored-by: Thomas Parnell <tom.parnell@gmail.com>
 2026-06-16 22:50:30 +00:00
Stan WozniakandGitHub 520828789c Apply LRU policy only to proper cache entries (#42656) 
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com>
 2026-06-16 21:49:15 +00:00
Mohammad Miadh AngkadGitHubMatthew Bonanni
 		9d4dc4ca2f [Kernel] Support GLM-5 dimensions for TRT-LLM ragged MLA prefill (#43525) 
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-06-16 20:49:47 +00:00
FedericoandGitHub b9684d99e9 [Bugfix] Gemma4: skip forced JSON for required/named tool choice (#45795) 
Signed-off-by: Federico Iezzi <fiezzi@google.com>
 2026-06-16 20:38:29 +00:00
Nick HillandGitHub d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
Song ZhixinGitHubNick Hill
 		8e27a9c215 [PERF] Fuse multi-group block table staged writes (#44944) 
Signed-off-by: jesse <szxfml@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 10:53:27 -07:00
alexbi29GitHubAlex Bilichenko
 		7d567172fc [Bugfix] Fix Qwen3 prompt tool-call reasoning false positive (#45763) 
Signed-off-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
Co-authored-by: Alex Bilichenko <alexbi29@users.noreply.github.com>
 2026-06-16 17:48:01 +00:00
ChaunceyandGitHub f00e163f35 [Frontend] Add Streaming Parser Engine and new MinimaxM2 Parser (#45701) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-06-16 13:38:17 -04:00
Dao007foreverGitHubClaude Opus 4.8
 		44b2512767 [KV Connector][Mooncake] Add cache_prefix to namespace store keys (#45767) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 10:24:20 -07:00
kourosh hakhamaneshiGitHubClaude Opus 4.8
 		188c68798e [KVConnector][MoRIIO] Allow overriding the advertised host IP (#45488) 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 17:18:37 +00:00
Ting SUNGitHubShengqi Chen
 		c45f681932 [Bugfix][Core] Fall back when numactl --membind is blocked in constrained containers (#45438) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-16 09:49:56 -07:00
Tiezhen WANGGitHubClaude
 		89e8645a9e [Model] Remove Dots1ForCausalLM (#45637) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-17 00:32:18 +08:00
Wentao YeandGitHub 88a9cdd439 [Model Runner V2] Enable GraniteMOE for MRv2 by default (#45461) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-16 09:31:32 -07:00
Micah WilliamsonandGitHub 6f612fbedf [ROCm][CI] Patch conftest to resolve occasional OOMs (#45722) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-06-16 10:00:15 -05:00
yifei wuGitHubHarry Mellor
 		a52205bccf [Model] Add HrmTextForCausalLM (Hierarchical Reasoning Model — Text) (#43098) 
Signed-off-by: Wuyifei <wuyifei@me.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-16 22:41:41 +08:00
Carl YandGitHub eb04c769d3 feat: MLA prefill enable FA4 fp8 output (#43050) 
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com>
 2026-06-16 07:10:59 -07:00
Xiaohong (Sean) ChenGitHubYanan Cao
 		ce3ef17bec [Kernel][Helion][1/N] Add Helion kernel for rms_norm_per_block_quant (#36895) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
 2026-06-16 22:09:52 +08:00
Francesco FuscoGitHubClaude
 		ced32bb474 [Perf] Add VLLM_TRITON_FORCE_FIRST_CONFIG to skip Triton autotuning (#42425) 
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-16 15:16:45 +02:00
Netanel HaberGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		c5e5c33fcd [Bugfix][MoE] Restore routed output unpadding before shared expert add (#45707) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 16:06:28 +03:00
Andreas KaratzasandGitHub 7e179e4bc0 [ROCm][CI] Gate incompatible HF references on Transformers v5 (#41532) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-16 20:34:11 +08:00
Lalithnarayan CGitHubTyler Michael SmithClaudeTyler Michael Smith
 		405c7cf283 [ZenCPU] Add zencpu Platform Runtime Logging and Docs (#42726) 
Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-06-16 08:23:12 -04:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		3f53e2138f [Refactor] Remove Fp8OnlineLinearMethod as scheduled (#45463) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 04:35:58 -07:00
Hank HanandGitHub d53f4593ce [KV Connector][Mooncake] Pipeline-parallel support for PD-disaggregated serving with Mooncake connector (#44528) 
Signed-off-by: hanhan.hank <hanhan.hank@bytedance.com>
Signed-off-by: Hank Han <hanhan7630@outlook.com>
 2026-06-16 04:35:38 -07:00
Shanshan ShenGitHubRoger WangIsotr0py
 		ad32608e24 [MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR (#43586) 
Signed-off-by: shen-shanshan <467638484@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-16 04:35:20 -07:00
Thien TranandGitHub b2cfae777d Add Triton recompile detection (#45631) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-06-16 18:25:28 +08:00
wangxiyuanandGitHub 3f1ff1ff14 [Misc]Clean up useless test (#45792) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-06-16 09:53:08 +00:00
joshua abrahamGitHubjoshuaOr Ozeri
 		7ad894c86a [Bugfix] Prevent cuMemcpyBatchAsync segfault with MTP and KV offloading (#44784) 
Signed-off-by: joshua <joshua.abraham@multicorewareinc.com>
Co-authored-by: joshua <joshua.abraham@multicorewareinc.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-16 07:58:39 +00:00
Li, JiangandGitHub a7fdfeef72 [CPU] Support Gemma Diffusion (#45690) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-16 14:39:56 +08:00
Taneem IbrahimandGitHub 81d8f4ebac [Misc] Added validation for Cohere /v2/embed input field exclusivity (#45640) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-06-16 05:42:43 +00:00
Andrew BarnesGitHubOpenAI Codex
 		a9a8a32dcd Register parsed config classes before tokenizer init (#40299) 
Signed-off-by: Bortlesboat <bortstheboat@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-16 05:33:08 +00:00
gitbisectorGitHubClaudeCyrus Leung
 		9d808e2309 [Core] Use fastsafetensors ParallelLoader for weight loading (#40183) 
Signed-off-by: Git Bisector <gitbisector@gmail.com>
Signed-off-by: gitbisector <gitbisector@gmail.com>
Signed-off-by: git bisector <gitbisector@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-06-15 22:32:05 -07:00
Ben BrowningandGitHub f3858d5422 [Frontend] [Parser] Migrate Nemotron V3 to streaming parser engine (#45755) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
 2026-06-16 05:31:21 +00:00
Luciano MartinsGitHubLuciano Martins
 		6607a80dab [Bugfix][Gemma4] Fix offline parser truncation, adjust_request token leak, and chat template sync (#45553) 
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
 2026-06-16 04:31:53 +00:00
Isotr0pyandGitHub e3cfea2e1b [Multimodal] Add Qwen3-VL video loader (#44412) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-16 03:45:34 +00:00
xx-thomasandGitHub b00e76ff72 [Misc][Model] add io processor for query/document embeddings from ColBERT (jinaai/jina-colbert-v2) (#45210) 
Signed-off-by: thomas <thomas.varghese@columbia.edu>
 2026-06-16 01:32:32 +00:00
llxGitHubNick HillJiangyun Zhu
 		d467a2a7f2 [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer (#45357) 
Signed-off-by: llx-08 <2596671364@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-06-15 21:36:09 +00:00
Ben BrowningGitHubFlora Feng
 		76a373eff4 [Frontend] Replace legacy Gemma4 parsers with engine-based implementation (#45588) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
 2026-06-15 21:34:07 +00:00
Jee Jee LiGitHubNick HillWoosuk Kwon
 		eacff17c8d [Model Runner V2][Bugfix] Fix MRV2 LoRA warmup (#35536) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-06-15 13:17:23 -07:00
Flora FengandGitHub cd9078fe59 [Frontend] Skip structural tags for auto tool_choice without strict mode (#45600) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-06-15 19:55:31 +00:00
Wentao YeandGitHub e18fe932ca [Perf] Optimize DSv4 prefill chunk planning, 4.0% E2E Throughput Improvement (#45061) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-15 19:50:21 +00:00
yzong-rhGitHubBen Browning
 		51ec5cf08f [Bugfix] Chat Completions Harmony Refactor Clean up (#45464) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <bbrownin@redhat.com>
 2026-06-15 14:45:19 -04:00