karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 2 Packages Activity
17,137 Commits 354 Branches 175 Tags
cbfaaeceeb432f906ac074e3ede6cb63168eb897
Commit Graph
2317 Commits
Author SHA1 Message Date
Yongye ZhuGitHubClaude Opus 4.8
 		cb7eb5c9b4 [MiniMax M3] Relocate sparse attention into the model dir + port decode top-k (#12) 
Move the MiniMax M3 sparse-attention backend and Triton kernels out of the
shared attention tree into the model definition (mirroring deepseek_v4), split
the kernels by role, and tighten the backend, then port the dedicated split-K
decode top-k from the sglang reference.

Structure:
- vllm/v1/attention/backends/minimax_m3_sparse.py
    -> vllm/models/minimax_m3/common/sparse_attention.py
- vllm/v1/attention/ops/minimax_m3_sparse_ops.py split into
    common/ops/index_topk.py  (index-score + top-k kernels)
    common/ops/sparse_attn.py (block-sparse GQA attention kernels)
  Pure-Triton/cross-platform, so under common/ (not nvidia/).
- registry enum + doc generator RELEVANT_PATTERNS repointed; the auto-generated
  attention_backends.md now has a dedicated "MiniMax M3 Sparse Attention" section.

Backend cleanups:
- Drop redundant metadata fields (sparse-selection params, decode max_query_len,
  decode cu_seqlens_q/context_lens) and read them from the layer/impl instead.
- Impl ctor takes explicit named args instead of kwargs.get lookups.
- Cudagraph support UNIFORM_SINGLE_TOKEN_DECODE -> UNIFORM_BATCH; reorder
  threshold and decode buffer scale by 1 + num_speculative_tokens.
- Unify the per-request context-length buffer (max_num_batched_tokens).
- Declare bf16-only KV cache.

Decode top-k (ported from sglang minimax_sparse_ops/decode):
- Split-K top-k: per-chunk partial top-k (_topk_index_partial_kernel) + merge
  (_topk_index_merge_kernel), replacing the single-program prefill top-k reuse.
- init/local block forcing moved into the decode score kernel (matches sglang).
- Verified against a torch reference (selected-block sets match across varied
  seq lengths, with/without init+local forcing, and short sequences).

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-05-31 16:19:23 -04:00
Yongye ZhuandClaude Opus 4.8 a3028cebbf [MiniMax M3] Add sparse-attention backend + merged attention layer 
Scaffold the lightning-indexer sparse-attention path:

- MiniMaxM3SparseBackend (registered as MINIMAX_M3_SPARSE): block-sparse GQA
  backend; get_kv_cache_shape serves both the main K/V cache and the
  single-vector index-key side cache.
- MiniMaxM3IndexerCache: side KV cache for per-token index keys, key-only so it
  uses a single-vector MLAAttentionSpec rather than a K+V FullAttentionSpec.
- MiniMaxM3SparseMetadata (+ prefill/decode sub-metadata) and its builder,
  splitting the batch via split_decodes_and_prefills.
- MiniMaxM3SparseImpl: subclasses AttentionImplBase so it can take a custom
  forward(query, index_query, kv_cache, index_kv_cache); no alibi / sliding
  window / logits soft cap. forward is a stub pending the kernel port.

MiniMaxM3SparseAttention is merged into a single AttentionLayerBase: it owns the
projections, per-head QK norm and RoPE, binds the backend + impl, registers the
main K/V cache, and holds the index cache. Its forward computes q/k/v and the
index q/k, pre-inserts K/V and index-K into their caches, then calls the sparse
impl with only the queries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Yongye Zhu <yongye@inferact.ai>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-05-31 16:02:56 +00:00
Bugen ZhaoandGitHub 50c80d7923 [Governance] Add @BugenZhao as Rust frontend code owner (#44047) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-05-30 22:23:54 +08:00
XiaoranGitHubXiaoran Chen
 		3becc5db40 [ROCm] Add attention sink support to AITer flash attention backend (#43817) 
Signed-off-by: Xiaoran Chen <xiaoran@fb.com>
Co-authored-by: Xiaoran Chen <xiaoran@fb.com>
 2026-05-30 18:13:18 +08:00
Ilya MarkovGitHubMarkov IlyaTyler Michael Smith
 		4aaba00f92 [EPLB] Make async EPLB default (#43219) 
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-05-29 18:07:16 +00:00
Chunyang WenandGitHub f191d5630e docs: clarify ITL acronym in optimization docs (#43922) 
Signed-off-by: chunyang.wen <chunyang.wen@gmail.com>
 2026-05-29 07:40:05 -07:00
Harry MellorandGitHub 0585b5ba2e Skip docs build if PR doesn't affect docs (#43972) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-29 12:09:52 +00:00
Kunshang JiandGitHub 30c6289b8e [XPU] fix xpu install document triton-xpu version (#43947) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-05-29 02:05:12 -07:00
ltd0924GitHubluotingdanIsotr0pyYu HuangJee Jee Li
 		b690b2bb67 [Model]Support Step-3.7-Flash (#43859) 
Signed-off-by: luotingdan <luotingdan@stepfun.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: luotingdan <luotingdan@stepfun.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Yu Huang <yuhuang@nvidia.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-05-28 17:01:48 -07:00
Harry MellorandGitHub 085ac221a3 Deprecate JAISLMHeadModel (#43784) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-28 18:29:12 +00:00
MaciejBalaNVGitHubIsotr0pyIsotr0pymergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Roger Wang
 		9aa131f944 Add Cosmos3 Reasoner model (#43356) 
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Signed-off-by: MaciejBalaNV <mbala@nvidia.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-28 09:43:55 -07:00
Animesh TrivediandGitHub bfb9ebc211 [Feature] Add support for timed trace replay in vllm bench serve to replay Moonshot and Alibaba workload traces (#39795) 
Signed-off-by: Animesh Trivedi <Animesh.Trivedi@ibm.com>
 2026-05-28 03:31:34 -07:00
JINO ROHITandGitHub e1814f822d minor docs: fix incorrect example path (#43830) 
Signed-off-by: JINO-ROHIT <find.jinorohit@gmail.com>
 2026-05-27 22:58:43 -07:00
Chunyang WenandGitHub 49a3510266 [Docs] Fix the duplicate doc icon issue (#43546) 
Signed-off-by: chunyang.wen <chunyang.wen@gmail.com>
 2026-05-27 16:09:58 +00:00
Ashwin GiridharanGitHubChauncey
 		52a31ccecc [Bugfix] Map reasoning_effort to enable_thinking in chat template kwargs (#43401) 
Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Signed-off-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-05-27 05:39:49 -07:00
Mohammad Miadh AngkadandGitHub 158289e0fc [Docs] Fix MLA prefill backend default docs (#43697) 
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
 2026-05-27 10:13:22 +00:00
Aditya SinghGitHubClaudemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ad464e16c0 [Doc] Add Ascend NPU tab to the quickstart installation guide (#43550) 
Signed-off-by: Aditya Singh <adisin650@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-27 08:41:29 +00:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		c02c758ea4 [Deprecation] Deprecate functions as scheduled for v0.21.0 (#43358) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-26 19:56:21 -07:00
Jee Jee LiandGitHub 6e503868ca [Kernel] Porting fuse_minimax_qk_norm to manual fusion (#43410) 
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-05-26 13:16:03 -07:00
Simon DanielssonGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		d565357a90 [Docs][ROCm] MoRI-IO Connector Usage Guide (#43603) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Signed-off-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-26 21:52:30 +08:00
Thibault CastellsandGitHub 5d09f471f4 [Misc] Support interleaved custom image benchmark datasets (#43636) 
Signed-off-by: ThibaultCastells <thib.castells@icloud.com>
 2026-05-26 03:37:25 -07:00
Roy WangandGitHub 0c942c69d6 [Doc] Add section on escalating stalled contributions (#43568) 
Signed-off-by: esmeetu <jasonailu87@gmail.com>
 2026-05-25 14:11:01 +08:00
Nguyễn Thế DuyGitHubClaude
 		3df1c7c43e [Docker] Non-root support for vllm-openai; add opt-in vllm-openai-nonroot target (#40275) 
Signed-off-by: TheDuyIT <nduy250299@gmail.com>
Signed-off-by: dtnguyen <dtnguyen@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-25 13:45:31 +08:00
wang.yuqiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		1b26fa361e [Docs] Reorganize offline inference docs. (#43552) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-25 13:44:39 +08:00
Dao007foreverGitHubClaude
 		0902d8e62f [KV Connector] Keep MooncakeStore full hits block-aligned (#43494) 
Signed-off-by: Dao Le <daole@inferact.ai>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-23 23:15:03 -07:00
HolegotsandGitHub 8737e4a857 [Docs] Fix stale version number in token_classify.md (#43489) 
Signed-off-by: holegots <ikun3.1415927@gmail.com>
 2026-05-23 10:42:20 -07:00
HolegotsandGitHub 7c2ff1f819 [Docs] Fix stale version number in token_embed.md (#43488) 
Signed-off-by: holegots <ikun3.1415927@gmail.com>
 2026-05-23 10:06:56 -07:00
Duncan MossandGitHub 552bbe6f4e [Attention] Add head_dim=512 support for FlashInfer trtllm attention backend (#38822) 2026-05-22 20:27:35 -04:00
Benjamin ChislettGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		4e2eba28be [Perf] Optimize hidden state extraction logic (#37374) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-22 18:23:08 -04:00
Yongye ZhuandGitHub 843715739b [Refactor] Extract DeepSeek V4 sparse MLA impl into model folder (#43149) 2026-05-22 10:06:31 -07:00
wang.yuqiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		2380bfc210 [Docs] Note image preprocessing difference between qwen_vl_utils and vllm. (#43393) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-22 01:43:14 -07:00
Isotr0pyandGitHub ba369b7eb5 [CI] Fix dockerfile dependency graph failure for pre-commit (#43378) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-05-22 10:26:05 +08:00
Nick HillandGitHub 0f66623b0d [Frontend] Rework fastokens integration (#43168) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-21 15:36:58 -07:00
Nick HillGitHubBugen Zhao
 		f2ace1d57d [Frontend][RFC] Rust front-end integration (#40848) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-05-21 12:24:48 +08:00
Douglas LehrandGitHub 452baa860b Add dllehr-amd to CODEOWNERS and committers list (#42772) 
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com>
 2026-05-20 16:10:44 -05:00
KebeandGitHub 19cf334207 [Feature] Support manually enabling the cumem allocator (#33648) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2026-05-20 08:58:30 -04:00
Ray WangGitHubRui Wang
 		87e31455b0 [Doc] Sync CLI guide with actual help modes and launch subcommand (#40326) 
Signed-off-by: Rui Wang <raygorous@gmail.com>
Co-authored-by: Rui Wang <raygorous@gmail.com>
 2026-05-20 02:32:03 -07:00
Nicolò LucchesiandGitHub 40651c0207 [Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-20 09:02:36 +02:00
Nicolò LucchesiandGitHub 7e4bc2cecb [Docs][PD][NIXL] Lease extension mechanism for blocks on P (#43099) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-20 08:58:25 +02:00
Terrence ZhaoandGitHub 5774aaed0c [Cohere] Enable Cohere MoE (#43143) 
Signed-off-by: Terrencezzj <terrence@cohere.ai>
 2026-05-19 19:32:06 -07:00
Fadi ArafehandGitHub be16785998 [CPU][DOC] Fix installation commands for Arm CPUs (#43115) 
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
 2026-05-19 23:31:15 +00:00
Dao007foreverGitHubClaude
 		aed2eb355a [Docs] Fix MooncakeStoreConnector role in disaggregated example (#42994) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-19 11:14:43 -07:00
Wang YiwenandGitHub 1c6158083a [Model] Openvla support (#42654) 
Signed-off-by: Wang Yiwen <121547057+yiwen101@users.noreply.github.com>
 2026-05-19 08:17:42 -07:00
wang.yuqiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Cyrus LeungHarry Mellor
 		257af77bc2 [Docs] Reorganize online serving docs. (#41907) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-19 14:43:18 +08:00
Gracie Guo (UX)GitHubGracie Guowang.yuqi
 		9fd8487d2f [Docs] Add SVG images for pooling models. (#42626) 
Signed-off-by: Gracie Guo <gracieguo@Gracies-MacBook-Pro.local>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Gracie Guo <gracieguo@Gracies-MacBook-Pro.local>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-05-18 22:50:38 -07:00
Mohammad Miadh AngkadandGitHub da03e549b3 [UX] Add a persistent cache for FlashInfer autotuning (#42537) 
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
 2026-05-18 20:25:37 -07:00
Blanc SwanandGitHub 4a39b4f553 [Model] Add Apertus Tool Parser (#41154) 
Signed-off-by: Blanc <swan.blanc@infomaniak.com>
 2026-05-18 11:20:04 -04:00
Jee Jee LiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Roger Wang
 		7d5b033782 [LoRA] Support 2D and 3D MoE LoRA adapter at the same time (#42242) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-18 15:22:26 +08:00
Harry MellorandGitHub c1f7854342 Improve logging when docs build is skipped (#42929) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-18 06:33:32 +00:00
SoyaazzandGitHub 990f49bdcb [MM][CG] Enable encoder Cudagraph for Step3VL (#42224) 
Signed-off-by: JisoLya <523420504@qq.com>
Signed-off-by: Soyaazz <523420504@qq.com>
 2026-05-17 20:19:13 -07:00