Karylab-cklius/vllm
1
0
Fork 1
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
16,206 Commits 352 Branches 175 Tags
019a41f00a9de88447abde1baa4e0d2bd34b4368
Commit Graph
16206 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Tyler Michael SmithandClaude Opus 4.6 019a41f00a Add DeepEP v2 lifecycle test to test_mnnvl_alltoall 
Test DeepEPV2All2AllManager init, ElasticBuffer handle creation
and caching, SM calculation, and destroy/re-create cycle.
Skipped when DeepEP v2 or NCCL >= 4.30.4 is not available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-29 22:58:10 -04:00
Tyler Michael SmithandClaude Opus 4.6 515e36da11 Fix FP8 dispatch test to match production behavior 
use_fp8_dispatch requires the ElasticBuffer to receive FP8 input.
In production, this is ensured by pre-quantizing via
moe_kernel_quantize_input when is_block_quantized=True.

The test was parametrizing use_fp8_dispatch independently of dtype,
allowing bf16 input with use_fp8_dispatch=True which triggers a
buffer size assertion in DeepEP v2.

Fix:
- Derive use_fp8_dispatch from dtype (True only for FP8 weights)
- Add block_shape=[128, 128] to quant config for FP8 to enable
  the block quantization path that pre-quantizes input

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-29 22:56:16 -04:00
Tyler Michael SmithandClaude Opus 4.6 a2a4b00f83 Add DeepEP v2 (ElasticBuffer) all2all backend for MoE EP 
Add a new `deepep_v2` all2all backend that uses the DeepEP v2
ElasticBuffer API (NCCL GIN backend). This provides a unified
dispatch/combine interface that works for both intra-node and
inter-node expert parallelism with analytical SM calculation.

Key changes:
- New DeepEPV2PrepareAndFinalize class using do_expand=True for
  per-expert-contiguous layout with weighted reduction in combine
- DeepEPV2All2AllManager with ElasticBuffer handle caching and
  theoretical SM calculation via get_theoretical_num_sms()
- NCCL >= 4.30.4 version gating in has_deep_ep_v2() since the
  GIN backend requires a newer NCCL than PyTorch typically bundles
- FP8 block-quantized dispatch support
- DBO (micro-batching) support with async prepare/finalize
- Environment variables: VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE,
  VLLM_DEEPEP_V2_PREFER_OVERLAP, VLLM_DEEPEP_V2_ALLOW_MULTIPLE_REDUCTION
- Update DeepEP install script to pin v2.0 release (b306af06af)
- Comprehensive multi-process test suite

Usage: --all2all-backend=deepep_v2 --enable-expert-parallel

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-29 21:02:26 -04:00
Kevin H. LuuGitHubClaude
 		0ab67c0222 [CI] Add key field to all test_areas pipeline steps (#41201) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-29 16:59:16 -07:00
Rohan PotdarandGitHub 3795d7acf4 [ROCm][Bugfix][GPTOSS]: fix input_ids and expert_map args for quark w4a8 gptoss (#41165) 
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
 2026-04-29 16:39:01 -07:00
Nick HillandGitHub 18599bfdf2 [Ci][BugFix] Fix slow DP tests due to bad teardown logic (#41166) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 19:31:00 -04:00
Thien TranandGitHub 296741d025 [DSv4] Use cvt PTX for FP32->FP4 conversion (#41015) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-04-29 16:16:40 -07:00
UranusGitHubClaude Opus 4.6Woosuk Kwon
 		a966aaed30 [Bugfix][MLA] Size arange_buffer to max_num_batched_tokens to prevent CUDA IMA (#39277) 
Signed-off-by: UranusSeven <109661872+UranusSeven@users.noreply.github.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-04-29 16:14:50 -07:00
Hemanth AcharyaandGitHub 6841f5dc77 [ROCm] Add env flags to disable dynamic MXFP4 quant and enable AITER tuned GEMMs for Attention Projection Layers (#39987) 
Signed-off-by: Hemanth Acharya <heachary@amd.com>
 2026-04-29 16:07:46 -07:00
roikoren755andGitHub c2fb013312 [Bugfix][Compile] Fix gc.collect/empty_cache patch arity in CUDAGraphWrapper (#41235) 
Signed-off-by: Roi Koren <roik@nvidia.com>
 2026-04-29 21:59:18 +00:00
Rishi PuriGitHubClaudeStefano Castagnetta
 		ccfb620c62 Create tests/distributed/test_mnnvl_alltoall.py (#35241) 
Signed-off-by: Rishi Puri <riship@nvidia.com>
Signed-off-by: Claude <claude@anthropic.com>
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-29 21:56:56 +00:00
Aaron HaoGitHubJunjie ZhangNick Hill
 		0335316a9b [BUG] Two phase pause to prevent deadlock (#39366) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 17:51:03 -04:00
Rohan PotdarandGitHub 944e138bcf [ROCm][Bugfix]: W4A4 MOE using emulation instead of AITER on MXFP4-supported hardware (#41175) 
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
 2026-04-29 16:39:03 -05:00
Luochao WangGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		b58669cb42 [Perf][Spec Decode] Avoid per-step numpy allocation in prepare_next_t… (#41043) 
Signed-off-by: wangluochao902 <wangluochao902@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-29 14:20:13 -07:00
Isotr0pyandGitHub 1628239eb2 [Multimodal][Render] Skip mm processor initialization and warmup for text-only mode (#41246) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-04-29 14:16:19 -07:00
yzong-rhandGitHub 93da1fe97a [CI] Add temperature to bfcl eval, default greedy (#41059) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
 2026-04-29 14:01:57 -07:00
Andrew BarnesandGitHub 169988a3c0 [ROCm] Use quant_dtype in per_token_quant instead of hardcoded FP8 (#39121) 
Signed-off-by: Bortlesboat <bortstheboat@gmail.com>
 2026-04-29 20:46:01 +00:00
ChaunceyGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		faab189554 [Feature]: IndexCache support for DSA models (#37735) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-29 15:15:35 -04:00
Laith SakkaandGitHub 6f20f81cbf Replace shape_invariants with simpler apprach in dynamic_arg_dims utilizing shape_id property. (#36194) 
Signed-off-by: Laith Sakka <lsakka@meta.com>
 2026-04-29 18:32:15 +00:00
daniserebandGitHub d1a75e303d Fix timeout when using LoRA adapters with Nemotron Super (#40916) 
Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>
 2026-04-30 01:39:49 +08:00
Cyrus LeungandGitHub 4a42aba380 [CI/Build] Enable FP8 on NVIDIA Thor (#39712) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-04-29 09:48:52 -07:00
Avshalom ManevichandGitHub a80d6f150c better logging for large uncachable items (#41145) 
Signed-off-by: h-avsha <avshalom.manevich@hcompany.ai>
 2026-04-29 09:48:47 -07:00
Terrence ZhaoandGitHub 91a2d39014 [Models] Cohere MoE (#40817) 
Signed-off-by: Terrencezzj <terrence@cohere.ai>
 2026-04-29 15:54:54 +00:00
Frederik GossenandGitHub a05848e255 [Bugfix] Report compile time for in-memory cache hit path (#41023) 
Signed-off-by: Frederik Gossen <frgossen@meta.com>
 2026-04-29 15:32:03 +00:00
Wentao YeGitHubNick Hill
 		51fda1ba44 [Model Runner v2] Fix block table IMA issue (#40648) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 08:30:33 -07:00
Wentao YeandGitHub 39a7f4f4e2 [Perf] Optimize AllPool.forward by slicing first, 51% faster in the method level benchmark (#41163) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-04-29 08:11:04 -07:00
Artem PerevedentsevandGitHub b92ef9ec5a [Perf] Enable FlashInfer top-k/top-p sampler by default (#40376) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
 2026-04-29 19:10:34 +04:00
Lalithnarayan CGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		5560cac7e2 [Bugfix][CPU] Backport PT cpp codegen indirect_assert scalar-mask fix (#40973) 
Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-29 10:21:55 -04:00
pmaybankGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		5b39b268f5 hf_name argument for vllm bench throughput CLI (#41012) 
Signed-off-by: Philip Maybank <pmaybank@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-29 12:57:58 +00:00
Tianmu LiGitHubClaudeLi, Jiang <jiang1.li@intel.com>
 		22524f7a92 [Feat] CPU fp8 attn for AMX/AVX-512 (#39445) 
Signed-off-by: Li, Tianmu <tianmu.li@intel.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-04-29 20:43:21 +08:00
Jee Jee LiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		9d8ad5b408 [Bugfix] Fix repeated DSv4 RoPE cache initialization (#41148) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-29 20:29:55 +08:00
Jared WenGitHubsfeng33Chauncey
 		11b69129e2 [Frontend] Add defer_loading and tool_reference support for Anthropic and OpenAI APIs (#40190) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-04-29 04:35:50 -07:00
Bugen ZhaoandGitHub 33f36d4260 [DSV4] Support max reasoning effort (#40982) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-04-29 11:03:47 +00:00
Ronen SchafferandGitHub 37e288214b [KV Offload] Tighten keys type from Iterable to Sequence in OffloadingManager (#41200) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-04-29 13:50:42 +03:00
Rohit Kumar SinghGitHubCyrus Leung
 		5371d6fb40 Fix PP in Gemma4 (#40786) 
Signed-off-by: Rohit kumar Singh <rksingh@habana.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-04-29 03:17:51 -07:00
Jiangyun ZhuandGitHub 6d7d4da99e [Bugfix] BailingMoeV2.5: rotate full qk_rope_head_dim in MLA RoPE (#41185) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-04-29 18:08:55 +08:00
AlecGitHubOpenAI Codex
 		3f1a4bb639 build: embed image provenance metadata in vLLM containers (#40653) 
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-04-29 03:07:41 -07:00
ChaunceyandGitHub 762022cafb [Bugfix] DSV32/V4 add missing type conversion for non-streaming tool calls (#41198) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 09:55:07 +00:00
ChaunceyandGitHub 3885d340a4 [Frontend]Responses API supports Tool/Function calling with streaming with named tool/function (#41110) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 09:11:27 +00:00
haosdentandGitHub ef70057ca7 [CI][CPU] Split CPU-Distributed Tests into per-scenario labels (#41203) 
Signed-off-by: haosdent <haosdent@gmail.com>
 2026-04-29 01:28:45 -07:00
Shengqi ChenGitHubClaude
 		e48cb85185 [CI/Build] Auto-detect manylinux ABI tag for nightly wheels (#41149) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-29 00:37:14 -07:00
ChaunceyandGitHub 92879e12ba [CI] fix test_rotary_embedding_opcheck format error (#41202) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 00:32:37 -07:00
rishitdholakia13GitHubNick HillCyrus Leung
 		68dd7db810 [Reasoning] Support for speculative decoding with thinking budget (#34668) 
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com>
Signed-off-by: rishitdholakia13 <123388671+rishitdholakia13@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-04-29 06:14:52 +00:00
Itay EtelisGitHubItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
Jee Jee LiandGitHub a269744e9f [Bugfix] Fix rope (#41113) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-04-28 22:42:35 -07:00
Wei ZhaoGitHubWei Zhao
 		8b49cf3a37 [Bugfix] Fix max_num_batched_token not captured in cuda graph (#40734) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: Wei Zhao (Engrg-Hardware 1) <weizha@login-bia02.bia.clusters.nvidia.com>
 2026-04-28 21:33:06 -07:00
Jiangyun ZhuandGitHub 2ae73c758c [Bugfix] fix inductor error for dpsk v4 (#41135) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-04-28 21:18:46 -07:00
Fadi ArafehandGitHub d95d03c719 [BugFix][CPU] fix error on CPU runner shutdown (#41034) 
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
 2026-04-28 21:08:35 -07:00
Wei ZhaoandGitHub 803b9d7881 [Bugfix] Fix Deepseek V4 import error due to AOT compile cache loading (#41090) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
 2026-04-28 21:08:16 -07:00
Walter Beller-MoralesandGitHub 1312f07531 [Feature] add cohere reasoning and tool parsers (#40422) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-04-28 21:07:53 -07:00