karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 2 Packages Activity
17,642 Commits 354 Branches 175 Tags
e3b4fdaf5db1aed53675c8467bc4862ba53bbfff
Commit Graph
308 Commits
Author SHA1 Message Date
Alexander MatveevandAlexander Matveev e3b4fdaf5d perf: add push-based allreduce for small tensor reductions 
Port SGLang's push-based 2-buffer allreduce protocol into vLLM as a new
communicator backend for small-message reductions. The push protocol
eliminates the two explicit cross-GPU NVLink barrier round-trips used by
the existing barrier-based CustomAllreduce, replacing them with a
sentinel-based data arrival detection mechanism and double-buffered epoch
alternation.

Key advantages over the barrier-based approach:
- Zero barriers: data arrival IS the synchronization (positive-zero sentinel)
- Single NVLink round-trip instead of two barrier exchanges + remote reads
- All SMs active (SM_count CTAs vs 2 CTAs) for higher NVLink bandwidth
- No cudaMemcpy to IPC staging buffer in eager mode
- PDL (griddepcontrol) support for kernel overlap on sm_90+

The new PushAllReduce is inserted in the CudaCommunicator dispatch chain
above the existing CustomAllreduce for messages below a size threshold
(~720 KB at TP=8). Larger messages continue to use the barrier-based
path. The existing CustomAllreduce code is not modified.

Measured results on DeepSeek-V4-Pro (61 layers, TP=8, 8x NVIDIA B200,
BS=1, decode with ISL=4, OSL=33024):
- Throughput: +2.14% (84.06 vs 82.30 tokens/s)
- TPOT: -2.09% (11.90 vs 12.15 ms/token)

Correctness verified via lm_eval gsm8k 5-shot with no regression
(exact_match delta within statistical noise).

The feature can be disabled at runtime via VLLM_DISABLE_PUSH_ALLREDUCE=1
to fall back to the barrier-based path.

Signed-off-by: Alexander Matveev <amatveev@redhat.com>
 2026-06-15 16:23:12 -04:00
Tiezhen WANGGitHubClaude
 		7df4fe1bd7 [Model] Remove XverseForCausalLM (#45638) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-14 22:09:00 -07:00
Tiezhen WANGGitHubClaude
 		c9340e6f35 [Model] Remove InternLMForCausalLM registry alias (#45128) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-11 20:02:51 +00:00
Harry MellorandGitHub 03878d1c22 Deprecations for v0.23 and v0.24 (#44992) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-11 14:35:38 +00:00
Wentao YeandGitHub 85a0ffae42 [CI Bug] Remove qwen test ValueError: No example model defined for Qwen/Qwen-7B-Chat (#45194) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-10 20:11:00 -07:00
Ilya MarkovandGitHub 6471ec75bd [EPLB] Reject NCCL-based EPLB communicators with async EPLB (#44978) 
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
 2026-06-10 19:51:27 +00:00
Yongye ZhuandGitHub 2ba68d9bf7 [Test] Fix one-sided MNNVL alltoall test workspace under-reservation (#44946) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-11 00:43:12 +08:00
Harry MellorandGitHub 3cc9fecd58 Deprecated 1st generation Qwen and QwenVL models (#45131) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-10 14:55:33 +00:00
wang.yuqiandGitHub 996222f4bf [CI] Reorganize entrypoints CI (#44947) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-06-09 00:46:11 -07:00
Tyler Michael SmithGitHubClaude Opus 4.6
 		e2f993dc41 [WideEP] Integrate DeepEP v2 (#41183) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-06-08 18:07:29 -07:00
bnellnmandGitHub dc68bd8c41 [MoE Refactor] FusedMoE/MoERunner inversion refactor (#41184) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-06-08 10:42:58 -04:00
Tushar JainGitHubTushar Jain
 		38fd2405f3 use split_group for pytorch process group creation (#41980) 
Signed-off-by: Tushar Jain <tushar00jain@users.noreply.github.com>
Co-authored-by: Tushar Jain <tushar00jain@users.noreply.github.com>
 2026-06-04 14:36:07 -04:00
Ilya MarkovGitHubMarkov IlyaNicolò LucchesiTyler Michael SmithTyler Michael Smith
 		4f423bd5bc [EPLB] Nixl communicator optimization. Zero-copy transfers (#41633) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-04 03:40:34 +00:00
Siddharth BedekarandGitHub 0917a009d3 Fix sparse NCCL weight transfer test construction (#44345) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
 2026-06-02 21:51:21 +00:00
Nick HillandGitHub cab5c9a2a9 [Core] Move max_concurrent_batches to VllmConfig (#44274) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 08:57:25 -07:00
Siddharth BedekarGitHubOpenAI Codex
 		266b9d9c64 [Frontend][Core] Add sparse NCCL weight transfer support for in-place updates (#40096) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-01 15:37:30 -04:00
Ilya MarkovGitHubMarkov IlyaTyler Michael Smith
 		4aaba00f92 [EPLB] Make async EPLB default (#43219) 
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-05-29 18:07:16 +00:00
Nick HillGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7e53283b1c [Core] Cleanup KVConnector handling with PP + fix MRV2 (#43732) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-28 13:12:03 -07:00
Harry MellorandGitHub 085ac221a3 Deprecate JAISLMHeadModel (#43784) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-28 18:29:12 +00:00
Andreas KaratzasandGitHub 445ded18c1 [ROCm][CI] Extend ROCm quick reduce coverage (#40990) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-26 21:57:13 +08:00
Wentao YeandGitHub 33d7cbe02c [Model Runner v2] Force v1 runner for tests (#43233) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-23 16:37:24 -07:00
Sumanth R HegdeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		3cb83c9592 Add model to WeightTransferEngine.__init__ (#42922) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-22 17:52:15 -07:00
akii96andGitHub bde560ed6e [ROCm] Add QuickReduce min-size override and codec threshold (#41675) 
Signed-off-by: <>
 2026-05-20 17:46:51 -05:00
Aaron HaoandGitHub 73dd2f33b7 [bug] fix WeightTransferConfig.backend to allow for all strings (#43121) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
 2026-05-19 21:01:29 -04:00
tomeras91andGitHub f54721bcc3 [Bugfix][MoE] FlashInfer one-sided: workspace union across heterogeneous layers (#42976) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2026-05-19 14:43:04 -04:00
Aaron HaoandGitHub e0a45f1455 [Feat][RL] IPC weight sync optimizations: multigpu support and chunked packed tensors (#37476) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: hao-aaron <ahao@anyscale.com>
 2026-05-15 22:53:06 +08:00
bnellnmGitHubRobert Shaw
 		d9b4990783 [MoE Refactor] EPLB refactoring for FusedMoE (#41055) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-05-12 14:16:31 -04:00
Yan Ru PeiandGitHub bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
bnellnmGitHubRobert ShawRobert Shaw
 		206eaed08d [MoE Refactor] Move expert map related code into ExpertMapManager class (#41046) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com>
 2026-05-12 09:18:27 -04:00
sungsoo haandGitHub 4f7bde572a [Kernel] Pack output and LSE in DCP A2A (#41160) 2026-05-01 09:01:17 -04:00
Rishi PuriGitHubClaudeStefano Castagnetta
 		ccfb620c62 Create tests/distributed/test_mnnvl_alltoall.py (#35241) 
Signed-off-by: Rishi Puri <riship@nvidia.com>
Signed-off-by: Claude <claude@anthropic.com>
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-29 21:56:56 +00:00
wang.yuqiandGitHub a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00
Sage MooreandGitHub 62b1bbe470 [EPLB] Remove asyncio infrastructure from Async EPLB (#40730) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-24 00:21:15 +00:00
liuzhenweiandGitHub 4a79262e0f [UT][Hardware] let torchrun example tests use the default backend (#39879) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-04-23 16:22:28 +08:00
Matthew BonanniGitHubTyler Michael Smith
 		96a85c5750 [Startup][UX] Enable CUDAGraph memory profiling by default (#38284) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-21 18:16:59 -04:00
Sage MooreandGitHub 3173441b0f [EPLB] Consolidate is_unchanged/is_received_locally into TransferMetadata (#37341) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-20 21:12:42 +00:00
Nicolò LucchesiandGitHub 304d5ba1a0 [Bugfix][CI] Fix tests/distributed/test_torchrun_example_moe.py (#40349) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-20 11:05:44 -07:00
Sage MooreGitHubTyler Michael Smith
 		3461c8b027 [EPLB] Refactor Async EPLB synchronization logic (#37601) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-04-20 17:05:41 +00:00
Ilya MarkovandGitHub 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
 2026-04-20 10:24:23 +00:00
Sumanth R HegdeGitHubRobert ShawCyrus Leung
 		adf9bb3c57 [CI] Add weight transfer tests to CI (#39821) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-04-16 15:51:45 -04:00
Martin HickeyGitHubOr Ozeri
 		cc07dad789 [HMA] [KVEvent] Enable GPU-side KV events for HMA (#37688) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-12 10:01:02 +03:00
Jeffrey WangandGitHub ab79863e6c Remove MQ multi-node tests (#38934) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-04-03 20:00:08 +00:00
Jeffrey WangandGitHub de5e6c44c6 [Feat][Executor] Introduce RayExecutorV2 (#36836) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-04-01 14:34:29 -07:00
wliao2GitHubHarry MellorKunshang Ji
 		4dfad17ed1 replace cuda_device_count_stateless() to current_platform.device_count() (#37841) 
Signed-off-by: Liao, Wei <wei.liao@intel.com>
Signed-off-by: wliao2 <wei.liao@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-31 22:32:54 +08:00
Ilya MarkovGitHubMarkov Ilya
 		abdbb68386 [EPLB] Add alternative communication for EPLB weight exchange (#33176) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
 2026-03-31 08:17:12 -04:00
Sage MooreandGitHub 497e234d38 [EPLB] Cleanup the transfer logic for the various eplb maps (#34520) 
Signed-off-by: Sage Moore <sagmoore@redhat.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-03-27 10:18:46 +01:00
Flora FengandGitHub 9040151fe1 [V0 Deprecation] Deprecate --disable-frontend-multiprocessing (#37612) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-20 11:31:43 +08:00
Sage MooreGitHubTyler Michael Smith
 		c32a58cc2a [EPLB] Simplify EPLB rearrange by only returning one map (#36267) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-03-18 20:34:00 -04:00
Isotr0pyandGitHub a836524d20 [Chore] Replace all base64 usages with faster pybase64 package (#37290) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-03-17 14:44:19 +00:00
Flora FengandGitHub 384dc7f77b [Refactor] Relocate completion and chat completion tests (#37125) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-17 11:31:23 +08:00