karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,650 Commits 354 Branches 175 Tags
perf/push-allreduce-2buffer
Commit Graph
310 Commits
Author SHA1 Message Date
Alexander MatveevandClaude Opus 4.6 fa6b6a83ec Apply pre-commit formatting and use torch.accelerator API 
- Apply ruff, clang-format formatting fixes
- Replace torch.cuda.set_device/device_count/synchronize with
  torch.accelerator equivalents per project convention

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>

Signed-off-by:  <>
 2026-06-15 18:21:59 -04:00
Alexander MatveevandClaude Opus 4.6 e0601e1b94 Address review: bind test sockets to localhost instead of all interfaces 
Fix CodeQL security warning by binding test helper sockets to
"localhost" instead of "" (all interfaces). These sockets are only
used for finding a free port for torch distributed init in tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>
 2026-06-15 18:09:24 -04:00
Alexander MatveevandAlexander Matveev 5fd9bd4f04 perf: add push-based allreduce for small tensor reductions 
Port SGLang's push-based 2-buffer allreduce protocol into vLLM as a new
communicator backend for small-message reductions. The push protocol
eliminates the two explicit cross-GPU NVLink barrier round-trips used by
the existing barrier-based CustomAllreduce, replacing them with a
sentinel-based data arrival detection mechanism and double-buffered epoch
alternation.

Key advantages over the barrier-based approach:
- Zero barriers: data arrival IS the synchronization (positive-zero sentinel)
- Single NVLink round-trip instead of two barrier exchanges + remote reads
- All SMs active (SM_count CTAs vs 2 CTAs) for higher NVLink bandwidth
- No cudaMemcpy to IPC staging buffer in eager mode
- PDL (griddepcontrol) support for kernel overlap on sm_90+

The new PushAllReduce is inserted in the CudaCommunicator dispatch chain
above the existing CustomAllreduce for messages below a size threshold
(~720 KB at TP=8). Larger messages continue to use the barrier-based
path. The existing CustomAllreduce code is not modified.

Measured results on DeepSeek-V4-Pro (61 layers, TP=8, 8x NVIDIA B200,
BS=1, decode with ISL=4, OSL=33024):
- Throughput: +2.14% (84.06 vs 82.30 tokens/s)
- TPOT: -2.09% (11.90 vs 12.15 ms/token)

Correctness verified via lm_eval gsm8k 5-shot with no regression
(exact_match delta within statistical noise).

The feature can be disabled at runtime via VLLM_DISABLE_PUSH_ALLREDUCE=1
to fall back to the barrier-based path.

Signed-off-by: Alexander Matveev <amatveev@redhat.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>
 2026-06-15 18:09:24 -04:00
Tiezhen WANGGitHubClaude
 		7df4fe1bd7 [Model] Remove XverseForCausalLM (#45638) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-14 22:09:00 -07:00
Tiezhen WANGGitHubClaude
 		c9340e6f35 [Model] Remove InternLMForCausalLM registry alias (#45128) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-11 20:02:51 +00:00
Harry MellorandGitHub 03878d1c22 Deprecations for v0.23 and v0.24 (#44992) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-11 14:35:38 +00:00
Wentao YeandGitHub 85a0ffae42 [CI Bug] Remove qwen test ValueError: No example model defined for Qwen/Qwen-7B-Chat (#45194) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-10 20:11:00 -07:00
Ilya MarkovandGitHub 6471ec75bd [EPLB] Reject NCCL-based EPLB communicators with async EPLB (#44978) 
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
 2026-06-10 19:51:27 +00:00
Yongye ZhuandGitHub 2ba68d9bf7 [Test] Fix one-sided MNNVL alltoall test workspace under-reservation (#44946) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-06-11 00:43:12 +08:00
Harry MellorandGitHub 3cc9fecd58 Deprecated 1st generation Qwen and QwenVL models (#45131) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-10 14:55:33 +00:00
wang.yuqiandGitHub 996222f4bf [CI] Reorganize entrypoints CI (#44947) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-06-09 00:46:11 -07:00
Tyler Michael SmithGitHubClaude Opus 4.6
 		e2f993dc41 [WideEP] Integrate DeepEP v2 (#41183) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-06-08 18:07:29 -07:00
bnellnmandGitHub dc68bd8c41 [MoE Refactor] FusedMoE/MoERunner inversion refactor (#41184) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-06-08 10:42:58 -04:00
Tushar JainGitHubTushar Jain
 		38fd2405f3 use split_group for pytorch process group creation (#41980) 
Signed-off-by: Tushar Jain <tushar00jain@users.noreply.github.com>
Co-authored-by: Tushar Jain <tushar00jain@users.noreply.github.com>
 2026-06-04 14:36:07 -04:00
Ilya MarkovGitHubMarkov IlyaNicolò LucchesiTyler Michael SmithTyler Michael Smith
 		4f423bd5bc [EPLB] Nixl communicator optimization. Zero-copy transfers (#41633) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-04 03:40:34 +00:00
Siddharth BedekarandGitHub 0917a009d3 Fix sparse NCCL weight transfer test construction (#44345) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
 2026-06-02 21:51:21 +00:00
Nick HillandGitHub cab5c9a2a9 [Core] Move max_concurrent_batches to VllmConfig (#44274) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 08:57:25 -07:00
Siddharth BedekarGitHubOpenAI Codex
 		266b9d9c64 [Frontend][Core] Add sparse NCCL weight transfer support for in-place updates (#40096) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-01 15:37:30 -04:00
Ilya MarkovGitHubMarkov IlyaTyler Michael Smith
 		4aaba00f92 [EPLB] Make async EPLB default (#43219) 
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-05-29 18:07:16 +00:00
Nick HillGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7e53283b1c [Core] Cleanup KVConnector handling with PP + fix MRV2 (#43732) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-28 13:12:03 -07:00
Harry MellorandGitHub 085ac221a3 Deprecate JAISLMHeadModel (#43784) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-28 18:29:12 +00:00
Andreas KaratzasandGitHub 445ded18c1 [ROCm][CI] Extend ROCm quick reduce coverage (#40990) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-26 21:57:13 +08:00
Wentao YeandGitHub 33d7cbe02c [Model Runner v2] Force v1 runner for tests (#43233) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-23 16:37:24 -07:00
Sumanth R HegdeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		3cb83c9592 Add model to WeightTransferEngine.__init__ (#42922) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-22 17:52:15 -07:00
akii96andGitHub bde560ed6e [ROCm] Add QuickReduce min-size override and codec threshold (#41675) 
Signed-off-by: <>
 2026-05-20 17:46:51 -05:00
Aaron HaoandGitHub 73dd2f33b7 [bug] fix WeightTransferConfig.backend to allow for all strings (#43121) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
 2026-05-19 21:01:29 -04:00
tomeras91andGitHub f54721bcc3 [Bugfix][MoE] FlashInfer one-sided: workspace union across heterogeneous layers (#42976) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2026-05-19 14:43:04 -04:00
Aaron HaoandGitHub e0a45f1455 [Feat][RL] IPC weight sync optimizations: multigpu support and chunked packed tensors (#37476) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: hao-aaron <ahao@anyscale.com>
 2026-05-15 22:53:06 +08:00
bnellnmGitHubRobert Shaw
 		d9b4990783 [MoE Refactor] EPLB refactoring for FusedMoE (#41055) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-05-12 14:16:31 -04:00
Yan Ru PeiandGitHub bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
bnellnmGitHubRobert ShawRobert Shaw
 		206eaed08d [MoE Refactor] Move expert map related code into ExpertMapManager class (#41046) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com>
 2026-05-12 09:18:27 -04:00
sungsoo haandGitHub 4f7bde572a [Kernel] Pack output and LSE in DCP A2A (#41160) 2026-05-01 09:01:17 -04:00
Rishi PuriGitHubClaudeStefano Castagnetta
 		ccfb620c62 Create tests/distributed/test_mnnvl_alltoall.py (#35241) 
Signed-off-by: Rishi Puri <riship@nvidia.com>
Signed-off-by: Claude <claude@anthropic.com>
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-29 21:56:56 +00:00
wang.yuqiandGitHub a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00
Sage MooreandGitHub 62b1bbe470 [EPLB] Remove asyncio infrastructure from Async EPLB (#40730) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-24 00:21:15 +00:00
liuzhenweiandGitHub 4a79262e0f [UT][Hardware] let torchrun example tests use the default backend (#39879) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-04-23 16:22:28 +08:00
Matthew BonanniGitHubTyler Michael Smith
 		96a85c5750 [Startup][UX] Enable CUDAGraph memory profiling by default (#38284) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-21 18:16:59 -04:00
Sage MooreandGitHub 3173441b0f [EPLB] Consolidate is_unchanged/is_received_locally into TransferMetadata (#37341) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-20 21:12:42 +00:00
Nicolò LucchesiandGitHub 304d5ba1a0 [Bugfix][CI] Fix tests/distributed/test_torchrun_example_moe.py (#40349) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-20 11:05:44 -07:00
Sage MooreGitHubTyler Michael Smith
 		3461c8b027 [EPLB] Refactor Async EPLB synchronization logic (#37601) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-04-20 17:05:41 +00:00
Ilya MarkovandGitHub 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
 2026-04-20 10:24:23 +00:00
Sumanth R HegdeGitHubRobert ShawCyrus Leung
 		adf9bb3c57 [CI] Add weight transfer tests to CI (#39821) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-04-16 15:51:45 -04:00
Martin HickeyGitHubOr Ozeri
 		cc07dad789 [HMA] [KVEvent] Enable GPU-side KV events for HMA (#37688) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-12 10:01:02 +03:00
Jeffrey WangandGitHub ab79863e6c Remove MQ multi-node tests (#38934) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-04-03 20:00:08 +00:00
Jeffrey WangandGitHub de5e6c44c6 [Feat][Executor] Introduce RayExecutorV2 (#36836) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-04-01 14:34:29 -07:00
wliao2GitHubHarry MellorKunshang Ji
 		4dfad17ed1 replace cuda_device_count_stateless() to current_platform.device_count() (#37841) 
Signed-off-by: Liao, Wei <wei.liao@intel.com>
Signed-off-by: wliao2 <wei.liao@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-31 22:32:54 +08:00
Ilya MarkovGitHubMarkov Ilya
 		abdbb68386 [EPLB] Add alternative communication for EPLB weight exchange (#33176) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
 2026-03-31 08:17:12 -04:00
Sage MooreandGitHub 497e234d38 [EPLB] Cleanup the transfer logic for the various eplb maps (#34520) 
Signed-off-by: Sage Moore <sagmoore@redhat.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-03-27 10:18:46 +01:00
Flora FengandGitHub 9040151fe1 [V0 Deprecation] Deprecate --disable-frontend-multiprocessing (#37612) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-20 11:31:43 +08:00
Sage MooreGitHubTyler Michael Smith
 		c32a58cc2a [EPLB] Simplify EPLB rearrange by only returning one map (#36267) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-03-18 20:34:00 -04:00