karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,650 Commits 354 Branches 175 Tags
perf/push-allreduce-2buffer
Commit Graph
183 Commits
Author SHA1 Message Date
Alexander MatveevandClaude Opus 4.6 fa6b6a83ec Apply pre-commit formatting and use torch.accelerator API 
- Apply ruff, clang-format formatting fixes
- Replace torch.cuda.set_device/device_count/synchronize with
  torch.accelerator equivalents per project convention

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>

Signed-off-by:  <>
 2026-06-15 18:21:59 -04:00
Alexander MatveevandAlexander Matveev 5fd9bd4f04 perf: add push-based allreduce for small tensor reductions 
Port SGLang's push-based 2-buffer allreduce protocol into vLLM as a new
communicator backend for small-message reductions. The push protocol
eliminates the two explicit cross-GPU NVLink barrier round-trips used by
the existing barrier-based CustomAllreduce, replacing them with a
sentinel-based data arrival detection mechanism and double-buffered epoch
alternation.

Key advantages over the barrier-based approach:
- Zero barriers: data arrival IS the synchronization (positive-zero sentinel)
- Single NVLink round-trip instead of two barrier exchanges + remote reads
- All SMs active (SM_count CTAs vs 2 CTAs) for higher NVLink bandwidth
- No cudaMemcpy to IPC staging buffer in eager mode
- PDL (griddepcontrol) support for kernel overlap on sm_90+

The new PushAllReduce is inserted in the CudaCommunicator dispatch chain
above the existing CustomAllreduce for messages below a size threshold
(~720 KB at TP=8). Larger messages continue to use the barrier-based
path. The existing CustomAllreduce code is not modified.

Measured results on DeepSeek-V4-Pro (61 layers, TP=8, 8x NVIDIA B200,
BS=1, decode with ISL=4, OSL=33024):
- Throughput: +2.14% (84.06 vs 82.30 tokens/s)
- TPOT: -2.09% (11.90 vs 12.15 ms/token)

Correctness verified via lm_eval gsm8k 5-shot with no regression
(exact_match delta within statistical noise).

The feature can be disabled at runtime via VLLM_DISABLE_PUSH_ALLREDUCE=1
to fall back to the barrier-based path.

Signed-off-by: Alexander Matveev <amatveev@redhat.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>
 2026-06-15 18:09:24 -04:00
Chris LeonardGitHubShengqi Chen
 		a014dddbaa [11b/n] Migrate Machete kernels to torch stable ABI (#45304) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-12 10:36:49 +00:00
Chris LeonardandGitHub 7021be66e8 [11a/n] Migrate Marlin kernels to torch stable ABI (#45176) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
 2026-06-11 21:22:37 -07:00
Chris LeonardGitHubShengqi Chen
 		6e64c1bab1 [10c/n] Migrate MoE kernels to torch stable ABI (#44565) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-10 23:02:26 -07:00
TJianandGitHub aa6fb8a329 [Bugfix] [ROCm] [Critical] fallback to regular abi for ROCm (#44648) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-06-05 15:51:17 +00:00
Chris LeonardandGitHub 56aff0dd15 [10/n] Migrate cuda_view and silu_and_mul_per_block_quant kernels to torch stale ABI. (#44334) 2026-06-04 20:14:43 -07:00
Yongye ZhuandGitHub b5235fca2e [DSv4] Adding TRTLLM gen attention kernel (#43827) 2026-06-04 07:35:09 -07:00
Chris LeonardGitHubShengqi Chen
 		59d0236193 [10b/n] Migrate custom all-reduce, DeepSeek V4 fused MLA, MiniMax reduce-RMS, and MXFP8 MoE to libtorch stable ABI (#44365) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-04 00:29:46 +08:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		22a58640b4 [9/n] Migrate attention and cache kernels to torch stable ABI (continued) (#43717) 
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-29 04:44:45 +00:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		284e6f543d [8/n] Migrate merge_attn_states, mamba, sampler to torch stable ABI (continued) (#43361) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-27 09:35:24 -07:00
Yongye ZhuandGitHub 6ab6ffb428 [Feat][DSV4] Fuse q pad into deepseek v4 fused kernel (#43162) 2026-05-26 05:12:54 -10:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		a7be0f342d [7/n] Migrate pos_encoding and norm kernels to libtorch stable ABI (continued) (#43209) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-23 13:20:00 +08:00
Chris LeonardGitHubMikayla GawareckiShengqi Chen
 		07aeaf9d4d [6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued) (#42663) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Signed-off-by: Chris Leonard <chleonar@redhat.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-05-20 00:18:12 -07:00
Wentao YeandGitHub 00e20e76f7 [Refactor] Remove dead cuda kernels (#42767) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-18 11:14:21 -07:00
Chris LeonardGitHubMikayla Gawarecki
 		85b2fecab7 [5/n] Migrate CUTLASS MLA, hadamard, awq, allspark and DSV3 fused a gemm to torch stable ABI (continued) (#42339) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
Co-authored-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
 2026-05-13 07:24:39 +00:00
Itay EtelisGitHubItay EtelisOr OzeriItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		00b0618a03 Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps (#39306) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <etelis2019@gmail.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <etelis2019@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-10 05:57:09 +03:00
Hexiang WangGitHubganyitjtanaatjtanaavllm
 		628c436301 [New Model][ROCm] Add AMD support for DeepSeek V4 (#40871) 
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: tjtanaavllm <tunjian.tan@amd.com>
Co-authored-by: ganyi <ygan@amd.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaavllm <tunjian.tan@amd.com>
 2026-05-05 08:55:37 -07:00
Yongye ZhuandGitHub 706a04d34b [DSV4] Add silu clamp limit to shared expert (#40950) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2026-04-27 00:37:43 -07:00
+8
Yifan QiaoGitHubYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
Kaicheng YangGitHubClaude Sonnet 4.6
 		4beeb0689c fused qknorm+rope kernel optimization for SM9.0 (#37376) 
Signed-off-by: EricccYang <yangyang4991@gmail.com>
Signed-off-by: Kaicheng Yang <53411596+EricccYang@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 19:58:37 -07:00
Jee Jee LiandGitHub ecd1ea1363 [Kernel] Porting the TRTLLM minimax_allreduce_rms kernels (#37045) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-04-11 00:20:20 +08:00
Roberto L. CastroGitHubClaude Sonnet 4.5Lucas Wilkinson
 		b55d830ec7 [Perf][Kernel] Persistent TopK scheduler: unified CUDAGraph-safe kernel with dynamic per-row dispatch - DeepSeek-V3.2 DSA decode (#37421) 
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-04-08 13:35:57 -04:00
Gregory ShtrasbergandGitHub 56c976c1b5 [ROCm] Enable fused_silu_mul_block_quant on ROCm (#38817) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2026-04-08 11:23:32 -05:00
Itay EtelisGitHubItay EtelisOr Ozeri
 		4a06e1246e [Perf] Batch KV cache swap copies via cuMemcpyBatchAsync (#38460) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-03 03:13:23 +00:00
Carl YandGitHub 3bc2734dd0 [Kernel] Fuse FP8 output quantization into merge_attn_states (#36518) 
Signed-off-by: Carl You <4531192+carlyou@users.noreply.github.com>
 2026-04-03 01:47:04 +00:00
Gregory ShtrasbergGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>tjtanaavllm
 		3aab680e3e [ROCm][Bugfix] Fix ROCm runtime failure due to missing symbol (#38750) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: tjtanaavllm <tunjian.tan@amd.com>
 2026-04-01 21:30:11 -07:00
MonishverandGitHub c09ad767cd Feature/silu block quant fusion v1 (#32996) 
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
 2026-04-01 18:50:43 +00:00
Olya KozlovaandGitHub 598190aac3 [fix] Remove trtllm ragged mla prefills (#36540) 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
 2026-03-31 12:30:27 -07:00
mikaylagawareckiandGitHub 7c080dd3c5 [4/n] Migrate FP4/W4A8 CUTLASS kernels to torch stable ABI (#37503) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
 2026-03-31 10:21:13 -07:00
SandishKumarHNGitHubLucas WilkinsonMatthew Bonanni
 		bcc6f67447 [Bugfix] Use null block (0) for padded block table entries (#35431) 
Signed-off-by: SandishKumarHN <sandish@fb.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-03-30 14:02:51 -07:00
mikaylagawareckiandGitHub ab1a6a43fa [3/n] Migrate cutlass/scaled_mm_entry.cu torch stable ABI (#37221) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
 2026-03-30 11:20:13 -07:00
mikaylagawareckiandGitHub bf4cc9ed2d [2/n] Migrate per_token_group_quant to torch stable ABI (#36058) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
 2026-03-25 10:15:13 -07:00
Kyle SayersandGitHub 38364a7e32 [Sparse24] [Deprecation] Remove Sparse24 CT integration and kernels (#36799) 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
 2026-03-23 16:03:29 -04:00
mikaylagawareckiandGitHub 8b10e4fb31 [1/n] Migrate permute_cols to libtorch stable ABI (#31509) 
Signed-off-by: Mikayla Gawarecki <mikaylagawarecki@gmail.com>
 2026-03-19 11:27:26 -04:00
Michael GoinandGitHub 09e4576f65 [Kernel] Add non-gated support for NVFP4 CUTLASS MoE (#37320) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-03-17 18:12:04 -04:00
Terry GaoandGitHub 3e6a1e1686 [Custom Ops] Add functional + out variant for scaled_fp4_quant (#34389) 
Signed-off-by: tianrengao <terrygao87@gmail.com>
 2026-03-16 18:51:46 -04:00
Roberto L. CastroandGitHub 580864d81e [Attention][Perf][Kernel] Replace torch.cat with vectorized CUDA kernel MLA query concat - DeepSeek-V3.2 (#34917) 
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
 2026-03-09 09:50:36 -07:00
EdalatiAliGitHubMichael Goin
 		cb21972a97 [Kernel] Integrate SM100 MXFP8 blockscaled grouped MM and quant kernels (#34448) 
Signed-off-by: EdalatiAli <aliedalati@cohere.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-03-01 23:31:19 -08:00
Asaf GardinandGitHub bbf81f9a92 [Mamba1] - Kernel Level Chunk Alignment for Prefix Caching (#34798) 
Signed-off-by: Josephasafg <ajgard7@gmail.com>
 2026-03-01 20:40:23 +08:00
Tyler Michael SmithGitHubClaude Opus 4.6
 		eb19955c37 [WideEP] Remove pplx all2all backend (#33724) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 14:30:10 -08:00
Michael GoinandGitHub 3ef9fd0f98 [Bugfix] Fix DSV3 kernels breaking _C and _moe_C on unsupported arches (#35123) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-02-23 17:11:27 -08:00
Robert ShawGitHubRobert Shaw
 		6874638bc4 [Model Bash] DeepSeek R1 BF16 Min Latency QKV A GEMM (0.5% E2E Speedup) (#34758) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2026-02-18 07:42:36 -08:00
ElizaWszolaGitHubLuka Govedič
 		a88b3be7c4 [Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales (#33255) 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-02-17 23:35:04 -08:00
Roberto L. CastroGitHubClaude Sonnet 4.5
 		afdce12c89 [Perf][Kernel] Add faster topKperRow decode kernel for DeepSeek-V3.2 sparse attention (#33680) 
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-10 10:29:52 -05:00
Robert ShawGitHubRobert ShawHarry Mellor
 		af9b69f977 [Quantization][Deprecation] Remove Marlin 24 (#32688) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-01-28 15:54:59 +00:00
Roberto L. CastroandGitHub fcb9df99bd [Perf][Kernel] Optimize FP4 quantization kernels (SM100F) (#32520) 
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
 2026-01-24 18:45:27 -07:00
Michael GoinandGitHub 4561f13985 [Refactor] Rename gptq_marlin to marlin to match MoE (#32952) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-23 16:48:12 -05:00
Or OzeriandGitHub 421012b63a OffloadingConnector: Support kernel_block_size != block_size (#30692) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-22 12:30:04 +00:00
Wentao YeandGitHub eebc58df0c [Refactor] Remove unused cutlass moe problem size function (#32047) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-01-18 12:46:59 -08:00