karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
17,650 Commits 354 Branches 175 Tags
perf/push-allreduce-2buffer
Commit Graph
5937 Commits
Author SHA1 Message Date
Alexander MatveevandClaude Opus 4.6 fa6b6a83ec Apply pre-commit formatting and use torch.accelerator API 
- Apply ruff, clang-format formatting fixes
- Replace torch.cuda.set_device/device_count/synchronize with
  torch.accelerator equivalents per project convention

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>

Signed-off-by:  <>
 2026-06-15 18:21:59 -04:00
Alexander MatveevandClaude Opus 4.6 e0601e1b94 Address review: bind test sockets to localhost instead of all interfaces 
Fix CodeQL security warning by binding test helper sockets to
"localhost" instead of "" (all interfaces). These sockets are only
used for finding a free port for torch distributed init in tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>
 2026-06-15 18:09:24 -04:00
Alexander MatveevandAlexander Matveev 5fd9bd4f04 perf: add push-based allreduce for small tensor reductions 
Port SGLang's push-based 2-buffer allreduce protocol into vLLM as a new
communicator backend for small-message reductions. The push protocol
eliminates the two explicit cross-GPU NVLink barrier round-trips used by
the existing barrier-based CustomAllreduce, replacing them with a
sentinel-based data arrival detection mechanism and double-buffered epoch
alternation.

Key advantages over the barrier-based approach:
- Zero barriers: data arrival IS the synchronization (positive-zero sentinel)
- Single NVLink round-trip instead of two barrier exchanges + remote reads
- All SMs active (SM_count CTAs vs 2 CTAs) for higher NVLink bandwidth
- No cudaMemcpy to IPC staging buffer in eager mode
- PDL (griddepcontrol) support for kernel overlap on sm_90+

The new PushAllReduce is inserted in the CudaCommunicator dispatch chain
above the existing CustomAllreduce for messages below a size threshold
(~720 KB at TP=8). Larger messages continue to use the barrier-based
path. The existing CustomAllreduce code is not modified.

Measured results on DeepSeek-V4-Pro (61 layers, TP=8, 8x NVIDIA B200,
BS=1, decode with ISL=4, OSL=33024):
- Throughput: +2.14% (84.06 vs 82.30 tokens/s)
- TPOT: -2.09% (11.90 vs 12.15 ms/token)

Correctness verified via lm_eval gsm8k 5-shot with no regression
(exact_match delta within statistical noise).

The feature can be disabled at runtime via VLLM_DISABLE_PUSH_ALLREDUCE=1
to fall back to the barrier-based path.

Signed-off-by: Alexander Matveev <amatveev@redhat.com>
Signed-off-by: Alexander Matveev <alexm-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com>
 2026-06-15 18:09:24 -04:00
Flora FengandGitHub cd9078fe59 [Frontend] Skip structural tags for auto tool_choice without strict mode (#45600) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-06-15 19:55:31 +00:00
Wentao YeandGitHub e18fe932ca [Perf] Optimize DSv4 prefill chunk planning, 4.0% E2E Throughput Improvement (#45061) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-15 19:50:21 +00:00
yzong-rhGitHubBen Browning
 		51ec5cf08f [Bugfix] Chat Completions Harmony Refactor Clean up (#45464) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <bbrownin@redhat.com>
 2026-06-15 14:45:19 -04:00
Ronen SchafferGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7e612a0f06 [KV Offloading] Implement reset_cache for TieringOffloadingManager (#44541) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-15 18:42:53 +00:00
+1
youkaichaoGitHubOpenAI CodexIsotr0pyThien TranBugen ZhaoJee Jee LiRoger WangfunctionstackxYongye ZhuJee Jee Li
 		0a1c5034f5 [Model] Add MiniMax M3 support (#45381) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-06-16 01:01:25 +08:00
RoyWangandGitHub a3195fab7b [AMD][Bugfix][Quantization] Honor fused-name match in is_layer_skipped (#43981) 2026-06-15 09:37:52 -07:00
Flora FengandGitHub 0d80979644 [Chore] Consolidate reasoning/tool parser attributes into unified Parser in chat serving (#45548) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-06-15 11:16:45 -04:00
SaddssandGitHub 588db18362 [Bugfix] Two-phase KV allocation for cross-group prefix cache hits (supersedes #33775) (#44409) 
Signed-off-by: Saddss <2872669061@qq.com>
 2026-06-15 22:39:59 +08:00
Xin HeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kunshang Ji
 		5ed15f42b9 Fix the E8M0 scale computation in the MXFP4 (W4A4) MOE CUTLASS kernel (#43557) 
Signed-off-by: Xin He <xin3.he@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-06-15 06:04:54 -07:00
Juan Pérez de AlgabaandGitHub b997071ec4 (security) Enforce audio upload size limit before full file materialization (#45510) 
Signed-off-by: jperezde <jperezde@redhat.com>
 2026-06-15 10:25:24 +00:00
Yejing LaiandGitHub 9872921c5f [XPU] skip UT test_with_ngram_gpu_spec_decoding (#44423) 
Signed-off-by: Lai, Yejing <yejing.lai@intel.com>
 2026-06-15 08:46:30 +00:00
Ting SUNandGitHub 48df95c43e [Feature][Frontend] Report multimodal token counts in usage.prompt_tokens_details (#45458) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
 2026-06-15 05:20:58 +00:00
Tiezhen WANGGitHubClaude
 		7df4fe1bd7 [Model] Remove XverseForCausalLM (#45638) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-14 22:09:00 -07:00
Ben BrowningGitHubFlora Feng
 		c4a3f9d137 [Frontend] Add Streaming Parser Engine and new Qwen3 Parser (#45413) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
 2026-06-15 11:59:05 +08:00
Li, JiangandGitHub 8760f972ca [CPU] Refine CPU attention frontend (#45391) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-06-14 19:26:54 -07:00
Chaojun ZhangandGitHub 2725c84aae [XPU] Enable sequence parallel support for XPU (#38608) 
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com>
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com>
 2026-06-14 19:26:46 -07:00
Ting SUNandGitHub 3d6ce816f0 [Bugfix][Model] Validate runai_streamer model_loader_extra_config (#45291) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
 2026-06-14 19:23:30 -07:00
Taneem IbrahimandGitHub 2c764c089a Added real /v1/embeddings support for messages + chat_template_kw (#45173) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-06-15 09:08:10 +08:00
Amanzhol SalykovandGitHub 725c3bc808 [ROCm][Perf] Enable W4A16 FlyDSL MoE (#44400) 
Signed-off-by: amd-asalykov <asalykov@amd.com>
Signed-off-by: Amanzhol Salykov <asalykov@amd.com>
 2026-06-14 00:14:39 -07:00
Ekagra RanjanGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry MellorBenjamin Chislett
 		4ef4492e9b [V1][Spec Decode] Add Dynamic SD (#32374) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2026-06-14 00:14:27 -07:00
achyuthan.sGitHubShengqi Chen
 		54bbf51668 [Bugfix] nightly Docker images crash with ImportError: AnthropicOutputConfig since May 28 (#44795) 
Signed-off-by: achyuthan.s <113010327+Achyuthan-S@users.noreply.github.com>
Signed-off-by: Achyuthan S <achyuthan.sivasankar@gmail.com>
Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
 2026-06-13 21:45:29 -07:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		71b961dd35 [Perf] SM90 cutlass fp8 mm supports odd M by swap_ab, 180~290% kernel performance improvement (#44572) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-13 12:05:45 -07:00
WEI CHENG CHIUGitHubClaude
 		521b88c29e [Bugfix] Reject structured outputs for diffusion decoders with a clear error (#45468) 
Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-13 12:04:01 -07:00
Juan Pérez de AlgabaandGitHub 470229c37e [Security] Fix DoS via prompt_embeds on M-RoPE models (#45252) 
Signed-off-by: jperezde <jperezde@redhat.com>
 2026-06-13 10:17:38 +00:00
Juan Pérez de AlgabaGitHubHarry Mellor
 		2b3006076c [Security] Add timeout guard for regex compilation in structured outp… (#45118) 
Signed-off-by: jperezde <jperezde@redhat.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-13 09:52:56 +00:00
WEI CHENG CHIUandGitHub 5b2943f5a6 [Bugfix] Return the tokenizer from maybe_make_thread_pool so it survives pickling (#45460) 
Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
 2026-06-13 06:01:35 +00:00
Martin HickeyGitHubIsotr0py
 		43f0e024bc [Render] Add /derender endpoints for disaggregated postprocessing (#43606) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-13 13:55:33 +08:00
Andreas KaratzasandGitHub 1033ffac2e [CI] Wait for SSL cert refresher events in the test (#45489) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-13 04:57:18 +00:00
WEI CHENG CHIUandGitHub 17ee5b1ac5 [Bugfix] Set type/role explicitly in streaming message_start event (#45376) 
Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
 2026-06-13 01:40:50 +00:00
Nick HillandGitHub 1a369783e9 [BugFix] Avoid prematurely freeing cached mm encoder outputs (#45347) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-12 15:39:40 -07:00
Fangzhou AiGitHubClaude Fable 5
 		badddd254f [ROCm][DSV4][Perf] Fuse inverse-RoPE and cache bf16 wo_a in o-projection (#45103) 
Signed-off-by: Fangzhou Ai <fangzhouai@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 15:57:09 -05:00
Michael GoinGitHubClaude
 		c90650088d Add the QuantizedActivation linear-kernel contract (#44260) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-12 13:48:15 -07:00
Michael GoinandGitHub 9eaacb23ec [Kernel] Consolidate Marlin thread-tile padding across all dense Marlin paths (#45295) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-06-12 13:46:21 -07:00
Wentao YeGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		78739c1946 [Model Runner v2] Migration from v1 to v2, with Qwen and DSv2 MOE models [3/N] (#42667) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-12 20:44:52 +00:00
Flora FengandGitHub 6e4a547176 [Refactor] Deprecate ResponsesParser wrapper, inline parsing into ParsableContext (#45431) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-06-12 16:15:41 -04:00
Ryan RockGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		aab639c705 [Core][AMD] Propagate shutdown timeout to MultiprocExecutor (#43154) 
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-12 15:13:31 -05:00
Isotr0pyandGitHub 6635279d8a [Migration] Migrate GGUF quantization support to plugin (#39612) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-12 12:02:21 -07:00
Jonas I. LiechtiandGitHub d6fd7ce8da [Model][Dflash] Enable Dflash support for Qwen3NextForCausalLM targets (#45319) 
Signed-off-by: Jonas I. Liechti <j-i-l@t4d.ch>
 2026-06-12 10:30:09 -07:00
Xiaohong (Sean) ChenGitHubYanan Cao
 		272c16953e [Kernel][Helion][1/N] Add Helion kernel for dynamic_per_token_scaled_fp8_quant (#33790) 
Signed-off-by: Sean Chen <seachen@redhat.com>
Co-authored-by: Yanan Cao <gmagogsfm@gmail.com>
 2026-06-12 12:50:06 -04:00
ChaunceyandGitHub 3b8fc3fe6d [Frontend] Support strict mode for tool calling with ResponsesAPI (#45396) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-06-12 10:59:59 -04:00
Srinivas KrovvidiGitHubsrinivas_oo7
 		9ff278b1d2 [Core][KV Connector] fix scheduler KV connector stats aggregation (#43877) 
Fixes scheduler-side KV connector stats collection so that:

1. update_connector_output() runs before scheduler-side stats are collected.
2. worker-side and scheduler-side KV connector stats are aggregated when both are present.
3. scheduler-only KV connector stats are still emitted when no worker-side stats exist.

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
 2026-06-12 14:51:55 +00:00
Guan-Ming (Wesley) ChiuandGitHub c7aa3d2630 [Core] Support structured outputs for beam search (#35022) 
Signed-off-by: Guan-Ming (Wesley) Chiu <guanmingchiu@gmail.com>
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
 2026-06-12 06:56:25 -07:00
Thillai ChithambaramGitHubMark McLoughlin
 		4171ae406c [V1][Metrics] Add MLA attention metrics for DeepSeek MFU estimation (#39457) 
Signed-off-by: Thillai Chithambaram <thillaichithambaram.a@gmail.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-06-12 14:28:40 +01:00
Ethan FengandGitHub b7f9b6ab27 [Metrics] Add group-aware KV cache capacity to vllm:cache_config_info (#42206) 
The startup log already reports the correct group-aware KV cache capacity for
hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`.

This PR adds kv_cache_size_tokens and kv_cache_max_concurrency.

Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
 2026-06-12 11:49:44 +00:00
Tiezhen WANGGitHubClaudeIsotr0py
 		f1e13f7df9 [Model] Remove Mono-InternVL (InternLM2VEForCausalLM) (#45129) 
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-12 10:41:09 +00:00
snadampalGitHubNicolò Lucchesi
 		88ed636218 [KV Connector]: Support KV push from Prefill to Decode node using Nixl KV Connector (#35264) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-06-12 10:38:41 +00:00
ChaunceyGitHubcjackalmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		2043258dec [Frontend] Support strict mode for tool calling (#45003) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-12 07:51:48 +00:00