karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
15,002 Commits 354 Branches 175 Tags
eadf848ea00860e583da3a7da17ff113ad2ca267
Commit Graph
4865 Commits
Author SHA1 Message Date
Tyler Michael SmithandClaude Opus 4.6 a827e39037 [Test] Trim test_fused_quant_layernorm parametrization 
Reduce the test matrix by trimming:
- NUM_TOKENS_HIDDEN_SIZES: 19 → 4 combos (keep small, misaligned,
  medium-aligned, large-misaligned)
- GROUP_SIZES: drop [1, 64] (redundant with [1, 128])
- Inline group_size/tma_alignment combos to 3 meaningful cases
  instead of full Cartesian product (7)

Covers the same code paths (per-token, per-block, TMA alignment)
with fewer redundant combinations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-03-28 13:12:28 -04:00
Tyler Michael SmithandClaude Opus 4.6 f8499127aa [Test] Trim test_layernorm parametrization for faster CI 
Reduce the test matrix from 864 to 216 cases (~4x speedup) by trimming
redundant hidden sizes (keep 8, 769, 8192 — covers small, misaligned,
large) and token counts (keep 7, 4096 — small, large), and quant scales
(keep 0.01, 10.0 — extreme ends).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-03-28 13:12:28 -04:00
Tyler Michael SmithandClaude Opus 4.6 0d98729a49 [Kernel] Add zero-cost NaN/Inf detection to RMSNorm kernels 
Add per-token NaN/Inf detection to all RMSNorm CUDA kernels by
piggybacking on the existing variance reduction. NaN/Inf propagates
through sum-of-squares naturally, so a single isnan(variance) ||
isinf(variance) check on thread 0 after the CUB reduction detects
it -- zero additional kernel launches, memory reads, or register
pressure. Each block writes to its own int8 flag slot (no atomics).

Controlled by VLLM_NAN_DETECT=1. When enabled:
- Bypasses Oink/batch-invariant paths (with warning)
- Reports per-token, per-layer NaN/Inf with layer names
- Distinguishes real-token errors from padding-token warnings
- CUDA-graph compatible (fixed flag buffer address)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-03-28 13:12:28 -04:00
Thillai ChithambaramandGitHub 828f862acb [Bugfix] Expand quantization method support in perf metrics (#37231) 
Signed-off-by: Thillai Chithambaram <thillaichithambaram.a@gmail.com>
 2026-03-18 23:54:19 +00:00
Andy LoandGitHub 577df69b26 [Bugfix] Fix KV scales inconsistency in fp8 MLA & FlashInfer kv_cache_dtype "auto" leading to gibberish (#37054) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2026-03-18 23:07:29 +00:00
Wentao YeandGitHub 0d81a1fe61 [V0 Deprecation] Deprecate virtual engine (#37195) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-18 14:30:14 -07:00
Or OzeriandGitHub 5dd8df0701 [kv_offload+HMA][2/N]: Support multiple KV groups in GPULoadStoreSpec (#36642) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 19:26:40 +02:00
Xin YangandGitHub b1169d7be8 [Kernel] Add gpt-oss Router GEMM kernel (#37205) 
Signed-off-by: Xin Yang <xyangx@amazon.com>
 2026-03-18 08:15:56 -07:00
elvischenvandGitHub 296839a1b0 [Perf] Eliminate padding and slicing op for GPT-OSS with Flashinfer MXFP4 MXFP8 MoE (#30647) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2026-03-18 15:01:26 +00:00
Cyrus LeungandGitHub 99267c23ca [2/3] Refactor InternVL-based processors (#37324) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-18 22:22:19 +08:00
Or OzeriandGitHub 525f2eeb0b [kv_offload+HMA][6/N]: Split offloading_connector.py (#37405) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 14:42:46 +01:00
Yufeng HeGitHubYufeng HeIsotr0py
 		918b7890a1 [Bugfix] Fix base64 JPEG video frames returning empty metadata (#37301) 
Signed-off-by: Yufeng He <40085740+universeplayer@users.noreply.github.com>
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Yufeng He <40085740+universeplayer@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-03-18 13:40:03 +00:00
Andy LoandGitHub 98b09ddc27 [NIXL][Bugfix] metrics & testing minor bug (#36051) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2026-03-18 14:39:14 +01:00
ChaunceyandGitHub b322b197f1 [Build] Bump python openai version (#32316) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-03-18 18:20:10 +08:00
Andreas KaratzasandGitHub eaf7c9b976 [CI] Fix PaddleOCR-VL HF test failure due to create_causal_mask API rename (#37328) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-18 09:44:12 +00:00
Karan BansalGitHubkaranb192
 		fad09e8a1f fix(glm47): improve tool call parsing and content normalization (#37386) 
Signed-off-by: karanb192 <karan@example.com>
Co-authored-by: karanb192 <karan@example.com>
 2026-03-18 08:12:21 +00:00
Or OzeriGitHubNicolò Lucchesi
 		fcf0687b27 [kv_offload+HMA][0/N]: Support block-level preemption handling (#34805) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-03-18 08:49:53 +02:00
liuzhenweiGitHubKunshang Ji
 		86b7e3c95a [XPU] skip unsupported ut and update test_nixl_connector (#37179) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-18 13:32:59 +08:00
Andreas KaratzasandGitHub ce2ef42fd3 [CI] Stabilize test_cpu_offloading by waiting for async offload before cache reset (#37335) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-18 05:26:20 +00:00
Andreas KaratzasandGitHub 8b6325758c [ROCm][CI] Add ROCM_EXTRA_ARGS to audio_in_video test server fixture (#37349) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-18 04:55:40 +00:00
gxd3andGitHub a0dd1995c7 [Hardware][TPU] Add supports_async_scheduling() method to Executor interface so that it can be extended for Executor implementations. (#36924) 
Signed-off-by: Guangxiang Du <gxd@google.com>
 2026-03-18 12:53:28 +08:00
Andreas KaratzasandGitHub 58cde5c026 [ROCm][CI] Skip trtllm kvfp8 dequant tests on ROCm (#37330) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-18 11:12:26 +08:00
Yanan CaoGitHubClaude Opus 4.6
 		ff9fbc9aff [Kernel][Helion] [16/N] Refactor register_kernel API to be more Dynamo-friendly (#36705) 
Signed-off-by: Yanan Cao <gmagogsfm@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 01:23:35 +00:00
Michael GoinandGitHub 09e4576f65 [Kernel] Add non-gated support for NVFP4 CUTLASS MoE (#37320) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-03-17 18:12:04 -04:00
Yong Hoon ShinandGitHub de35c06c66 Make KV connector metadata build overridable via plugin (#37336) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2026-03-17 21:29:06 +00:00
Athrael SojuGitHubwang.yuqi
 		c0745a851a [Model] Add ColQwen3.5 4.5B support (#36887) 
Signed-off-by: Athrael Soju <athrael.soju@gmail.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-03-17 21:17:02 +00:00
Ekagra RanjanandGitHub b5ca9c3557 [Models] Cohere ASR (#35809) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
 2026-03-17 21:04:17 +00:00
Cyrus LeungandGitHub 51f0acda79 [Model] Remove unused handle_oov_mm_token (#37321) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-17 19:44:52 +00:00
Andrey TalmanandGitHub 68f783a727 [Torch 2.11] Guard torch._C._cpu attribute checks for forward compatibility (#35673) 
Signed-off-by: atalman <atalman@fb.com>
 2026-03-17 18:47:59 +00:00
Andreas KaratzasandGitHub 4ed51308c8 [CI] Fix GPU memory leak when RemoteOpenAIServer fails to start in __init__ (#37230) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-17 09:08:08 -07:00
Isotr0pyandGitHub a836524d20 [Chore] Replace all base64 usages with faster pybase64 package (#37290) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-03-17 14:44:19 +00:00
BhoomitGitHubHarry Mellor
 		3717a4dd47 [Misc][LoRA] Add --lora-target-modules to restrict LoRA to specific modules (#34984) 
Signed-off-by: Bhoomit Vasani <bhoomit.2010@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-17 14:36:41 +00:00
Harry MellorandGitHub ecfcdd2ce4 Fix Phi3 test that fails with Transformers v5 (#37298) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-17 14:29:24 +00:00
SageandGitHub 59192dfd39 [Frontend] Complete OpenAI render delegation (#37287) 
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
 2026-03-17 13:53:55 +00:00
Cyrus LeungandGitHub f340324335 [1/2] Move InternVL-based processors (#37260) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-17 21:50:56 +08:00
ViacheslavandGitHub 293f036e6d Add gigachat 3.1 tool parser + fix gigachat3 tool parser (#36664) 
Signed-off-by: Viacheslav Barinov <viacheslav.teh@gmail.com>
 2026-03-17 12:03:20 +00:00
SageandGitHub 00f8e0d211 [Frontend] Delegate tokenization serving preprocessing to OpenAIServingRender (#37266) 
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
 2026-03-17 11:22:54 +00:00
Augusto YaoandGitHub 9c7cab5ebb [Feature]: Support for multiple embedding types in a single inference call (#35829) 
Signed-off-by: augusto.yjh <augusto.yjh@antgroup.com>
 2026-03-17 17:05:42 +08:00
ChaunceyandGitHub 132bfd45b6 [Bugfix][ResponsesAPI] Fix crash when tool_choice=required exceeds max_output_tokens (#37258) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-03-17 08:54:52 +00:00
Benjamin ChislettandGitHub 8a680463fa [Bugfix] Fix NemotronH MTP + Chunked Prefill (#35447) 2026-03-17 07:07:33 +01:00
Flora FengandGitHub 3e3d320c1b [Refactor] Relocate responses API tests (#37241) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-17 05:14:52 +00:00
Flora FengandGitHub 384dc7f77b [Refactor] Relocate completion and chat completion tests (#37125) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-17 11:31:23 +08:00
Flora FengandGitHub f04d5226f8 [CI] Fix flaky tool_use chat completion tests with deterministic seed (#37027) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-03-17 03:24:34 +00:00
Vadim GimpelsonGitHubPavani Majety
 		6c1cfbad32 Support non-contiguous KV cache in TRTLLM fp8 dequant kernel (#36867) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Pavani Majety <pavanimajety@gmail.com>
 2026-03-16 17:48:42 -07:00
Harry HuangandGitHub 45f526d652 [BugFix] Correct max memory usage for multiple KV-cache groups (#36030) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
 2026-03-17 00:38:52 +00:00
Walter Beller-MoralesandGitHub 061980c36a [Feature][Frontend] add support for Cohere Embed v2 API (#37074) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-03-16 19:55:53 -04:00
Ben BrowningandGitHub 7a49742b88 [CI/Build] Add common tool call parser test suite (#27599) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
 2026-03-16 19:46:20 -04:00
Terry GaoandGitHub 3e6a1e1686 [Custom Ops] Add functional + out variant for scaled_fp4_quant (#34389) 
Signed-off-by: tianrengao <terrygao87@gmail.com>
 2026-03-16 18:51:46 -04:00
Andreas KaratzasandGitHub 4f9b14c21c [CI] Stabilize multinode DP internal LB completion tests (#36356) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-16 15:40:23 -07:00
EdalatiAliandGitHub e5b807607c [Quant][Feature] Support online MXFP8 quantization for MoE and dense models (#35448) 
Signed-off-by: EdalatiAli <aliedalati@cohere.com>
 2026-03-16 18:07:39 -04:00