karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
15,015 Commits 354 Branches 175 Tags
tms/fix-nan
Commit Graph
106 Commits
Author SHA1 Message Date
Tyler Michael SmithandClaude Sonnet 4.5 dcbedb7661 [Bugfix] Add NaN masking to NVFP4 quantization to prevent output contamination 
## Problem
NaN values in input tensors (e.g., from attention softmax 0/0) cause NaN
block scales during FP4 quantization, which then contaminate 100% of the
token's output during GEMM.

## Solution
Mask NaN→0 before quantization in apply_nvfp4_linear(). This prevents
NaN from contaminating block scales while preserving clean data.

## Cost
~19us per layer (~0.6ms for 32-layer model, ~50% overhead on quantization).
Cannot fuse into custom CUDA op without kernel changes.

## Tests
- test_nvfp4_nan_block_contamination.py: Demonstrates bug (NaN→100% output)
- test_nvfp4_nan_integration.py: Validates fix through production code path
- test_nvfp4_nan_propagation.py: Comprehensive multi-scenario coverage
- All existing NVFP4 tests pass (no regression)

## Future Work
TODO in code notes proper fixes:
1. Integrate NaN check into scaled_fp4_quant CUDA kernel (zero-cost)
2. Fix upstream attention to not produce NaN
3. Integrate with check_tensor infrastructure

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-03-29 00:07:26 -04:00
Terry GaoandGitHub 3e6a1e1686 [Custom Ops] Add functional + out variant for scaled_fp4_quant (#34389) 
Signed-off-by: tianrengao <terrygao87@gmail.com>
 2026-03-16 18:51:46 -04:00
Kunshang JiandGitHub 53ec16a705 [Hardware] Replace torch.cuda.device_count/current_device/set_device API (#36145) 
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-12 07:57:47 -07:00
Hashem HashemiandGitHub 721ae79f50 Improvements to wvSplitKrc skinny GEMM solution (#34304) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-03-10 09:14:27 -07:00
Kunshang JiandGitHub 66a2209645 [Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-05 10:36:39 +00:00
Harry MellorandGitHub 17dc9c7fc9 [CI] Bump mypy version (#34950) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-04 20:55:11 +00:00
Hashem HashemiandGitHub 7600642eae Add padding support to wvSplitK solution for skinny GEMMs (#33762) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-02-28 09:02:05 +00:00
Michael GoinandGitHub 4fec53cfcb [CI] Actually run tests/kernels/quantization/test_block_fp8.py in CI (#34274) 2026-02-26 17:58:03 -07:00
Elizabeth ThomasGitHubClaude Opus 4.6Michael Goin
 		c97234c08b fix(mxfp4): Disable monolithic path for TRITON backend with EP (#34270) 
Signed-off-by: Elizabeth Thomas <email2eliza@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-02-25 13:33:42 -08:00
Kunshang JiandGitHub 8ad54a991b [Platform] Add current_platform.num_compute_units interface (#35042) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-02-24 22:22:49 -08:00
BadrBasowidGitHubvllmellmTJian
 		6af03f2394 [Refactor] [1/N] Reorganize kernel abstraction directory (#34055) 
Signed-off-by: BadrBasowid <badr.basowid@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-02-24 06:47:22 +00:00
rasmithandGitHub 2b84ac669c [CI][AMD][BugFix] Use torch.testing.assert_close instead of assert torch.allclose in test_rocm_skinny_gemms.py (#34181) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2026-02-18 23:10:19 +00:00
Hashem HashemiandGitHub 1b3540e6c6 Threshold fix wvSplitk for occasional CI fails (#34013) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-02-11 03:59:14 +00:00
Hashem HashemiandGitHub ed17f54c8b Perf tuning and expansion of cases covered for wvSplitKrc (#33493) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-02-07 05:33:11 -08:00
Hashem HashemiandGitHub d5c4800112 Adds padding and perf improvements to wvSplitK_fp8 (#33527) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-02-05 22:16:02 +00:00
Roy WangandGitHub 63c0889416 [Misc] Fix flashinfer related tests (#33462) 
Signed-off-by: esmeetu <jasonailu87@gmail.com>
 2026-01-31 16:10:24 -05:00
Gregory ShtrasbergandGitHub 31aedfe7d6 [Bugfix][ROCm] Fixing the skinny gemm dispatch logic from #32831 (#33366) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2026-01-30 19:05:23 -06:00
Robert ShawGitHubRobert ShawHarry Mellor
 		af9b69f977 [Quantization][Deprecation] Remove Marlin 24 (#32688) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-01-28 15:54:59 +00:00
Roberto L. CastroandGitHub fcb9df99bd [Perf][Kernel] Optimize FP4 quantization kernels (SM100F) (#32520) 
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
 2026-01-24 18:45:27 -07:00
Michael GoinandGitHub 4561f13985 [Refactor] Rename gptq_marlin to marlin to match MoE (#32952) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-23 16:48:12 -05:00
Xin YangandGitHub d08b356ee0 [Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper (#32619) 
Signed-off-by: Xin Yang <xyangx@amazon.com>
 2026-01-22 15:47:04 -05:00
Robert ShawGitHubRobert Shaw
 		85f55c943c [Quantization][Deprecation] Deprecate HQQ (#32681) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2026-01-21 09:32:40 -05:00
vllmellmandGitHub 148117ea2e [Refactor] Make FP8 Linear Ops use kernel abstraction (#27814) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
 2026-01-20 14:48:20 +08:00
Hashem HashemiandGitHub 7a1030431a Atomics Reduce Counting Optimization for SplitK Skinny GEMMs. (#29843) 
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
 2026-01-16 11:45:04 -06:00
rasmithandGitHub 3c2685645e [CI][AMD][Quantization][BugFix] Fix fp8 max in quant_utils.py and update test_fp8_quant.::test_static_fp8_quant_group_2d to use correct fp8 dtype and adjust atol/rtol (#32201) 
Signed-off-by: Randall Smith <ransmith@amd.com>
 2026-01-15 05:04:34 +00:00
Roberto L. CastroandGitHub 8ef50d9a6b [Kernel][Performance] Enable smaller Scaling Factor tiling for NVFP4 small-batch decoding (#30885) 
Signed-off-by: LopezCastroRoberto <roberto.lopez.castro@udc.es>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
 2026-01-13 15:22:53 -08:00
Lucas WilkinsonandGitHub 0a0aa07747 [Quant] Make static quant support all group shapes (#30833) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-09 12:49:27 -08:00
Lucas WilkinsonGitHubLuka Govedič
 		6cdf015c3c [Misc] Fix Current vLLM config is not set. warnings, assert to avoid issues in the future (#31747) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-01-08 15:20:49 -08:00
Kate ChengGitHubJhao-Ting Chen
 		cc6dafaef2 [Perf][Kernels] Enable FlashInfer DeepGEMM swapAB on SM90 (for W8A8 Linear Op) (#29213) 
Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com>
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
Co-authored-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
 2026-01-07 10:53:54 -05:00
Kevin McKayGitHubClaude Sonnet 4.5
 		4614c5a539 [Bugfix][Hardware][AMD] Consolidate FP8 min/max values helper function (#31106) 
Signed-off-by: c0de128 <kevin.mckay@outlook.com>
Signed-off-by: Kevin McKay <kevin@example.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-01-07 06:55:03 +00:00
wangxiyuanandGitHub bb4337b34c [Platform] Deprecate seed_everything (#31659) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-01-04 18:34:04 -08:00
Matthias GehreGitHubWentao Ye
 		e9add129ad [Bugfix] awq_gemm: fix argument order swap (#30364) 
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2025-12-14 18:15:37 +08:00
rasmithGitHubRandall Smith
 		08f8a5627e [CI/Build][Kernel][BugFix][AMD] Fix per_token_group_quant_fp8 to use correct fp8 min/max values and update atol/rtol in test_quantfp8_group_functionality (#30292) 
Signed-off-by: Randall Smith <ransmith@amd.com>
Co-authored-by: Randall Smith <ransmith@amd.com>
 2025-12-12 18:41:56 -05:00
shivamprandGitHub cd7740ac5c [ROCm] Enable Triton ScaledMM fallback + kernel selection fix (#26668) 
Signed-off-by: Shivam <shivampr.dev@gmail.com>
Signed-off-by: Shivam <shivamprasad91@gmail.com>
 2025-12-12 13:28:20 -05:00
rasmithGitHubRandall SmithMichael Goingemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		ba80926681 [CI/Build][AMD] Skip test_cutlass_w4a8_moe tests on ROCm sine they require cutlass_pack_scale_fp8 (#30508) 
Signed-off-by: Randall Smith <ransmith@amd.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Randall Smith <ransmith@amd.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-12 01:02:19 +00:00
rasmithGitHubRandall Smith
 		b75f826fca [CI/Build][AMD] Skip quantization kernels tests that require CUTLASS or e4m3fn when not supported by platform (#30020) 
Signed-off-by: Randall Smith <ransmith@amd.com>
Co-authored-by: Randall Smith <ransmith@amd.com>
 2025-12-10 02:28:37 +00:00
czhu-cohereandGitHub f6227c22ab [Kernel]Support W4A8 Grouped GEMM on Hopper (#29691) 
Signed-off-by: czhu-cohere <conway.zhu@cohere.com>
 2025-12-08 19:29:06 -08:00
rasmithGitHubRandall Smith
 		62079d8600 [CI/Build][AMD] Skip marlin, machete, and hadacore tests since these require _C functions not defined for ROCm (#30109) 
Signed-off-by: Randall Smith <ransmith@amd.com>
Co-authored-by: Randall Smith <ransmith@amd.com>
 2025-12-06 12:54:17 +08:00
Jinzhen LinGitHubMichael GoinMichael Goin
 		1656ad3704 [Kernel][Quantization] add w4a8 support for marlin kernel (#24722) 
Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin@redhat.com>
 2025-11-29 07:19:33 -08:00
Alexander MatveevGitHubmgoin
 		3aaa94ac99 [Performance] Reduce DeepGEMM N dim restriction from 128 to 64 multiplier (#28687) 
Signed-off-by: Alexander Matveev <amatveev@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-11-19 15:47:13 -08:00
Cyrus LeungandGitHub 638e4196d1 [Misc] Make SchedulerConfig.max_model_len init-only (#28733) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-15 01:59:31 -08:00
wangxiyuanandGitHub 2dacd57394 [platform] Move get_cu_count to utils (#27005) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-11-13 08:48:47 +08:00
Pavani MajetyandGitHub 72b1c2ae2c [Bugfix] Use latency MOE backend as default for Flashinfer and other misc fixes (#27439) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
 2025-11-07 04:18:39 -08:00
Yeshwanth NandGitHub 71b1c8b667 [Chore]:Extract math and argparse utilities to separate modules (#27188) 
Signed-off-by: Yeshwanth Surya <yeshsurya@gmail.com>
Signed-off-by: Yeshwanth N <yeshsurya@gmail.com>
Signed-off-by: yeshsurya <yeshsurya@gmail.com>
 2025-10-26 04:03:32 -07:00
Xiangyu LiandGitHub 5cc6bddb6e [Kernel] Add GPTQv2 format support for low-bit or asymmetric quantization, by adapting gptq_gemm (#26092) 2025-10-23 23:26:13 -04:00
dongbo910220GitHubWentao YeCyrus Leung
 		3ae082c373 [Chore] Separate out optional dependency checks from vllm.utils (#27207) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
Signed-off-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-10-22 10:44:21 -04:00
Harry MellorandGitHub 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Roberto L. CastroGitHubAndrei PanferovMichael Goin
 		96ad65b7fe [Transform] [Quantization] Add QuTLASS support to vLLM (#24440) 
Signed-off-by: LopezCastroRoberto <roberto.lopez.castro@udc.es>
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
Signed-off-by: Andrei Panferov <andrei@panferov.org>
Co-authored-by: Andrei Panferov <andrei@panferov.org>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-10-10 09:43:40 -07:00
elvischenvandGitHub 5e49c3e777 Bump Flashinfer to v0.4.0 (#26326) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2025-10-08 23:58:44 -07:00
Harry MellorandGitHub b893d661b1 Fix per file ruff ignores related to simplification (#26259) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 20:31:53 +00:00