karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity

[Bugfix] Preserve FP8 indexer WK pairs across incremental load_weights (#46168)

Browse Source 
Signed-off-by: lcheng <lcheng321@gatech.edu>
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai>
This commit is contained in:
lcheng
2026-07-06 09:16:46 -07:00
committed by GitHub
co-authored by Isotr0py Nicolò Lucchesi NickLucche
parent 3ee9eea928
commit 24dd2aec81
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vllm/model_executor/models/deepseek_v2.py
+4 -1
View File
@@ -1511,7 +1511,10 @@ class DeepseekV2Model(nn.Module):
("qkv_proj", "v_proj", "v"),
]
# Fused indexer wk + weights_proj (shard 0 = wk, shard 1 = weights_proj)
_pending_wk_fp8: dict = {} # When WK is in FP8, we dequant to BF16 for fusion
_pending_wk_fp8 = getattr(self, "_pending_indexer_wk_fp8", None)
if _pending_wk_fp8 is None:
self._pending_indexer_wk_fp8 = _pending_wk_fp8 = {}
indexer_fused_mapping = [
("wk_weights_proj", "wk", 0),
("wk_weights_proj", "weights_proj", 1),