karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity

add warning when FP8 KV cache misses prefill query quantization (#39752)

Browse Source 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-lyris02.lyris.clusters.nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Albert Cheng
2026-04-14 14:43:05 -04:00
committed by GitHub
co-authored by Albert Cheng Michael Goin gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent be0c855ebd
commit dc8df110bc
1 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vllm/model_executor/layers/attention/mla_attention.py
+13
View File
@@ -1443,6 +1443,19 @@ class MLACommonMetadataBuilder(AttentionMetadataBuilder[M]):
scope="local",
)
return model_dtype
elif (
is_quantized_kv_cache(vllm_config.cache_config.cache_dtype)
and backend_supports_prefill_query_quantization()
):
logger.warning_once(
"FP8 KV cache is enabled but prefill queries are not "
"quantized to FP8. For long-context workloads (ISL >= 4K), "
"enabling FP8 prefill attention can significantly optimize "
"prefill latency. To enable, add: "
'--attention-config \'{"use_prefill_query_quantization"'
": true}'",
scope="local",
)
return model_dtype