Yongye ZhuandClaude Opus 4.7 964c6eb485 [Attention][TokenSpeed MLA] Fix decode FP8 numerics: pass output_scale and assert FP8 Q ...

The decode kernel needs both bmm scales to recover correct outputs from an
FP8 KV cache: bmm1 (softmax_scale = scale * q_scale * k_scale) and bmm2
(output_scale = k_scale, since V is stored as V_real / k_scale). We were
only passing bmm1, which left bmm2 = 1.0 and produced silently wrong output.

Also assert query dtype is float8_e4m3fn on entry to forward_mqa.
supports_quant_query_input=True (inherited from MLACommonImpl) tells the
upstream pipeline to FP8-quantize Q via _decode_concat_quant_fp8_op; the
kernel is shape-specialized for FP8 Q + FP8 KV, so any other dtype here
means the upstream quant path didn't run and the kernel will produce
garbage. Failing loud beats failing silent.

Verified: gsm8k matches reference with TOKENSPEED_MLA decode +
FLASH_ATTN prefill on Kimi-K2.5-NVFP4 / TP=4 / B200.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>

2026-05-06 02:49:09 +00:00

..

arch_overview.md

Allow markdownlint to run locally (#36398)

2026-03-08 20:05:24 -07:00

attention_backends.md

[Attention][TokenSpeed MLA] Fix decode FP8 numerics: pass output_scale and assert FP8 Q

2026-05-06 02:49:09 +00:00

cuda_graphs_multimodal.md

[MM][CG] Support ViT CG for Qwen2.5-VL (#40830)

2026-05-02 11:10:14 +08:00

cuda_graphs.md

[torch.compile] Refactor Attention Quant Fusion Pass and Remove Boilerplate (#37373)

2026-03-31 14:15:50 -04:00

custom_op.md

[Docs] Adds vllm-musa to custom_op.md (#37840)

2026-03-25 11:54:36 +00:00

dbo.md

[MoE Refactor] Create MK for TRTLLM Kernels (#32564)

2026-03-03 10:39:50 -08:00

debug_vllm_compile.md

[vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823)

2026-05-01 23:41:15 -04:00

fused_moe_modular_kernel.md

[Feature]: Remove Chunking From FusedMoE (#34086)

2026-03-12 14:24:38 -04:00

fusions.md

[compile] mla + group fp8 fusion (#38877)

2026-04-21 23:16:58 -04:00

huggingface_integration.md

[Misc] Update TokenizerLike interface and move get_cached_tokenizer (#29730)

2025-11-30 14:59:47 +08:00

hybrid_kv_cache_manager.md

[doc] Hybrid KV Cache Manager design doc (#22688)

2025-08-26 20:19:05 +00:00

io_processor_plugins.md

add io_process_plugin for sparse embedding (#34214)

2026-02-28 09:16:37 +00:00

logits_processors.md

[Bugfix][CI] fix typos (#34934)

2026-03-05 17:05:46 +00:00

lora_resolver_plugins.md

[Misc] HF Hub LoRA Resolver (#20320)

2026-01-26 13:56:32 +00:00

metrics.md

[Examples] Resettle Observability examples. (#40123)

2026-04-17 03:13:31 -07:00

mm_processing.md

[1/N] Reorganize multimodal processing code (#32327)

2026-01-14 15:25:31 +00:00

model_runner_v2.md

[Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085)

2026-03-05 10:36:39 +00:00

moe_kernel_features.md

[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided (#40960)

2026-04-30 15:33:12 -07:00

multiprocessing.md

[Docs] Update design/multiprocessing.md (#30677)

2026-03-04 17:58:59 +00:00

optimization_levels.md

[ROCm] Add MLA dual RMS norm fusion (Q, KV) pass for DeepSeek/Kimi-K2 (#39242)

2026-04-20 14:56:27 +00:00

p2p_nccl_connector.md

[V0 Deprecation] Remove VLLM_USE_V1 from docs and scripts (#26336)

2025-10-07 16:46:44 +08:00

paged_attention.md

[doc] fix broken links (#32158)

2026-01-12 10:18:38 +00:00

plugin_system.md

[Deprecation][1/2] Remove items deprecated in v0.18 (#36470)

2026-03-09 03:43:23 -07:00

prefix_caching.md

docs: prefix caching seems quite outdated (#28784)

2026-01-19 11:49:52 -08:00

torch_compile_multimodal.md

[torch.compile][BE][Multimodal] Remove requirement to set_model_tag to avoid cache conflict (#37345)

2026-03-19 17:26:12 +00:00

torch_compile.md

[Frontend] Remap -O to -cc commandline flag (#29557)

2025-11-28 21:51:12 +00:00

vllm_ir.md

[vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823)

2026-05-01 23:41:15 -04:00