forked from Karylab-cklius/vllm
Tyler Michael SmithandClaude Sonnet 4.5
dcbedb7661
## Problem NaN values in input tensors (e.g., from attention softmax 0/0) cause NaN block scales during FP4 quantization, which then contaminate 100% of the token's output during GEMM. ## Solution Mask NaN→0 before quantization in apply_nvfp4_linear(). This prevents NaN from contaminating block scales while preserving clean data. ## Cost ~19us per layer (~0.6ms for 32-layer model, ~50% overhead on quantization). Cannot fuse into custom CUDA op without kernel changes. ## Tests - test_nvfp4_nan_block_contamination.py: Demonstrates bug (NaN→100% output) - test_nvfp4_nan_integration.py: Validates fix through production code path - test_nvfp4_nan_propagation.py: Comprehensive multi-scenario coverage - All existing NVFP4 tests pass (no regression) ## Future Work TODO in code notes proper fixes: 1. Integrate NaN check into scaled_fp4_quant CUDA kernel (zero-cost) 2. Fix upstream attention to not produce NaN 3. Integrate with check_tensor infrastructure Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>