On MIG partitions, NVML device-level queries fail with NVMLError_NoPermission.
This causes PyTorch's CUDACachingAllocator (via expandable_segments) to crash
with an internal assert before the real torch.cuda.OutOfMemoryError can surface,
and causes NVML memory queries to report the full GPU's memory (e.g. 141 GB for
H200) instead of the MIG partition's memory (e.g. 16 GB).
This change:
1. Adds MIG detection in vllm/platforms/cuda.py that checks CUDA_VISIBLE_DEVICES
for MIG UUIDs and probes NVML for permission errors, then falls back to
NonNvmlCudaPlatform which uses torch.cuda APIs that correctly report MIG
partition memory.
2. Patches test utilities (wait_for_gpu_memory_to_clear, _get_gpu_memory_used)
to fall back to torch.cuda.mem_get_info() when NVML fails with permission
errors on MIG.
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
