154 lines
2.2 KiB
Markdown
154 lines
2.2 KiB
Markdown
(quantization-supported-hardware)=
|
||
|
||
# Supported Hardware
|
||
|
||
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
|
||
|
||
:::{list-table}
|
||
:header-rows: 1
|
||
:widths: 20 8 8 8 8 8 8 8 8 8 8
|
||
|
||
- * Implementation
|
||
* Volta
|
||
* Turing
|
||
* Ampere
|
||
* Ada
|
||
* Hopper
|
||
* AMD GPU
|
||
* Intel GPU
|
||
* x86 CPU
|
||
* AWS Inferentia
|
||
* Google TPU
|
||
- * AWQ
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
- * GPTQ
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
- * Marlin (GPTQ/AWQ/FP8)
|
||
* ❌
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * INT8 (W8A8)
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ✅︎
|
||
* ❌
|
||
* ✅︎
|
||
- * FP8 (W8A8)
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * BitBLAS (GPTQ)
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * AQLM
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * bitsandbytes
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * DeepSpeedFP
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * GGUF
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
- * modelopt
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎
|
||
* ✅︎︎
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
* ❌
|
||
:::
|
||
|
||
- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
|
||
- ✅︎ indicates that the quantization method is supported on the specified hardware.
|
||
- ❌ indicates that the quantization method is not supported on the specified hardware.
|
||
|
||
:::{note}
|
||
This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
|
||
|
||
For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
|
||
:::
|