vllm/docs

hide

hide
navigation toc

Welcome to vLLM

{ align="center" alt="vLLM Light" class="logo-light" width="60%" } { align="center" alt="vLLM Dark" class="logo-dark" width="60%" }

Easy, fast, and cheap LLM serving for everyone

<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

Where to get started with vLLM depends on the type of user. If you are looking to:

• Run open-source models on vLLM, we recommend starting with the Quickstart Guide
• Build applications with vLLM, we recommend starting with the User Guide
• Build vLLM, we recommend starting with Developer Guide

For information about the development of vLLM, see:

• Roadmap
• Releases

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests
• Fast model execution with CUDA/HIP graph
• Quantization: GPTQ, AWQ, INT4, INT8, and FP8
• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
• Speculative decoding
• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs, Gaudi® accelerators and GPUs, IBM Power CPUs, TPU, and AWS Trainium and Inferentia Accelerators.
• Prefix caching support
• Multi-LoRA support

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)
• vLLM paper (SOSP 2023)
• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.
• vLLM Meetups