vllm-frontend-rs
This is a Rust drop-in alternative frontend for vLLM. The current goal is to rebuild the northbound serving layer in Rust while still talking to the core Python vLLM engine process(es) via ZMQ over the existing engine boundary.
It should still be considered experimental, and is not feature-complete. We are working to add more functionality from the python front-end.
See https://github.com/Inferact/vllm-frontend-rs for the original commit history before it was moved into the main vllm repo.
Architecture
The component is organized as a Cargo workspace with several crates, layered bottom-up:
┌─────────────────────────────────┐
│ vllm-cmd / vllm-rs │ CLI entrypoint:
│ │ Python vLLM frontend subprocess
│ │ Rust managed-engine serve mode
├─────────────────────────────────┤
│ vllm-server │ OpenAI-compatible HTTP API (axum)
├─────────────────────────────────┤
│ vllm-chat │ Chat completions: template rendering,
│ │ structured assistant events,
│ │ reasoning & tool parsing
├─────────────────────────────────┤
│ vllm-text │ Tokenizer & incremental detokenizer
├─────────────────────────────────┤
│ vllm-llm │ Thin token-in/token-out facade over
│ │ the engine client
├─────────────────────────────────┤
│ vllm-engine-core-client │ ZMQ transport + MessagePack protocol
│ │ for the headless vLLM engine
└─────────────────────────────────┘
vllm-rs integrates into Python vllm as a Rust frontend subprocess.
Python owns process startup and launches the Rust API server as a Python-supervised worker, while
passing the inherited listening socket and transport addresses into vllm-rs.
For example:
VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3-0.6B
External Engine
vllm-rs serve can be run standalone with --data-parallel-size-local 0 when the Python engines
are started elsewhere and this node should run only the Rust frontend. The frontend still uses
the global --data-parallel-size to determine how many engines it expects to join the shared handshake.
vllm-rs serve Qwen/Qwen3-0.6B \
--headless \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 62100 \
--data-parallel-size 1 \
--data-parallel-size-local 1
Then start the Rust frontend-only server:
vllm-rs serve Qwen/Qwen3-0.6B \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 62100 \
--data-parallel-size 1 \
--data-parallel-size-local 0
For frontend development with a single engine, add --engine-session to the
frontend command:
vllm-rs serve Qwen/Qwen3-0.6B \
--data-parallel-address 127.0.0.1 \
--data-parallel-rpc-port 62100 \
--data-parallel-size 1 \
--engine-session /tmp/vllm-rs-engine-session.json
When the session file does not exist, the frontend performs the normal
handshake and writes the engine transport state to it. Stop the frontend only,
then run the same command again to bind the saved endpoints and reconnect to
the already loaded EngineCore. --engine-session implies frontend-only
external-engine mode, so --data-parallel-size-local 0 is not required.
Reattach is currently a development-only DP=1 workflow. It does not preserve
HTTP request state: if the frontend exits with active requests, their clients
disconnect, EngineCore continues them, and the replacement frontend discards
their stale outputs while accepting new work. Prefer restarting while idle to
avoid wasted engine work. Run only one frontend for a session at a time, and
delete the session file whenever EngineCore is restarted.
To build the vllm-rs in isolation:
# from the local checkout
./build_rust.sh
Example Request
After either startup path, you can use any OpenAI-compatible client:
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"stream": true
}'