From dd0d74cd921ed8b3651fc5bbc5ce2b1296842fcd Mon Sep 17 00:00:00 2001 From: Nils Matteson Date: Tue, 7 Jul 2026 16:05:07 -0600 Subject: [PATCH] [Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374) Signed-off-by: Nils Matteson Signed-off-by: Nils Matteson Co-authored-by: Nils Matteson Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> --- docs/configuration/optimization.md | 8 ++++++++ vllm/v1/worker/gpu_worker.py | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/configuration/optimization.md b/docs/configuration/optimization.md index 0f407fbc97f..efa0f8b9046 100644 --- a/docs/configuration/optimization.md +++ b/docs/configuration/optimization.md @@ -16,6 +16,14 @@ vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow user For more information, see the [optimization level documentation](../design/optimization_levels.md). +## Faster Startup + +Beyond the optimization levels, three mechanisms reduce time-to-first-token on repeated boots of the same (model, config, hardware) combination: + +- **Reuse the compile cache.** vLLM persists `torch.compile` artifacts under `VLLM_CACHE_ROOT` (default `~/.cache/vllm`), and the cache directory can be copied between machines or baked into a container image; see the [torch.compile design doc](../design/torch_compile.md). Set `VLLM_FORCE_AOT_LOAD=1` to fail loudly instead of silently recompiling when the cache misses (any change to the model, config, relevant `VLLM_*` environment variables, torch build, or GPU model invalidates it). +- **Skip memory profiling with `--kv-cache-memory`.** On startup, vLLM logs the exact `--kv-cache-memory` value that reproduces the current allocation. Passing it back on the next boot skips the memory-profiling measurement and the CUDA-graph memory estimation pass. Note that this has performance implications: the KV cache is sized to exactly the given value instead of being measured, so a conservative value caps batch concurrency (and therefore throughput), while an optimistic one fails at allocation time. The value is only valid on the same GPU with the same initial free memory; if a boot OOMs after hardware or co-tenant changes, remove the flag to re-profile. +- **Serve without CUDA graphs using `--enforce-eager`.** Skips both compilation and CUDA-graph capture for the fastest possible startup, at the cost of steady-state decode performance. Useful for development loops and for measuring how much of a boot is compile/capture. + ## Preemption Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests. diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py index 57922c71439..341607b1b3a 100644 --- a/vllm/v1/worker/gpu_worker.py +++ b/vllm/v1/worker/gpu_worker.py @@ -832,7 +832,7 @@ class Worker(WorkerBase): f"{format_gib(self.available_kv_cache_memory_bytes)} GiB." ) - logger.debug(msg) + logger.info(msg) if self.use_v2_model_runner: # V2: Run full execute_model + sample_tokens to JIT compile triton kernels.