forked from Karylab-cklius/vllm
997132911e
Signed-off-by: Florian Woerner <florian.woerner@onmyown.io> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
613 B
613 B
llm-d
vLLM can be deployed with llm-d, a Kubernetes-native distributed inference serving stack providing well-lit paths for anyone to serve large generative AI models at scale. It helps achieve the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators and infrastructure providers.
You can use vLLM with llm-d directly by following the official guides or via KServe's LLMInferenceService.