Seiji Eicher and GitHub
6370e53f24
[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving ( #48145 )
...
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
2026-07-29 10:02:57 +02:00
Philip Pesic and GitHub
dc1be79031
Add CachePolicyFactory for pluggable/external eviction policies ( #49114 )
...
Signed-off-by: Philip Pesic <philippesic06@gmail.com >
2026-07-29 07:34:55 +03:00
nvbfalk and GitHub
60915c972c
[Feature] Add VidCom2 video token pruning ( #47750 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-07-28 03:20:17 +00:00
8de50e46d4
[Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback ( #49376 )
...
Signed-off-by: harjoth <harjoth.khara@gmail.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-07-26 20:53:28 -07:00
da3a252fd1
[KVOffload][P2P] Generic P2P secondary tier: peer lookup and serving via ParentManager ( #48021 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Signed-off-by: liranschour <liranschour@users.noreply.github.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-26 11:45:47 +03:00
94682b79f4
[multimodal] Make PyNvVideoCodec decoder concurrency configurable ( #49753 )
...
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-24 23:31:41 -07:00
Andreas Karatzas and GitHub
16aca639b7
[ROCm] Upgrade NIXL and UCX ( #49251 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
2026-07-21 22:30:02 -05:00
f007cceb42
[KV Offload] Support self-describing KV events with TieringOffloadingSpec ( #48679 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-20 16:41:58 +03:00
5245c80564
[Doc] Document blocks_per_chunk in the KV offloading guide ( #49100 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
2026-07-20 09:48:43 +03:00
426d48bfa1
[KV Offload] Add optional tier locality to FS/OBJ KV events ( #48281 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-17 10:31:52 +03:00
ab0a20d151
[Docs] Add Phi-3.5-mini-instruct to batch invariance tested models ( #46396 )
...
Signed-off-by: Yuval Luria <yuvalluria@users.noreply.github.com >
Co-authored-by: Yuval Luria <yuvalluria@users.noreply.github.com >
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com >
2026-07-16 18:05:58 -04:00
615834ee58
[KVOffload][P2P] Well-known default host/port env vars and per-DP-rank control port ( #47636 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-15 15:22:56 +03:00
e23b19309b
Deepstream video backend ( #42424 )
...
Signed-off-by: Viranjan Pagar <vpagar@nvidia.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-07-10 02:23:30 -07:00
5715fde12c
[Feature][Parser] Support include_reasoning param for non-Harmony models ( #44301 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-07-10 15:34:02 +08:00
2d814a0082
[kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers ( #47923 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-10 06:17:23 +03:00
93e2ab7111
Disable dynamic speculative decoding when DP is enabled ( #45963 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-07-07 12:52:41 +00:00
Harry Mellor and GitHub
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
700e882eab
Add TorchCodec as a video decoding backend ( #46609 )
...
Signed-off-by: Nicolas Hug <contact@nicolas-hug.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-07-06 19:58:51 -07:00
86db6c3070
[Frontend] add per-request timing metrics field to response body of Chat/Completions APIs ( #46768 )
...
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com >
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: GPT-5.5 <noreply@cursor.com >
2026-07-06 17:48:29 -07:00
d9c1767cd4
[INC][ARK] Direct Register Custom Op for ARK ( #46361 )
...
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:45:50 +08:00
1a308c449c
[XPU] Add W8A8 FP8 linear kernel with multi-granularity quant support ( #43645 )
...
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-04 18:10:01 +08:00
07516fda67
[MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs ( #45953 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-03 23:58:27 -04:00
67ff0ae30f
Support nvfp4 kv with kv-cache-dtype-skip-layers sliding_window ( #42890 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-04 02:29:13 +00:00
3af8789559
[Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) ( #38174 )
...
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com >
Signed-off-by: Wonderful <wandanfeng0802@gmail.com >
Co-authored-by: Wan_DF <wonderful199082@126.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-07-02 01:34:20 -07:00
bec232a914
Secondary tier implementation for PD disaggregation ( #42285 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Signed-off-by: liranschour <liranschour@users.noreply.github.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-06-30 07:51:44 +03:00
33f50773cb
[Doc] Fix typos, grammar, and broken commands across docs ( #46398 )
...
Signed-off-by: MichaelCaoo <a992033227@163.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-06-23 02:01:22 +00:00
a9f7b2d41c
[feature][kv_offload] Self-describing KV events for OffloadingConnector ( #43468 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-06-22 07:27:46 +00:00
d14e551a53
[Model] Remove MiniMaxText01, MiniMaxVL01, MiniMaxForCausalLM ( #45993 )
...
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-22 15:20:46 +08:00
Samuel Shen and GitHub
c9135db27c
[Docs] Update stale LMCache examples ( #45762 )
...
Signed-off-by: Samuel Shen <slshen@tensormesh.ai >
2026-06-19 03:21:36 +00:00
35e4dd4a69
[KV Connector][Mooncake] Async lookup to reduce scheduler overhead ( #45659 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-18 21:44:02 +00:00
8d4f54966c
fix(quantization): Fix AWQ dequantize on Intel XPU and refactor AutoAWQ config ( #42727 )
...
Signed-off-by: Alex <alex.tech.lab@outlook.com >
Signed-off-by: AlexHuang <jihuihuang@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-18 20:12:28 +08:00
Angelo Ruocco and GitHub
ee0fd6984a
docs, kv_offloading: add docs for selective offload ( #45279 )
...
Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com >
2026-06-17 14:58:00 +03:00
44b2512767
[KV Connector][Mooncake] Add cache_prefix to namespace store keys ( #45767 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-16 10:24:20 -07:00
Flora Feng and GitHub
cd9078fe59
[Frontend] Skip structural tags for auto tool_choice without strict mode ( #45600 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-15 19:55:31 +00:00
4ef4492e9b
[V1][Spec Decode] Add Dynamic SD ( #32374 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-06-14 00:14:27 -07:00
Isotr0py and GitHub
6635279d8a
[Migration] Migrate GGUF quantization support to plugin ( #39612 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-12 12:02:21 -07:00
Sai Sridhar Tarra and GitHub
a30addc754
[Docs][KV Connector][NIXL] document KV Transfer stat logging and Prometheus metrics ( #44055 )
...
Signed-off-by: Sai Sridhar <tarrasridhar1154@gmail.com >
2026-06-12 15:39:11 +00:00
2043258dec
[Frontend] Support strict mode for tool calling ( #45003 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
ebc6ef971a
Hidden states extraction improvements ( #43805 )
...
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-11 09:44:45 -04:00
89c6a41001
[Bench] Add BFCL dataset for vllm bench serve tool-calling workloads ( #42457 )
...
Signed-off-by: Li Zhang <lzhanga@amazon.com >
Co-authored-by: Li Zhang <lzhanga@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-06-09 23:59:18 -07:00
Change72 and GitHub
320c52b134
[Bench] benchmark_serving_multi_turn: make non-standard conversation_id payload opt-in ( #43756 )
...
Signed-off-by: Change72 <cguo51@asu.edu >
2026-06-09 19:41:56 -07:00
3d119f78f7
[Docs] Add KV offloading usage guide (single- and multi-tier) ( #44415 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-09 19:20:23 +03:00
Mohammad Miadh Angkad and GitHub
fff9210b2a
[CI/Docs] Remove stale disagg prefill links ( #44918 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-06-09 03:05:53 -07:00
Lanze Liu and GitHub
4128605ad4
[Docs] Remove broken link to deleted disaggregated_prefill.sh ( #44929 )
...
Signed-off-by: Lanze Liu <lanzetech@gmail.com >
2026-06-09 01:40:06 +00:00
Nicolò Lucchesi and GitHub
5add018beb
[Connector] Remove P2pNcclConnector ( #44854 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-06-08 18:58:29 +08:00
Daoyuan Li and GitHub
f6a708ab2b
[Doc] Add Llama-3.2-3B-Instruct to batch-invariance tested models ( #44435 )
...
Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com >
2026-06-05 16:04:32 -07:00
Harry Mellor and GitHub
a80af24356
Speed up docs build ( #44635 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-05 14:51:44 +00:00
Nicolò Lucchesi and GitHub
d98b8f371c
[NixlConnector] Initiate deprecation cycle for kv_both role ( #43874 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-06-05 11:08:17 +02:00
Fadi Arafeh and GitHub
3da29aa4a5
[DOC] Add INT8 W4A8 docs and Arm's supported quantization schemes ( #34894 )
...
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com >
2026-06-04 16:27:17 +00:00
a248b45d05
[Model] Add Gemma4 Unified (encoder-free) support ( #44429 )
...
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com >
2026-06-03 12:01:39 -07:00
