|
166a8e954b
|
[Model] Add Inkling multi-depth MTP support [5/N]
Extend the merged Inkling MTP=1 implementation to multiple checkpoint depths with multi-step speculative decoding and KV-cache plumbing.
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-07-16 21:17:29 +00:00 |
|
|
+1
|
fb5ec0dc9e
|
[Model] Add Inkling MTP=1 support [3/N] (#48869)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-07-16 13:27:21 -07:00 |
|
|
|
971dac2caa
|
[Bugfix][KV-transfer] MoRIIO: retry RDMA send-queue-full backpressure instead of failing the read (#47495)
Signed-off-by: Edwin Lim <edwin.lim@mangoboost.io>
Signed-off-by: Edwin Lim <edwinlim0919@gmail.com>
Signed-off-by: harishk-mangoboost <harish.kambhampaty@mangoboost.io>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: harishk-mangoboost <harish.kambhampaty@mangoboost.io>
|
2026-07-16 20:02:27 +00:00 |
|
|
Shangdi YuandGitHub
|
efa2e424f6
|
[Helion] Fix degenerate scale_ub in kernel input generators (#48868)
Signed-off-by: Shangdi Yu <shangdiy@meta.com>
|
2026-07-16 19:57:45 +00:00 |
|
|
|
02bf9c7907
|
Fix Quark mxfp4 quantized model loading issue under mtp (#46757)
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
|
2026-07-16 14:58:47 -04:00 |
|
|
Woosuk KwonandGitHub
|
f61163e6c7
|
[Model] Add Hopper FA4 relative attention for Inkling (#48858)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-07-16 11:26:52 -07:00 |
|
|
Wentao YeandGitHub
|
626c90b2d5
|
[Refactor] Move fla to third party (#48500)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-07-16 19:22:36 +01:00 |
|
|
+1
|
251f7e478e
|
[Model] Add PW CUDA graph support for Inkling [2/N] (#48822)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-07-16 10:28:56 -07:00 |
|
|
|
ce65385618
|
[KV Offload] Split tiering_lookup_delay into sync/async histograms (#47679)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
|
2026-07-16 20:03:45 +03:00 |
|
|
music-dinoandGitHub
|
7d56fe2adc
|
[ROCm][CI] Avoid HIP init at config time via lazy aiter import in Quark OCP-MX (#48015)
Signed-off-by: Dino Music <Dino.Music@amd.com>
|
2026-07-16 16:47:08 +00:00 |
|
|
Zhongdongming DaiandGitHub
|
75bdad40b5
|
[Bug][Quantization] Fix humming is_layer_skipped for compressed-tensors "re:" ignore entries (#48507)
Signed-off-by: Zhongdongming Dai <zhongdongmin@nvidia.com>
|
2026-07-16 07:47:42 -07:00 |
|
|
|
d08eebad16
|
[Perf][MoE] Write FlashInfer combine into final output (#47156)
Signed-off-by: snordmann <snordmann@nvidia.com>
Co-authored-by: Codex <codex@openai.com>
|
2026-07-16 17:21:03 +03:00 |
|
|
wang.yuqiandGitHub
|
3e90d015ba
|
[Frontend] Overlap preprocessing and computation for pooling models offline inference (#47699)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
|
2026-07-16 14:00:20 +00:00 |
|
|
|
7cd1d57b74
|
[CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds (#47442)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-07-16 13:51:41 +00:00 |
|
|
|
b8168e33e0
|
[ROCm][Perf][DSV4] Enable split sparse decode on gfx942 (#46275)
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-07-16 12:28:20 +00:00 |
|
|
ovidiusmandGitHub
|
d803b44dbe
|
[NIXL] Bump nixl to 1.3.1 (#47559)
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
|
2026-07-16 14:24:33 +02:00 |
|
|
|
530852f959
|
[KV Connector] Fix PD async scheduling race condition for hybrid attn models (#48481)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: llx-08 <2596671364@qq.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-16 11:42:37 +01:00 |
|
|
Nicolò LucchesiandGitHub
|
a317bc5739
|
[Misc][Nixl] Unify _logical_to_remote_kernel_block_ids (#48717)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
|
2026-07-16 18:36:25 +08:00 |
|
|
|
a9531edfa6
|
[KV Offload] Define clean backend configuration boundary (#48150)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <cguo51@asu.edu>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-16 13:27:05 +03:00 |
|
|
ReidandGitHub
|
8c3393f373
|
[Bugfix][Rust Frontend] Limit chat top_logprobs in responses (#48134)
Signed-off-by: reidliu41 <reid201711@gmail.com>
|
2026-07-16 09:41:44 +00:00 |
|
|
Ilia YastrebovandGitHub
|
9f8cbfd8eb
|
Vectorize prep xfer list creation (#48209)
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
|
2026-07-16 11:39:05 +02:00 |
|
|
|
ea1d65fe6d
|
[Rust Frontend] Add Seed-OSS tool parser (#47741)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
|
2026-07-16 17:28:02 +08:00 |
|
|
|
f44f3d6f79
|
[Rust Frontend] Wait for mock engine endpoints before ZMQ connect (#47965)
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
|
2026-07-16 09:20:31 +00:00 |
|
|
|
cc706b05a5
|
[Bugfix][Rust Frontend] Detokenizer: avoid leaking prompt on zero-generated-token completions (#47707)
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: xiaguan <751080330@qq.com>
|
2026-07-16 09:11:21 +00:00 |
|
|
Thien TranandGitHub
|
85e296950c
|
BF16x3 router GEMM (#47973)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
|
2026-07-16 17:04:20 +08:00 |
|
|
ReidandGitHub
|
dc9f845ddc
|
[Rust Frontend] Fix mock engine test shutdown race (#48738)
Signed-off-by: reidliu41 <reid201711@gmail.com>
|
2026-07-16 07:27:43 +00:00 |
|
|
Elvir CrnčevićandGitHub
|
12f2c515a7
|
[Bugfix] Fix offloading set_ overflow for packed non-uniform KV caches (#48530)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
|
2026-07-16 10:04:39 +03:00 |
|
|
+1
|
6570c9800c
|
[Model] Add Inkling model support [1/N] (#48799)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-07-15 23:40:07 -07:00 |
|
|
|
8bfd683901
|
[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-07-15 23:49:23 -06:00 |
|
|
Micah WilliamsonandGitHub
|
7dc2698632
|
[ROCm][CI] Set "highest" matmul precision for reference hf_runner in test_bert_for_masked_lm (#48784)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
|
2026-07-16 05:49:15 +00:00 |
|
|
ErenAta16andGitHub
|
59b964f37d
|
fix(lora): validate LoRA rank is positive in PEFTHelper (#48437)
Signed-off-by: ErenAta16 <erena6466@gmail.com>
|
2026-07-16 05:22:52 +00:00 |
|
|
|
6a9f24aa8c
|
[ROCm][CI] Fix cuda graph mem profile issue (#48764)
Signed-off-by: charlifu <charlifu@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
|
2026-07-16 04:23:18 +00:00 |
|
|
|
ba47bb5be1
|
Bump flashinfer version to 0.6.14 (#47669)
Signed-off-by: AmeenP <ameenp360@gmail.com>
Signed-off-by: AmeenP <ameen@primeintellect.ai>
Co-authored-by: AmeenP <ameen@primeintellect.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Pavani Majety <pmajety@nvidia.com>
|
2026-07-15 21:00:40 -07:00 |
|
|
|
df8a0900df
|
[BugFix] Don't apply weight in batch-invariant RMSNorm when has_weight=False (#48741)
Signed-off-by: Josephasafg <ajgard7@gmail.com>
Co-authored-by: Michael Gokhman <michael.gokhman@yahoo.com>
|
2026-07-16 11:59:03 +08:00 |
|
|
|
2db39c7049
|
[Bugfix][Spec Decode] Fix eagle3 first-layer qkv_proj prefix for quantized drafts (#48068)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-07-16 11:34:41 +08:00 |
|
|
rongfu.lengandGitHub
|
3935829f89
|
[Docs] fix error key name (#48802)
Signed-off-by: rongfu.leng <lenronfu@gmail.com>
|
2026-07-16 03:16:16 +00:00 |
|
|
qli88andGitHub
|
7746961277
|
[CI] Fix flaky lora test (#47375)
Signed-off-by: Qiang Li <qiang.li2@amd.com>
Signed-off-by: qli88 <qiang.li2@amd.com>
|
2026-07-16 02:23:03 +00:00 |
|
|
qli88andGitHub
|
5de1add806
|
[feature]Add int4 quantization support for emulation moe backend (#48451)
Signed-off-by: Qiang Li <qiang.li2@amd.com>
|
2026-07-16 02:00:12 +00:00 |
|
|
Mike GandGitHub
|
915dffaa5f
|
[Attention] Mirror Triton KV dtype checks in MLA (#47060)
Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com>
|
2026-07-16 01:54:52 +00:00 |
|
|
nemanjaudovicandGitHub
|
81e13a0591
|
[Compilation] Skip x.size(dim) in _decompose_size_nodes (#42543)
Signed-off-by: nemanjaudovic <nudovic@amd.com>
|
2026-07-15 17:59:14 -07:00 |
|
|
BRIJ RAJ KISHOREandGitHub
|
f95e3f0edb
|
[Tests] Gate Step3VL under Transformers v5 (#44349)
Signed-off-by: brijrajk <22271048+brijrajk@users.noreply.github.com>
|
2026-07-15 17:59:10 -07:00 |
|
|
|
5a65ba5f17
|
[Refactor] Move iteration logging to the frontend (#46647)
Signed-off-by: maxyanghu <hyoung2991@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Shang Wang <shangw@nvidia.com>
|
2026-07-15 17:59:05 -07:00 |
|
|
|
9d1c695be5
|
[XPU] Add DSpark speculative decoding support for DeepSeek-V4 (#47677)
Signed-off-by: Ma Jian <jian1.ma@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-07-15 17:59:02 -07:00 |
|
|
|
3c1bc1fc0d
|
[ROCm][Perf] Optimize sparse attention prefill kernel for DeepSeek-V4 (#48519)
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
|
2026-07-15 17:58:59 -07:00 |
|
|
Michael GoinandGitHub
|
3a5e88e629
|
[Bugfix] Fix local speculators with dots in the name from classifying as custom_class (#48754)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2026-07-15 17:58:06 -07:00 |
|
|
|
0becb7486b
|
[BugFix][MLA] Support kv_cache_dtype_skip_layers for MLA attention (#47309)
Signed-off-by: liuruikang <liuruikang.cs@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-16 00:06:11 +00:00 |
|
|
Wentao YeandGitHub
|
2dab187f75
|
[Perf] Optimize fused_topk_bias for DSv4, 1.5~2x kernel performance improvement (#47463)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-07-15 23:40:55 +00:00 |
|
|
Giuseppe GrossiandGitHub
|
015b0320de
|
Add giuseppegrossi to rocm label auto cc action (#48643)
Signed-off-by: giuseppegrossi <ggrossi@amd.com>
|
2026-07-15 16:38:56 -07:00 |
|
|
|
4238b011a7
|
[Feature] Migrate moe sp support to non-torch compiled path for GLM5.2 (#47881)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-07-15 23:33:15 +00:00 |
|
|
kliuaeandGitHub
|
eb33ff34dd
|
[ROCm][Perf] DSv4 two-stage compressor kernel for HCA prefill (#47718)
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com>
|
2026-07-15 23:31:51 +00:00 |
|