karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity
18,237 Commits 354 Branches 175 Tags
dependabot/github_actions/actions/setup-python-6.3.0
Commit Graph
74 Commits
Author SHA1 Message Date
liranschourGitHubOr OzeriOr Ozeri
 		bec232a914 Secondary tier implementation for PD disaggregation (#42285) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-30 07:51:44 +03:00
Varun Sundar RabindranathGitHubVarun Sundar Rabindranath
 		c8fb2963bd [FS-Offloading] Batch Lookup in C (#46713) 
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
 2026-06-29 09:28:32 -07:00
Ronen SchafferGitHubOr Ozeri
 		bb61177e49 [KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-24 18:06:08 +03:00
Rui YinGitHubOr Ozeri
 		cf9fd6457e Fix KV offload request-finished lifecycle contract (#46284) 
Signed-off-by: test test <2260891073@qq.com>
Signed-off-by: Rui Yin <2260891073@qq.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-24 15:42:40 +03:00
Martin HickeyandGitHub f237e16b41 [KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-06-24 14:44:24 +03:00
Itay EtelisGitHubItay EtelisItay Etelis
 		7d47cff933 [Bugfix][KV Offload] Fix swap_blocks_batch on the default stream (#46379) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
 2026-06-23 05:45:27 -07:00
Srinivas KrovvidiGitHubsrinivas_oo7
 		091bc1026e [KV Offloading] Add tiering metric plumbing (#45959) 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
 2026-06-23 15:10:36 +03:00
AlexHuangandGitHub 9a938df64e [Test][KV Offloading] Add unit tests for OffloadingSpecFactory and SecondaryTierFactory (#46355) 
Signed-off-by: Alex <alex.tech.lab@outlook.com>
 2026-06-22 17:45:04 +03:00
Varun Sundar RabindranathGitHubVarun Sundar Rabindranath
 		68567ef2df [CPUOffloadingManager] Maintain evictable list in LRUCachePolicy (#46216) 
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
 2026-06-22 06:54:44 +00:00
Varun Sundar RabindranathGitHubVarun Sundar Rabindranath
 		3b4a76b63f [KV-Offloading] : Expose CPU cache usage metric (#45737) 
Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
 2026-06-20 21:21:55 +00:00
Itay EtelisGitHubItay Etelis
 		ea6078fe6a [KV Connector][Offloading] Disable parallel-agnostic fs-tier cache on V2 model runner (#46044) 
Signed-off-by: Itay Etelis <etelis2019@gmail.com>
Co-authored-by: Itay Etelis <etelis2019@gmail.com>
 2026-06-18 20:43:35 +03:00
Varun Sundar RabindranathGitHubVarun Sundar Rabindranath
 		ed938ad7db [CPUOffloading] Guard CPU eviction check (#45757) 
Signed-off-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
 2026-06-18 05:34:59 +03:00
Ronen SchafferGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7e612a0f06 [KV Offloading] Implement reset_cache for TieringOffloadingManager (#44541) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-15 18:42:53 +00:00
Itay EtelisGitHubItay Etelis
 		c3662b36ea [KV offload] Parallel-agnostic fs-tier cache for single full-attention group (#44733) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
 2026-06-11 15:48:37 +03:00
Effi OferGitHubOr Ozeri
 		af65e08fc5 KV-Cache multi-tier offloading async batched lookup (#44193) 
Signed-off-by: Effi Ofer <effi.ofer@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-10 14:59:30 +00:00
Srinivas KrovvidiGitHubsrinivas_oo7Srinivasoo7mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Or Ozeri
 		9dfc313bdc Feature/offloading manager stats (#35669) 
Signed-off-by: Sriusa4414@gmail.com
Signed-off-by: srinivas_oo7 <Sriusa4414@gmail.com>
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-06-10 12:44:55 +00:00
Effi OferandGitHub 6a894574bf Add objectstore as a secondary tier to multi-tier kv cache offloading (#41968) 
Signed-off-by: Effi Ofer <effi.ofer@gmail.com>
 2026-06-05 18:05:41 +03:00
Varun Sundar RabindranathGitHubvarun sundar rabindranath
 		3d76f395e3 [SharedOffloadRegion] Align blocks to page-size (#43689) 
Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>
 2026-06-03 14:25:57 +03:00
Itay EtelisGitHubItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		1fa9ea09f6 [Perf] Triton fast path for small CPU→GPU swap_blocks_batch in the offloading connector (#42212) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-03 13:38:17 +03:00
Rotem ShavittandGitHub 3f0a91bb96 Nit Changes in Tiered KV Offload (#44293) 
Signed-off-by: Rotem Shavitt <rshavitt@gmail.com>
 2026-06-02 21:53:21 -07:00
Ronen SchafferandGitHub 2a2b5ca791 [KV Offload] Add on_schedule_end() hook to separate step lifecycle from event draining (#44206) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-06-02 13:42:52 +03:00
Ronen SchafferGitHubOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		a3ed5ab10c [KV Offload] Add per-request offloading policy via on_new_request lifecycle hook (#43205) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-28 20:45:18 +00:00
Ronen SchafferandGitHub 4bfa0f2b14 [KV Offload] Rename SecondaryTierManager.get_finished() to get_finished_jobs() (#43870) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-28 16:00:18 +00:00
Rotem ShavittandGitHub 1b16f2ddc9 change name of fs_python secondary tier to fs. (#43600) 
Signed-off-by: Rotem Shavitt <rshavitt@gmail.com>
 2026-05-28 07:05:48 +00:00
Rotem ShavittGitHubOr Ozeri
 		d0a100c87a File system secondary tier implemented in python (#41735) 
Signed-off-by: Rotem Shavitt <rshavitt@gmail.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-05-24 18:14:44 +00:00
Ronen SchafferandGitHub 4f940896a3 [KV Offload] Pass OffloadingSpec instead of VllmConfig to secondary tiers (#43076) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-20 03:32:08 +00:00
Ronen SchafferandGitHub 84747489de Tier offload followup (#42529) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-18 19:41:58 +00:00
Ronen SchafferandGitHub 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 17:21:43 +03:00
Ronen SchafferandGitHub 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 11:11:10 +00:00
Martin HickeyGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		4955990f1b [kv_offload] Move FilterReusedOffloadingManager logic to CPUOffloadingManager (#41727) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-11 18:09:29 +03:00
Ronen SchafferandGitHub e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-10 15:09:25 +03:00
Ronen SchafferandGitHub f39bcf1e30 [KV Offload] Return None from lookup() for in-flight blocks (#41795) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-06 17:31:21 +03:00
Martin HickeyGitHubOr Ozeri
 		c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Andreas KaratzasandGitHub 95995bbef8 [ROCm][Engine] Fix GPU memory leaks in engine shutdown and test workaround for async KV prefix cache reset (#38503) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-04-25 05:25:20 +00:00
Or OzeriGitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		aad88f8486 [kv_offload+HMA][8/N]: Support multi-group worker transfer (#38453) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-22 08:44:00 +03:00
Or OzeriGitHubNicolò Lucchesi
 		f774ba028a [kv_offload+HMA][4/N]: Support sliding window lookup (#36645) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-20 12:53:51 +03:00
omerpaz95andGitHub 4353c9cb4a [KV Offload] Pass request context (#39185) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-04-19 08:54:59 +03:00
Or OzeriandGitHub 235e1f930a [kv_offload+HMA][3/N]: Remove block_size from KVEvents (#36644) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-15 11:53:19 +03:00
omerpaz95GitHubOr Ozeri
 		be0c855ebd [KV Offload] Unified memory layout for offloading workers (#37206) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-14 21:33:33 +03:00
Or OzeriandGitHub 512c5eb455 [kv_offload+HMA][5/N]: Track group block hashes and block IDs (#37109) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-08 19:50:28 +03:00
Ronen SchafferandGitHub 7c139ab23f [KV Offload] Clean up ARC/LRU refactoring leftovers: group ARC tests and fix stale comment (#38217) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-04-07 15:14:45 +03:00
wliao2andGitHub 32e0c0bfa2 refactor hard coded device string in test files under tests/v1 and tests/lora (#37566) 
Signed-off-by: Liao, Wei <wei.liao@intel.com>
 2026-04-03 11:21:47 +08:00
Or OzeriandGitHub 7cc302dd87 [kv_offload+HMA][7/N]: Support register_kv_caches for hybrid models (#37853) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-27 08:38:33 +03:00
Ronen SchafferandGitHub e3c6c10cad [KV Offload] Refactor CPU offloading: pluggable CachePolicy, remove Backend abstraction, restructure into cpu/ package (#37874) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-03-24 07:02:51 +02:00
Or OzeriandGitHub 5dd8df0701 [kv_offload+HMA][2/N]: Support multiple KV groups in GPULoadStoreSpec (#36642) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 19:26:40 +02:00
Andreas KaratzasandGitHub ce2ef42fd3 [CI] Stabilize test_cpu_offloading by waiting for async offload before cache reset (#37335) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-18 05:26:20 +00:00
Srinivasoo7GitHubsrinivas_oo7Srinivasoo7Or Ozeri
 		106ff69c4e feat(kv-offload): Strategy A — StoreReusedOffloadingManager gates CPU stores on reuse frequency (#35342) 
Signed-off-by: srinivas_oo7 <Sriusa4414@gmail.com>
Signed-off-by: Sriusa4414@gmail.com
Signed-off-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-10 14:43:40 +00:00
Ronen SchafferandGitHub bb6888b8b1 [Bugfix][CPUOffloadingManager] Prevent eviction of already-stored blocks in LRU/ARC prepare_store() (#35846) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-03-04 14:25:33 +02:00
omerpaz95GitHubOmer Paz
 		7227d06156 [Metrics] [KVConnector] Add Offloading Connector metrics (#27942) 
Added queries and hits metrics for the Offloading Connector.

Also added timing metrics for store and load operations, which take the
average time it takes to load/store, per-token.

The metrics are available from Prometheus and from the StatLogger.

Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Omer Paz <Omer.Paz@ibm.com>
 2026-01-27 13:34:49 +00:00
Or OzeriandGitHub 421012b63a OffloadingConnector: Support kernel_block_size != block_size (#30692) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-22 12:30:04 +00:00