karylab_agents/vllm

Fork 0

forked from Karylab-cklius/vllm

Files

T

History

+13

7c6729b769 [Model] Add Kimi K3 support: model files and kernels [1/N] (#50089 )

Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>

2026-07-29 14:10:58 +08:00

attention_benchmarks

[Attention] Integrate FlashAttention 4 SM100 headdim 256 support (#42669 )

2026-07-27 18:25:50 +00:00

auto_tune

Allow markdownlint to run locally (#36398 )

2026-03-08 20:05:24 -07:00

cutlass_benchmarks

[Sparse24] [Deprecation] Remove Sparse24 CT integration and kernels (#36799 )

2026-03-23 16:03:29 -04:00

fused_kernels

[Kernel] Fuse FP8 output quantization into merge_attn_states (#36518 )

2026-04-03 01:47:04 +00:00

kernels

[Model] Add Kimi K3 support: model files and kernels [1/N] (#50089 )

2026-07-29 14:10:58 +08:00

multi_turn

[Bench] benchmark_serving_multi_turn: make non-standard conversation_id payload opt-in (#43756 )

2026-06-09 19:41:56 -07:00

overheads

[Chore]:Extract math and argparse utilities to separate modules (#27188 )

2025-10-26 04:03:32 -07:00

replayssm

[Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode (#48018 )

2026-07-24 09:39:49 -07:00

structured_schemas

benchmarks: simplify test jsonschema (#14567 )

2025-03-11 13:39:30 +00:00

__init__.py

[vLLM IR] Add IR op testing and benchmarking infrastructure (#40167 )

2026-04-21 00:23:03 +00:00

backend_request_func.py

Bump Transformers version to 5.10.4 (#41359 )

2026-07-07 05:13:28 -07:00

benchmark_batch_invariance.py

[Chore] Update more locations to use attention_config.backend (#31153 )

2025-12-22 19:19:50 -08:00

benchmark_block_pool.py

[Chore]:Extract math and argparse utilities to separate modules (#27188 )

2025-10-26 04:03:32 -07:00

benchmark_hash.py

[Core] Add xxHash as a high-performance hash option for accelerating prefix caching (#29163 )

2025-12-03 16:06:57 +00:00

benchmark_hidden_state_extraction.py

Hidden states extraction improvements (#43805 )

2026-06-11 09:44:45 -04:00

benchmark_latency.py

[CI/Build][Doc] Fully deprecate old bench scripts for serving / throughput / latency (#24411 )

2025-09-09 10:02:35 +00:00

benchmark_long_document_qa_throughput.py

[Mypy] Better fixes for the mypy issues in vllm/config (#37902 )

2026-03-25 06:14:43 -07:00

benchmark_ngram_proposer.py

[Cleanup] Remove obsolete spec decoding compatibility logic (#32003 )

2026-01-09 05:44:18 +00:00

benchmark_pin_memory.py

[Bug Fix] Allow pinned memory for WSL2 (#41496 )

2026-06-16 05:56:26 +00:00

benchmark_prefix_block_hash.py

[Core] Add xxHash as a high-performance hash option for accelerating prefix caching (#29163 )

2025-12-03 16:06:57 +00:00

benchmark_prefix_caching.py

[Mypy] Better fixes for the mypy issues in vllm/config (#37902 )

2026-03-25 06:14:43 -07:00

benchmark_prioritization.py

[Mypy] Better fixes for the mypy issues in vllm/config (#37902 )

2026-03-25 06:14:43 -07:00

benchmark_serving_structured_output.py

[Misc] Add common random prefix option to structured-output serving benchmark (#41632 )

2026-05-16 00:44:48 +00:00

benchmark_serving.py

[CI/Build][Doc] Fully deprecate old bench scripts for serving / throughput / latency (#24411 )

2025-09-09 10:02:35 +00:00

benchmark_throughput.py

[CI/Build][Doc] Fully deprecate old bench scripts for serving / throughput / latency (#24411 )

2025-09-09 10:02:35 +00:00

benchmark_topk_topp.py

Revert "[Platform] Replace torch.cuda.Event with torch.Event (#47140 )" (#47668 )

2026-07-06 14:18:33 +01:00

benchmark_utils.py

[Refactor] Remove dead or duplicate func utils or variables (#35318 )

2026-02-26 10:57:56 -05:00

kv_cache_watermark.sh

[Core] Add kvcache watermark to reduce preemptions (#44594 )

2026-06-11 08:27:31 -07:00

README.md

[Docs] Update link to Benchmark CLI documentation (#33254 )

2026-02-06 16:00:59 +00:00

run_structured_output_benchmark.sh

[CI][BugFix] ShellCheck cleanup to remove baseline and preserve runtime behavior (#34514 )

2026-02-17 12:22:56 +00:00

sonnet.txt

feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark (#3277 )

2024-03-27 13:39:26 -07:00

README.md

Benchmarks

This directory used to contain vLLM's benchmark scripts and utilities for performance testing and evaluation.

Serving benchmarks: Scripts for testing online inference performance (latency, throughput)
Throughput benchmarks: Scripts for testing offline batch inference performance
Specialized benchmarks: Tools for testing specific features like structured output, prefix caching, long document QA, request prioritization, and multi-modal inference
Dataset utilities: Framework for loading and sampling from various benchmark datasets (ShareGPT, HuggingFace datasets, synthetic data, etc.)

Usage

For detailed usage instructions, examples, and dataset information, see the Benchmark CLI documentation.

For full CLI reference see:

README.md

Benchmarks

Contents

Usage