Fix embed scaling + CUDA graphs in Transformers modelling backend (#48010 )

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> (cherry picked from commit 56da398dac)
Allow FlashInfer A2A backends for TRTLLM FP8 MoE Modular (#46661 )
2026-07-08 21:12:19 -07:00 · 2026-07-08 21:12:19 -07:00 · 2026-07-08 21:12:19 -07:00 · 2026-07-08 21:12:19 -07:00 · 2026-07-08 21:12:19 -07:00 · 2026-07-08 21:12:19 -07:00
181 changed files with 5130 additions and 569 deletions
@@ -17,12 +17,14 @@ steps:
  - tests/kernels/test_awq_int4_to_int8.py
  - tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
  - tests/kernels/mamba/cpu/test_cpu_gdn_ops.py
+  - tests/kernels/mamba/test_cpu_short_conv.py
  commands:
    - |
      bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
      pytest -x -v -s tests/kernels/attention/test_cpu_attn.py
      pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py
      pytest -x -v -s tests/kernels/moe/test_cpu_quant_fused_moe.py
+      pytest -x -v -s tests/kernels/mamba/test_cpu_short_conv.py
      pytest -x -v -s tests/kernels/test_onednn.py
      pytest -x -v -s tests/kernels/test_awq_int4_to_int8.py
      pytest -x -v -s tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
@@ -45,7 +45,7 @@ steps:
    agent_tags:
      label: production
      gpu: 1+
-      mem: 16+
+      mem: 24+
    no_plugin: true
    env:
      REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -38,7 +38,7 @@ steps:
  agent_tags:
    label: production
    gpu: 1+
-    mem: 16+
+    mem: 24+
  no_plugin: true
  working_dir: "."
  env:
@@ -82,7 +82,7 @@ steps:
  agent_tags:
    label: production
    gpu: 1+
-    mem: 24+
+    mem: 16+
  no_plugin: true
  working_dir: "."
  env:
@@ -134,7 +134,7 @@ steps:
  agent_tags:
    label: production
    gpu: 2+
-    mem: 24+
+    mem: 16+
  no_plugin: true
  working_dir: "."
  env:
@@ -9,7 +9,7 @@ steps:
  agent_tags:
    label: production
    gpu: 2+
-    mem: 24+
+    mem: 16+
  no_plugin: true
  working_dir: "."
  env:
@@ -9,7 +9,7 @@ steps:
  agent_tags:
    label: production
    gpu: 1+
-    mem: 16+
+    mem: 24+
  no_plugin: true
  working_dir: "."
  env:
@@ -81,7 +81,7 @@ steps:
  agent_tags:
    label: production
    gpu: 1+
-    mem: 16+
+    mem: 24+
  no_plugin: true
  working_dir: "."
  env:
@@ -105,7 +105,7 @@ steps:
  agent_tags:
    label: production
    gpu: 1+
-    mem: 16+
+    mem: 24+
  no_plugin: true
  working_dir: "."
  env:
@@ -81,7 +81,7 @@ steps:
    agent_tags:
      label: production
      gpu: 1+
-      mem: 16+
+      mem: 24+
    no_plugin: true
    env:
      REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -39,7 +39,8 @@ function cpu_tests() {
    pytest -x -v -s tests/kernels/core/test_cpu_activation.py
    pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py
    pytest -x -v -s tests/kernels/mamba/cpu/test_cpu_gdn_ops.py
-    pytest -x -v -s tests/kernels/moe/test_cpu_int4_moe.py"
+    pytest -x -v -s tests/kernels/moe/test_cpu_int4_moe.py
+    pytest -x -v -s tests/kernels/mamba/test_cpu_short_conv.py"

  # skip tests requiring model downloads if HF_TOKEN is not set
  # due to rate-limits
@@ -63,7 +64,6 @@ function cpu_tests() {
    set -e
    pytest -x -v -s tests/quantization/test_compressed_tensors.py::test_compressed_tensors_w8a8_logprobs"

-
  # basic online serving
  docker exec cpu-test bash -c '
    set -e
@@ -8,7 +8,12 @@ if [[ "$MODE" != "style-clippy" && "$MODE" != "test" ]]; then
  exit 2
 fi

-ROOT_DIR="$(git rev-parse --show-toplevel)"
+if ROOT_DIR="$(git rev-parse --show-toplevel 2>/dev/null)"; then
+  :
+else
+  SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
+  ROOT_DIR="$(cd -- "${SCRIPT_DIR}/../.." && pwd -P)"
+fi
 cd "$ROOT_DIR"

 export CARGO_TERM_COLOR="${CARGO_TERM_COLOR:-always}"
@@ -404,6 +404,36 @@ steps:
  - pytest -v -s transformers_utils
  - pytest -v -s config

+#------------------------------------------------------------  mi250 · rust  -----------------------------------------------------------#
+
+- label: Rust Frontend Cargo Style + Clippy # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
+  agent_pool: mi250_1
+  no_gpu: true
+  working_dir: "/vllm-workspace"
+  source_file_dependencies:
+  - rust/
+  - rust-toolchain.toml
+  - .buildkite/test_areas/rust_frontend_cargo.yaml
+  - .buildkite/scripts/run-rust-frontend-cargo-ci.sh
+  commands:
+  - bash .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy
+
+- label: Rust Frontend Cargo Tests # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
+  agent_pool: mi250_1
+  no_gpu: true
+  working_dir: "/vllm-workspace"
+  source_file_dependencies:
+  - rust/
+  - rust-toolchain.toml
+  - .buildkite/test_areas/rust_frontend_cargo.yaml
+  - .buildkite/scripts/run-rust-frontend-cargo-ci.sh
+  commands:
+  - bash .buildkite/scripts/run-rust-frontend-cargo-ci.sh test
+
 #-----------------------------------------------------------  mi250 · docker  ----------------------------------------------------------#

 - label: Docker Build Metadata (ROCm) # TBD
@@ -1930,6 +1960,118 @@ steps:
  - pip install "vllm-gguf-plugin >= 0.0.2"
  - pytest -v -s plugins_tests/gguf

+#-------------------------------------------------------  mi300 · rust_frontend  -------------------------------------------------------#
+
+- label: Rust Frontend OpenAI Coverage # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
+  agent_pool: mi300_1
+  optional: true
+  working_dir: "/vllm-workspace/tests"
+  source_file_dependencies:
+  - rust/
+  - vllm/benchmarks/
+  - vllm/entrypoints/openai/
+  - vllm/entrypoints/serve/
+  - vllm/v1/sample/
+  - tests/utils.py
+  - tests/benchmarks/test_serve_cli.py
+  - tests/entrypoints/openai/chat_completion/test_chat_completion.py
+  - tests/entrypoints/openai/completion/test_shutdown.py
+  - tests/v1/sample/test_logprobs_e2e.py
+  - vllm/platforms/rocm.py
+  commands:
+  - export VLLM_USE_RUST_FRONTEND=1
+  - export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  - pytest -v -s benchmarks/test_serve_cli.py -k "not insecure and not (test_bench_serve and not test_bench_serve_chat)"
+  - pytest -v -s entrypoints/openai/chat_completion/test_chat_completion.py -k "not test_invalid_json_schema and not test_invalid_regex"
+  - pytest -v -s entrypoints/openai/completion/test_shutdown.py -k "not engine_failure and not test_abort_timeout_exits_quickly"
+  - pytest -v -s v1/sample/test_logprobs_e2e.py -k "test_prompt_logprobs_e2e_server"
+
+- label: Rust Frontend Serve Admin Coverage # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
+  agent_pool: mi300_1
+  optional: true
+  working_dir: "/vllm-workspace/tests"
+  source_file_dependencies:
+  - rust/
+  - vllm/entrypoints/openai/
+  - vllm/entrypoints/serve/
+  - vllm/v1/engine/
+  - tests/utils.py
+  - tests/entrypoints/serve/disagg/test_serving_tokens.py
+  - tests/entrypoints/serve/instrumentator/test_basic.py
+  - tests/entrypoints/serve/instrumentator/test_metrics.py
+  - vllm/platforms/rocm.py
+  commands:
+  - export VLLM_USE_RUST_FRONTEND=1
+  - export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  - pytest -v -s entrypoints/serve/instrumentator/test_basic.py -k "not show_version and not server_load"
+  - pytest -v -s entrypoints/serve/disagg/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow"
+  - pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist"
+
+- label: Rust Frontend Core Correctness # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
+  agent_pool: mi300_1
+  optional: true
+  working_dir: "/vllm-workspace/tests"
+  source_file_dependencies:
+  - rust/
+  - vllm/entrypoints/openai/
+  - tests/utils.py
+  - tests/entrypoints/openai/correctness/test_lmeval.py
+  - vllm/platforms/rocm.py
+  commands:
+  - export VLLM_USE_RUST_FRONTEND=1
+  - export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  - pytest -s entrypoints/openai/correctness/test_lmeval.py::test_lm_eval_accuracy_v1_engine
+
+- label: Rust Frontend Tool Use # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
+  agent_pool: mi300_1
+  optional: true
+  working_dir: "/vllm-workspace/tests"
+  source_file_dependencies:
+  - rust/
+  - vllm/entrypoints/openai/
+  - vllm/tool_parsers/
+  - tests/utils.py
+  - tests/tool_use/
+  - vllm/platforms/rocm.py
+  commands:
+  - export VLLM_USE_RUST_FRONTEND=1
+  - export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  - pytest -v -s tool_use --ignore=tool_use/mistral --models llama3.2 -k "not test_response_format_with_tool_choice_required and not test_parallel_tool_calls_false and not test_tool_call_and_choice"
+
+- label: Rust Frontend Distributed # TBD
+  timeout_in_minutes: 180
+  mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
+  agent_pool: mi300_4
+  num_gpus: 4
+  optional: true
+  working_dir: "/vllm-workspace/tests"
+  source_file_dependencies:
+  - rust/
+  - vllm/distributed/
+  - vllm/engine/
+  - vllm/executor/
+  - vllm/v1/engine/
+  - vllm/v1/worker/
+  - tests/utils.py
+  - tests/v1/distributed/test_external_lb_dp.py
+  - tests/v1/distributed/test_hybrid_lb_dp.py
+  - tests/v1/distributed/test_internal_lb_dp.py
+  - vllm/platforms/rocm.py
+  commands:
+  - export VLLM_USE_RUST_FRONTEND=1
+  - export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  - TP_SIZE=1 DP_SIZE=4 pytest -v -s v1/distributed/test_internal_lb_dp.py -k "not 4 and not server_info"
+  - TP_SIZE=1 DP_SIZE=2 pytest -v -s v1/distributed/test_external_lb_dp.py -k "not 4 and not server_info"
+  - TP_SIZE=1 DP_SIZE=4 pytest -v -s v1/distributed/test_hybrid_lb_dp.py -k "not 4 and not server_info"
+
 #-------------------------------------------------------  mi300 · quantization  --------------------------------------------------------#

 - label: Quantization # TBD
@@ -150,9 +150,9 @@ steps:
  commands:
    - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/moe-refactor-dp-ep/config-b200.txt

- label: LM Eval Humming (A100 - TEMPORARY)
-  key: lm-eval-humming-a100
-  timeout_in_minutes: 30
+- label: LM Eval Humming f16 (A100 - TEMPORARY)
+  key: lm-eval-humming-f16-a100
+  timeout_in_minutes: 120
  device: a100
  optional: true
  num_devices: 1
@@ -160,13 +160,29 @@ steps:
  - vllm/model_executor/layers/quantization/humming.py
  - vllm/model_executor/layers/quantization/utils/humming_utils.py
  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
-  - vllm/model_executor/layers/fused_moe/oracle/mxfp4.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
  commands:
  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config.txt

- label: LM Eval Humming (H100 - TEMPORARY)
-  key: lm-eval-humming-h100
-  timeout_in_minutes: 30
+- label: LM Eval Humming Act int8 (A100 - TEMPORARY)
+  key: lm-eval-humming-act-a100
+  timeout_in_minutes: 120
+  device: a100
+  optional: true
+  num_devices: 1
+  source_file_dependencies:
+  - vllm/model_executor/layers/quantization/humming.py
+  - vllm/model_executor/layers/quantization/utils/humming_utils.py
+  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
+  commands:
+  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-int8.txt
+
+- label: LM Eval Humming f16 (H100 - TEMPORARY)
+  key: lm-eval-humming-f16-h100
+  timeout_in_minutes: 120
  device: h100
  optional: true
  num_devices: 1
@@ -174,14 +190,30 @@ steps:
  - vllm/model_executor/layers/quantization/humming.py
  - vllm/model_executor/layers/quantization/utils/humming_utils.py
  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
-  - vllm/model_executor/layers/fused_moe/oracle/mxfp4.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
  commands:
  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config.txt
-  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-fp8.txt

- label: LM Eval Humming (B200 - TEMPORARY)
-  key: lm-eval-humming-b200
-  timeout_in_minutes: 30
+- label: LM Eval Humming Act fp8/int8 (H100 - TEMPORARY)
+  key: lm-eval-humming-act-h100
+  timeout_in_minutes: 120
+  device: h100
+  optional: true
+  num_devices: 1
+  source_file_dependencies:
+  - vllm/model_executor/layers/quantization/humming.py
+  - vllm/model_executor/layers/quantization/utils/humming_utils.py
+  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
+  commands:
+  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-fp8.txt
+  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-int8.txt
+
+- label: LM Eval Humming f16 (B200 - TEMPORARY)
+  key: lm-eval-humming-f16-b200
+  timeout_in_minutes: 120
  device: b200-k8s
  optional: true
  num_devices: 1
@@ -189,10 +221,26 @@ steps:
  - vllm/model_executor/layers/quantization/humming.py
  - vllm/model_executor/layers/quantization/utils/humming_utils.py
  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
-  - vllm/model_executor/layers/fused_moe/oracle/mxfp4.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
  commands:
  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config.txt
+
+- label: LM Eval Humming Act fp8/int8 (B200 - TEMPORARY)
+  key: lm-eval-humming-act-b200
+  timeout_in_minutes: 120
+  device: b200-k8s
+  optional: true
+  num_devices: 1
+  source_file_dependencies:
+  - vllm/model_executor/layers/quantization/humming.py
+  - vllm/model_executor/layers/quantization/utils/humming_utils.py
+  - vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py
+  - vllm/model_executor/layers/fused_moe/oracle/
+  - vllm/model_executor/kernels/linear/
+  commands:
  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-fp8.txt
+  - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/humming/config-act-int8.txt

 - label: LM Eval TurboQuant KV Cache
  key: lm-eval-turboquant-kv-cache
@@ -17,7 +17,7 @@ else()
  FetchContent_Declare(
    fmha_sm100
    GIT_REPOSITORY https://github.com/vllm-project/MSA.git
-    GIT_TAG fee783153f3efe57e3e933c5cb7e267a7cebcfb5
+    GIT_TAG 2e63ec37a0fc29bc20f39cd1a52e0f5affc33a73
    GIT_PROGRESS TRUE
    CONFIGURE_COMMAND ""
    BUILD_COMMAND ""
@@ -252,6 +252,8 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/doc
 COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
 COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/.buildkite /.buildkite
 COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/pyproject.toml /pyproject.toml
+COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/rust /rust
+COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/rust-toolchain.toml /rust-toolchain.toml
 COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/vllm/v1 /vllm_v1

 # RIXL/UCX build stages
@@ -543,6 +545,8 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.h
 COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
 COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/.buildkite /.buildkite
 COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/pyproject.toml /pyproject.toml
+COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/rust /rust
+COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/rust-toolchain.toml /rust-toolchain.toml
 COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/vllm/v1 /vllm_v1

 # -----------------------
@@ -576,6 +580,7 @@ RUN apt-get update -q -y && apt-get install -q -y --no-install-recommends \
    libibverbs1 \
    ibverbs-providers \
    ibverbs-utils \
+    unzip \
    pkg-config ffmpeg libavcodec-dev libavformat-dev libavutil-dev \
    libswscale-dev libavdevice-dev libavfilter-dev libswresample-dev \
    && rm -rf /var/lib/apt/lists/*
@@ -816,6 +816,44 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
    export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
    ```

+#### Video Decoding Backend
+
+vLLM decodes video bytes into frames using a selectable decoding backend. Three
+backends are supported:
+
+- `opencv` (default): OpenCV-based decoder.
+- `pyav`: PyAV decoder.
+- `torchcodec`: TorchCodec (PyTorch-native) decoder.
+
+All three backends are ultimately backed by FFmpeg. `torchcodec` lets
+you choose which FFmpeg version is used while `opencv` and `pyav` rely on
+whichever FFmpeg build they were linked against.
+
+Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
+
+```bash
+vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
+  --media-io-kwargs '{"video": {"backend": "torchcodec"}}'
+```
+
+**TorchCodec-specific parameters:**
+
+The following parameters only apply to the `torchcodec` backend:
+
+- `num_ffmpeg_threads`: Number of FFmpeg decoding threads. `0` (default) relies
+  on the FFmpeg default, which is `min(cpu_count + 1, 16)`. This allows you to
+  control thread over-subscription.
+- `seek_mode`: Seek mode for the decoder. `"exact"` (default) guarantees
+  frame-accurate sampling by scanning the file when the decoder is created.
+  `"approximate"` skips that scan for faster decoder creation, at the cost of
+  relying on the file's metadata (which may yield less accurate seeking).
+
+```bash
+# Example: TorchCodec with approximate seek mode and 4 FFmpeg threads
+vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
+  --media-io-kwargs '{"video": {"backend": "torchcodec", "seek_mode": "approximate", "num_ffmpeg_threads": 4}}'
+```
+
 #### Video Frame Recovery

 For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.
@@ -0,0 +1,127 @@
+# Per-Request Metrics
+
+vLLM can return per-request timing metrics directly in API responses.
+This is useful for billing, SLA monitoring, and latency analysis at the
+individual request level, as a complement to the server-aggregated Prometheus
+metrics exposed at `/metrics`.
+
+## Enabling
+
+Start the server with `--enable-per-request-metrics`:
+
+```bash
+vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-per-request-metrics
+```
+
+When this flag is set, supported API responses include metrics for each
+attributable request.
+
+!!! note
+    At high concurrency, enabling per-request metrics computation may introduce
+    non-negligible CPU overhead. Benchmark your specific workload to evaluate the
+    impact before enabling in production.
+
+## Response Format
+
+When per-request metrics are enabled, the response includes a `metrics` object:
+
+```json
+{
+  "id": "chatcmpl-abc123",
+  "object": "chat.completion",
+  "model": "meta-llama/Llama-3.1-8B-Instruct",
+  "choices": [ ... ],
+  "usage": {
+    "prompt_tokens": 42,
+    "completion_tokens": 128,
+    "total_tokens": 170
+  },
+  "metrics": {
+    "time_to_first_token_ms": 85.2,
+    "generation_time_ms": 1240.5,
+    "queue_time_ms": 12.3,
+    "mean_itl_ms": 9.1,
+    "tokens_per_second": 103.2
+  }
+}
+```
+
+| Field | Description |
+| --- | --- |
+| `time_to_first_token_ms` | Time from when the request was scheduled until the first output token was generated (TTFT). |
+| `generation_time_ms` | Decode time: time from the first output token to the last output token. Excludes both queue wait and prefill/TTFT. |
+| `queue_time_ms` | Time the request spent waiting in the scheduler queue before processing began. |
+| `mean_itl_ms` | Mean inter-token latency (average time between successive output tokens) during the decode phase. `null` for single-token responses. |
+| `tokens_per_second` | Overall output token throughput: all generated tokens over the inference interval (scheduling to last output token). Unlike `generation_time_ms`, this includes the prefill phase, so it reflects end-to-end generation speed rather than pure decode speed. |
+
+All fields are `null` if the underlying timing data is not available for that
+request.
+
+!!! note
+    Timing metrics describe a single generation stream, so they are only
+    returned when the request maps to exactly one. They are suppressed (the
+    `metrics` object is `null`) for requests with `n > 1`, because the
+    underlying timing data reflects only one of the `n` sequences and cannot be
+    accurately attributed to the request as a whole. Token usage
+    (`prompt_tokens`, `completion_tokens`) remains accurate in these cases.
+    Per-request metrics also require server-side statistics logging, which is
+    on by default. vLLM rejects `--enable-per-request-metrics` when
+    `--disable-log-stats` is also set.
+
+## Example Request
+
+=== "Non-streaming"
+
+    ```python
+    from openai import OpenAI
+
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
+
+    response = client.chat.completions.create(
+        model="meta-llama/Llama-3.1-8B-Instruct",
+        messages=[{"role": "user", "content": "What is the capital of France?"}],
+    )
+
+    print(response.usage)
+    print(response.model_extra.get("metrics"))
+    ```
+
+=== "Streaming"
+
+    In streaming responses, metrics are attached to the final usage chunk (the
+    chunk sent after all content chunks). That chunk is only emitted when usage
+    reporting is enabled with `stream_options.include_usage: true` or forced
+    server-side with `--enable-force-include-usage`. Without forced usage, a
+    streaming client must set `stream_options.include_usage: true` to receive
+    metrics.
+
+    ```python
+    from openai import OpenAI
+
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
+
+    stream = client.chat.completions.create(
+        model="meta-llama/Llama-3.1-8B-Instruct",
+        messages=[{"role": "user", "content": "What is the capital of France?"}],
+        stream=True,
+        stream_options={"include_usage": True},
+    )
+
+    for chunk in stream:
+        if chunk.usage:
+            print("Usage:", chunk.usage)
+            print("Metrics:", chunk.model_extra.get("metrics"))
+    ```
+
+## Completions API
+
+Per-request metrics are also available on the `/v1/completions` endpoint using
+the same `metrics` response field. As with `n > 1`, metrics are omitted for
+requests with multiple prompts, because the timing data cannot be attributed to
+a single prompt's generation.
+
+## Relationship to Prometheus Metrics
+
+The `metrics` response field provides per-request values for a single request.
+The `/metrics` Prometheus endpoint exposes server-level histograms (e.g.
+`vllm:time_to_first_token_seconds`) that aggregate across all requests.
@@ -595,6 +595,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
 | `MolmoForCausalLM` | Molmo | T + I<sup>+</sup> | `allenai/Molmo-7B-D-0924`, `allenai/Molmo-7B-O-0924`, etc. | ✅︎ | ✅︎ |
 | `Molmo2ForConditionalGeneration` | Molmo2 | T + I<sup>+</sup> / V | `allenai/Molmo2-4B`, `allenai/Molmo2-8B`, `allenai/Molmo2-O-7B`, `allenai/MolmoWeb-4B`<sup>^</sup>, `allenai/MolmoWeb-8B`<sup>^</sup> | ✅︎ | ✅︎ |
 | `MossAudioModel` | MOSS-Audio | T + A<sup>+</sup> | `OpenMOSS-Team/MOSS-Audio-4B-Instruct`, `OpenMOSS-Team/MOSS-Audio-4B-Thinking`, `OpenMOSS-Team/MOSS-Audio-8B-Instruct`, `OpenMOSS-Team/MOSS-Audio-8B-Thinking` | ✅︎ | ✅︎ |
+| `MossTranscribeDiarizeForConditionalGeneration` | MOSS-Transcribe-Diarize | T + A | `OpenMOSS-Team/MOSS-Transcribe-Diarize` | | ✅︎ |
 | `Moondream3ForCausalLM` | Moondream3 | T + I | `moondream/moondream3-preview` | | ✅︎ |
 | `NVLM_D_Model` | NVLM-D 1.0 | T + I<sup>+</sup> | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |
 | `OpenCUAForConditionalGeneration` | OpenCUA-7B | T + I<sup>E+</sup> | `xlangai/OpenCUA-7B` | ✅︎ | ✅︎ |
@@ -697,6 +698,7 @@ Speech2Text models trained specifically for Automatic Speech Recognition.
 | `GlmAsrForConditionalGeneration` | GLM-ASR | `zai-org/GLM-ASR-Nano-2512` | ✅︎ | ✅︎ |
 | `GraniteSpeechForConditionalGeneration` | Granite Speech | `ibm-granite/granite-4.0-1b-speech`, `ibm-granite/granite-speech-3.3-2b`, etc. | ✅︎ | ✅︎ |
 | `GraniteSpeechPlusForConditionalGeneration` | Granite Speech Plus | `ibm-granite/granite-speech-4.1-2b-plus` | ✅︎ | ✅︎ |
+| `MossTranscribeDiarizeForConditionalGeneration` | MOSS-Transcribe-Diarize | `OpenMOSS-Team/MOSS-Transcribe-Diarize` | | ✅︎ |
 | `Qwen3ASRForConditionalGeneration` | Qwen3-ASR | `Qwen/Qwen3-ASR-1.7B`, etc. | ✅︎ | ✅︎ |
 | `Qwen3OmniMoeThinkerForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct`, etc. | | ✅︎ |
 | `VoxtralForConditionalGeneration` | Voxtral (Mistral format) | `mistralai/Voxtral-Mini-3B-2507`, `mistralai/Voxtral-Small-24B-2507`, etc. | ✅︎ | ✅︎ |
@@ -15,6 +15,9 @@ torchaudio; platform_machine != "s390x" and platform_machine != "riscv64"
 # required for the image processor of phi3v, this must be updated alongside torch
 torchvision; platform_machine != "s390x"  and platform_machine != "riscv64"

+# required for the torchcodec video decoding backend
+torchcodec >= 0.14; platform_machine != "s390x" and platform_machine != "riscv64" and platform_machine != "ppc64le"
+
 # Intel Extension for PyTorch, only for x86_64 CPUs
 intel-openmp==2024.2.1; platform_machine == "x86_64"

@@ -8,6 +8,7 @@ torch==2.11.0
 torchaudio==2.11.0
 # These must be updated alongside torch
 torchvision==0.26.0 # Required for phi3v processor. See https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
+torchcodec >= 0.14
 PyNvVideoCodec==2.1.0
 # FlashInfer should be updated together with the Dockerfile
 flashinfer-python==0.6.13
@@ -28,4 +29,4 @@ quack-kernels>=0.3.3
 tokenspeed-mla==0.1.2; platform_system == "Linux"

 # Humming kernels for quantization gemm
-humming-kernels[cu13]==0.1.6
+humming-kernels[cu13]==0.1.10
@@ -1134,6 +1134,8 @@ torchaudio==2.11.0+cpu
    #   -r requirements/test/cuda.in
    #   encodec
    #   vocos
+torchcodec==0.14.0+cpu
+    # via -r requirements/test/cuda.in
 torchvision==0.26.0+cpu
    # via
    #   -r requirements/test/cuda.in
@@ -13,6 +13,7 @@ pytest-cov
 # testing utils
 albumentations # required for Nemotron Parse in test_common.py
 av  # required for audio_in_video tests
+torchcodec >= 0.14 # required for torchcodec video backend tests
 backoff # required for phi4mm test
 blobfile # required for kimi-vl test
 httpx
@@ -1233,6 +1233,10 @@ torchaudio==2.11.0+cu130
    #   -r requirements/test/cuda.in
    #   encodec
    #   vocos
+torchcodec==0.14.0+cu130
+    # via
+    #   -c requirements/cuda.txt
+    #   -r requirements/test/cuda.in
 torchvision==0.26.0+cu130
    # via
    #   -c requirements/cuda.txt
@@ -15,6 +15,7 @@ numba == 0.65.0 # Required for N-gram speculative decoding
 torch==2.12.0
 torchaudio
 torchvision
+torchcodec >= 0.14 # Required for the torchcodec video decoding backend

 auto_round_lib>=0.14.0
 vllm_xpu_kernels @ https://github.com/vllm-project/vllm-xpu-kernels/releases/download/v0.1.10.1/vllm_xpu_kernels-0.1.10.1-cp38-abi3-manylinux_2_28_x86_64.whl
@@ -172,6 +172,9 @@ impl ChatLlm {
    pub async fn chat(&self, mut request: ChatRequest) -> Result<ChatEventStream> {
        request.validate()?;

+        // Stamp before rendering so render and tokenize count toward TTFT/e2e.
+        let arrival_time = vllm_llm::current_unix_timestamp_secs();
+
        let output_processor = self.backend.new_chat_output_processor(
            &mut request,
            NewChatOutputProcessorOptions {
@@ -210,6 +213,7 @@ impl ChatLlm {
            data_parallel_rank: request.data_parallel_rank,
            reasoning_parser_kwargs,
            lora_request: request.lora_request,
+            arrival_time: Some(arrival_time),
        };
        let decoded_stream = self.text.generate(text_request).await?.map_err(Error::from).boxed();

@@ -130,6 +130,19 @@ impl RoundtripCase {
        }
    }

+    /// DeepSeek V3.2 DSML tool-call format.
+    fn deepseek_v32() -> Self {
+        Self {
+            model_id: "deepseek-ai/DeepSeek-V3.2-Exp",
+            assistant_stop_suffix: "<｜end▁of▁sentence｜>",
+            tool_call_parser: ParserSelection::Auto,
+            reasoning_parser: ParserSelection::Auto,
+            thinking_behavior: ThinkingBehavior::Toggleable { default: false },
+            json_fmt: compact_json_fmt(),
+            sort_json_keys: false,
+        }
+    }
+
    /// GLM-4.7 XML-like argument format with `<think>` reasoning tags.
    fn glm47() -> Self {
        Self {
@@ -235,6 +248,7 @@ roundtrip_tests! {
    qwen35 => [reasoning_and_content, tool_call_mix],
    minimax_m25 => [reasoning_and_content, tool_call_mix],
    deepseek_v4 => [reasoning_and_content, tool_call_mix],
+    deepseek_v32 => [tool_call_mix],
    glm47 => [reasoning_and_content, tool_call_mix],
    seed_oss => [reasoning_and_content],
    step3p5 => [reasoning_and_content],
@@ -11,6 +11,21 @@ use serde_tuple::{Deserialize_tuple, Serialize_tuple};
 /// <https://github.com/vllm-project/vllm/blob/5a0a8fc1ea7542394ff315138bd5677b7b53bca1/vllm/v1/serial_utils.py#L41-L43>
 const CUSTOM_TYPE_RAW_VIEW: i8 = 3;

+#[derive(Serialize)]
+#[serde(rename = "_ExtStruct")]
+struct MsgpackExtRef<'a>((i8, ByteSlice<'a>));
+
+struct ByteSlice<'a>(&'a [u8]);
+
+impl Serialize for ByteSlice<'_> {
+    fn serialize<S>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        serializer.serialize_bytes(self.0)
+    }
+}
+
 #[easy_ext::ext(ShapeExt)]
 impl [usize] {
    /// Returned the total number of elements implied by this shape, or `None`
@@ -184,7 +199,7 @@ impl Serialize for WireArrayData {
        match self {
            Self::AuxIndex(index) => serializer.serialize_u64(*index as u64),
            Self::RawView(bytes) => {
-                Value::Ext(CUSTOM_TYPE_RAW_VIEW, bytes.clone()).serialize(serializer)
+                MsgpackExtRef((CUSTOM_TYPE_RAW_VIEW, ByteSlice(bytes))).serialize(serializer)
            }
        }
    }
@@ -194,6 +209,21 @@ impl Serialize for WireArrayData {
 mod tests {
    use super::*;

+    #[test]
+    fn raw_view_serializes_as_msgpack_ext() {
+        let bytes = vec![1, 2, 3, 4];
+        let encoded =
+            rmp_serde::to_vec_named(&WireArrayData::RawView(bytes.clone())).expect("encode");
+        let expected = rmp_serde::to_vec_named(&Value::Ext(CUSTOM_TYPE_RAW_VIEW, bytes.clone()))
+            .expect("encode expected");
+
+        assert_eq!(encoded, expected);
+        assert_eq!(
+            rmpv::decode::read_value(&mut std::io::Cursor::new(encoded)).expect("decode"),
+            Value::Ext(CUSTOM_TYPE_RAW_VIEW, bytes)
+        );
+    }
+
    #[test]
    fn constructors_build_raw_view_tensors() {
        let f32_tensor = WireNdArray::from_f32(vec![2], vec![1.0, 2.5]).unwrap();
@@ -14,6 +14,7 @@ pub use output::{
    GenerateOutputStreamExt, GeneratePromptInfo, TokenUsage,
 };
 pub use request::GenerateRequest;
+pub use request_metrics::current_unix_timestamp_secs;
 pub use vllm_engine_core_client::protocol::logprobs::{Logprobs, PositionLogprobs, TokenLogprob};

 use crate::inflight::InflightRequests;
@@ -1,5 +1,4 @@
 use std::collections::BTreeMap;
-use std::time::{SystemTime, UNIX_EPOCH};

 use uuid::Uuid;
 use vllm_engine_core_client::protocol::lora::LoraRequest;
@@ -8,6 +7,7 @@ use vllm_engine_core_client::protocol::request::{EngineCoreRequest, ReasoningPar
 use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams;

 use crate::error::{Error, Result};
+use crate::request_metrics::current_unix_timestamp_secs;

 /// Tokenized decoder-only generate request accepted by [`crate::Llm`].
 ///
@@ -30,8 +30,9 @@ pub struct GenerateRequest {
    pub mm_features: Option<MmFeatures>,
    /// Unix timestamp, in seconds, when this request arrived at the frontend.
    ///
-    /// When omitted, the Rust frontend fills it immediately before sending the
-    /// request to engine-core, matching Python's default arrival-time behavior.
+    /// Stamped at the frontend entry, before render and tokenization, to match
+    /// Python's renderer-entry arrival_time. When omitted, it is filled as a
+    /// fallback before the request is sent to engine-core.
    pub arrival_time: Option<f64>,
    /// Optional salt used to partition prefix-cache entries for this request.
    pub cache_salt: Option<String>,
@@ -122,13 +123,6 @@ impl PreparedGenerateRequest {
    }
 }

-fn current_unix_timestamp_secs() -> f64 {
-    SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .expect("system clock is before unix epoch")
-        .as_secs_f64()
-}
-
 #[cfg(test)]
 mod tests {
    use std::collections::BTreeMap;
@@ -330,7 +330,7 @@ fn diff_or_zero(end: f64, start: f64) -> f64 {
 ///
 /// Original Python request timestamp source:
 /// <https://github.com/vllm-project/vllm/blob/bc2c0c86efb28e77677a3cfb8687e976914a313a/vllm/v1/metrics/stats.py#L206-L216>
-pub(crate) fn current_unix_timestamp_secs() -> f64 {
+pub fn current_unix_timestamp_secs() -> f64 {
    SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .expect("system clock is before unix epoch")
@@ -94,6 +94,7 @@ pub fn to_text_request(
        data_parallel_rank: None,
        reasoning_parser_kwargs: None,
        lora_request: None,
+        arrival_time: None,
    })
 }

@@ -70,6 +70,7 @@ pub(super) fn prepare_generate_request(
        data_parallel_rank: ctx.data_parallel_rank,
        reasoning_parser_kwargs: None,
        lora_request: lora_resolution.lora_request.clone(),
+        arrival_time: None,
    };

    Ok(PreparedRequest {
@@ -144,6 +144,7 @@ pub(super) fn prepare_completion_request(
        data_parallel_rank: ctx.data_parallel_rank,
        reasoning_parser_kwargs: None,
        lora_request: lora_resolution.lora_request.clone(),
+        arrival_time: None,
    };

    Ok(PreparedRequest {
@@ -132,6 +132,10 @@ impl TextLlm {
    ) -> Result<(TextRequest, GenerateOutputStream)> {
        request.validate()?;

+        if request.arrival_time.is_none() {
+            request.arrival_time = Some(vllm_llm::current_unix_timestamp_secs());
+        }
+
        let tokenizer = self.backend.tokenizer();
        let prompt_token_ids = match take(&mut request.prompt) {
            Prompt::Text(text) => tokenizer.encode(&text, request.add_special_tokens)?,
@@ -18,7 +18,7 @@ use crate::request::{SamplingParams, TextRequest};
 /// One text request after it has been lowered into the raw generate boundary.
 #[derive(Debug)]
 pub struct PreparedTextRequest {
-    /// The original high-level request, preserved for response-side metadata
+    /// The high-level request fields still needed for response-side metadata
    /// and decoding options.
    pub text_request: TextRequest,
    /// The southbound request ready to be sent to `vllm-llm`.
@@ -28,7 +28,7 @@ pub struct PreparedTextRequest {
 /// Convert a high-level [`TextRequest`] into one lower-level
 /// [`GenerateRequest`] ready for the `llm` crate.
 pub fn lower_text_request(
-    request: TextRequest,
+    mut request: TextRequest,
    prompt_token_ids: Vec<u32>,
    sampling_hints: SamplingHints,
    sampling_limits: SamplingLimits,
@@ -40,7 +40,10 @@ pub fn lower_text_request(
    let generate_request = GenerateRequest {
        request_id: request.request_id.clone(),
        prompt_token_ids,
-        mm_features: request.mm_features.clone(),
+        // Align with Python's response path: decoded output state does not retain
+        // `mm_features`; move them to the engine request to avoid cloning large
+        // multimodal tensor payloads.
+        mm_features: request.mm_features.take(),
        sampling_params: lower_sampling_params(
            request.sampling_params.clone(),
            sampling_hints,
@@ -53,7 +56,7 @@ pub fn lower_text_request(
        data_parallel_rank: request.data_parallel_rank,
        reasoning_parser_kwargs: request.reasoning_parser_kwargs.clone(),
        lora_request: request.lora_request.clone(),
-        arrival_time: None,
+        arrival_time: request.arrival_time,
        trace_headers: None,
    };

@@ -307,6 +310,7 @@ mod tests {
    use std::collections::{BTreeSet, HashMap};

    use serial_test::file_serial;
+    use vllm_engine_core_client::protocol::multimodal::{MmFeatureSpec, PlaceholderRange};
    use vllm_tokenizer::test_utils::TestTokenizer;

    use super::*;
@@ -574,6 +578,35 @@ mod tests {
        .assert_debug_eq(&params);
    }

+    #[test]
+    fn lower_text_request_moves_multimodal_features_to_generate_request() {
+        let features = vec![MmFeatureSpec {
+            data: None,
+            modality: "image".to_string(),
+            identifier: "image-1".to_string(),
+            mm_position: PlaceholderRange {
+                offset: 2,
+                length: 4,
+                is_embed: None,
+            },
+            mm_hash: Some("hash-1".to_string()),
+        }];
+        let mut request = sample_request();
+        request.mm_features = Some(features.clone());
+
+        let prepared = lower_text_request(
+            request,
+            vec![1, 2, 3],
+            sample_sampling_hints(),
+            sample_sampling_limits(),
+            &stub_tokenizer(),
+        )
+        .unwrap();
+
+        assert_eq!(prepared.generate_request.mm_features, Some(features));
+        assert_eq!(prepared.text_request.mm_features, None);
+    }
+
    #[test]
    fn lower_text_request_uses_union_vocab_for_prompt_token_ids() {
        lower_text_request(
@@ -1110,6 +1143,44 @@ mod tests {
        assert_eq!(prepared.generate_request.request_id, "text-1");
    }

+    #[test]
+    fn lower_text_request_passes_arrival_time_through() {
+        let request = TextRequest {
+            arrival_time: Some(42.5),
+            ..sample_request()
+        };
+
+        let prepared = lower_text_request(
+            request,
+            vec![1, 2, 3],
+            sample_sampling_hints(),
+            sample_sampling_limits(),
+            &stub_tokenizer(),
+        )
+        .unwrap();
+
+        assert_eq!(prepared.generate_request.arrival_time, Some(42.5));
+    }
+
+    #[test]
+    fn lower_text_request_leaves_arrival_time_unset_when_absent() {
+        let request = TextRequest {
+            arrival_time: None,
+            ..sample_request()
+        };
+
+        let prepared = lower_text_request(
+            request,
+            vec![1, 2, 3],
+            sample_sampling_hints(),
+            sample_sampling_limits(),
+            &stub_tokenizer(),
+        )
+        .unwrap();
+
+        assert_eq!(prepared.generate_request.arrival_time, None);
+    }
+
    #[test]
    fn resolve_max_tokens_user_smaller_than_model_limit() {
        let result = resolve_max_tokens(Some(50), None, 200, 100);
@@ -187,6 +187,12 @@ pub struct TextRequest {
    /// LoRA adapter selected for this request.
    #[serde(default)]
    pub lora_request: Option<LoraRequest>,
+    /// Wall-clock unix timestamp (seconds) when this request arrived at the
+    /// frontend, stamped before render/tokenize to match Python's
+    /// renderer-entry arrival_time. When unset, it is stamped before
+    /// tokenization.
+    #[serde(default)]
+    pub arrival_time: Option<f64>,
 }

 impl TextRequest {
@@ -205,6 +211,7 @@ impl TextRequest {
            data_parallel_rank: None,
            reasoning_parser_kwargs: None,
            lora_request: None,
+            arrival_time: None,
        }
    }

@@ -0,0 +1,106 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Tests for SpeculativeConfig.compose_draft_hf_overrides.
+
+Callable ``hf_overrides`` on the target model config (e.g. the
+``dummy_hf_overrides`` shrink used by ``tests/models/test_initialization.py``)
+must also be applied when building the draft ``ModelConfig``. Otherwise a
+draft belonging to a large target model is instantiated at full size even
+when the target itself is shrunk — which is what kept spec-decode archs like
+``EagleMistralLarge3ForCausalLM`` stuck at ``is_available_online=False``
+("TODO: revert once figuring out OOM in CI").
+"""
+
+import functools
+
+import pytest
+from transformers import PretrainedConfig
+
+from vllm.config.speculative import SpeculativeConfig
+
+
+def _make_hf_config(**kwargs) -> PretrainedConfig:
+    defaults = dict(
+        architectures=["LlamaForCausalLM"],
+        model_type="llama",
+        num_hidden_layers=64,
+    )
+    defaults.update(kwargs)
+    return PretrainedConfig(**defaults)
+
+
+@pytest.mark.cpu_test
+def test_dict_overrides_are_not_forwarded_to_draft():
+    """Dict overrides are target-specific key patches; the draft must get
+    only the architecture-mapping override."""
+    composed = SpeculativeConfig.compose_draft_hf_overrides(
+        {"max_position_embeddings": 1234}
+    )
+    assert composed is SpeculativeConfig.hf_config_override
+
+
+@pytest.mark.cpu_test
+def test_none_overrides_fall_back_to_arch_mapping():
+    composed = SpeculativeConfig.compose_draft_hf_overrides(None)
+    assert composed is SpeculativeConfig.hf_config_override
+
+
+@pytest.mark.cpu_test
+def test_callable_overrides_reach_the_draft_config():
+    """A callable override (config-to-config transform) composes with the
+    architecture-mapping override and is applied to the draft config."""
+
+    def shrink(hf_config: PretrainedConfig) -> PretrainedConfig:
+        hf_config.num_hidden_layers = 1
+        return hf_config
+
+    composed = SpeculativeConfig.compose_draft_hf_overrides(shrink)
+    assert composed is not SpeculativeConfig.hf_config_override
+
+    out = composed(_make_hf_config())
+    # The shrink transform must have been applied to the draft config.
+    assert out.num_hidden_layers == 1
+
+
+@pytest.mark.cpu_test
+def test_arch_mapping_applies_before_callable_override():
+    """The static arch-mapping override runs first, so the user callable
+    observes (and may adjust) the post-mapping config."""
+    seen_architectures: list[str] = []
+
+    def record(hf_config: PretrainedConfig) -> PretrainedConfig:
+        seen_architectures.append(hf_config.architectures[0])
+        return hf_config
+
+    composed = SpeculativeConfig.compose_draft_hf_overrides(record)
+
+    # MiMo is one of the arch-mapped model types: hf_config_override
+    # rewrites architectures to ["MiMoMTPModel"].
+    mimo = _make_hf_config(
+        architectures=["MiMoForCausalLM"],
+        model_type="mimo",
+        num_nextn_predict_layers=1,
+    )
+    composed(mimo)
+    assert seen_architectures == ["MiMoMTPModel"]
+
+
+def _module_level_shrink(hf_config: PretrainedConfig) -> PretrainedConfig:
+    hf_config.num_hidden_layers = 1
+    return hf_config
+
+
+@pytest.mark.cpu_test
+def test_composed_override_is_picklable():
+    """The draft ``ModelConfig`` is sent to spawned engine-core processes, so
+    the composed override must be picklable. A nested local closure is not
+    (it raised ``Can't get local object`` on DFlashDraftModel); a
+    ``functools.partial`` over a module-referenceable static method is.
+    Guard against regressing to a closure."""
+    composed = SpeculativeConfig.compose_draft_hf_overrides(_module_level_shrink)
+
+    assert isinstance(composed, functools.partial)
+    assert composed.func is SpeculativeConfig._apply_composed_hf_override
+
+    out = composed(_make_hf_config())
+    assert out.num_hidden_layers == 1
@@ -191,7 +191,7 @@ class TestNCCLEngineParsing:
        return NCCLWeightTransferEngine(
            config,
            create_mock_vllm_config(),
-            "cuda",
+            torch.device("cuda"),
            MagicMock(spec=torch.nn.Module),
        )

@@ -240,21 +240,30 @@ class TestEngineRegistry:
    def test_create_engine_nccl(self):
        config = WeightTransferConfig(backend="nccl")
        engine = WeightTransferEngineFactory.create_engine(
-            config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+            config,
+            create_mock_vllm_config(),
+            torch.device("cuda"),
+            MagicMock(spec=torch.nn.Module),
        )
        assert isinstance(engine, NCCLWeightTransferEngine)

    def test_create_engine_ipc(self):
        config = WeightTransferConfig(backend="ipc")
        engine = WeightTransferEngineFactory.create_engine(
-            config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+            config,
+            create_mock_vllm_config(),
+            torch.device("cuda"),
+            MagicMock(spec=torch.nn.Module),
        )
        assert isinstance(engine, IPCWeightTransferEngine)

    def test_create_engine_sparse_nccl(self):
        config = WeightTransferConfig(backend="sparse_nccl")
        engine = WeightTransferEngineFactory.create_engine(
-            config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+            config,
+            create_mock_vllm_config(),
+            torch.device("cuda"),
+            MagicMock(spec=torch.nn.Module),
        )
        assert isinstance(engine, SparseNCCLWeightTransferEngine)

@@ -264,7 +273,7 @@ class TestEngineRegistry:
            WeightTransferEngineFactory.create_engine(
                config,
                create_mock_vllm_config(),
-                "cuda",
+                torch.device("cuda"),
                MagicMock(spec=torch.nn.Module),
            )

@@ -284,7 +293,7 @@ class TestSparseNCCLPatchApplication:
    def _make_engine(self, model):
        config = WeightTransferConfig(backend="sparse_nccl")
        return SparseNCCLWeightTransferEngine(
-            config, create_mock_vllm_config(), "cpu", model
+            config, create_mock_vllm_config(), torch.device("cpu"), model
        )

    def _make_model(self, numel: int = 8):
@@ -382,7 +391,10 @@ def test_nccl_receive_weights_without_init_raises():

    config = WeightTransferConfig(backend="nccl")
    engine = NCCLWeightTransferEngine(
-        config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+        config,
+        create_mock_vllm_config(),
+        torch.device("cuda"),
+        MagicMock(spec=torch.nn.Module),
    )

    update_info = NCCLWeightTransferUpdateInfo(
@@ -400,7 +412,10 @@ def test_sparse_nccl_receive_weights_without_init_raises():

    config = WeightTransferConfig(backend="sparse_nccl")
    engine = SparseNCCLWeightTransferEngine(
-        config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+        config,
+        create_mock_vllm_config(),
+        torch.device("cuda"),
+        MagicMock(spec=torch.nn.Module),
    )

    update_info = SparseNCCLWeightTransferUpdateInfo(
@@ -495,7 +510,9 @@ def inference_receive_tensor(
    vllm_config.model_config = MagicMock()

    recorder = Recorder()
-    engine = NCCLWeightTransferEngine(config, vllm_config, "cuda", recorder)
+    engine = NCCLWeightTransferEngine(
+        config, vllm_config, torch.device("cuda"), recorder
+    )
    # Transport-only test: bypass the set_current_vllm_config context that
    # receive_weights enters, since vllm_config here is a mock.
    import vllm.config as _vllm_config_mod
@@ -664,7 +681,9 @@ def inference_receive_sparse_tensor(
        num_updates_list=[3],
    )

-    engine = SparseNCCLWeightTransferEngine(config, vllm_config, "cuda", model)
+    engine = SparseNCCLWeightTransferEngine(
+        config, vllm_config, torch.device("cuda"), model
+    )
    from vllm.distributed.weight_transfer.nccl_common import (
        NCCLWeightTransferInitInfo,
    )
@@ -879,7 +898,7 @@ class TestIPCEngineParsing:
        return IPCWeightTransferEngine(
            config,
            create_mock_vllm_config(),
-            "cuda",
+            torch.device("cuda"),
            MagicMock(spec=torch.nn.Module),
        )

@@ -1068,7 +1087,9 @@ def inference_receive_ipc_tensor(
    vllm_config.model_config = MagicMock()

    recorder = Recorder()
-    engine = IPCWeightTransferEngine(config, vllm_config, "cuda", recorder)
+    engine = IPCWeightTransferEngine(
+        config, vllm_config, _get_ray_assigned_device(), recorder
+    )
    # Transport-only test: bypass the set_current_vllm_config context that
    # receive_weights enters, since vllm_config here is a mock.
    import vllm.config as _vllm_config_mod
@@ -1173,7 +1194,10 @@ def test_ipc_receive_weights_missing_gpu_uuid_raises():

    config = WeightTransferConfig(backend="ipc")
    engine = IPCWeightTransferEngine(
-        config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module)
+        config,
+        create_mock_vllm_config(),
+        torch.device("cuda:0"),
+        MagicMock(spec=torch.nn.Module),
    )

    dummy_tensor = torch.ones(10, 10, device="cuda:0")
@@ -2,6 +2,7 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
 import asyncio
 import json
+from collections.abc import AsyncIterator
 from contextlib import suppress
 from dataclasses import dataclass, field
 from typing import Any
@@ -19,6 +20,7 @@ from tests.entrypoints.openai.utils import (
 from tests.utils import RemoteOpenAIServer
 from vllm._aiter_ops import is_aiter_found_and_supported
 from vllm.config import MultiModalConfig
+from vllm.entrypoints.generate.base.serving import build_per_request_timing_metrics
 from vllm.entrypoints.openai.chat_completion.protocol import (
    ChatCompletionRequest,
    ChatCompletionResponse,
@@ -50,9 +52,17 @@ from vllm.tokenizers import get_tokenizer
 from vllm.tokenizers.mistral import MistralTokenizer
 from vllm.tokenizers.registry import cached_tokenizer_from_config
 from vllm.v1.engine.async_llm import AsyncLLM
+from vllm.v1.metrics.stats import RequestStateStats

 GPT_OSS_MODEL_NAME = "openai/gpt-oss-20b"
 GPT_OSS_SPECULATOR_NAME = "RedHatAI/gpt-oss-20b-speculator.eagle3"
+_PER_REQUEST_STATS = RequestStateStats(
+    queued_ts=1.0,
+    scheduled_ts=1.5,
+    first_token_ts=2.0,
+    last_token_ts=3.0,
+    num_generation_tokens=2,
+)


@pytest.fixture(scope="module")
@@ -608,6 +618,169 @@ def _build_serving_chat(
    return serving_chat


+def _build_minimal_metrics_serving_chat(
+    enable_per_request_metrics: bool,
+    enable_force_include_usage: bool = False,
+) -> OpenAIServingChat:
+    serving = OpenAIServingChat.__new__(OpenAIServingChat)
+    serving.response_role = "assistant"
+    serving.parser_cls = None
+    serving.enable_auto_tools = False
+    serving.enable_prompt_tokens_details = False
+    serving.enable_log_outputs = False
+    serving.enable_log_deltas = False
+    serving.enable_force_include_usage = enable_force_include_usage
+    serving.request_logger = None
+    serving.system_fingerprint = None
+    serving.enable_per_request_metrics = enable_per_request_metrics
+    return serving
+
+
+def _make_metrics_request_output(
+    metrics: RequestStateStats | None = _PER_REQUEST_STATS,
+    token_ids: tuple[int, ...] = (100, 101),
+) -> RequestOutput:
+    return RequestOutput(
+        request_id="test-id",
+        prompt="Test prompt",
+        prompt_token_ids=[1, 2, 3],
+        prompt_logprobs=None,
+        outputs=[
+            CompletionOutput(
+                index=0,
+                text="Hello",
+                token_ids=list(token_ids),
+                cumulative_logprob=None,
+                logprobs=None,
+                finish_reason="stop",
+            )
+        ],
+        finished=True,
+        metrics=metrics,
+    )
+
+
+async def _single_request_output(
+    request_output: RequestOutput,
+) -> AsyncIterator[RequestOutput]:
+    yield request_output
+
+
+async def _collect_metrics_stream_chunks(
+    serving: OpenAIServingChat,
+    request: ChatCompletionRequest,
+) -> list[dict[str, Any]]:
+    chunks: list[dict[str, Any]] = []
+    async for line in serving.chat_completion_stream_generator(
+        request,
+        _single_request_output(_make_metrics_request_output()),
+        "chatcmpl-test-id",
+        "test-model",
+        conversation=[{"role": "user", "content": "Test"}],
+        tokenizer=MagicMock(),
+        request_metadata=RequestResponseMetadata(request_id="chatcmpl-test-id"),
+    ):
+        line = line.strip()
+        if not line.startswith("data: "):
+            continue
+        payload = line[len("data: ") :]
+        if payload != "[DONE]":
+            chunks.append(json.loads(payload))
+    return chunks
+
+
+def test_build_per_request_timing_metrics_valid_timestamps():
+    metrics = build_per_request_timing_metrics(
+        _PER_REQUEST_STATS, num_generation_tokens=10
+    )
+
+    assert metrics.time_to_first_token_ms == pytest.approx(500.0)
+    assert metrics.generation_time_ms == pytest.approx(1000.0)
+    assert metrics.queue_time_ms == pytest.approx(500.0)
+    assert metrics.mean_itl_ms == pytest.approx(1000.0 / 9, rel=1e-4)
+    assert metrics.tokens_per_second == pytest.approx(10.0 / 1.5, rel=1e-4)
+
+
+@pytest.mark.asyncio
+async def test_chat_per_request_metrics_follow_server_flag():
+    request = ChatCompletionRequest(
+        model="test-model",
+        messages=[{"role": "user", "content": "Test prompt"}],
+        max_tokens=10,
+        stream=False,
+    )
+    request_output = _make_metrics_request_output()
+
+    disabled_serving = _build_minimal_metrics_serving_chat(
+        enable_per_request_metrics=False
+    )
+    disabled_response = await disabled_serving.chat_completion_full_generator(
+        request,
+        _single_request_output(request_output),
+        "chatcmpl-test-id",
+        "test-model",
+        conversation=[{"role": "user", "content": "Test"}],
+        tokenizer=MagicMock(),
+        request_metadata=RequestResponseMetadata(request_id="chatcmpl-test-id"),
+    )
+    assert disabled_response.metrics is None
+
+    enabled_serving = _build_minimal_metrics_serving_chat(
+        enable_per_request_metrics=True
+    )
+    enabled_response = await enabled_serving.chat_completion_full_generator(
+        request,
+        _single_request_output(request_output),
+        "chatcmpl-test-id",
+        "test-model",
+        conversation=[{"role": "user", "content": "Test"}],
+        tokenizer=MagicMock(),
+        request_metadata=RequestResponseMetadata(request_id="chatcmpl-test-id"),
+    )
+    assert enabled_response.metrics is not None
+    assert enabled_response.metrics.time_to_first_token_ms == pytest.approx(500.0)
+
+
+@pytest.mark.asyncio
+async def test_chat_per_request_metrics_suppressed_for_n_greater_than_one():
+    serving = _build_minimal_metrics_serving_chat(enable_per_request_metrics=True)
+    response = await serving.chat_completion_full_generator(
+        ChatCompletionRequest(
+            model="test-model",
+            messages=[{"role": "user", "content": "Test prompt"}],
+            max_tokens=10,
+            stream=False,
+            n=2,
+        ),
+        _single_request_output(_make_metrics_request_output()),
+        "chatcmpl-test-id",
+        "test-model",
+        conversation=[{"role": "user", "content": "Test"}],
+        tokenizer=MagicMock(),
+        request_metadata=RequestResponseMetadata(request_id="chatcmpl-test-id"),
+    )
+    assert response.metrics is None
+
+
+@pytest.mark.asyncio
+async def test_chat_streaming_metrics_ride_on_usage_chunk():
+    serving = _build_minimal_metrics_serving_chat(enable_per_request_metrics=True)
+    chunks = await _collect_metrics_stream_chunks(
+        serving,
+        ChatCompletionRequest(
+            model="test-model",
+            messages=[{"role": "user", "content": "Test prompt"}],
+            max_tokens=10,
+            stream=True,
+            stream_options={"include_usage": True},
+        ),
+    )
+
+    usage_chunks = [chunk for chunk in chunks if chunk.get("usage")]
+    assert usage_chunks
+    assert usage_chunks[-1]["metrics"]["time_to_first_token_ms"] == pytest.approx(500.0)
+
+
@dataclass
 class MockEngine:
    model_config: MockModelConfig = field(default_factory=MockModelConfig)
@@ -183,27 +183,39 @@ async def test_thinking_token_budget_mixed_requests(client: openai.AsyncOpenAI):
 async def test_thinking_token_budget_limits_reasoning(client: openai.AsyncOpenAI):
    """Test that thinking_token_budget limits the number of reasoning tokens.

-    Counts non-empty streaming ``delta.reasoning`` chunks (coarse proxy; each
-    chunk may represent multiple decode tokens — see
-    ``_count_reasoning_decode_token_ids_between_markers`` and the Qwen3.5 MTP
-    test for id-based checks).
+    Counts reasoning decode tokens by id, which is robust to how tokens are
+    grouped into streamed chunks (a single chunk can carry several tokens under
+    async scheduling / stream_interval > 1). Counting chunks under-counts.
    """

-    reasoning_token_count = 0
+    tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME)
+    start_ids = list(tokenizer.encode(REASONING_START_STR, add_special_tokens=False))
+    end_ids = list(tokenizer.encode(REASONING_END_STR, add_special_tokens=False))
+
+    prompt_token_ids: list[int] = []
+    decode_token_ids: list[int] = []
    stream = await client.chat.completions.create(
        model=MODEL_NAME,
        messages=MESSAGES,
        max_tokens=100,
        stream=True,
-        extra_body={"thinking_token_budget": THINK_BUDGET},
+        extra_body={"thinking_token_budget": THINK_BUDGET, "return_token_ids": True},
    )
    async for chunk in stream:
-        delta = chunk.choices[0].delta
-        if getattr(delta, "reasoning", None):
-            reasoning_token_count += 1
+        if not chunk.choices:
+            continue
+        if getattr(chunk, "prompt_token_ids", None):
+            prompt_token_ids = list(chunk.prompt_token_ids)
+        delta_ids = getattr(chunk.choices[0], "token_ids", None)
+        if delta_ids:
+            decode_token_ids.extend(delta_ids)

+    reasoning_token_count = _count_reasoning_decode_token_ids_between_markers(
+        prompt_token_ids + decode_token_ids, start_ids, end_ids
+    )
+    assert reasoning_token_count is not None, "missing reasoning start marker in ids"
    assert reasoning_token_count == THINK_BUDGET, (
-        f"reasoning tokens ({reasoning_token_count}) exceeded "
+        f"reasoning tokens ({reasoning_token_count}) != "
        f"thinking_token_budget ({THINK_BUDGET})"
    )

@@ -11,7 +11,10 @@ from pydantic import ValidationError
 from vllm.config.multimodal import MultiModalConfig
 from vllm.entrypoints.openai.completion.protocol import CompletionRequest
 from vllm.entrypoints.openai.completion.serving import OpenAIServingCompletion
-from vllm.entrypoints.openai.engine.protocol import GenerationError
+from vllm.entrypoints.openai.engine.protocol import (
+    GenerationError,
+    RequestResponseMetadata,
+)
 from vllm.entrypoints.openai.models.protocol import BaseModelPath
 from vllm.entrypoints.openai.models.serving import OpenAIServingModels
 from vllm.entrypoints.scale_out.render.serving import ServingRender
@@ -20,9 +23,17 @@ from vllm.renderers.hf import HfRenderer
 from vllm.renderers.online_renderer import OnlineRenderer
 from vllm.tokenizers.registry import cached_tokenizer_from_config
 from vllm.v1.engine.async_llm import AsyncLLM
+from vllm.v1.metrics.stats import RequestStateStats

 MODEL_NAME = "openai-community/gpt2"
 MODEL_NAME_SHORT = "gpt2"
+_PER_REQUEST_STATS = RequestStateStats(
+    queued_ts=1.0,
+    scheduled_ts=1.5,
+    first_token_ts=2.0,
+    last_token_ts=3.0,
+    num_generation_tokens=2,
+)
 BASE_MODEL_PATHS = [
    BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME),
    BaseModelPath(name=MODEL_NAME_SHORT, model_path=MODEL_NAME_SHORT),
@@ -93,6 +104,39 @@ def _build_serving_completion(engine: AsyncLLM) -> OpenAIServingCompletion:
    )


+def _build_minimal_metrics_serving_completion(
+    enable_per_request_metrics: bool,
+) -> OpenAIServingCompletion:
+    serving = OpenAIServingCompletion.__new__(OpenAIServingCompletion)
+    serving.enable_prompt_tokens_details = False
+    serving.system_fingerprint = None
+    serving.enable_per_request_metrics = enable_per_request_metrics
+    return serving
+
+
+def _make_metrics_request_output(
+    metrics: RequestStateStats | None = _PER_REQUEST_STATS,
+) -> RequestOutput:
+    return RequestOutput(
+        request_id="test-id",
+        prompt="Test prompt",
+        prompt_token_ids=[1, 2, 3],
+        prompt_logprobs=None,
+        outputs=[
+            CompletionOutput(
+                index=0,
+                text="Hello",
+                token_ids=[100, 101],
+                cumulative_logprob=None,
+                logprobs=None,
+                finish_reason="stop",
+            )
+        ],
+        finished=True,
+        metrics=metrics,
+    )
+
+
 def _build_renderer(model_config: MockModelConfig):
    return HfRenderer(
        MockVllmConfig(model_config, parallel_config=MockParallelConfig()),
@@ -100,6 +144,58 @@ def _build_renderer(model_config: MockModelConfig):
    )


+def test_completion_per_request_metrics_follow_server_flag():
+    request = CompletionRequest(model=MODEL_NAME, prompt="Test prompt", max_tokens=10)
+    request_output = _make_metrics_request_output()
+
+    disabled_serving = _build_minimal_metrics_serving_completion(
+        enable_per_request_metrics=False
+    )
+    disabled_response = disabled_serving.request_output_to_completion_response(
+        [request_output],
+        request,
+        "cmpl-test-id",
+        0,
+        MODEL_NAME,
+        None,
+        RequestResponseMetadata(request_id="cmpl-test-id"),
+    )
+    assert disabled_response.metrics is None
+
+    enabled_serving = _build_minimal_metrics_serving_completion(
+        enable_per_request_metrics=True
+    )
+    enabled_response = enabled_serving.request_output_to_completion_response(
+        [request_output],
+        request,
+        "cmpl-test-id",
+        0,
+        MODEL_NAME,
+        None,
+        RequestResponseMetadata(request_id="cmpl-test-id"),
+    )
+    assert enabled_response.metrics is not None
+    assert enabled_response.metrics.time_to_first_token_ms == pytest.approx(500.0)
+
+
+def test_completion_per_request_metrics_suppressed_for_multiple_prompts():
+    serving = _build_minimal_metrics_serving_completion(enable_per_request_metrics=True)
+    response = serving.request_output_to_completion_response(
+        [_make_metrics_request_output(), _make_metrics_request_output()],
+        CompletionRequest(
+            model=MODEL_NAME,
+            prompt=["Test prompt", "Another prompt"],
+            max_tokens=10,
+        ),
+        "cmpl-test-id",
+        0,
+        MODEL_NAME,
+        None,
+        RequestResponseMetadata(request_id="cmpl-test-id"),
+    )
+    assert response.metrics is None
+
+
@pytest.mark.asyncio
 async def test_completion_error_non_stream():
    """test finish_reason='error' returns 500 InternalServerError (non-streaming)"""
@@ -206,6 +206,14 @@ def test_chat_template_validation_for_sad_paths(serve_parser):
        validate_parsed_serve_args(args)


+def test_per_request_metrics_requires_log_stats(serve_parser):
+    args = serve_parser.parse_args(
+        args=["--enable-per-request-metrics", "--disable-log-stats"]
+    )
+    with pytest.raises(ValueError):
+        validate_parsed_serve_args(args)
+
+
@pytest.mark.parametrize(
    "cli_args, expected_middleware",
    [
@@ -0,0 +1,10 @@
+model_name: "nm-testing/Qwen2-1.5B-Instruct-FP8W8"
+accuracy_threshold: 0.55
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,8 @@
+model_name: "nm-testing/Qwen2-1.5B-Instruct-FP8W8"
+accuracy_threshold: 0.55
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
@@ -0,0 +1,10 @@
+model_name: "mgoin/Qwen3-0.6B-MXFP8"
+accuracy_threshold: 0.39
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,8 @@
+model_name: "mgoin/Qwen3-0.6B-MXFP8"
+accuracy_threshold: 0.39
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
@@ -0,0 +1,13 @@
+model_name: "QuixiAI/Qwen3-30B-A3B-AWQ"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+  --dtype bfloat16
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,13 @@
+model_name: "QuixiAI/Qwen3-30B-A3B-AWQ"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+  --dtype bfloat16
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,11 @@
+model_name: "QuixiAI/Qwen3-30B-A3B-AWQ"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+  --dtype bfloat16
@@ -0,0 +1,11 @@
+model_name: "nm-testing/Qwen3-30B-A3B-FP8-block"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,9 @@
+model_name: "nm-testing/Qwen3-30B-A3B-FP8-block"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -0,0 +1,11 @@
+model_name: "nm-testing/Qwen3-30B-A3B-Fp8-v1"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,9 @@
+model_name: "nm-testing/Qwen3-30B-A3B-Fp8-v1"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -0,0 +1,11 @@
+model_name: "Qwen/Qwen3-30B-A3B-GPTQ-Int4"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,11 @@
+model_name: "Qwen/Qwen3-30B-A3B-GPTQ-Int4"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,9 @@
+model_name: "Qwen/Qwen3-30B-A3B-GPTQ-Int4"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -0,0 +1,11 @@
+model_name: "RedHatAI/Qwen3-30B-A3B-Instruct-2507-quantized.w8a8"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,9 @@
+model_name: "RedHatAI/Qwen3-30B-A3B-Instruct-2507-quantized.w8a8"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -5,8 +5,7 @@ num_fewshot: 5
 server_args: >-
  --enforce-eager
  --max-model-len 8192
-  --tensor-parallel-size 1
  --quantization humming
-  --kernel-config.enable_flashinfer_autotune=False
+  --linear-backend humming
 env:
  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -5,6 +5,5 @@ num_fewshot: 5
 server_args: >-
  --enforce-eager
  --max-model-len 8192
-  --tensor-parallel-size 1
  --quantization humming
-  --kernel-config.enable_flashinfer_autotune=False
+  --linear-backend humming
@@ -0,0 +1,9 @@
+model_name: "nvidia/Qwen3-30B-A3B-NVFP4"
+accuracy_threshold: 0.86
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -0,0 +1,13 @@
+model_name: "Qwen/Qwen3-30B-A3B"
+accuracy_threshold: 0.80
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --tensor-parallel-size 1
+  --quantization humming
+  --kernel-config.enable_flashinfer_autotune=False
+env:
+  VLLM_HUMMING_ONLINE_QUANT_CONFIG: '{"dtype":"int5","hadamard_block_size":-1}'
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,13 @@
+model_name: "Qwen/Qwen3-30B-A3B"
+accuracy_threshold: 0.80
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --tensor-parallel-size 1
+  --quantization humming
+  --kernel-config.enable_flashinfer_autotune=False
+env:
+  VLLM_HUMMING_ONLINE_QUANT_CONFIG: '{"dtype":"int5","hadamard_block_size":-1}'
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,12 @@
+model_name: "Qwen/Qwen3-30B-A3B"
+accuracy_threshold: 0.80
+num_questions: 1319
+num_fewshot: 5
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --tensor-parallel-size 1
+  --quantization humming
+  --kernel-config.enable_flashinfer_autotune=False
+env:
+  VLLM_HUMMING_ONLINE_QUANT_CONFIG: '{"dtype":"int5","hadamard_block_size":-1}'
@@ -0,0 +1,12 @@
+model_name: "Qwen/Qwen3.5-35B-A3B-FP8"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,10 @@
+model_name: "Qwen/Qwen3.5-35B-A3B-FP8"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -0,0 +1,12 @@
+model_name: "Qwen/Qwen3.5-35B-A3B"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --quantization experts_int8
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,10 @@
+model_name: "Qwen/Qwen3.5-35B-A3B"
+accuracy_threshold: 0.90
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --quantization experts_int8
@@ -0,0 +1,10 @@
+model_name: "RedHatAI/Qwen3.5-4B-quantized.w4a16"
+accuracy_threshold: 0.82
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -0,0 +1,11 @@
+model_name: "RedHatAI/Qwen3.5-4B-quantized.w4a16"
+accuracy_threshold: 0.82
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
+env:
+  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"int8"}'
@@ -0,0 +1,9 @@
+model_name: "RedHatAI/Qwen3.5-4B-quantized.w4a16"
+accuracy_threshold: 0.82
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --linear-backend humming
@@ -0,0 +1,10 @@
+model_name: "RedHatAI/Qwen3.6-35B-A3B-NVFP4"
+accuracy_threshold: 0.91
+num_questions: 1319
+num_fewshot: 5
+gen_prefix: " <think>\n\n</think>\n"
+server_args: >-
+  --enforce-eager
+  --max-model-len 8192
+  --moe-backend humming
+  --linear-backend humming
@@ -1,2 +1,9 @@
 gpt-oss-20b-humming-act-fp8.yaml
 Qwen3-30B-A3B-MXFP4A16-humming-act-fp8.yaml
+Qwen2-1.5B-Instruct-FP8W8-humming-act-fp8.yaml
+Qwen3-0.6B-MXFP8-humming-act-fp8.yaml
+Qwen3-30B-A3B-Fp8-v1-humming-act-fp8.yaml
+Qwen3-30B-A3B-FP8-block-humming-act-fp8.yaml
+Qwen3-30B-A3B-GPTQ-Int4-humming-act-fp8.yaml
+Qwen3-30B-A3B-AWQ-humming.yaml
+Qwen3.5-35B-A3B-FP8-humming-act-fp8.yaml
@@ -0,0 +1,4 @@
+Qwen3-30B-A3B-Instruct-2507-quantized.w8a8-humming-act-int8.yaml
+Qwen3-30B-A3B-GPTQ-Int4-humming-act-int8.yaml
+Qwen3-30B-A3B-AWQ-humming.yaml
+Qwen3.5-35B-A3B-experts-int8-humming-act-int8.yaml
@@ -0,0 +1,3 @@
+Qwen3-30B-A3B-int5wc-hadamard-humming.yaml
+Qwen3-30B-A3B-int5wc-hadamard-humming-act-fp8.yaml
+Qwen3-30B-A3B-int5wc-hadamard-humming-act-int8.yaml
@@ -1,2 +1,13 @@
 gpt-oss-20b-humming.yaml
 Qwen3-30B-A3B-MXFP4A16-humming.yaml
+Qwen2-1.5B-Instruct-FP8W8-humming.yaml
+Qwen3-0.6B-MXFP8-humming.yaml
+Qwen3.6-35B-A3B-NVFP4-humming.yaml
+Qwen3-30B-A3B-Fp8-v1-humming.yaml
+Qwen3-30B-A3B-FP8-block-humming.yaml
+Qwen3-30B-A3B-GPTQ-Int4-humming.yaml
+Qwen3-30B-A3B-Instruct-2507-quantized.w8a8-humming.yaml
+Qwen3-30B-A3B-NVFP4-humming.yaml
+Qwen3-30B-A3B-AWQ-humming.yaml
+Qwen3.5-35B-A3B-FP8-humming.yaml
+Qwen3.5-35B-A3B-experts-int8-humming.yaml
@@ -5,7 +5,6 @@ num_fewshot: 5
 server_args: >-
  --enforce-eager
  --max-model-len 8192
-  --tensor-parallel-size 1
  --moe-backend humming
 env:
  VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}'
@@ -5,5 +5,4 @@ num_fewshot: 5
 server_args: >-
  --enforce-eager
  --max-model-len 8192
-  --tensor-parallel-size 1
  --moe-backend humming
@@ -146,6 +146,7 @@ async def call_vllm_chat_api(
 def _build_gsm8k_prompts(
    num_questions: int = 1319,
    num_shots: int = 5,
+    gen_prefix: str = "",
 ) -> tuple[list[str], list[int]]:
    """Build few-shot GSM8K completion prompts and ground-truth labels."""
    if num_questions == 0:
@@ -157,14 +158,15 @@ def _build_gsm8k_prompts(
    for i in range(num_shots):
        few_shot_examples += (
            f"Question: {train_data[i]['question']}\n"
-            f"Answer: {train_data[i]['answer']}\n\n"
+            f"Answer:{gen_prefix} {train_data[i]['answer']}\n\n"
        )

    prompts = []
    labels = []
    for i in range(num_questions):
        prompts.append(
-            few_shot_examples + f"Question: {test_data[i]['question']}\nAnswer:"
+            few_shot_examples
+            + f"Question: {test_data[i]['question']}\nAnswer:{gen_prefix}"
        )
        labels.append(get_answer_value(test_data[i]["answer"]))

@@ -213,6 +215,7 @@ def evaluate_gsm8k(
    temperature: float = 0.0,
    seed: int | None = 42,
    request_timeout_seconds: float = 600,
+    gen_prefix: str = "",
 ) -> dict[str, float | int]:
    """
    Evaluate GSM8K accuracy using vLLM serve endpoint.
@@ -220,7 +223,7 @@ def evaluate_gsm8k(
    Returns dict with accuracy, invalid_rate, latency, etc.
    """
    base_url = f"{host}:{port}"
-    prompts, labels = _build_gsm8k_prompts(num_questions, num_shots)
+    prompts, labels = _build_gsm8k_prompts(num_questions, num_shots, gen_prefix)
    num_questions = len(prompts)

    async def run_async_evaluation():
@@ -278,6 +281,7 @@ def evaluate_gsm8k_offline(
    num_shots: int = 5,
    max_tokens: int = 256,
    temperature: float = 0.0,
+    gen_prefix: str = "",
 ) -> dict[str, float | int]:
    """Evaluate GSM8K accuracy using an offline vllm.LLM object.

@@ -286,7 +290,7 @@ def evaluate_gsm8k_offline(
    """
    from vllm import SamplingParams

-    prompts, labels = _build_gsm8k_prompts(num_questions, num_shots)
+    prompts, labels = _build_gsm8k_prompts(num_questions, num_shots, gen_prefix)

    sampling_params = SamplingParams(
        temperature=temperature,
@@ -70,6 +70,7 @@ def run_gsm8k_eval(eval_config: dict, server_url: str) -> dict:
        host=host,
        port=port,
        request_timeout_seconds=request_timeout_seconds,
+        gen_prefix=eval_config.get("gen_prefix", ""),
    )

    return results
@@ -150,7 +150,7 @@ def _reference_index_topk(
        num_blocks = (seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE
        pages = block_table[req_id, :num_blocks]
        k = index_kv_cache[pages].reshape(num_blocks * BLOCK_SIZE, -1)
-        score = torch.einsum("qhd,kd->hqk", q.float(), k.float()) * sm_scale
+        score = sm_scale * torch.einsum("qhd,kd->hqk", q.float(), k.float())

        q_pos = prefix_len + torch.arange(q_len, device=idx_q.device)
        k_pos = torch.arange(k.shape[0], device=idx_q.device)
@@ -621,11 +621,12 @@ def test_decode_index_topk_fp8(num_idx_heads: int):
        init_blocks=init_blocks,
        local_blocks=local_blocks,
        num_kv_heads=num_idx_heads,
-        sm_scale=head_dim**-0.5,
        decode_query_len=decode_query_len,
+        max_decode_query_len=decode_query_len,
    )
    # Reference from the DEQUANTIZED fp8 values (the kernel computes the fp8 QK
-    # in fp32, so it must match an fp32 matmul of the same e4m3 values).
+    # in fp32 with no scaling, so it must match an unscaled fp32 matmul of the
+    # same e4m3 values).
    expected = _reference_index_topk(
        idx_q.float(),
        index_kv_cache.float(),
@@ -636,7 +637,6 @@ def test_decode_index_topk_fp8(num_idx_heads: int):
        topk,
        init_blocks,
        local_blocks,
-        head_dim**-0.5,
    )
    _assert_topk_indices_equal_unordered(actual, expected)

@@ -0,0 +1,189 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import pytest
+import torch
+
+from vllm.config import CompilationConfig, VllmConfig
+from vllm.forward_context import set_forward_context
+from vllm.model_executor.layers.mamba.short_conv import ShortConv
+from vllm.model_executor.layers.utils import dispatch_cpu_unquantized_gemm
+from vllm.platforms import current_platform
+from vllm.v1.attention.backends.short_conv_attn import ShortConvAttentionMetadata
+
+if not current_platform.is_cpu():
+    pytest.skip("skipping CPU-only tests", allow_module_level=True)
+
+
+@pytest.fixture(autouse=True)
+def mock_dist():
+    with (
+        patch(
+            "vllm.model_executor.layers.linear.get_tensor_model_parallel_rank",
+            return_value=0,
+        ),
+        patch(
+            "vllm.model_executor.layers.linear.get_tensor_model_parallel_world_size",
+            return_value=1,
+        ),
+        patch(
+            "vllm.distributed.parallel_state.model_parallel_is_initialized",
+            return_value=True,
+        ),
+        patch(
+            "vllm.distributed.parallel_state.get_tp_group",
+            return_value=MagicMock(rank_in_group=0),
+        ),
+    ):
+        yield
+
+
+@pytest.fixture
+def vllm_config():
+    # ShortConv only needs compilation_config from the current vLLM config, so a
+    # minimal config (model_config=None) avoids mocking ModelConfig and the
+    # associated VllmConfig validation churn.
+    return VllmConfig(compilation_config=CompilationConfig())
+
+
+def test_short_conv_forward_native_prefill(vllm_config):
+    prefix = "test_layer"
+    config = SimpleNamespace(conv_L_cache=4, conv_bias=True)
+    dim = 16
+
+    from vllm.config import set_current_vllm_config
+
+    with set_current_vllm_config(vllm_config):
+        layer = ShortConv(config=config, dim=dim, layer_idx=0, prefix=prefix)
+
+    layer.to("cpu")
+    # vLLM Linear layers allocate weights with torch.empty (uninitialized).
+    # On ARM these come back as zero-filled pages, so in_proj output is zero and
+    # the prefill state stays zero. Seed + init to make the test platform-safe.
+    torch.manual_seed(0)
+    for p in layer.parameters():
+        torch.nn.init.normal_(p)
+    dispatch_cpu_unquantized_gemm(layer.in_proj, remove_weight=False)
+    dispatch_cpu_unquantized_gemm(layer.out_proj, remove_weight=False)
+
+    # Mock AttentionMetadata
+    num_prefills = 1
+    num_prefill_tokens = 5
+    query_start_loc_p = torch.tensor([0, 5], dtype=torch.int32)
+    state_indices_tensor_p = torch.tensor([0], dtype=torch.int32)
+
+    # ShortConvAttentionMetadata
+    attn_metadata = ShortConvAttentionMetadata(
+        num_prefills=num_prefills,
+        num_prefill_tokens=num_prefill_tokens,
+        num_decodes=0,
+        num_decode_tokens=0,
+        num_reqs=1,
+        query_start_loc_p=query_start_loc_p,
+        has_initial_states_p=torch.tensor([False]),
+        state_indices_tensor_p=state_indices_tensor_p,
+        state_indices_tensor_d=torch.empty((0, 1), dtype=torch.int32),
+        num_accepted_tokens=None,
+        query_start_loc_d=None,
+        block_idx_last_scheduled_token=None,
+        block_idx_first_scheduled_token_p=None,
+        block_idx_last_computed_token=None,
+        block_idx_last_scheduled_token_prev_step=None,
+        num_computed_tokens_p=None,
+        seq_lens=torch.tensor([5]),
+    )
+
+    # Mock KV cache
+    # conv_state shape (num_blocks, L_cache - 1, dim)
+    conv_state = torch.zeros((1, config.conv_L_cache - 1, dim))
+    layer.kv_cache = (conv_state,)
+
+    hidden_states = torch.randn((num_prefill_tokens, dim))
+    output = torch.zeros_like(hidden_states)
+
+    attn_metadata_dict = {prefix: attn_metadata}
+    with set_forward_context(attn_metadata=attn_metadata_dict, vllm_config=vllm_config):
+        layer.forward_native(hidden_states, output)
+
+    # Check if KV cache was updated
+    assert not torch.allclose(conv_state, torch.zeros_like(conv_state))
+
+
+def test_short_conv_forward_native_decode(vllm_config):
+    prefix = "test_layer_decode"
+    config = SimpleNamespace(conv_L_cache=4, conv_bias=True)
+    dim = 16
+
+    from vllm.config import set_current_vllm_config
+
+    with set_current_vllm_config(vllm_config):
+        layer = ShortConv(config=config, dim=dim, layer_idx=0, prefix=prefix)
+
+    layer.to("cpu")
+    torch.manual_seed(0)
+    for p in layer.parameters():
+        torch.nn.init.normal_(p)
+    dispatch_cpu_unquantized_gemm(layer.in_proj, remove_weight=False)
+    dispatch_cpu_unquantized_gemm(layer.out_proj, remove_weight=False)
+
+    # Mock AttentionMetadata for 2 decode requests
+    num_decodes = 2
+    state_indices_tensor_d = torch.tensor([0, 1], dtype=torch.int32)
+
+    attn_metadata = ShortConvAttentionMetadata(
+        num_prefills=0,
+        num_prefill_tokens=0,
+        num_decodes=num_decodes,
+        num_decode_tokens=num_decodes,
+        num_reqs=num_decodes,
+        query_start_loc_p=None,
+        has_initial_states_p=None,
+        state_indices_tensor_p=torch.empty((0,), dtype=torch.int32),
+        state_indices_tensor_d=state_indices_tensor_d,
+        num_accepted_tokens=None,
+        query_start_loc_d=torch.tensor([0, 1, 2], dtype=torch.int32),
+        block_idx_last_scheduled_token=None,
+        block_idx_first_scheduled_token_p=None,
+        block_idx_last_computed_token=None,
+        block_idx_last_scheduled_token_prev_step=None,
+        num_computed_tokens_p=None,
+        seq_lens=torch.tensor([1, 1]),
+    )
+
+    # Mock KV cache (2 blocks for 2 requests)
+    conv_state = torch.randn((2, config.conv_L_cache - 1, dim))
+    layer.kv_cache = (conv_state,)
+
+    hidden_states = torch.randn((num_decodes, dim))
+    output = torch.zeros_like(hidden_states)
+
+    old_conv_state = conv_state.clone()
+
+    attn_metadata_dict = {prefix: attn_metadata}
+    with set_forward_context(attn_metadata=attn_metadata_dict, vllm_config=vllm_config):
+        layer.forward_native(hidden_states, output)
+
+    # Check if KV cache was updated
+    assert not torch.allclose(conv_state, old_conv_state)
+
+
+def test_dispatch_cpu_unquantized_gemm_conv_layer():
+    # Convolution layers have >2D weights; dispatch should skip them gracefully.
+    # Shape/dtype are AMX-pack safe (bf16, width==4, dim % block_size == 0) so
+    # the AMX prepack branch does not raise on AMX-capable CPUs.
+    class MockConvLayer(torch.nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.weight = torch.nn.Parameter(
+                torch.randn(32, 1, 4, dtype=torch.bfloat16)
+            )
+            self.bias = torch.nn.Parameter(torch.randn(32, dtype=torch.bfloat16))
+
+    layer = MockConvLayer()
+    # The ndim != 2 guard returns early without raising.
+    dispatch_cpu_unquantized_gemm(layer, remove_weight=False)
+    # No cpu_linear set — conv layers are handled elsewhere.
+    assert not hasattr(layer, "cpu_linear")
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

+from types import SimpleNamespace
+
 import pytest
 import torch

@@ -8,8 +10,7 @@ from vllm.platforms import current_platform

 if not current_platform.is_device_capability_family(120):
    pytest.skip(
-        reason="FlashInfer CuteDSL SM12x MoE requires SM120 "
-        "(RTX Pro 6000 / DGX Spark).",
+        reason="FlashInfer B12x MoE requires SM120 (RTX Pro 6000 / DGX Spark).",
        allow_module_level=True,
    )

@@ -18,8 +19,8 @@ from vllm.utils.flashinfer import has_flashinfer_b12x_moe
 if not has_flashinfer_b12x_moe():
    pytest.skip(
        reason=(
-            "FlashInfer cute_dsl_fused_moe_nvfp4 / convert_sf_to_mma_layout "
-            "not available in installed FlashInfer (needs PRs #3051 and #3066)."
+            "FlashInfer B12xMoEWrapper not available in installed "
+            "FlashInfer (needs PR #3080)."
        ),
        allow_module_level=True,
    )
@@ -40,7 +41,6 @@ from vllm.model_executor.layers.fused_moe.config import nvfp4_moe_quant_config
 from vllm.model_executor.layers.fused_moe.experts.flashinfer_b12x_moe import (
    FlashInferB12xExperts,
 )
-from vllm.utils.flashinfer import flashinfer_convert_sf_to_mma_layout
 from vllm.utils.torch_utils import set_random_seed

 # Dimensions chosen to satisfy FP4 alignment requirements (k multiple of 256,
@@ -59,7 +59,7 @@ def _reorder_gate_up_to_up_gate(
 ) -> tuple[torch.Tensor, torch.Tensor]:
    """Swap gate and up-projection halves along dim=1 to [up, gate] order.

-    The SM12x kernel expects weights in [up (w3), gate (w1)] order while the
+    The B12x kernel expects weights in [up (w3), gate (w1)] order while the
    BF16 reference uses [gate (w1), up (w3)].  This replicates the reordering
    done at model-load time by ``prepare_nvfp4_moe_layer_for_fi_or_cutlass``.
    """
@@ -70,6 +70,22 @@ def _reorder_gate_up_to_up_gate(
    )


+def _process_b12x_weights(
+    experts: FlashInferB12xExperts,
+    w1_scale: torch.Tensor,
+    w2_scale: torch.Tensor,
+    w1_scale_2: torch.Tensor,
+    w2_scale_2: torch.Tensor,
+) -> None:
+    layer = SimpleNamespace(
+        w13_weight_scale=w1_scale,
+        w13_weight_scale_2=w1_scale_2,
+        w2_weight_scale=w2_scale,
+        w2_weight_scale_2=w2_scale_2,
+    )
+    experts.process_weights_after_loading(layer)
+
+
@pytest.mark.parametrize("m,n,k", MNK_FACTORS)
@pytest.mark.parametrize("e", [8, 16])
@pytest.mark.parametrize("topk", [1, 2, 4])
@@ -174,22 +190,12 @@ def test_flashinfer_b12x_moe(
            moe_config=moe_config,
            quant_config=quant_config,
        )
-        # In production, process_weights_after_loading computes these after
-        # normalizing block scales. In the test the scales are already in final
-        # form (global_scale=1.0), so we compute the MMA layouts directly.
-        num_experts_w1, m1, k1_sf = w1_blockscale.shape
-        experts.w1_sf_mma = flashinfer_convert_sf_to_mma_layout(
-            w1_blockscale.reshape(num_experts_w1 * m1, k1_sf),
-            m=m1,
-            k=k1_sf * 16,
-            num_groups=num_experts_w1,
-        )
-        num_experts_w2, m2, k2_sf = w2_blockscale.shape
-        experts.w2_sf_mma = flashinfer_convert_sf_to_mma_layout(
-            w2_blockscale.reshape(num_experts_w2 * m2, k2_sf),
-            m=m2,
-            k=k2_sf * 16,
-            num_groups=num_experts_w2,
+        _process_b12x_weights(
+            experts,
+            w1_blockscale,
+            w2_blockscale,
+            ones_e,
+            ones_e,
        )

        kernel = mk.FusedMoEKernel(
@@ -224,5 +230,134 @@ def test_flashinfer_b12x_moe(
        torch.testing.assert_close(sm12x_output, torch_output, atol=2e-1, rtol=2e-1)


+@pytest.mark.parametrize("m,n,k", MNK_FACTORS)
+@pytest.mark.parametrize("e", [8, 16])
+@pytest.mark.parametrize("topk", [1, 2, 4])
+@pytest.mark.parametrize("dtype", [torch.bfloat16])
+@torch.inference_mode()
+def test_flashinfer_b12x_moe_relu2(
+    m: int,
+    n: int,
+    k: int,
+    e: int,
+    topk: int,
+    dtype: torch.dtype,
+    workspace_init,
+):
+    """Test FlashInferB12xExperts with ReLU2 (non-gated) activation.
+
+    ReLU2 is used by Nemotron-H style models.  Unlike the gated SiLU
+    path, w1 has shape [E, N, K] (not [E, 2N, K]) and the activation
+    is relu(x)^2 without a gate/up split.
+    """
+    set_random_seed(7)
+    with set_current_vllm_config(
+        VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1))
+    ):
+        a = torch.randn((m, k), device="cuda", dtype=dtype) / 10
+
+        # Non-gated: w1 shape is (e, n, k), not (e, 2n, k).
+        w1_bf16 = torch.randn((e, n, k), device="cuda", dtype=dtype) / 15
+        w2_bf16 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 15
+
+        gs = torch.ones(1, device="cuda", dtype=torch.float32)
+        sf_vec_size = 16
+
+        # W1: no gate/up reordering for non-gated.
+        w1_flat = w1_bf16.reshape(e * n, k)
+        w1_q_flat, w1_sf_flat = fp4_quantize(
+            w1_flat,
+            global_scale=gs,
+            sf_vec_size=sf_vec_size,
+            is_sf_swizzled_layout=True,
+        )
+        w1_q = w1_q_flat.view(e, n, k // 2)
+        w1_blockscale = w1_sf_flat.view(e, n, w1_sf_flat.shape[1])
+
+        w2_flat = w2_bf16.reshape(e * k, n)
+        w2_q_flat, w2_sf_flat = fp4_quantize(
+            w2_flat,
+            global_scale=gs,
+            sf_vec_size=sf_vec_size,
+            is_sf_swizzled_layout=True,
+        )
+        w2_q = w2_q_flat.view(e, k, n // 2)
+        w2_blockscale = w2_sf_flat.view(e, k, w2_sf_flat.shape[1])
+
+        ones_e = torch.ones(e, device="cuda", dtype=torch.float32)
+
+        quant_config = nvfp4_moe_quant_config(
+            g1_alphas=ones_e,
+            g2_alphas=ones_e,
+            a1_gscale=ones_e,
+            a2_gscale=ones_e,
+            w1_scale=w1_blockscale,
+            w2_scale=w2_blockscale,
+        )
+
+        moe_config = make_dummy_moe_config(
+            num_experts=e,
+            experts_per_token=topk,
+            hidden_dim=k,
+            intermediate_size=n,
+            in_dtype=dtype,
+            activation=MoEActivation.RELU2_NO_MUL,
+        )
+
+        experts = FlashInferB12xExperts(
+            moe_config=moe_config,
+            quant_config=quant_config,
+        )
+        _process_b12x_weights(
+            experts,
+            w1_blockscale,
+            w2_blockscale,
+            ones_e,
+            ones_e,
+        )
+
+        kernel = mk.FusedMoEKernel(
+            maybe_make_prepare_finalize(
+                moe=moe_config,
+                quant_config=quant_config,
+                allow_new_interface=True,
+                use_monolithic=False,
+            ),
+            experts,
+            inplace=False,
+        )
+
+        score = torch.randn((m, e), device="cuda", dtype=dtype)
+        topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False)
+
+        b12x_output = kernel.apply(
+            hidden_states=a,
+            w1=w1_q,
+            w2=w2_q,
+            topk_weights=topk_weights,
+            topk_ids=topk_ids,
+            global_num_experts=e,
+            activation=MoEActivation.RELU2_NO_MUL,
+            apply_router_weight_on_input=False,
+            expert_map=None,
+        )
+
+        torch_output = torch_moe(
+            a,
+            w1_bf16,
+            w2_bf16,
+            score,
+            topk,
+            activation=MoEActivation.RELU2_NO_MUL,
+        )
+
+        torch.testing.assert_close(
+            b12x_output,
+            torch_output,
+            atol=2e-1,
+            rtol=2e-1,
+        )
+
+
 if __name__ == "__main__":
    test_flashinfer_b12x_moe(16, 128, 256, 8, 2, torch.bfloat16)
@@ -55,6 +55,7 @@ def make_dummy_moe_config(
    intermediate_size: int = 1,
    in_dtype: torch.dtype = torch.bfloat16,
    max_num_tokens: int = 512,
+    activation: MoEActivation = MoEActivation.SILU,
 ) -> FusedMoEConfig:
    """
    This is a dummy config for the mk constructor interface
@@ -73,7 +74,7 @@ def make_dummy_moe_config(
        else num_experts,
        num_logical_experts=num_experts,
        moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(),
-        activation=MoEActivation.SILU,
+        activation=activation,
        in_dtype=in_dtype,
        device="cuda",
        routing_method=RoutingMethodType.TopK,
@@ -1129,6 +1129,11 @@ _MULTIMODAL_EXAMPLE_MODELS = {
        },
        trust_remote_code=True,
    ),
+    "MossTranscribeDiarizeForConditionalGeneration": _HfExamplesInfo(
+        "OpenMOSS-Team/MOSS-Transcribe-Diarize",
+        trust_remote_code=True,
+        is_available_online=False,
+    ),
    "HfMoondream": _HfExamplesInfo(
        "moondream/moondream3-preview",
        tokenizer="moondream/starmie-v1",
@@ -1492,8 +1497,6 @@ _SPECULATIVE_DECODING_EXAMPLE_MODELS = {
    "EagleMistralLarge3ForCausalLM": _HfExamplesInfo(
        "mistralai/Mistral-Large-3-675B-Instruct-2512",
        speculative_model="mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle",
-        # TODO: revert once figuring out OOM in CI
-        is_available_online=False,
    ),
    "LlamaForCausalLMEagle3": _HfExamplesInfo(
        "Qwen/Qwen3-8B",
@@ -258,7 +258,8 @@ def _apply_qkv_fuser_with_stubs(module: nn.Module, fuser: QKVFuser):
        merged.weight.copy_(torch.cat([q.weight, k.weight, v.weight], dim=0))
        if q.bias is not None:
            merged.bias.copy_(torch.cat([q.bias, k.bias, v.bias], dim=0))
-    merged.split_sizes = [q.out_features, k.out_features, v.out_features]
+    merged.output_sizes = [q.out_features, k.out_features, v.out_features]
+    merged.tp_size = 1
    setattr(module, fuser.merged_name, merged)
    for name in (fuser.q_name, fuser.k_name, fuser.v_name):
        delattr(module, name)
@@ -332,7 +333,8 @@ def test_detects_and_rewrites_qkv(attn_cls, kv_heads):
    # original semantics (branches, kwargs, attribute reads)
    code = fuser.fused_forward.__code__
    names = code.co_names
-    assert "qkv_proj" in names and "split_sizes" in names and "o_proj" in names
+    assert "qkv_proj" in names and "output_sizes" in names and "o_proj" in names
+    assert "tp_size" in names
    assert not {"q_proj", "k_proj", "v_proj"} & set(names)
    if attn_cls is FakeAttention:
        assert "update" in names  # the cache branch survives
@@ -768,6 +768,118 @@ def test_pyav_backend_returns_target_frames_not_keyframes():
        )


+# ============================================================================
+# TorchCodec Backend Tests
+# ============================================================================
+
+
+def test_torchcodec_backend_loads_frames(
+    dummy_video_path, monkeypatch: pytest.MonkeyPatch
+):
+    """Test that the torchcodec codec backend can load frames."""
+    pytest.importorskip("torchcodec")
+    with monkeypatch.context() as m:
+        m.setenv("VLLM_VIDEO_LOADER_BACKEND", "opencv")
+
+        with open(dummy_video_path, "rb") as f:
+            video_data = f.read()
+
+        loader = VIDEO_LOADER_REGISTRY.load("opencv")
+        frames, metadata = loader.load_bytes(
+            video_data, num_frames=8, backend="torchcodec"
+        )
+
+        assert frames.ndim == 4
+        assert frames.shape[3] == 3  # RGB
+        assert frames.shape[0] == 8
+        assert frames.shape[0] == len(metadata["frames_indices"])
+        assert metadata["video_backend"] == "torchcodec"
+        assert "total_num_frames" in metadata
+        assert "fps" in metadata
+        assert "duration" in metadata
+
+
+def test_torchcodec_dynamic_backend_loads_frames(
+    dummy_video_path, monkeypatch: pytest.MonkeyPatch
+):
+    """Test that the torchcodec codec with dynamic sampling can load frames."""
+    pytest.importorskip("torchcodec")
+    with monkeypatch.context() as m:
+        m.setenv("VLLM_VIDEO_LOADER_BACKEND", "opencv_dynamic")
+
+        with open(dummy_video_path, "rb") as f:
+            video_data = f.read()
+
+        loader = VIDEO_LOADER_REGISTRY.load("opencv_dynamic")
+        frames, metadata = loader.load_bytes(
+            video_data, fps=2, max_duration=10, backend="torchcodec"
+        )
+
+        assert frames.ndim == 4
+        assert frames.shape[3] == 3  # RGB
+        assert frames.shape[0] > 0
+        assert frames.shape[0] == len(metadata["frames_indices"])
+        assert metadata["video_backend"] == "torchcodec_dynamic"
+
+
+def test_torchcodec_backend_rejects_frame_recovery(dummy_video_path):
+    """frame_recovery is OpenCV-only; torchcodec must reject it."""
+    pytest.importorskip("torchcodec")
+    with open(dummy_video_path, "rb") as f:
+        video_data = f.read()
+
+    loader = VIDEO_LOADER_REGISTRY.load("opencv")
+    with pytest.raises(AssertionError):
+        loader.load_bytes(
+            video_data, num_frames=8, backend="torchcodec", frame_recovery=True
+        )
+
+
+def test_torchcodec_backend_returns_target_frames_not_keyframes():
+    """Regression test: torchcodec must return the requested frames, not the
+    GOP keyframe they seek back to.
+
+    Mirrors ``test_pyav_backend_returns_target_frames_not_keyframes``: a long
+    GOP (single keyframe at frame 0) with a per-frame green-channel marker.
+    With ``seek_mode="exact"`` torchcodec resolves each index to the exact
+    frame, so the returned markers must be distinct, ordered, and match the
+    requested indices.
+    """
+    pytest.importorskip("torchcodec")
+    num_frames = 50
+    num_sampled = 4
+    height, width = 64, 64
+
+    video_bytes = create_long_gop_video(
+        num_frames=num_frames, width=width, height=height
+    )
+
+    loader = VIDEO_LOADER_REGISTRY.load("opencv")
+    frames, metadata = loader.load_bytes(
+        video_bytes, num_frames=num_sampled, backend="torchcodec"
+    )
+    assert frames.shape == (num_sampled, height, width, 3)
+
+    requested = list(metadata["frames_indices"])
+    assert len(requested) == num_sampled
+
+    actual = [int(f[height // 2, width // 2, 1]) for f in frames]
+
+    assert len(set(actual)) == num_sampled, (
+        f"torchcodec returned only {len(set(actual))} distinct frames for "
+        f"{num_sampled} requested indices: markers={actual}, "
+        f"requested={requested}. Keyframe-snap regression."
+    )
+
+    assert actual == sorted(actual), f"Returned frames out of order: markers={actual}"
+
+    for marker, want_idx in zip(actual, requested):
+        assert abs(marker - want_idx) <= 10, (
+            f"Frame mismatch: requested index {want_idx}, "
+            f"got marker {marker} (tolerance ±10)"
+        )
+
+
@pytest.mark.parametrize(
    "loader_key, kwargs, expected_num_frames",
    [
@@ -854,6 +966,42 @@ def test_pyav_backend_returns_target_frames_not_keyframes():
            120,
            id="glm46v-pyav-60s",
        ),
+        # uniform sampling + torchcodec codec (same frame counts as opencv)
+        pytest.param(
+            "opencv",
+            {"num_frames": 32, "backend": "torchcodec"},
+            32,
+            id="torchcodec-num_frames",
+        ),
+        pytest.param(
+            "opencv", {"fps": 2, "backend": "torchcodec"}, 120, id="torchcodec-fps"
+        ),
+        pytest.param(
+            "opencv",
+            {"num_frames": 500, "fps": 2, "backend": "torchcodec"},
+            120,
+            id="torchcodec-num_frames_wins_fps",
+        ),
+        # dynamic sampling + torchcodec codec
+        pytest.param(
+            "opencv_dynamic",
+            {"fps": 1, "max_duration": 60, "backend": "torchcodec"},
+            60,
+            id="torchcodec_dynamic-within_max_duration",
+        ),
+        pytest.param(
+            "opencv_dynamic",
+            {"fps": 2, "max_duration": 30, "backend": "torchcodec"},
+            60,
+            id="torchcodec_dynamic-exceeds_max_duration",
+        ),
+        # glm46v dynamic FPS + torchcodec codec
+        pytest.param(
+            "glm46v",
+            {"backend": "torchcodec"},
+            120,
+            id="glm46v-torchcodec-60s",
+        ),
    ],
 )
 def test_video_loader_frames_sampling(
@@ -864,6 +1012,8 @@ def test_video_loader_frames_sampling(
    expected_num_frames: int,
 ):
    """Test video loader frames sampling functionality."""
+    if kwargs.get("backend") == "torchcodec":
+        pytest.importorskip("torchcodec")
    monkeypatch.setenv("VLLM_VIDEO_LOADER_BACKEND", loader_key)
    loader = VIDEO_LOADER_REGISTRY.load(loader_key)

@@ -133,6 +133,49 @@ def test_modelopt_mixed_precision_quantizes_parallel_lm_head():
    assert isinstance(method, ModelOptNvFp4LinearMethod)


+def test_modelopt_mixed_precision_infers_fused_gate_up_projection():
+    from vllm.model_executor.layers.linear import LinearBase
+
+    config = _mixed_precision_config(
+        {
+            "model.layers.0.mlp.gate_proj": {"quant_algo": "NVFP4"},
+            "model.layers.0.mlp.up_proj": {"quant_algo": "NVFP4"},
+        }
+    )
+
+    fake_layer = MagicMock(spec=LinearBase)
+    with patch(
+        "vllm.model_executor.layers.quantization.modelopt.init_nvfp4_linear_kernel"
+    ):
+        method = config.get_quant_method(fake_layer, "model.layers.0.mlp.gate_up_proj")
+
+    assert isinstance(method, ModelOptNvFp4LinearMethod)
+
+
+@pytest.mark.parametrize(
+    ("quantized_prefix", "missing_prefix"),
+    [
+        ("model.layers.0.mlp.gate_proj", "model.layers.0.mlp.down_proj"),
+        ("model.layers.0.self_attn.o_proj", "model.layers.0.self_attn.qkv_proj"),
+    ],
+)
+def test_modelopt_mixed_precision_does_not_infer_missing_sibling_linear(
+    quantized_prefix, missing_prefix
+):
+    from vllm.model_executor.layers.linear import LinearBase
+
+    config = _mixed_precision_config(
+        {
+            quantized_prefix: {"quant_algo": "NVFP4"},
+        }
+    )
+
+    fake_layer = MagicMock(spec=LinearBase)
+    method = config.get_quant_method(fake_layer, missing_prefix)
+
+    assert isinstance(method, UnquantizedLinearMethod)
+
+
 def test_vocab_parallel_embedding_weight_loader_accepts_scalar_scale():
    holder = Mock()
    scale = torch.nn.Parameter(torch.empty(1))
@@ -649,6 +649,30 @@ def test_nested_hf_overrides():
    assert model_config.hf_config.vision_config.hidden_size == 512


+def test_model_class_overrides_registers_target():
+    """`model_class_overrides` redirects an architecture to a custom class."""
+    from vllm.model_executor.models import ModelRegistry
+
+    arch = "_TestModelClassOverrideArch"
+    target = "vllm.model_executor.models.llama:LlamaForCausalLM"
+    assert arch not in ModelRegistry.models
+
+    model_config = ModelConfig(
+        "facebook/opt-125m",
+        model_class_overrides={arch: target},
+    )
+    try:
+        # Accessing `.registry` is the chokepoint that applies the overrides;
+        # it has already run during construction.
+        registered = model_config.registry.models[arch]
+        assert registered.module_name == "vllm.model_executor.models.llama"
+        assert registered.class_name == "LlamaForCausalLM"
+        # Idempotent: a second access does not re-register or error out.
+        assert model_config.registry.models[arch] is registered
+    finally:
+        ModelRegistry.models.pop(arch, None)
+
+
@pytest.mark.skipif(
    current_platform.is_rocm(), reason="Encoder Decoder models not supported on ROCm."
 )
@@ -2,13 +2,21 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project


+import json
+
 import pytest

 from tests.tool_parsers.common_tests import (
    ToolParserTestConfig,
    ToolParserTests,
 )
-from tests.tool_parsers.utils import run_tool_extraction
+from tests.tool_parsers.utils import (
+    run_tool_extraction,
+    run_tool_extraction_streaming,
+    split_string_into_token_deltas,
+)
+from vllm.tokenizers import get_tokenizer
+from vllm.tool_parsers.granite_tool_parser import GraniteToolParser


 class TestGraniteToolParser(ToolParserTests):
@@ -116,3 +124,38 @@ I'll get that information.""",
            f"Expected 1 tool call from string format, got {len(tool_calls)}"
        )
        assert tool_calls[0].function.name == "get_weather"
+
+
+# granite emits arguments before name and its own tokenizer (not gpt2) is used
+# here so the token boundaries match production; get_tokenizer only fetches the
+# small tokenizer files, not the model weights.
+@pytest.fixture(scope="module")
+def granite_tokenizer():
+    return get_tokenizer(tokenizer_name="ibm-granite/granite-3.1-8b-instruct")
+
+
+@pytest.mark.parametrize("chunk_size", [2, 3, 4, 5])
+def test_streaming_parallel_calls_batched_deltas(granite_tokenizer, chunk_size):
+    """A batched delta (multiple tokens) spanning the boundary between two
+    parallel calls must not drop the first call's name. granite streams
+    arguments before name, so the name only completes as the next call appears.
+    """
+    parser = GraniteToolParser(granite_tokenizer)
+    model_output = (
+        '<|tool_call|> [{"arguments": {"city": "Tokyo"}, "name": "get_weather"}, '
+        '{"arguments": {"timezone": "Asia/Tokyo"}, "name": "get_time"}]'
+    )
+    token_deltas = split_string_into_token_deltas(granite_tokenizer, model_output)
+    batched = [
+        "".join(token_deltas[i : i + chunk_size])
+        for i in range(0, len(token_deltas), chunk_size)
+    ]
+    reconstructor = run_tool_extraction_streaming(
+        parser, batched, assert_one_tool_per_delta=False
+    )
+    names = [tc.function.name for tc in reconstructor.tool_calls]
+    assert names == ["get_weather", "get_time"]
+    # trailing args of the final call are flushed by the serving layer
+    assert json.loads(reconstructor.tool_calls[0].function.arguments) == {
+        "city": "Tokyo"
+    }
@@ -144,6 +144,7 @@ def stream_delta_message_generator(
    mistral_tokenizer: TokenizerLike,
    model_output: str | None,
    tools: list[tuple[str, str]] | None,
+    chunk_size: int = 1,
 ) -> Generator[DeltaMessage, None, None]:
    if (
        isinstance(mistral_tokenizer, MistralTokenizer)
@@ -182,15 +183,13 @@ def stream_delta_message_generator(
    previous_tokens = None
    prefix_offset = 0
    read_offset = 0
+    pending_text = ""
+    pending_token_ids: list[int] = []
    for i, delta_token in enumerate(all_token_ids):
-        delta_token_ids = [delta_token]
-        previous_token_ids = all_token_ids[:i]
-        current_token_ids = all_token_ids[: i + 1]
-
        (new_tokens, delta_text, new_prefix_offset, new_read_offset) = (
            detokenize_incrementally(
                tokenizer=mistral_tokenizer,
-                all_input_ids=current_token_ids,
+                all_input_ids=all_token_ids[: i + 1],
                prev_tokens=previous_tokens,
                prefix_offset=prefix_offset,
                read_offset=read_offset,
@@ -198,27 +197,39 @@ def stream_delta_message_generator(
                spaces_between_special_tokens=True,
            )
        )
+        previous_tokens = (
+            previous_tokens + new_tokens if previous_tokens else new_tokens
+        )
+        prefix_offset = new_prefix_offset
+        read_offset = new_read_offset

-        current_text = previous_text + delta_text
+        # Buffer tokens so each streamed delta can carry ``chunk_size`` tokens,
+        # reproducing the multi-token deltas produced by async scheduling /
+        # stream_interval > 1.
+        pending_text += delta_text
+        pending_token_ids.append(delta_token)
+        if len(pending_token_ids) < chunk_size and i != len(all_token_ids) - 1:
+            continue
+
+        previous_token_ids = all_token_ids[: i + 1 - len(pending_token_ids)]
+        current_token_ids = all_token_ids[: i + 1]
+        current_text = previous_text + pending_text

        delta_message = mistral_tool_parser.extract_tool_calls_streaming(
            previous_text,
            current_text,
-            delta_text,
+            pending_text,
            previous_token_ids,
            current_token_ids,
-            delta_token_ids,
+            pending_token_ids,
            request=_DUMMY_REQUEST,
        )
        if delta_message:
            yield delta_message

        previous_text = current_text
-        previous_tokens = (
-            previous_tokens + new_tokens if previous_tokens else new_tokens
-        )
-        prefix_offset = new_prefix_offset
-        read_offset = new_read_offset
+        pending_text = ""
+        pending_token_ids = []


@pytest.mark.parametrize(
@@ -1572,3 +1583,39 @@ def test_grammar_from_tool_parser_set_by_adjust_request(
    request = _make_request()
    result = mistral_tool_parser.adjust_request(request)
    assert result._grammar_from_tool_parser is True
+
+
+@pytest.mark.parametrize("chunk_size", [2, 3, 4, 5])
+def test_streaming_pre_v11_parallel_calls_batched_deltas(
+    mistral_pre_v11_tool_parser, mistral_pre_v11_tokenizer, chunk_size
+):
+    """A batched delta spanning the boundary between two parallel calls must
+    keep them on distinct indices (the bug collapsed both onto index 0)."""
+    model_output = (
+        '[TOOL_CALLS] [{"name": "add", "arguments": {"a": 3.5, "b": 4}}, '
+        '{"name": "get_current_weather", "arguments": '
+        '{"city": "San Francisco", "state": "CA", "unit": "celsius"}}]'
+    )
+    names: list[str] = []
+    args: list[str] = []
+    idx = -1
+    for delta_message in stream_delta_message_generator(
+        mistral_pre_v11_tool_parser,
+        mistral_pre_v11_tokenizer,
+        model_output,
+        tools=None,
+        chunk_size=chunk_size,
+    ):
+        for tool_call in delta_message.tool_calls or []:
+            if tool_call.index != idx:
+                idx = tool_call.index
+                args.append("")
+            if tool_call.function and tool_call.function.name:
+                names.append(tool_call.function.name)
+            if tool_call.function and tool_call.function.arguments:
+                args[tool_call.index] += tool_call.function.arguments
+
+    assert names == ["add", "get_current_weather"]
+    assert len(args) == 2
+    # trailing args of the final call are flushed by the serving layer
+    assert json.loads(args[0]) == {"a": 3.5, "b": 4}
@@ -115,14 +115,12 @@ async def test_parallel_tool_calls(
            assert not role_name or role_name == "assistant"
            role_name = "assistant"

-        # if a tool call is streamed make sure there's exactly one
-        # (based on the request parameters
+        # a chunk may carry >1 tool-call delta at a parallel-call boundary
        streamed_tool_calls = chunk.choices[0].delta.tool_calls

-        if streamed_tool_calls and len(streamed_tool_calls) > 0:
-            # make sure only one diff is present - correct even for parallel
-            assert len(streamed_tool_calls) == 1
-            tool_call = streamed_tool_calls[0]
+        for tool_call in streamed_tool_calls or []:
+            # deltas arrive in non-decreasing index order
+            assert tool_call.index >= tool_call_idx

            # if a new tool is being called, set up empty arguments
            if tool_call.index != tool_call_idx:
@@ -28,6 +28,7 @@ hybrid_ssm_configs=(
  # GDN (Qwen3.5)
  "VLLM_SSM_CONV_STATE_LAYOUT=DS GPU_MEMORY_UTILIZATION=0.8 MODEL_NAMES=Qwen/Qwen3.5-0.8B"
  "VLLM_SSM_CONV_STATE_LAYOUT=DS PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 GPU_MEMORY_UTILIZATION=0.8 MODEL_NAMES=Qwen/Qwen3.5-0.8B"
+  "VLLM_SSM_CONV_STATE_LAYOUT=DS ENFORCE_EAGER=0 GPU_MEMORY_UTILIZATION=0.8 MODEL_NAMES=Qwen/Qwen3.5-0.8B VLLM_SERVE_EXTRA_ARGS=--spec-method,mtp,--spec-tokens,1"
  # Mamba1 (Jamba)
  "VLLM_SSM_CONV_STATE_LAYOUT=DS GPU_MEMORY_UTILIZATION=0.8 MODEL_NAMES=ai21labs/AI21-Jamba2-3B VLLM_SERVE_EXTRA_ARGS=--max-model-len,8192"
 )
@@ -76,6 +76,7 @@ DECODER_TP_SIZE=${DECODER_TP_SIZE:-1}
 GPU_MEMORY_UTILIZATION=${GPU_MEMORY_UTILIZATION:-0.2}
 PREFILL_BLOCK_SIZE=${PREFILL_BLOCK_SIZE:-128}
 DECODE_BLOCK_SIZE=${DECODE_BLOCK_SIZE:-128}
+ENFORCE_EAGER=${ENFORCE_EAGER:-1}
 # Comma-separated extra args for vllm serve (e.g. --max-model-len,2048)
 VLLM_SERVE_EXTRA_ARGS=${VLLM_SERVE_EXTRA_ARGS:-}

@@ -157,11 +158,13 @@ run_tests_for_model() {
    VLLM_NIXL_SIDE_CHANNEL_PORT=$SIDE_CHANNEL_PORT \
    vllm serve $model_name \
    --port $PORT \
-    --enforce-eager \
    --block-size ${PREFILL_BLOCK_SIZE} \
    --gpu-memory-utilization $GPU_MEMORY_UTILIZATION \
    --tensor-parallel-size $PREFILLER_TP_SIZE \
    --kv-transfer-config '$KV_CONFIG_P'"
+    if [[ "$ENFORCE_EAGER" == "1" ]]; then
+      BASE_CMD="${BASE_CMD} --enforce-eager"
+    fi
    if [[ -n "$VLLM_SERVE_EXTRA_ARGS" ]]; then
      IFS=',' read -r -a extra_args <<< "$VLLM_SERVE_EXTRA_ARGS"
      for arg in "${extra_args[@]}"; do
@@ -206,10 +209,12 @@ run_tests_for_model() {
    VLLM_NIXL_SIDE_CHANNEL_PORT=$SIDE_CHANNEL_PORT \
    vllm serve $model_name \
    --port $PORT \
-    --enforce-eager \
    --block-size ${DECODE_BLOCK_SIZE} \
    --gpu-memory-utilization $GPU_MEMORY_UTILIZATION \
    --kv-transfer-config '$KV_CONFIG_D'"
+    if [[ "$ENFORCE_EAGER" == "1" ]]; then
+      BASE_CMD="${BASE_CMD} --enforce-eager"
+    fi
    if [[ -n "$VLLM_SERVE_EXTRA_ARGS" ]]; then
      IFS=',' read -r -a extra_args <<< "$VLLM_SERVE_EXTRA_ARGS"
      for arg in "${extra_args[@]}"; do
@@ -147,6 +147,7 @@ LinearBackend = Literal[
    "flashinfer_cudnn",
    "flashinfer_b12x",
    "marlin",
+    "humming",
    "triton",
    "deep_gemm",
    "torch",
@@ -74,6 +74,12 @@ else:

 logger = init_logger(__name__)

+# Process-local record of which (arch, target) model-class overrides have been
+# registered in *this* process. Must not live on ModelConfig: that instance is
+# pickled to each worker, so an instance flag would arrive already "registered"
+# while the worker's own global ModelRegistry is still untouched.
+_REGISTERED_MODEL_CLASS_OVERRIDES: set[tuple[str, str]] = set()
+
 RunnerOption = Literal["auto", RunnerType]
 ConvertType = Literal["none", "embed", "classify"]
 ConvertOption = Literal["auto", ConvertType]
@@ -274,6 +280,13 @@ class ModelConfig:
    hf_overrides: HfOverrides = field(default_factory=dict)
    """If a dictionary, contains arguments to be forwarded to the Hugging Face
    config. If a callable, it is called to update the HuggingFace config."""
+    model_class_overrides: dict[str, str] = field(default_factory=dict)
+    """Override the model class used for one or more architectures, mapping the
+    architecture name to a `"module:class"` target (the same format accepted by
+    `ModelRegistry.register_model`). This registers the target class at runtime,
+    e.g. `{"GlmMoeDsaForCausalLM":
+    "vllm.models.deepseek_v32.nvidia.model:DeepseekV32ForCausalLM"}`. This
+    argument is for development and debugging purposes only."""
    generation_config: str = "auto"
    """The folder path to the generation config. Defaults to `"auto"`, the
    generation config will be loaded from model path. If set to `"vllm"`, no
@@ -812,8 +825,34 @@ class ModelConfig:

    @property
    def registry(self):
+        self._maybe_register_model_class_overrides()
        return me_models.ModelRegistry

+    def _maybe_register_model_class_overrides(self) -> None:
+        # Apply ``model_class_overrides`` here because this property is the
+        # single chokepoint through which every model-class inspect/resolve
+        # passes, in both the engine front-end and every worker process. The
+        # guard is process-local (see ``_REGISTERED_MODEL_CLASS_OVERRIDES``), so
+        # each worker re-registers into its own ModelRegistry exactly once
+        # rather than trusting a pickled-in instance flag.
+        if not self.model_class_overrides:
+            return
+        pending = [
+            (arch, target)
+            for arch, target in self.model_class_overrides.items()
+            if (arch, target) not in _REGISTERED_MODEL_CLASS_OVERRIDES
+        ]
+        if not pending:
+            return
+        logger.warning_once(
+            "Applying model_class_overrides %s. This is intended for "
+            "development/debugging.",
+            str(self.model_class_overrides),
+        )
+        for arch, target in pending:
+            me_models.ModelRegistry.register_model(arch, target)
+            _REGISTERED_MODEL_CLASS_OVERRIDES.add((arch, target))
+
    @property
    def architectures(self) -> list[str]:
        return self.model_arch_config.architectures
@@ -999,6 +1038,7 @@ class ModelConfig:
                "modelopt",
                "modelopt_fp4",
                "modelopt_mxfp8",
+                "mxfp8",
                "modelopt_mixed",
                # Ensure heavy backends are probed last to avoid unnecessary
                # imports during override detection (e.g., MXFP4 imports Triton)
@@ -652,6 +652,7 @@ class ParallelConfig:
            )
            and self.enable_expert_parallel
            and self.tensor_parallel_size > 1
+            and self.data_parallel_size > 1
        )

    @property
@@ -2,6 +2,8 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

 import copy
+import functools
+from collections.abc import Callable
 from typing import TYPE_CHECKING, Any, Literal, get_args

 from pydantic import Field, SkipValidation, field_validator, model_validator
@@ -588,6 +590,40 @@ class SpeculativeConfig:

        return hf_config

+    @staticmethod
+    def _apply_composed_hf_override(
+        target_hf_overrides: Callable[[PretrainedConfig], PretrainedConfig],
+        hf_config: PretrainedConfig,
+    ) -> PretrainedConfig:
+        hf_config = SpeculativeConfig.hf_config_override(hf_config)
+        return target_hf_overrides(hf_config)
+
+    @staticmethod
+    def compose_draft_hf_overrides(
+        target_hf_overrides: HfOverrides | None,
+    ) -> Callable[[PretrainedConfig], PretrainedConfig]:
+        """Build the ``hf_overrides`` for the draft ``ModelConfig``.
+
+        Callable overrides on the target are config-to-config transforms
+        (e.g. test harnesses shrinking ``num_hidden_layers``) and must also
+        reach the draft config — otherwise a draft belonging to a large
+        target is instantiated at full size even when the target is shrunk.
+        Dict overrides are target-specific key patches and are not applied
+        to the draft.
+
+        The composed override must stay picklable: the draft ``ModelConfig``
+        is sent to spawned engine-core processes, so a local closure would
+        fail with ``Can't get local object`` during pickling. Bind the
+        target via ``functools.partial`` over a module-referenceable static
+        method instead.
+        """
+        if not callable(target_hf_overrides):
+            return SpeculativeConfig.hf_config_override
+
+        return functools.partial(
+            SpeculativeConfig._apply_composed_hf_override, target_hf_overrides
+        )
+
    def __post_init__(self):
        # Note: "method" is a new parameter that helps to extend the
        # configuration of non-model-based proposers, and the "model" parameter
@@ -752,7 +788,12 @@ class SpeculativeConfig:
                if self.method == "medusa":
                    draft_hf_overrides = {"model_type": "medusa"}
                else:
-                    draft_hf_overrides = SpeculativeConfig.hf_config_override
+                    # Compose any callable hf_overrides set on the target so the
+                    # draft config receives the same transform (e.g. the test
+                    # shrink). Dict overrides stay target-only.
+                    draft_hf_overrides = SpeculativeConfig.compose_draft_hf_overrides(
+                        self.target_model_config.hf_overrides
+                    )
                self.draft_model_config = ModelConfig(
                    model=self.model,
                    runner="draft",
@@ -269,11 +269,17 @@ def _create_subgroups_split_group(
    must enter with the same ``split_ranks`` definition. Each rank receives
    the subgroup it belongs to.
    """
+    from vllm.distributed.utils import (
+        get_cpu_distributed_timeout_or_none,
+        get_distributed_timeout_or_none,
+    )
+
    device_backend_str = _device_backend_str(torch_distributed_backend)
    self_device_group = torch.distributed.split_group(
        split_ranks=group_ranks,
        group_desc=f"{group_name}:device",
        backend=device_backend_str,
+        timeout=get_distributed_timeout_or_none(),
    )
    # CPU subgroup: split_group requires the requested backend filter to
    # include the parent's default device type (= the device the parent PG
@@ -284,6 +290,7 @@ def _create_subgroups_split_group(
        split_ranks=group_ranks,
        group_desc=f"{group_name}:cpu",
        backend=f"cpu:gloo,{device_backend_str}",
+        timeout=get_cpu_distributed_timeout_or_none(),
    )
    return self_device_group, self_cpu_group

@@ -417,13 +424,19 @@ class GroupCoordinator:
                    self.rank_in_group = ranks.index(self.rank)
                    break
        else:
-            from vllm.distributed.utils import get_cpu_distributed_timeout_or_none
+            from vllm.distributed.utils import (
+                get_cpu_distributed_timeout_or_none,
+                get_distributed_timeout_or_none,
+            )

            timeout = get_cpu_distributed_timeout_or_none()
+            device_timeout = get_distributed_timeout_or_none()

            for ranks in group_ranks:
                device_group = torch.distributed.new_group(
-                    ranks, backend=torch_distributed_backend
+                    ranks,
+                    backend=torch_distributed_backend,
+                    timeout=device_timeout,
                )
                # a group with `gloo` backend, to allow direct coordination between
                # processes through the CPU.
@@ -504,10 +517,16 @@ class GroupCoordinator:
        This is a collective call: every world rank must invoke it. Used where we
        want to issue ops that can run concurrently with ops on `device_group`.
        """
+        from vllm.distributed.utils import get_distributed_timeout_or_none
+
+        device_timeout = get_distributed_timeout_or_none()
        sibling: ProcessGroup | None = None
        for ranks in self.group_ranks:
            pg = torch.distributed.new_group(
-                ranks, backend=self.torch_distributed_backend, group_desc=group_desc
+                ranks,
+                backend=self.torch_distributed_backend,
+                group_desc=group_desc,
+                timeout=device_timeout,
            )
            if self.rank in ranks:
                sibling = pg
@@ -533,6 +533,16 @@ def get_cpu_distributed_timeout_or_none() -> timedelta | None:
    return timedelta(seconds=timeout_seconds) if timeout_seconds is not None else None


+def get_distributed_timeout_or_none() -> timedelta | None:
+    from vllm.config import get_current_vllm_config_or_none
+
+    vllm_config = get_current_vllm_config_or_none()
+    if vllm_config is None:
+        return None
+    timeout_seconds = vllm_config.parallel_config.distributed_timeout_seconds
+    return timedelta(seconds=timeout_seconds) if timeout_seconds is not None else None
+
+
 def init_gloo_process_group(
    prefix_store: PrefixStore,
    group_rank: int,
@@ -616,6 +626,10 @@ def stateless_init_torch_distributed_process_group(
        gloo_timeout = get_cpu_distributed_timeout_or_none()
        if gloo_timeout is not None:
            timeout = gloo_timeout
+    else:
+        device_timeout = get_distributed_timeout_or_none()
+        if device_timeout is not None:
+            timeout = device_timeout

    if listen_socket is not None:
        store = create_tcp_store(
@@ -536,6 +536,9 @@ class EngineArgs:
    code_revision: str | None = ModelConfig.code_revision
    hf_token: bool | str | None = ModelConfig.hf_token
    hf_overrides: HfOverrides = get_field(ModelConfig, "hf_overrides")
+    model_class_overrides: dict[str, str] = get_field(
+        ModelConfig, "model_class_overrides"
+    )
    tokenizer_revision: str | None = ModelConfig.tokenizer_revision
    quantization: QuantizationMethods | str | None = ModelConfig.quantization
    quantization_config: "dict[str, Any] | QuantizationConfigArgs | None" = None
@@ -851,6 +854,9 @@ class EngineArgs:
        model_group.add_argument("--config-format", **model_kwargs["config_format"])
        model_group.add_argument("--hf-token", **model_kwargs["hf_token"])
        model_group.add_argument("--hf-overrides", **model_kwargs["hf_overrides"])
+        model_group.add_argument(
+            "--model-class-overrides", **model_kwargs["model_class_overrides"]
+        )
        model_group.add_argument("--pooler-config", **model_kwargs["pooler_config"])
        model_group.add_argument(
            "--generation-config", **model_kwargs["generation_config"]
@@ -1622,6 +1628,7 @@ class EngineArgs:
            code_revision=self.code_revision,
            hf_token=self.hf_token,
            hf_overrides=self.hf_overrides,
+            model_class_overrides=self.model_class_overrides,
            tokenizer_revision=self.tokenizer_revision,
            max_model_len=self.max_model_len,
            quantization=self.quantization,
@@ -130,6 +130,7 @@ async def init_generate_state(
        enable_force_include_usage=args.enable_force_include_usage,
        enable_log_outputs=args.enable_log_outputs,
        enable_log_deltas=args.enable_log_deltas,
+        enable_per_request_metrics=args.enable_per_request_metrics,
    )
    state.openai_serving_chat = (
        OpenAIServingChat(**_chat_kwargs) if "generate" in supported_tasks else None
@@ -150,6 +151,7 @@ async def init_generate_state(
            return_tokens_as_token_ids=args.return_tokens_as_token_ids,
            enable_prompt_tokens_details=args.enable_prompt_tokens_details,
            enable_force_include_usage=args.enable_force_include_usage,
+            enable_per_request_metrics=args.enable_per_request_metrics,
        )
        if "generate" in supported_tasks
        else None
@@ -18,6 +18,7 @@ from vllm.entrypoints.openai.completion.protocol import CompletionRequest
 from vllm.entrypoints.openai.engine.protocol import (
    ErrorResponse,
    GenerationError,
+    PerRequestTimingMetrics,
 )
 from vllm.entrypoints.openai.models.serving import OpenAIServingModels
 from vllm.entrypoints.openai.responses.protocol import ResponsesRequest
@@ -34,6 +35,7 @@ from vllm.tracing import (
    extract_trace_headers,
    log_tracing_disabled_warning,
 )
+from vllm.v1.metrics.stats import RequestStateStats

 logger = init_logger(__name__)

@@ -41,6 +43,61 @@ RequestT = TypeVar("RequestT", bound=AnyRequest)
 _T = TypeVar("_T")


+def build_per_request_timing_metrics(
+    metrics: RequestStateStats | None,
+    num_generation_tokens: int,
+) -> PerRequestTimingMetrics:
+    """Build per-request timing metrics from ``RequestStateStats``.
+
+    ``generation_time_ms`` is the decode interval only (first output token to
+    last output token); it excludes both queue wait and prefill/TTFT.
+    ``tokens_per_second`` is overall output throughput: all generated tokens
+    over the inference interval (scheduling to last output token), so it counts
+    the prefill/TTFT phase and is not simply the reciprocal of ``mean_itl_ms``.
+    Each field is left ``None`` when the timestamps it depends on are
+    unavailable.
+    """
+    if metrics is None:
+        return PerRequestTimingMetrics()
+
+    queued_ts = metrics.queued_ts
+    scheduled_ts = metrics.scheduled_ts
+    first_token_ts = metrics.first_token_ts
+    last_token_ts = metrics.last_token_ts
+
+    time_to_first_token_ms: float | None = None
+    generation_time_ms: float | None = None
+    queue_time_ms: float | None = None
+    mean_itl_ms: float | None = None
+    tokens_per_second: float | None = None
+
+    if scheduled_ts > 0 and first_token_ts > 0:
+        time_to_first_token_ms = (first_token_ts - scheduled_ts) * 1000
+
+    if first_token_ts > 0 and last_token_ts > 0:
+        generation_time_ms = (last_token_ts - first_token_ts) * 1000
+
+    if queued_ts > 0 and scheduled_ts > 0:
+        queue_time_ms = (scheduled_ts - queued_ts) * 1000
+
+    if first_token_ts > 0 and last_token_ts > 0 and num_generation_tokens > 1:
+        decode_time = last_token_ts - first_token_ts
+        mean_itl_ms = decode_time / (num_generation_tokens - 1) * 1000
+
+    if scheduled_ts > 0 and last_token_ts > 0:
+        inference_time_ms = (last_token_ts - scheduled_ts) * 1000
+        if inference_time_ms > 0:
+            tokens_per_second = num_generation_tokens / inference_time_ms * 1000
+
+    return PerRequestTimingMetrics(
+        time_to_first_token_ms=time_to_first_token_ms,
+        generation_time_ms=generation_time_ms,
+        queue_time_ms=queue_time_ms,
+        mean_itl_ms=mean_itl_ms,
+        tokens_per_second=tokens_per_second,
+    )
+
+
@dataclass(kw_only=True)
 class ServeContext(Generic[RequestT]):
    request: RequestT
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Harry Mellorandkhluu	8bbc0062b1	Fix embed scaling + CUDA graphs in Transformers modelling backend (#48010 ) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> (cherry picked from commit `56da398dac`)	2026-07-08 21:12:19 -07:00
Thien Tranandkhluu	65168ef45c	Allow FlashInfer A2A backends for TRTLLM FP8 MoE Modular (#46661 ) Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg> (cherry picked from commit `0d2f4e7c9c`)	2026-07-08 21:12:19 -07:00
Robert Shawandkhluu	51e3572d30	[Bug] Fix Batched DeepGEMM (#47884 ) Signed-off-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Co-authored-by: Robert Shaw <robertgshaw2-redhat@dgx-b200-02.mgmt.accl-001.lab.rdu2.dc.redhat.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit `572b25b03e`)	2026-07-08 21:12:19 -07:00
Canlin Guoandkhluu	654e9e5b90	[Model] Support MOSS-Transcribe-Diarize (#47729 ) Signed-off-by: gcanlin <canlinguosdu@gmail.com> (cherry picked from commit `285c08c036`)	2026-07-08 21:12:19 -07:00
Wei Zhaoandkhluu	15d7edbbcf	[Perf] Minimax M3 - Support cross-layer allreduce-norm fusion (#47631 ) Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> (cherry picked from commit `2afa3f7e95`)	2026-07-08 21:12:19 -07:00
Jeff (Junze) Maandkhluu	b53923c9ef	[Bugfix] DSV4 TP16 garbage output (#47493 ) Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> (cherry picked from commit `80eb01e93d`)	2026-07-08 21:12:19 -07:00
Yongye Zhuandkhluu	cf3df33687	[Minimax-M3] Using tok_sparse_select from MSA instead of triton kernels (#47502 ) Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit `9021589498`)	2026-07-08 21:12:19 -07:00
Dakai Anandkhluu	ae3665a6ab	[Bugfix] Fix PD disagg + MTP correctness for Qwen3.5(GDN) (#47466 ) Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> (cherry picked from commit `65dcde1695`)	2026-07-08 21:12:19 -07:00
Summer Yangandkhluu	f5ba5da988	[Perf] Use blocking CUDA events to avoid busy polling cuda driver lock (#47081 ) Signed-off-by: Jingyi Yang <girasoleyang@gmail.com> (cherry picked from commit `d3e69fd671`)	2026-07-08 21:12:19 -07:00
Wentao Yeandkhluu	7da76e4147	[CI Bug Fix] Temp fix for v3.2 accuracy (#47902 ) (cherry picked from commit `3f99883d97`)	2026-07-07 18:25:17 -07:00
Rahul Vishwakarmaandkhluu	b2dec4ac5a	[CPU][Bugfix] Fix flaky ShortConv prefill test on ARM (uninitialized weights) (#47848 ) Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com> (cherry picked from commit `f7efab58ec`)	2026-07-07 18:20:30 -07:00
Harry Mellorandkhluu	88550342ce	[CI] Fix Transformers modeling backend LoRA test (#47832 ) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> (cherry picked from commit `0ed05b6f82`)	2026-07-07 18:07:57 -07:00
danielafrimiandkhluu	ad49bb65d9	[BugFix] Fix ModelOpt mixed-precision quantization for sparse `quantized_layers` configs. (#47318 ) Signed-off-by: Daniel Afrimi <dafrimi@nvidia.com> Signed-off-by: <dafrimi@nvidia.com> (cherry picked from commit `0a2965b1b3`)	2026-07-07 18:05:03 -07:00
xiangdongandGitHub	6db31c8e76	[XPU][CI]Adjust memory request for tests in Intel GPU CI (#47758 ) Signed-off-by: zengxian <xiangdong.zeng@intel.com>	2026-07-07 05:56:40 +00:00
Humphrey GitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Cyrus Leung	cbe9c40f99	[Bugfix] Forward callable hf_overrides to the draft model config (#45352 ) Signed-off-by: HumphreySun98 <humphreysun98@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>	2026-07-06 21:12:38 -07:00
Andreas KaratzasandGitHub	2f71b2bd9f	[ROCm] Align mixed encoder-decoder KV cache views in V2 runner (#47685 ) Signed-off-by: Andreas Karatzas <akaratza@amd.com>	2026-07-07 12:09:22 +08:00
Jee Jee Li GitHubmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>	32ab064621	[UX] Add `model_class_overrides` for development and debugging (#47148 ) Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>	2026-07-07 11:43:08 +08:00
Guan-Ming ChiuandGitHub	c64c356990	[Perf] Bound DiffusionGemma sampler transient via request-tiled logits (#45672 ) Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>	2026-07-07 03:42:01 +00:00
Tahsin TunanandGitHub	34e6dfced8	[Rust Frontend] Stamp `arrival_time` at the frontend entry (#47787 ) Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>	2026-07-07 03:27:10 +00:00
ReidandGitHub	39a1d32b59	[Rust Frontend] Avoid extra copies for multimodal tensors (#47581 ) Signed-off-by: reidliu41 <reid201711@gmail.com>	2026-07-07 03:09:55 +00:00
Nicolas Hug GitHub Claude Isotr0py	700e882eab	Add TorchCodec as a video decoding backend (#46609 ) Signed-off-by: Nicolas Hug <contact@nicolas-hug.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>	2026-07-06 19:58:51 -07:00
jialoop-git GitHub Claude Opus 4.8	a4f019fa25	fix(distributed): propagate distributed_timeout_seconds to NCCL device groups (#45159 ) Signed-off-by: jialoop-git <joane8913456@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 02:52:51 +00:00
Rahul VishwakarmaandGitHub	9dd2465896	feat(cpu): add CPU support for Mamba ShortConv (#35059 ) Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com>	2026-07-07 10:47:08 +08:00
ReidandGitHub	a46c9329e5	[Rust Frontend] Add DeepSeek V3.2 roundtrip fixture (#47619 ) Signed-off-by: reidliu41 <reid201711@gmail.com>	2026-07-07 10:47:02 +08:00
Kyle SayersandGitHub	445321fab4	[Bugfix] [Quantization] Fix loading for CT DSV2 (#47780 )	2026-07-07 02:28:00 +00:00
YiSheng5 GitHub Kunshang Ji	69f3150981	[XPU] Fix PP accuracy on XPU device (#47253 ) Signed-off-by: yisheng <yi.sheng@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>	2026-07-07 09:17:27 +08:00
nv-nedelman-1 GitHub Simon Mo GPT-5.5	86db6c3070	[Frontend] add per-request timing `metrics` field to response body of Chat/Completions APIs (#46768 ) Signed-off-by: Nicholas Edelman <nedelman@nvidia.com> Signed-off-by: Simon Mo <simon@inferact.ai> Co-authored-by: Simon Mo <simon@inferact.ai> Co-authored-by: GPT-5.5 <noreply@cursor.com>	2026-07-06 17:48:29 -07:00
akii96 GitHub Andreas Karatzas	5769a7382c	[ROCm][CI][Bugfix] Fix flaky parallel tool-call streaming (test assertion + Mistral/Granite parsers) (#47550 ) Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>	2026-07-06 19:06:19 -04:00
Andreas KaratzasandGitHub	8484ca5d45	[ROCm][CI] Adding Rust parity (#47478 ) Signed-off-by: Andreas Karatzas <akaratza@amd.com>	2026-07-06 15:05:39 -07:00
Simon Danielsson GitHub nnyrhila	482e5524fe	[Bugfix][ROCm] Fix memory access fault in AITER MLA backend for DPA+FP8 KV (#47276 ) Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com> Co-authored-by: nnyrhila <niko.nyrhila@amd.com>	2026-07-06 21:30:02 +00:00
SherryC41 GitHub sherryC41 Matthew Bonanni	567a78432d	[Bugfix] Fix dp mtp hang (#40589 ) Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: sherryC41 <sherry.c.c41@gmail.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>	2026-07-06 21:08:17 +00:00
Jinzhen Lin GitHub mgoin	d891b9bd51	[Quantization] add humming moe backend to all dense/moe oracles (#41652 ) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: mgoin <mgoin64@gmail.com>	2026-07-06 13:36:07 -07:00
Zhiyi Hong GitHub yy-fighting Michael Goin	04adc8843b	[Bugfix]Fix DeepSeek-V4 fp8_ds_mla KV cache reshape (#47716 ) Co-authored-by: yy-fighting <23518844576@qq.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>	2026-07-06 12:56:44 -07:00
Harry MellorandGitHub	ae098abe3f	[CI] Fix some errors on `main` (#47726 ) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>	2026-07-06 19:40:23 +00:00
Andrii Skliar GitHub Andrii Skliar	b1384f5ec6	Enable B12x backend for non-gated MoEs (like Nemotron) (#43328 ) Signed-off-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com>	2026-07-06 12:40:07 -07:00
Nathaniel McVicar GitHub Harry Mellor	b136cc2c2c	[Bugfix][Model] Add stability window to DiffusionGemma to match HF stability_threshold semantics (#45965 ) Signed-off-by: Nathaniel McVicar <namcvica@microsoft.com> Signed-off-by: Nathaniel McVicar <Nathaniel.McVicar@microsoft.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>	2026-07-06 19:39:12 +00:00