karylab_agents/vllm
1
0
Fork 0
forked from Karylab-cklius/vllm
Code Pull Requests 2 Actions 1 Packages Activity

[Frontend] Support strict mode for tool calling (#45003)

Browse Source 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Chauncey
2026-06-12 07:51:48 +00:00
committed by GitHub
co-authored by cjackal mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent bd59c913bc
commit 2043258dec
29 changed files with 692 additions and 1956 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/tool_calling.md
+17 -7
View File
@@ -115,18 +115,28 @@ Whether vLLM enforces the tool parameter schema during generation depends on the
| --- | --- | --- |
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
| `"auto"` | No | The model generates freely. A tool-call parser extracts tool calls from the raw text. Arguments may be malformed or not match the schema. |
| `"auto"` | Depends on the parser | Model-specific structural-tag parsers can constrain tool-call arguments with structured outputs. Other parsers generate freely and extract tool calls from raw text. |
| `"none"` | N/A | No tool calls are produced. |
When schema conformance matters, prefer `tool_choice="required"` or named function calling over `"auto"`.
### Strict Mode
### Strict Mode (`strict` parameter)
Strict tool calling makes function-call arguments adhere to the function schema instead of relying only on best-effort parsing. vLLM implements strict tool calling for structural-tag based tool parsers by using the structured outputs backend under the hood.
The [OpenAI API](https://platform.openai.com/docs/guides/function-calling#strict-mode) supports a `strict` field on function definitions. When set to `true`, OpenAI uses constrained decoding to guarantee that tool-call arguments match the function schema, even in `tool_choice="auto"` mode.
For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
vLLM **does not implement** `strict` mode today. The `strict` field is accepted in requests (to avoid breaking clients that set it), but it has no effect on decoding behavior. In auto mode, argument validity depends entirely on the model's output quality and the parser's extraction logic.
* Set `additionalProperties` to `false` for each object in `parameters`.
* Mark all fields in `properties` as required.
* Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
Tracking issues: [#15526](https://github.com/vllm-project/vllm/issues/15526), [#16313](https://github.com/vllm-project/vllm/issues/16313).
vLLM controls structural-tag strict tool calling with the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable. It defaults to `true`.
```bash
VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
```
When this variable is `true`, structural-tag based tool parsers attach a structural tag to the request, so the structured outputs backend can constrain the model-specific tool-call format and function-call arguments. When it is `false`, vLLM does not attach structural tags for tool calling. In that case, `tool_choice="auto"` falls back to best-effort parser extraction from the raw model output, and no structural-tag constraint is applied.
This environment variable only affects structural-tag based tool calling. It does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
## Automatic Function Calling
@@ -146,7 +156,7 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
!!! note
With `tool_choice="auto"`, tool-call arguments are extracted from the model's raw text output by the selected parser. No schema-level constraint is applied during decoding, so arguments may occasionally be malformed or violate the function's parameter schema. See [Constrained Decoding Behavior](#constrained-decoding-behavior) for details.
With `tool_choice="auto"`, schema-level constraint depends on the selected parser and `VLLM_ENFORCE_STRICT_TOOL_CALLING`. Structural-tag parsers can enforce tool-call constraints when it is `true`; when it is `false`, or when the selected parser has no structural-tag support, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
### Hermes Models (`hermes`)
requirements/common.txt
+1 -1
View File
@@ -25,7 +25,7 @@ outlines_core == 0.2.14
# required for outlines backend disk cache
diskcache == 5.6.3
lark == 1.2.2
xgrammar >= 0.2.0, < 1.0.0; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
xgrammar >= 0.2.1, < 1.0.0; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
typing_extensions >= 4.10
filelock >= 3.16.1 # need to contain https://github.com/tox-dev/filelock/pull/317
partial-json-parser # used for parsing partial JSON outputs
requirements/test/rocm.txt
+1 -1
View File
@@ -1367,7 +1367,7 @@ word2number==1.1
# via lm-eval
wrapt==2.1.2
# via smart-open
xgrammar==0.2.0
xgrammar==0.2.1
# via
# -c requirements/common.txt
# -r requirements/test/../common.txt
tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py
+2 -1
View File
@@ -250,6 +250,7 @@ async def k2_client(k2_server):
@pytest.mark.asyncio
@pytest.mark.skip(reason="Skipping Kimi K2 tool ID test")
@pytest.mark.parametrize("model_name", [MODEL_NAME])
@pytest.mark.parametrize("stream", [True, False])
@pytest.mark.parametrize("tool_choice", ["required"])
@@ -442,7 +443,7 @@ async def test_named_tool_use(
if delta.role:
assert delta.role == "assistant"
assert delta.content is None or len(delta.content) == 0
if delta.tool_calls:
if delta.tool_calls and delta.tool_calls[0].function.arguments:
output.append(delta.tool_calls[0].function.arguments)
if chunk.choices[0].finish_reason is not None:
finish_reason_count += 1
tests/entrypoints/openai/responses/conftest.py
+1
View File
@@ -390,6 +390,7 @@ def server_with_store(default_server_args):
env_dict={
"VLLM_ENABLE_RESPONSES_API_STORE": "1",
"VLLM_SERVER_DEV_MODE": "1",
"VLLM_ENFORCE_STRICT_TOOL_CALLING": "0",
},
) as remote_server:
yield remote_server
tests/parser/test_parse.py
+22 -4
View File
@@ -2,13 +2,31 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import json
import os
import pytest
from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
from vllm.parser.abstract_parser import DelegatingParser
from vllm.reasoning.basic_parsers import BaseThinkingReasoningParser
from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser
_STRICT_TOOL_CALLING_ENV = "VLLM_ENFORCE_STRICT_TOOL_CALLING"
_STRICT_TOOL_CALLING_ENV_VALUE = os.environ.get(_STRICT_TOOL_CALLING_ENV)
os.environ[_STRICT_TOOL_CALLING_ENV] = "0"
from vllm.entrypoints.openai.chat_completion.protocol import ( # noqa: E402
ChatCompletionRequest,
)
from vllm.parser.abstract_parser import DelegatingParser # noqa: E402
from vllm.reasoning.basic_parsers import ( # noqa: E402
BaseThinkingReasoningParser,
)
from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser # noqa: E402
@pytest.fixture(scope="module", autouse=True)
def restore_strict_tool_calling_env():
yield
if _STRICT_TOOL_CALLING_ENV_VALUE is None:
os.environ.pop(_STRICT_TOOL_CALLING_ENV, None)
else:
os.environ[_STRICT_TOOL_CALLING_ENV] = _STRICT_TOOL_CALLING_ENV_VALUE
class ThinkReasoningParser(BaseThinkingReasoningParser):
tests/tool_parsers/test_qwen3coder_tool_parser.py
+13 -190
View File
@@ -3,6 +3,7 @@
import json
from collections.abc import Generator
from unittest.mock import MagicMock
import pytest
from openai.types.responses.function_tool import FunctionTool
@@ -19,15 +20,12 @@ from vllm.entrypoints.openai.engine.protocol import (
FunctionCall,
ToolCall,
)
from vllm.parser.abstract_parser import DelegatingParser
from vllm.tokenizers import TokenizerLike, get_tokenizer
from vllm.tokenizers.detokenizer_utils import detokenize_incrementally
from vllm.tool_parsers.qwen3coder_tool_parser import (
Qwen3CoderToolParser,
)
from vllm.tool_parsers.qwen3xml_tool_parser import (
Qwen3XMLToolParser,
StreamingXMLToolCallParser,
)
MODEL = "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8"
@@ -43,17 +41,8 @@ def qwen3_tool_parser(qwen3_tokenizer, sample_tools):
@pytest.fixture
def qwen3_xml_tool_parser(qwen3_tokenizer, sample_tools):
return Qwen3XMLToolParser(qwen3_tokenizer, tools=sample_tools)
@pytest.fixture(params=["xml"])
def qwen3_tool_parser_parametrized(qwen3_tool_parser, qwen3_xml_tool_parser, request):
"""Parameterized fixture that provides both parser types for testing"""
if request.param == "original":
return qwen3_tool_parser
else:
return qwen3_xml_tool_parser
def qwen3_tool_parser_parametrized(qwen3_tool_parser):
return qwen3_tool_parser
WEATHER_PARAMS = {
@@ -168,47 +157,6 @@ def assert_tool_calls(
)
def test_qwen3xml_deferred_array_parses_json_literals():
parser = StreamingXMLToolCallParser()
parser.set_tools(
[
ChatCompletionToolsParam(
type="function",
function={
"name": "AskUserQuestion",
"parameters": QUESTION_PARAMS,
},
)
]
)
delta = parser.parse_single_streaming_chunks(
"""<tool_call>
<function=AskUserQuestion>
<parameter=questions>
[{"question": "Pick a color", "multiSelect": false, "answer": null}]
</parameter>
</function>
</tool_call>"""
)
arguments = "".join(
tool_call.function.arguments or ""
for tool_call in delta.tool_calls or []
if tool_call.function and tool_call.function.arguments is not None
)
assert json.loads(arguments) == {
"questions": [
{
"question": "Pick a color",
"multiSelect": False,
"answer": None,
}
]
}
def stream_delta_message_generator(
qwen3_tool_parser,
qwen3_tokenizer: TokenizerLike,
@@ -523,7 +471,7 @@ hello world
</function>
</tool_call>"""
parser = Qwen3XMLToolParser(qwen3_tokenizer, tools=tools)
parser = Qwen3CoderToolParser(qwen3_tokenizer, tools=tools)
request = ChatCompletionRequest(model=MODEL, messages=[], tools=tools)
extracted_tool_calls = parser.extract_tool_calls(model_output, request=request)
@@ -1146,125 +1094,6 @@ TX
assert parsed_args["state"] == "TX"
def test_extract_tool_calls_complex_type_with_single_quote(
qwen3_tokenizer,
):
"""Test parameter type conversion based on tool schema"""
tools = [
ChatCompletionToolsParam(
type="function",
function={
"name": "test_types",
"parameters": {
"type": "object",
"properties": {
"int_param": {"type": "integer"},
"float_param": {"type": "float"},
"bool_param": {"type": "boolean"},
"str_param": {"type": "string"},
"obj_param": {"type": "object"},
},
},
},
)
]
model_output = """<tool_call>
<function=test_types>
<parameter=obj_param>
{'key': 'value'}
</parameter>
</function>
</tool_call>"""
parser = Qwen3XMLToolParser(qwen3_tokenizer, tools=tools)
request = ChatCompletionRequest(model=MODEL, messages=[], tools=tools)
extracted_tool_calls = parser.extract_tool_calls(model_output, request=request)
args = json.loads(extracted_tool_calls.tool_calls[0].function.arguments)
assert args["obj_param"] == {"key": "value"}
def test_extract_tool_calls_streaming_missing_opening_tag(
qwen3_tool_parser_parametrized, qwen3_tokenizer
):
"""Test streaming with missing opening <tool_call> tag
This tests that the streaming parser correctly handles
tool calls that start directly with <function=...>
"""
model_output = """I'll check the weather for you.
<function=get_current_weather>
<parameter=city>
Dallas
</parameter>
<parameter=state>
TX
</parameter>
<parameter=unit>
fahrenheit
</parameter>
</function>
</tool_call>"""
request = ChatCompletionRequest(model=MODEL, messages=[])
other_content = ""
tool_states = {}
for delta_message in stream_delta_message_generator(
qwen3_tool_parser_parametrized, qwen3_tokenizer, model_output, request
):
if delta_message.content:
other_content += delta_message.content
if delta_message.tool_calls:
for tool_call in delta_message.tool_calls:
idx = tool_call.index
if idx not in tool_states:
tool_states[idx] = {
"id": None,
"name": None,
"arguments": "",
"type": None,
}
if tool_call.id:
tool_states[idx]["id"] = tool_call.id
if tool_call.type:
assert tool_call.type == "function"
tool_states[idx]["type"] = tool_call.type
if tool_call.function:
if tool_call.function.name:
tool_states[idx]["name"] = tool_call.function.name
if tool_call.function.arguments is not None:
tool_states[idx]["arguments"] += tool_call.function.arguments
# Verify content was streamed
assert "I'll check the weather for you." in other_content
# Verify we got the tool call
assert len(tool_states) == 1
assert len(qwen3_tool_parser_parametrized.prev_tool_call_arr) == 1
state = tool_states[0]
assert state["id"] is not None
assert state["type"] == "function"
assert state["name"] == "get_current_weather"
# Verify arguments were parsed correctly despite missing opening tag
assert state["arguments"] is not None
args = json.loads(state["arguments"])
assert args["city"] == "Dallas"
assert args["state"] == "TX"
assert args["unit"] == "fahrenheit"
def test_malformed_xml_no_gt_delimiter(qwen3_tool_parser):
"""Regression: malformed XML without '>' must not crash (PR #36774)."""
model_output = (
@@ -1456,15 +1285,12 @@ def test_get_vllm_registry_structural_tag_returns_structural_tag(
@pytest.mark.parametrize("include_reasoning", [True, False])
def test_adjust_request_auto_uses_vllm_registry_structural_tag(
monkeypatch: pytest.MonkeyPatch,
qwen3_tool_parser: Qwen3CoderToolParser,
sample_tools: list[ChatCompletionToolsParam],
include_reasoning: bool,
) -> None:
monkeypatch.setattr(
"vllm.tool_parsers.abstract_tool_parser.VLLM_ENFORCE_STRICT_TOOL_CALLING",
True,
)
class TestParser(DelegatingParser):
tool_parser_cls = Qwen3CoderToolParser
request_tools = _as_chat_completion_tools(sample_tools)
req = ChatCompletionRequest(
messages=[],
@@ -1473,7 +1299,7 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
tool_choice="auto",
include_reasoning=include_reasoning,
)
out = qwen3_tool_parser.adjust_request(req)
out = TestParser(MagicMock(), tools=sample_tools).adjust_request(req)
assert out.structured_outputs is not None
assert out.structured_outputs.structural_tag is not None
assert isinstance(out.structured_outputs.structural_tag, str)
@@ -1482,14 +1308,11 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
def test_adjust_request_required_prefers_structural_tag(
monkeypatch: pytest.MonkeyPatch,
qwen3_tool_parser: Qwen3CoderToolParser,
sample_tools: list[ChatCompletionToolsParam],
) -> None:
monkeypatch.setattr(
"vllm.tool_parsers.abstract_tool_parser.VLLM_ENFORCE_STRICT_TOOL_CALLING",
True,
)
class TestParser(DelegatingParser):
tool_parser_cls = Qwen3CoderToolParser
request_tools = _as_chat_completion_tools(sample_tools)
req = ChatCompletionRequest(
messages=[],
@@ -1497,6 +1320,6 @@ def test_adjust_request_required_prefers_structural_tag(
tools=request_tools,
tool_choice="required",
)
out = qwen3_tool_parser.adjust_request(req)
out = TestParser(MagicMock(), tools=sample_tools).adjust_request(req)
assert out.structured_outputs is not None
assert out.structured_outputs.structural_tag is not None
tests/tool_parsers/test_qwen3xml_tool_parser.py
-72
View File
@@ -1,72 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import pytest
from tests.tool_parsers.common_tests import (
ToolParserTestConfig,
ToolParserTests,
)
class TestQwen3xmlToolParser(ToolParserTests):
@pytest.fixture
def test_config(self) -> ToolParserTestConfig:
return ToolParserTestConfig(
parser_name="qwen3_xml",
# Test data
no_tool_calls_output="This is a regular response without any tool calls.",
single_tool_call_output="<tool_call>\n<function=get_weather>\n<parameter=city>Tokyo</parameter>\n</function>\n</tool_call>",
parallel_tool_calls_output="<tool_call>\n<function=get_weather>\n<parameter=city>Tokyo</parameter>\n</function>\n</tool_call><tool_call>\n<function=get_time>\n<parameter=timezone>Asia/Tokyo</parameter>\n</function>\n</tool_call>",
various_data_types_output=(
"<tool_call>\n<function=test_function>\n"
"<parameter=string_field>hello</parameter>\n"
"<parameter=int_field>42</parameter>\n"
"<parameter=float_field>3.14</parameter>\n"
"<parameter=bool_field>true</parameter>\n"
"<parameter=null_field>null</parameter>\n"
'<parameter=array_field>["a", "b", "c"]</parameter>\n'
'<parameter=object_field>{"nested": "value"}</parameter>\n'
"</function>\n</tool_call>"
),
empty_arguments_output="<tool_call>\n<function=refresh>\n</function>\n</tool_call>",
surrounding_text_output=(
"Let me check the weather for you.\n\n"
"<tool_call>\n<function=get_weather>\n"
"<parameter=city>Tokyo</parameter>\n"
"</function>\n</tool_call>\n\n"
"I will get that information."
),
escaped_strings_output=(
"<tool_call>\n<function=test_function>\n"
'<parameter=quoted>He said "hello"</parameter>\n'
"<parameter=path>C:\\Users\\file.txt</parameter>\n"
"<parameter=newline>line1\nline2</parameter>\n"
"</function>\n</tool_call>"
),
malformed_input_outputs=[
"<tool_call><function=func>",
"<tool_call><function=></function></tool_call>",
],
# Expected results
single_tool_call_expected_name="get_weather",
single_tool_call_expected_args={"city": "Tokyo"},
parallel_tool_calls_count=2,
parallel_tool_calls_names=["get_weather", "get_time"],
# xfail markers - Qwen3XML has systematic streaming issues
xfail_streaming={
"test_single_tool_call_simple_args": (
"Qwen3XML streaming has systematic issues"
),
"test_parallel_tool_calls": "Qwen3XML streaming has systematic issues",
"test_various_data_types": "Qwen3XML streaming has systematic issues",
"test_empty_arguments": "Qwen3XML streaming has systematic issues",
"test_surrounding_text": "Qwen3XML streaming has systematic issues",
"test_escaped_strings": "Qwen3XML streaming has systematic issues",
"test_streaming_reconstruction": (
"Qwen3XML streaming reconstruction has known issues"
),
},
supports_typed_arguments=False,
)
tests/tool_parsers/test_structural_tag_registry.py
+314
View File
@@ -0,0 +1,314 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from xgrammar import StructuralTag
from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionNamedFunction,
ChatCompletionNamedToolChoiceParam,
ChatCompletionRequest,
ChatCompletionToolsParam,
)
from vllm.parser.abstract_parser import DelegatingParser
from vllm.tool_parsers.abstract_tool_parser import ToolParser
from vllm.tool_parsers.deepseekv3_tool_parser import DeepSeekV3ToolParser
from vllm.tool_parsers.deepseekv4_tool_parser import DeepSeekV4ToolParser
from vllm.tool_parsers.deepseekv31_tool_parser import DeepSeekV31ToolParser
from vllm.tool_parsers.deepseekv32_tool_parser import DeepSeekV32ToolParser
from vllm.tool_parsers.glm47_moe_tool_parser import Glm47MoeModelToolParser
from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser
from vllm.tool_parsers.kimi_k2_tool_parser import KimiK2ToolParser
from vllm.tool_parsers.llama_tool_parser import Llama3JsonToolParser
from vllm.tool_parsers.minimax_m2_tool_parser import MinimaxM2ToolParser
from vllm.tool_parsers.qwen3coder_tool_parser import Qwen3CoderToolParser
from vllm.tool_parsers.structural_tag_registry import (
SUPPORTED_STRUCTURAL_TAG_MODELS,
VLLM_BUILTIN_STRUCTURAL_TAG_MODELS,
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS,
_get_function_parameters,
get_model_structural_tag,
)
@pytest.fixture
def sample_tools() -> list[ChatCompletionToolsParam]:
return [
ChatCompletionToolsParam(
type="function",
function={
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
)
]
def test_supported_structural_tag_models_include_vllm_builtins():
assert SUPPORTED_STRUCTURAL_TAG_MODELS == (
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
)
assert "hermes" in VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_get_model_structural_tag_supports_all_xgrammar_builtins(
model: str,
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice="auto",
reasoning=False,
)
assert isinstance(tag, StructuralTag)
def test_get_model_structural_tag_supports_vllm_hermes(
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model="hermes",
tools=sample_tools,
tool_choice="required",
reasoning=False,
)
assert isinstance(tag, StructuralTag)
assert tag.model_dump() == {
"type": "structural_tag",
"format": {
"type": "tags_with_separator",
"tags": [
{
"type": "tag",
"begin": '<tool_call>\n{"name": "get_weather", "arguments": ',
"content": {
"type": "json_schema",
"json_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
"style": "json",
},
"end": "}\n</tool_call>",
},
{
"type": "tag",
"begin": '<tool_call>{"name": "get_weather", "arguments": ',
"content": {
"type": "json_schema",
"json_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
"style": "json",
},
"end": "}</tool_call>",
},
],
"separator": "",
"at_least_one": True,
"stop_after_first": False,
},
}
def test_hermes_required_tool_calls_use_empty_separator():
tools = [
ChatCompletionToolsParam(
type="function",
function={
"name": "get_weather",
"parameters": {"type": "object", "properties": {}},
},
),
ChatCompletionToolsParam(
type="function",
function={
"name": "get_time",
"parameters": {"type": "object", "properties": {}},
},
),
]
tag = get_model_structural_tag(
model="hermes",
tools=tools,
tool_choice="required",
reasoning=False,
)
assert tag is not None
assert tag.format.separator == ""
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_get_model_structural_tag_supports_named_tool_choice(
model: str,
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice=ChatCompletionNamedToolChoiceParam(
function=ChatCompletionNamedFunction(name="get_weather")
),
reasoning=False,
)
assert isinstance(tag, StructuralTag)
@pytest.mark.parametrize(
("parser_cls", "model"),
[
(DeepSeekV3ToolParser, "deepseek_r1"),
(DeepSeekV31ToolParser, "deepseek_v3_1"),
(DeepSeekV32ToolParser, "deepseek_v3_2"),
(DeepSeekV4ToolParser, "deepseek_v4"),
(Glm47MoeModelToolParser, "glm_4_7"),
(Hermes2ProToolParser, "hermes"),
(KimiK2ToolParser, "kimi"),
(Llama3JsonToolParser, "llama"),
(MinimaxM2ToolParser, "minimax"),
(Qwen3CoderToolParser, "qwen_3_coder"),
],
)
def test_tool_parsers_declare_matching_xgrammar_builtin_model(parser_cls, model):
assert parser_cls.structural_tag_model == model
assert not parser_cls.supports_required_and_named
def test_tool_parsers_without_structural_tag_support_required_and_named():
class NonStructuralTagToolParser(ToolParser):
pass
assert NonStructuralTagToolParser.structural_tag_model is None
assert NonStructuralTagToolParser.supports_required_and_named
def test_non_structural_tag_parser_uses_schema_constraints(
sample_tools: list[ChatCompletionToolsParam],
):
parser = ToolParser(MagicMock())
request = ChatCompletionRequest(
messages=[],
model="m",
tools=sample_tools,
tool_choice="required",
)
out = parser.adjust_request(request)
assert out.structured_outputs is not None
assert out.structured_outputs.json is not None
assert out.structured_outputs.structural_tag is None
def test_get_structural_tag_disables_reasoning(
monkeypatch: pytest.MonkeyPatch,
sample_tools: list[ChatCompletionToolsParam],
):
captured: list[bool] = []
def fake_get_model_structural_tag(*, reasoning: bool, **kwargs):
captured.append(reasoning)
return None
monkeypatch.setattr(
"vllm.tool_parsers.structural_tag_registry.get_model_structural_tag",
fake_get_model_structural_tag,
)
request = ChatCompletionRequest(
messages=[],
model="m",
tools=sample_tools,
tool_choice="auto",
)
parser = Qwen3CoderToolParser(MagicMock(), tools=sample_tools)
parser.get_structural_tag(request)
assert captured == [False]
def test_unified_parser_get_structural_tag_disables_reasoning(
monkeypatch: pytest.MonkeyPatch,
sample_tools: list[ChatCompletionToolsParam],
):
captured: list[bool] = []
def fake_get_model_structural_tag(*, reasoning: bool, **kwargs):
captured.append(reasoning)
return None
monkeypatch.setattr(
"vllm.tool_parsers.structural_tag_registry.get_model_structural_tag",
fake_get_model_structural_tag,
)
class TestParser(DelegatingParser):
tool_parser_cls = Qwen3CoderToolParser
request = ChatCompletionRequest(
messages=[],
model="m",
tools=sample_tools,
tool_choice="auto",
)
parser = TestParser(MagicMock(), tools=sample_tools)
parser.reasoning_parser = MagicMock(adjust_request=lambda request: request)
parser.adjust_request(request)
assert captured == [False]
def test_xgrammar_function_parameters_are_preserved(
monkeypatch: pytest.MonkeyPatch,
sample_tools: list[ChatCompletionToolsParam],
):
captured: list[list[dict]] = []
def fake_get_xgrammar_model_structural_tag(*, tools: list[dict], **kwargs):
captured.append(tools)
return None
monkeypatch.setattr(
"vllm.tool_parsers.structural_tag_registry.get_xgrammar_model_structural_tag",
fake_get_xgrammar_model_structural_tag,
)
get_model_structural_tag(
model="llama",
tools=sample_tools,
tool_choice="auto",
reasoning=False,
)
assert (
captured[0][0]["function"]["parameters"] == sample_tools[0].function.parameters
)
assert sample_tools[0].function.parameters is not None
def test_get_function_parameters_relaxes_function_strict_false():
function = SimpleNamespace(
parameters={"type": "object", "properties": {}},
strict=False,
)
assert _get_function_parameters(function) is True
vllm/entrypoints/openai/api_server.py
-14
View File
@@ -308,20 +308,6 @@ async def init_app_state(
) -> None:
vllm_config = engine_client.vllm_config
# Propagate enable_in_reasoning to the API-server process. The engine core
# runs in a separate process, so the contextvar that backs
# `get_current_vllm_config_or_none()` is None on this stack. Tool parsers
# call `get_enable_structured_outputs_in_reasoning()` during request
# handling and need to see the real flag, otherwise they silently fall
# back to False and mismatch the engine-side bitmask gating.
from vllm.tool_parsers.structural_tag_registry import (
set_enable_structured_outputs_in_reasoning,
)
set_enable_structured_outputs_in_reasoning(
vllm_config.structured_outputs_config.enable_in_reasoning
)
if args.tool_call_parser is not None:
from vllm.parser.metrics import init_parser_metrics
vllm/entrypoints/openai/chat_completion/batch_serving.py
+2 -2
View File
@@ -74,7 +74,7 @@ class OpenAIServingChatBatch(OpenAIServingChat):
if error_check_ret is not None:
return error_check_ret
tool_parser = render.tool_parser
parser = render.parser
tool_dicts: list[dict] | None = None
all_conversations: list[list[ConversationMessage]] = []
@@ -94,7 +94,7 @@ class OpenAIServingChatBatch(OpenAIServingChat):
default_template_content_format=render.chat_template_content_format,
default_template_kwargs=render.default_chat_template_kwargs,
tool_dicts=tool_dicts,
tool_parser=tool_parser,
parser=parser,
)
all_conversations.append(conversation)
all_engine_prompts.append(engine_prompts[0])
vllm/entrypoints/openai/responses/serving.py
+5 -8
View File
@@ -102,10 +102,9 @@ from vllm.logprobs import Logprob as SampleLogprob
from vllm.logprobs import SampleLogprobs
from vllm.lora.request import LoRARequest
from vllm.outputs import CompletionOutput
from vllm.parser import ParserManager
from vllm.parser import Parser, ParserManager
from vllm.sampling_params import SamplingParams, StructuredOutputsParams
from vllm.tokenizers import TokenizerLike
from vllm.tool_parsers import ToolParser
from vllm.utils import random_uuid
from vllm.utils.collection_utils import as_list
@@ -613,8 +612,7 @@ class OpenAIServingResponses(OpenAIServing):
default_template_content_format=self.chat_template_content_format,
default_template_kwargs=chat_template_kwargs,
tool_dicts=tool_dicts,
tool_parser=self.parser.tool_parser_cls if self.parser else None,
reasoning_parser=self.parser.reasoning_parser_cls if self.parser else None,
parser=self.parser,
)
return messages, engine_inputs
@@ -623,7 +621,7 @@ class OpenAIServingResponses(OpenAIServing):
request: ResponsesRequest,
messages: list[ResponseInputOutputItem],
tool_dicts: list[dict[str, Any]] | None,
tool_parser: type[ToolParser] | None,
parser: type[Parser] | None,
chat_template: str | None,
chat_template_content_format: ChatTemplateContentFormatOption,
):
@@ -638,8 +636,7 @@ class OpenAIServingResponses(OpenAIServing):
default_template_content_format=chat_template_content_format,
default_template_kwargs=chat_template_kwargs,
tool_dicts=tool_dicts,
tool_parser=tool_parser,
reasoning_parser=self.parser.reasoning_parser_cls if self.parser else None,
parser=parser,
)
return engine_inputs
@@ -707,7 +704,7 @@ class OpenAIServingResponses(OpenAIServing):
context.request,
context.parser.response_messages,
context.tool_dicts,
context.parser_cls.tool_parser_cls if context.parser_cls else None,
context.parser_cls,
context.chat_template,
context.chat_template_content_format,
)
vllm/entrypoints/serve/render/serving.py
+24 -28
View File
@@ -43,8 +43,7 @@ from vllm.inputs import (
tokens_input,
)
from vllm.logger import init_logger
from vllm.parser import ParserManager
from vllm.reasoning.abs_reasoning_parsers import ReasoningParser
from vllm.parser import Parser, ParserManager
from vllm.renderers import BaseRenderer, merge_kwargs
from vllm.renderers.inputs.preprocess import (
extract_prompt_components,
@@ -52,7 +51,6 @@ from vllm.renderers.inputs.preprocess import (
parse_model_prompt,
prompt_to_seq,
)
from vllm.tool_parsers import ToolParser
from vllm.utils import random_uuid
from vllm.utils.mistral import is_mistral_tokenizer, is_mistral_tool_parser
from vllm.utils.mistral import mt as _mt
@@ -89,16 +87,12 @@ class OpenAIServingRender:
self.trust_request_chat_template = trust_request_chat_template
self.enable_auto_tools = enable_auto_tools
self.exclude_tools_when_tool_choice_none = exclude_tools_when_tool_choice_none
self.tool_parser: type[ToolParser] | None = ParserManager.get_tool_parser(
self.parser: type[Parser] | None = ParserManager.get_parser(
tool_parser_name=tool_parser,
reasoning_parser_name=reasoning_parser,
enable_auto_tools=enable_auto_tools,
model_name=model_config.model,
)
self.reasoning_parser: type[ReasoningParser] | None = (
ParserManager.get_reasoning_parser(
reasoning_parser_name=reasoning_parser,
)
)
self.default_chat_template_kwargs: dict[str, Any] = (
default_chat_template_kwargs or {}
)
@@ -193,7 +187,7 @@ class OpenAIServingRender:
"""
tokenizer = self.renderer.tokenizer
tool_parser = self.tool_parser
tool_parser = self.parser.tool_parser_cls if self.parser is not None else None
if is_mistral_tokenizer(tokenizer):
# because of issues with pydantic we need to potentially
@@ -252,9 +246,8 @@ class OpenAIServingRender:
default_template_content_format=self.chat_template_content_format,
default_template_kwargs=self.default_chat_template_kwargs,
tool_dicts=tool_dicts,
tool_parser=tool_parser,
parser=self.parser,
skip_mm_cache=skip_mm_cache,
reasoning_parser=self.reasoning_parser,
)
else:
# For GPT-OSS.
@@ -526,8 +519,7 @@ class OpenAIServingRender:
default_template_content_format: ChatTemplateContentFormatOption,
default_template_kwargs: dict[str, Any] | None,
tool_dicts: list[dict[str, Any]] | None = None,
tool_parser: type[ToolParser] | None = None,
reasoning_parser: type[ReasoningParser] | None = None,
parser: type[Parser] | None = None,
*,
skip_mm_cache: bool = False,
) -> tuple[list[ConversationMessage], list[EngineInput]]:
@@ -567,14 +559,6 @@ class OpenAIServingRender:
skip_mm_cache=skip_mm_cache,
)
if reasoning_parser is not None:
tokenizer = renderer.get_tokenizer()
request = reasoning_parser(
tokenizer,
model_config=self.model_config,
chat_template_kwargs=chat_params.chat_template_kwargs,
).adjust_request(request=request)
# tool parsing is done only if a tool_parser has been set and if
# tool_choice is not "none" (if tool_choice is "none" but a tool_parser
# is set, we want to prevent parsing a tool_call hallucinated by the LLM
@@ -582,15 +566,22 @@ class OpenAIServingRender:
# Exception: Mistral grammar-capable tokenizers always call
# adjust_request — even for tool_choice="none" — so that the grammar
# factory can prevent special-token leakage.
if tool_parser is not None:
tool_choice = getattr(request, "tool_choice", "none")
if parser is not None:
tokenizer = renderer.get_tokenizer()
tool_parser = parser.tool_parser_cls
tool_choice = getattr(request, "tool_choice", "none")
is_mistral_grammar_eligible = (
is_mistral_tool_parser(tool_parser)
tool_parser is not None
and is_mistral_tool_parser(tool_parser)
and is_mistral_tokenizer(tokenizer)
and tokenizer.supports_grammar
)
if tool_choice != "none" or is_mistral_grammar_eligible:
should_adjust_request = (
parser.reasoning_parser_cls is not None
or tool_choice != "none"
or is_mistral_grammar_eligible
)
if should_adjust_request:
if not isinstance(request, ChatCompletionRequest | ResponsesRequest):
msg = (
"Tool usage is only supported "
@@ -598,8 +589,13 @@ class OpenAIServingRender:
f"but got {type(request).__name__}"
)
raise NotImplementedError(msg)
request = tool_parser(tokenizer, request.tools).adjust_request(
request=request
request = parser(
tokenizer,
request.tools,
model_config=self.model_config,
chat_template_kwargs=chat_params.chat_template_kwargs,
).adjust_request(
request=request,
)
return conversation, [engine_input]
vllm/envs.py
+6 -7
View File
@@ -200,6 +200,7 @@ if TYPE_CHECKING:
MOONCAKE_REQUESTER_LOCAL_HOSTNAME: str | None = None
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE: int = 163840
VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS: int = 1
VLLM_ENFORCE_STRICT_TOOL_CALLING: bool = True
VLLM_MQ_MAX_CHUNK_BYTES_MB: int = 16
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: int = 300
VLLM_KV_CACHE_LAYOUT: Literal["NHD", "HND"] | None = None
@@ -227,7 +228,6 @@ if TYPE_CHECKING:
VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS: bool = False
VLLM_SYSTEM_START_DATE: str | None = None
VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY: bool = False
VLLM_ENFORCE_STRICT_TOOL_CALLING: bool = False
VLLM_CUSTOM_SCOPES_FOR_PROFILING: bool = False
VLLM_NVTX_SCOPES_FOR_PROFILING: bool = False
VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES: bool = True
@@ -1536,6 +1536,11 @@ environment_variables: dict[str, Callable[[], Any]] = {
"VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS": lambda: int(
os.getenv("VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS", "1")
),
# Enforce function parameter schemas in structural-tag based tool calling.
"VLLM_ENFORCE_STRICT_TOOL_CALLING": lambda: os.getenv(
"VLLM_ENFORCE_STRICT_TOOL_CALLING", "True"
).lower()
in ("true", "1"),
# Control the max chunk bytes (in MB) for the rpc message queue.
# Object larger than this threshold will be broadcast to worker
# processes via zmq.
@@ -1659,12 +1664,6 @@ environment_variables: dict[str, Callable[[], Any]] = {
"VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY": lambda: bool(
int(os.getenv("VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY", "0"))
),
# When 1,the model structural tags will be used to enforce the model
# output conforming to the model's tool-calling format and schema.
# Default 0 (off).
"VLLM_ENFORCE_STRICT_TOOL_CALLING": lambda: bool(
int(os.getenv("VLLM_ENFORCE_STRICT_TOOL_CALLING", "0"))
),
# Add optional custom scopes for profiling, disable to avoid overheads
"VLLM_CUSTOM_SCOPES_FOR_PROFILING": lambda: bool(
int(os.getenv("VLLM_CUSTOM_SCOPES_FOR_PROFILING", "0"))
vllm/parser/abstract_parser.py
+36
View File
@@ -25,6 +25,7 @@ from vllm.entrypoints.openai.responses.protocol import ResponsesRequest
from vllm.logger import init_logger
from vllm.parser.metrics import record_tool_parser_invocation
from vllm.reasoning.abs_reasoning_parsers import ReasoningParser
from vllm.sampling_params import StructuredOutputsParams
from vllm.tokenizers import TokenizerLike
from vllm.tool_parsers.abstract_tool_parser import Tool, ToolParser
from vllm.tool_parsers.streaming import (
@@ -427,10 +428,45 @@ class DelegatingParser(Parser):
) -> ChatCompletionRequest | ResponsesRequest:
if self._reasoning_parser is not None:
request = self._reasoning_parser.adjust_request(request)
if self._tool_parser is not None:
request = self._apply_structural_tag(request)
if self._tool_parser is not None:
request = self._tool_parser.adjust_request(request)
return request
def _apply_structural_tag(
self, request: ChatCompletionRequest | ResponsesRequest
) -> ChatCompletionRequest | ResponsesRequest:
if (
not isinstance(request, ChatCompletionRequest)
or self._tool_parser is None
or self._tool_parser.structural_tag_model is None
or not request.tools
):
return request
need_tool_calling = (
request.tool_choice == "auto"
or request.tool_choice == "required"
or isinstance(request.tool_choice, ChatCompletionNamedToolChoiceParam)
)
if not need_tool_calling:
return request
structure_tag = self._tool_parser.get_structural_tag(
request,
reasoning=False,
)
if structure_tag is None:
return request
structural_tag = json.dumps(structure_tag.model_dump())
request.structured_outputs = StructuredOutputsParams(
structural_tag=structural_tag,
)
request.response_format = None
return request
def extract_reasoning_streaming(
self,
previous_text: str,
vllm/tool_parsers/__init__.py
+4 -4
View File
@@ -119,8 +119,8 @@ _TOOL_PARSERS_TO_REGISTER = {
"LongcatFlashToolParser",
),
"mimo": (
"qwen3xml_tool_parser",
"Qwen3XMLToolParser",
"qwen3coder_tool_parser",
"Qwen3CoderToolParser",
),
"minimax_m2": (
"minimax_m2_tool_parser",
@@ -159,8 +159,8 @@ _TOOL_PARSERS_TO_REGISTER = {
"Qwen3CoderToolParser",
),
"qwen3_xml": (
"qwen3xml_tool_parser",
"Qwen3XMLToolParser",
"qwen3coder_tool_parser",
"Qwen3CoderToolParser",
),
"seed_oss": (
"seed_oss_tool_parser",
vllm/tool_parsers/abstract_tool_parser.py
+35 -27
View File
@@ -6,6 +6,7 @@ import json
import os
from collections.abc import Callable, Sequence
from functools import cached_property
from typing import Any
from openai.types.responses import (
ResponseFormatTextJSONSchemaConfig,
@@ -13,8 +14,8 @@ from openai.types.responses import (
)
from openai.types.responses.function_tool import FunctionTool
import vllm.envs as envs
from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionNamedToolChoiceParam,
ChatCompletionRequest,
ChatCompletionToolsParam,
)
@@ -25,7 +26,6 @@ from vllm.entrypoints.openai.engine.protocol import (
from vllm.entrypoints.openai.responses.protocol import (
ResponsesRequest,
)
from vllm.envs import VLLM_ENFORCE_STRICT_TOOL_CALLING
from vllm.logger import init_logger
from vllm.sampling_params import (
StructuredOutputsParams,
@@ -57,6 +57,17 @@ class ToolParser:
# extract_tool_calls / extract_tool_calls_streaming methods for
# required/named tool_choice, treating them the same as "auto".
supports_required_and_named: bool = True
# xgrammar builtin structural tag model key. Subclasses set this when
# their parsed tool-call syntax matches a builtin xgrammar format.
structural_tag_model: str | None = None
def __init_subclass__(cls, **kwargs: Any) -> None:
super().__init_subclass__(**kwargs)
if (
cls.structural_tag_model is not None
and envs.VLLM_ENFORCE_STRICT_TOOL_CALLING
):
cls.supports_required_and_named = False
def __init__(
self,
@@ -112,32 +123,16 @@ class ToolParser:
if not request.tools:
return request
# Step 1 (highest priority for ChatCompletionRequest): apply
# vLLM-owned structural tag support for model-specific tool formats.
# Set structured output params when tool constraints are derived from
# the tool schema. Unified parsers handle model-specific structural
# tags before calling into the tool parser.
structured_outputs = getattr(request, "structured_outputs", None)
if (
isinstance(request, ChatCompletionRequest)
and VLLM_ENFORCE_STRICT_TOOL_CALLING
structured_outputs is not None
and structured_outputs.structural_tag is not None
):
need_tool_calling = (
request.tool_choice == "auto"
or request.tool_choice == "required"
or isinstance(request.tool_choice, ChatCompletionNamedToolChoiceParam)
)
if need_tool_calling:
structure_tag = self.get_structural_tag(request)
if structure_tag is not None:
if request.structured_outputs is None:
request.structured_outputs = StructuredOutputsParams(
structural_tag=json.dumps(structure_tag.model_dump()),
)
else:
request.structured_outputs.structural_tag = json.dumps(
structure_tag.model_dump()
)
return request
return request
# Step 2: set structured output params when tool constraints are
# derived from the tool schema.
json_schema_from_tool = get_json_schema_from_tools(
tool_choice=request.tool_choice, tools=request.tools
)
@@ -169,8 +164,21 @@ class ToolParser:
return request
def get_structural_tag(self, request: ChatCompletionRequest):
return None
def get_structural_tag(
self, request: ChatCompletionRequest, *, reasoning: bool = False
):
if self.structural_tag_model is None:
return None
if not envs.VLLM_ENFORCE_STRICT_TOOL_CALLING:
return None
from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
return get_model_structural_tag(
model=self.structural_tag_model,
tools=request.tools,
tool_choice=request.tool_choice,
reasoning=reasoning,
)
def extract_tool_calls(
self, model_output: str, request: ChatCompletionRequest
vllm/tool_parsers/deepseekv31_tool_parser.py
+2
View File
@@ -25,6 +25,8 @@ logger = init_logger(__name__)
class DeepSeekV31ToolParser(ToolParser):
structural_tag_model = "deepseek_v3_1"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/deepseekv32_tool_parser.py
+1
View File
@@ -53,6 +53,7 @@ class DeepSeekV32ToolParser(ToolParser):
tool_call_start_token: str = "<｜DSML｜function_calls>"
tool_call_end_token: str = "</｜DSML｜function_calls>"
structural_tag_model = "deepseek_v3_2"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/deepseekv3_tool_parser.py
+2
View File
@@ -28,6 +28,8 @@ logger = init_logger(__name__)
class DeepSeekV3ToolParser(ToolParser):
structural_tag_model = "deepseek_r1"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/deepseekv4_tool_parser.py
+1 -15
View File
@@ -1,14 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionRequest,
)
from vllm.tool_parsers.deepseekv32_tool_parser import DeepSeekV32ToolParser
from vllm.tool_parsers.structural_tag_registry import (
get_enable_structured_outputs_in_reasoning,
get_model_structural_tag,
)
class DeepSeekV4ToolParser(DeepSeekV32ToolParser):
@@ -21,11 +14,4 @@ class DeepSeekV4ToolParser(DeepSeekV32ToolParser):
tool_call_start_token: str = "<｜DSML｜tool_calls>"
tool_call_end_token: str = "</｜DSML｜tool_calls>"
def get_structural_tag(self, request: ChatCompletionRequest):
return get_model_structural_tag(
model="deepseek_v4",
tools=request.tools,
tool_choice=request.tool_choice,
reasoning=get_enable_structured_outputs_in_reasoning(),
)
structural_tag_model = "deepseek_v4"
vllm/tool_parsers/glm47_moe_tool_parser.py
+1
View File
@@ -24,6 +24,7 @@ logger = init_logger(__name__)
class Glm47MoeModelToolParser(Glm4MoeModelToolParser):
supports_required_and_named = False
structural_tag_model = "glm_4_7"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/hermes_tool_parser.py
+1
View File
@@ -32,6 +32,7 @@ logger = init_logger(__name__)
class Hermes2ProToolParser(ToolParser):
structural_tag_model = "hermes"
tool_call_start_token: str = "<tool_call>"
tool_call_end_token: str = "</tool_call>"
tool_call_regex = re.compile(
vllm/tool_parsers/kimi_k2_tool_parser.py
+2
View File
@@ -29,6 +29,8 @@ logger = init_logger(__name__)
class KimiK2ToolParser(ToolParser):
structural_tag_model = "kimi"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/llama_tool_parser.py
+1
View File
@@ -46,6 +46,7 @@ class Llama3JsonToolParser(ToolParser):
"""
bot_token: str = "<|python_tag|>"
structural_tag_model = "llama"
# Simple regex to find opening braces - we'll use JSON decoder for parsing
# This handles arbitrary nesting depth correctly
tool_call_start_regex: re.Pattern = re.compile(r"\{")
vllm/tool_parsers/minimax_m2_tool_parser.py
+2
View File
@@ -34,6 +34,8 @@ logger = init_logger(__name__)
class MinimaxM2ToolParser(ToolParser):
structural_tag_model = "minimax"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
vllm/tool_parsers/qwen3coder_tool_parser.py
+1 -14
View File
@@ -18,17 +18,12 @@ from vllm.entrypoints.openai.engine.protocol import (
FunctionCall,
ToolCall,
)
from vllm.envs import VLLM_ENFORCE_STRICT_TOOL_CALLING
from vllm.logger import init_logger
from vllm.tokenizers import TokenizerLike
from vllm.tool_parsers.abstract_tool_parser import (
Tool,
ToolParser,
)
from vllm.tool_parsers.structural_tag_registry import (
get_enable_structured_outputs_in_reasoning,
get_model_structural_tag,
)
from vllm.tool_parsers.utils import (
coerce_to_schema_type,
extract_types_from_schema,
@@ -39,7 +34,7 @@ logger = init_logger(__name__)
class Qwen3CoderToolParser(ToolParser):
supports_required_and_named: bool = not VLLM_ENFORCE_STRICT_TOOL_CALLING
structural_tag_model = "qwen_3_coder"
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
super().__init__(tokenizer, tools)
@@ -589,11 +584,3 @@ class Qwen3CoderToolParser(ToolParser):
return result
return None
def get_structural_tag(self, request: ChatCompletionRequest):
return get_model_structural_tag(
model="qwen_3_5",
tools=request.tools,
tool_choice=request.tool_choice,
reasoning=get_enable_structured_outputs_in_reasoning(),
)
vllm/tool_parsers/qwen3xml_tool_parser.py
-1300
View File
File diff suppressed because it is too large Load Diff
vllm/tool_parsers/structural_tag_registry.py
+195 -261
View File
@@ -1,14 +1,15 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# Model-specific structural tag builders adapted from XGrammar's
# builtin structural tag implementations:
# https://github.com/mlc-ai/xgrammar/blob/main/python/xgrammar/builtin_structural_tag.py
from collections.abc import Callable
from typing import Any, Literal
from xgrammar import StructuralTag
from xgrammar import StructuralTag, normalize_tool_choice
from xgrammar import get_model_structural_tag as get_xgrammar_model_structural_tag
from xgrammar.openai_tool_call_schema import (
BuiltinToolParam,
FunctionToolParam,
)
from xgrammar.structural_tag import (
AnyTextFormat,
ConstStringFormat,
@@ -24,23 +25,51 @@ from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionToolsParam,
)
SimplifiedToolChoice = Literal["auto", "required", "forced"]
ToolChoice = (
Literal["none", "auto", "required"] | ChatCompletionNamedToolChoiceParam | None
)
SimplifiedToolChoice = Literal["auto", "required", "forced"]
StructuralTagBuilder = Callable[
[list[ChatCompletionToolsParam], SimplifiedToolChoice, bool],
[
list[FunctionToolParam],
list[BuiltinToolParam],
SimplifiedToolChoice,
bool,
],
StructuralTag,
]
_structural_tag_registry: dict[str, StructuralTagBuilder] = {}
# Keep this list in sync with xgrammar.builtin_structural_tag. It is used for
# vLLM-side validation and for documenting the xgrammar builtin surface that
# can be requested by tool parsers through ``structural_tag_model``.
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset(
{
"llama",
"kimi",
"deepseek_r1",
"deepseek_v3_1",
"qwen_3_5",
"qwen_3_coder",
"qwen_3",
"harmony",
"deepseek_v3_2",
"glm_4_7",
"deepseek_v4",
}
)
VLLM_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset({"hermes"})
SUPPORTED_STRUCTURAL_TAG_MODELS = (
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
)
_VLLM_STRUCTURAL_TAG_REGISTRY: dict[str, StructuralTagBuilder] = {}
def register_model_structural_tag(name: str):
"""Register a vLLM-owned model-specific structural tag builder."""
def register_vllm_structural_tag(model: str):
"""Register a vLLM-owned structural tag builder."""
def decorator(func: StructuralTagBuilder) -> StructuralTagBuilder:
_structural_tag_registry[name] = func
_VLLM_STRUCTURAL_TAG_REGISTRY[model] = func
return func
return decorator
@@ -52,279 +81,184 @@ def get_model_structural_tag(
tool_choice: ToolChoice,
reasoning: bool,
) -> StructuralTag | None:
"""Build a structural tag from vLLM-owned model-specific builders."""
"""Build a structural tag with xgrammar's builtin model templates."""
builder = _structural_tag_registry.get(model)
if builder is None:
supported = list(_structural_tag_registry.keys())
raise ValueError(f"Unknown format type: {model}, supported types: {supported}")
normalized_tools, simplified_tool_choice = _normalize_tool_choice(
tools=tools,
tool_choice=tool_choice,
)
if not normalized_tools:
if not tools or tool_choice == "none":
return None
return builder(normalized_tools, simplified_tool_choice, reasoning)
dumped_tools = [_model_dump(tool) for tool in tools]
dumped_tool_choice = _model_dump(tool_choice)
if model in _VLLM_STRUCTURAL_TAG_REGISTRY:
function_tools, builtin_tools, simplified_tool_choice = normalize_tool_choice(
dumped_tools,
dumped_tool_choice,
)
return _VLLM_STRUCTURAL_TAG_REGISTRY[model](
function_tools,
builtin_tools,
simplified_tool_choice,
reasoning,
)
if model not in XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS:
supported = sorted(SUPPORTED_STRUCTURAL_TAG_MODELS)
raise ValueError(f"Unknown format type: {model}, supported types: {supported}")
return get_xgrammar_model_structural_tag(
model=model,
tools=dumped_tools,
tool_choice=dumped_tool_choice,
reasoning=reasoning,
)
def _normalize_tool_choice(
tools: list[ChatCompletionToolsParam] | None,
tool_choice: ToolChoice,
) -> tuple[list[ChatCompletionToolsParam], SimplifiedToolChoice]:
"""Normalize vLLM ChatCompletion tool_choice for structural tag builders."""
def _model_dump(value: Any) -> Any:
"""Convert vLLM/Pydantic request objects to xgrammar's dict protocol."""
if not tools:
return [], "auto"
if tool_choice is None or tool_choice == "none":
return [], "auto"
if tool_choice == "auto":
return tools, "auto"
if tool_choice == "required":
return tools, "required"
if isinstance(tool_choice, ChatCompletionNamedToolChoiceParam):
tool_name = tool_choice.function.name
filtered_tools = [tool for tool in tools if tool.function.name == tool_name]
if not filtered_tools:
raise ValueError(
f"The tool with name '{tool_name}' is not found in the tools list."
)
return filtered_tools, "forced"
raise ValueError(f"Unsupported tool_choice for structural tag: {tool_choice}")
if hasattr(value, "model_dump"):
return value.model_dump(exclude_none=True)
return value
def _get_function_parameters(function: Any) -> dict[str, Any] | bool:
"""Return the JSON schema used for constrained tool arguments."""
def _get_function_parameters(function) -> dict[str, Any] | bool:
if getattr(function, "strict", None) is False:
return True
if function.parameters is None:
return True
return function.parameters
return function.parameters if function.parameters is not None else True
_enable_structured_outputs_in_reasoning: bool = False
def _hermes_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]:
arguments_field_prefix = '", "arguments": '
formats = [
# <tool_call>
# {"name": "t1", "arguments": {"q": "v"}}
# </tool_call>
('<tool_call>\n{"name": "', "}\n</tool_call>"),
# <tool_call>{"name": "t1", "arguments": {"q": "v"}}</tool_call>
('<tool_call>{"name": "', "}</tool_call>"),
]
def set_enable_structured_outputs_in_reasoning(enabled: bool) -> None:
"""Publish the engine's ``enable_in_reasoning`` flag to tool parsers.
Called once during APIServer startup so request-time parsers can read
it without going through the EngineCore-only contextvar.
"""
global _enable_structured_outputs_in_reasoning
_enable_structured_outputs_in_reasoning = bool(enabled)
def get_enable_structured_outputs_in_reasoning() -> bool:
"""Whether structured outputs are active during the reasoning phase.
When ``True``, the structural tag will cover the reasoning part:
``<think>...</think>`` prefix (if available); when ``False`` (default), the tag only
constrains the post-reasoning suffix.
"""
return _enable_structured_outputs_in_reasoning
@register_model_structural_tag("deepseek_v4")
def get_deepseek_v4_structural_tag(
tools: list[ChatCompletionToolsParam],
tool_choice: SimplifiedToolChoice,
reasoning: bool,
) -> StructuralTag:
"""Build DeepSeek V4 structural tags."""
invoke_begin_prefix = '<｜DSML｜invoke name="'
invoke_begin_suffix = '">\n'
invoke_end = "</｜DSML｜invoke>\n"
tool_calls_prefix = "\n\n"
function_calls_begin = "<｜DSML｜tool_calls>\n"
function_calls_end = "</｜DSML｜tool_calls>"
function_calls_trigger = "<｜DSML｜tool_calls>"
think_tag_end = "</think>"
think_exclude_tokens = ["<think>", "</think>"]
xml_style = "deepseek_xml"
if tool_choice == "auto":
tags = []
for tool in tools:
function = tool.function
parameters = _get_function_parameters(function)
tags.append(
TagFormat(
begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
content=JSONSchemaFormat(
json_schema=parameters,
style=xml_style,
),
end=invoke_end,
)
)
if tags:
function_calling_tags = TagsWithSeparatorFormat(
tags=tags,
separator="\n",
at_least_one=True,
)
suffix_tag = TriggeredTagsFormat(
triggers=[function_calls_trigger],
tags=[
TagFormat(
begin=function_calls_begin,
content=function_calling_tags,
end=function_calls_end,
)
],
excludes=think_exclude_tokens,
)
else:
suffix_tag = AnyTextFormat(excludes=think_exclude_tokens)
elif tool_choice == "forced":
if not tools:
raise ValueError("Forced tool choice must resolve to exactly one tool.")
function = tools[0].function
suffix_tag = SequenceFormat(
elements=[
ConstStringFormat(value=tool_calls_prefix + function_calls_begin),
TagFormat(
begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
content=JSONSchemaFormat(
json_schema=_get_function_parameters(function),
style=xml_style,
),
end=invoke_end,
),
ConstStringFormat(value=function_calls_end),
]
)
elif tool_choice == "required":
tags = []
for tool in tools:
function = tool.function
parameters = _get_function_parameters(function)
tags.append(
TagFormat(
begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
content=JSONSchemaFormat(
json_schema=parameters,
style=xml_style,
),
end=invoke_end,
)
)
assert len(tags) > 0
suffix_tag = SequenceFormat(
elements=[
ConstStringFormat(value=tool_calls_prefix + function_calls_begin),
TagsWithSeparatorFormat(
tags=tags,
separator="\n",
at_least_one=True,
),
ConstStringFormat(value=function_calls_end),
]
)
if not reasoning:
return StructuralTag(format=suffix_tag)
prefix_tag = TagFormat(begin="", content=AnyTextFormat(), end=think_tag_end)
return StructuralTag(format=SequenceFormat(elements=[prefix_tag, suffix_tag]))
@register_model_structural_tag("qwen_3_5")
def get_qwen_3_5_structural_tag(
tools: list[ChatCompletionToolsParam],
tool_choice: SimplifiedToolChoice,
reasoning: bool,
) -> StructuralTag:
"""Build Qwen XML structural tags.
This format is used for Qwen3-Coder/Qwen3.5/Qwen3.6 and is compatible with
Qwen variants that use the same XML tool-call format.
"""
tool_call_begin_prefix = "<tool_call>\n<function="
tool_call_begin_suffix = ">\n"
tool_call_end = "\n</function>\n</tool_call>"
tool_call_trigger = "<tool_call>\n<function="
think_tag_end = "</think>"
think_suffix = "\n\n"
think_exclude_tokens = ["<think>", "</think>"]
if tool_choice == "auto":
tags = []
for tool in tools:
function = tool.function
parameters = _get_function_parameters(function)
tags.append(
TagFormat(
begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
content=JSONSchemaFormat(json_schema=parameters, style="qwen_xml"),
end=tool_call_end,
)
)
if tags:
suffix_tag = TriggeredTagsFormat(
triggers=[tool_call_trigger],
tags=tags,
excludes=think_exclude_tokens,
)
else:
suffix_tag = AnyTextFormat(excludes=think_exclude_tokens)
elif tool_choice == "forced":
if not tools:
raise ValueError("Forced tool choice must resolve to exactly one tool.")
function = tools[0].function
suffix_tag = TagFormat(
begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
return [
TagFormat(
begin=begin + tool.function.name + arguments_field_prefix,
content=JSONSchemaFormat(
json_schema=_get_function_parameters(function),
style="qwen_xml",
json_schema=_get_function_parameters(tool.function)
),
end=tool_call_end,
end=end,
)
for tool in tools
for begin, end in formats
]
elif tool_choice == "required":
tags = []
for tool in tools:
function = tool.function
parameters = _get_function_parameters(function)
tags.append(
TagFormat(
begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
content=JSONSchemaFormat(json_schema=parameters, style="qwen_xml"),
end=tool_call_end,
)
)
assert len(tags) > 0
@register_vllm_structural_tag("hermes")
def get_hermes_structural_tag(
tools: list[FunctionToolParam],
builtin_tools: list[BuiltinToolParam],
tool_choice: SimplifiedToolChoice,
reasoning: bool,
) -> StructuralTag:
del builtin_tools, reasoning
tool_call_trigger = "<tool_call>"
if tool_choice == "auto":
tags = _hermes_tool_tags(tools)
suffix_tag = (
TriggeredTagsFormat(triggers=[tool_call_trigger], tags=tags)
if tags
else AnyTextFormat()
)
elif tool_choice == "forced":
suffix_tag = TagsWithSeparatorFormat(
tags=tags,
tags=_hermes_tool_tags(tools),
separator="",
at_least_one=True,
stop_after_first=True,
)
else:
suffix_tag = TagsWithSeparatorFormat(
tags=_hermes_tool_tags(tools),
separator="",
at_least_one=True,
)
if not reasoning:
result = StructuralTag(format=suffix_tag)
else:
prefix_tag = SequenceFormat(
return StructuralTag(format=suffix_tag)
def _minimax_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]:
return [
TagFormat(
begin=f'<invoke name="{tool.function.name}">\n',
content=JSONSchemaFormat(
json_schema=_get_function_parameters(tool.function),
style="minimax_xml",
),
end="</invoke>\n",
)
for tool in tools
]
@register_vllm_structural_tag("minimax")
def get_minimax_structural_tag(
tools: list[FunctionToolParam],
builtin_tools: list[BuiltinToolParam],
tool_choice: SimplifiedToolChoice,
reasoning: bool,
) -> StructuralTag:
del builtin_tools, reasoning
tool_call_begin = "<minimax:tool_call>\n"
tool_call_end = "</minimax:tool_call>"
tool_call_trigger = "<minimax:tool_call>"
tags = _minimax_tool_tags(tools)
if tool_choice == "auto":
suffix_tag = (
TriggeredTagsFormat(
triggers=[tool_call_trigger],
tags=[
TagFormat(
begin=tool_call_begin,
content=TagsWithSeparatorFormat(
tags=tags,
separator="",
at_least_one=True,
),
end=tool_call_end,
)
],
excludes=["<think>", "</think>"],
)
if tags
else AnyTextFormat(excludes=["<think>", "</think>"])
)
elif tool_choice == "forced":
suffix_tag = SequenceFormat(
elements=[
TagFormat(begin="", content=AnyTextFormat(), end=think_tag_end),
ConstStringFormat(value=think_suffix),
ConstStringFormat(value="\n" + tool_call_begin),
TagsWithSeparatorFormat(
tags=tags,
separator="",
at_least_one=True,
stop_after_first=True,
),
ConstStringFormat(value=tool_call_end),
]
)
else:
suffix_tag = SequenceFormat(
elements=[
ConstStringFormat(value="\n" + tool_call_begin),
TagsWithSeparatorFormat(
tags=tags,
separator="",
at_least_one=True,
),
ConstStringFormat(value=tool_call_end),
]
)
result = StructuralTag(format=SequenceFormat(elements=[prefix_tag, suffix_tag]))
return result
return StructuralTag(format=suffix_tag)