wang.yuqiandGitHub 5895afd780 [Bugfix] The special_tokens in tokenizer should also be controlled by do_lower_case in encoder_config. (#20750) ...

Signed-off-by: wang.yuqi <noooop@126.com>

2025-07-18 09:10:47 +00:00

..

__init__.py

Asynchronous tokenization (#2879)

2024-03-15 23:37:01 +00:00

test_cached_tokenizer.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

test_detokenize.py

Support embedding models in V1 (#16188)

2025-06-18 21:36:33 -07:00

test_do_lower_case.py

[Bugfix] The special_tokens in tokenizer should also be controlled by do_lower_case in encoder_config. (#20750)

2025-07-18 09:10:47 +00:00

test_get_eos.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

test_mistral_tokenizer.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

test_tokenizer_group.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

test_tokenizer_registry.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

test_tokenizer.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00