Nick HillandGitHub a96197f564 [Perf] Simplify DeepseekV32 tokenizer, ensure fast detokenization used (#33855) ...

Signed-off-by: Nick Hill <nickhill123@gmail.com>

2026-02-08 07:16:34 +00:00

..

__init__.py

[Misc] Refactor tokenizer interface (#29693)

2025-11-29 04:02:21 -08:00

test_basic.py

[Perf] Simplify DeepseekV32 tokenizer, ensure fast detokenization used (#33855)

2026-02-08 07:16:34 +00:00

test_detokenize.py

[CI Failure] Disable mosaicml/mpt-7b and databricks/dbrx-instruct tests (#31182)

2025-12-22 15:40:35 -08:00

test_hf.py

[Misc] Update TokenizerLike interface and move get_cached_tokenizer (#29730)

2025-11-30 14:59:47 +08:00

test_mistral.py

Add Mistral Large 3 and Ministral 3 (#29757)

2025-12-02 10:29:00 +00:00

test_registry.py

[Refactor] TokenizerRegistry only uses lazy imports (#30609)

2025-12-13 23:16:22 +08:00