tokenizers v1 encodes 3–30× faster than v0.23, authors report

by

Published September 21, 2026. Authors Arthur Zucker, Simon Brandeis, Luc Georges and Lysandre describe a refactor of tokenizers that aims to remove tokenization as a performance bottleneck in large-scale ML workflows. The release-candidate v1 preserves the same token IDs, API, vocabulary and merge ranks as v0.23 while focusing on throughput and parallelism.

Measured improvements

The authors report that, across ten model families, v1’s encode path runs 3 to 30 times faster than v0.23 on a single Apple M4 Max core, with t5-base at the low end and gpt2 at the high end. They measured scaling at about 76% of linear when using eight workers. Benchmarks were run from the tokbench repository and include single-threaded, multi-threaded, latency, decoding throughput and memory measurements; the post details the rules used to ensure consistent comparisons.

Technical changes that reduced work

The post summarizes several coordinated changes. Tokenization still runs in four stages: normalization, pre-tokenization, the model stage and post-processing, and most speedups occur in the model stage. Key changes include:

  • bitcannon: hand-written splitters replace the regex engine for common grammars, using bitstream boolean operations and SIMD to find splits instead of a per-character scan. bitcannon covers GPT-2, cl100k, o200k, Tekken and DeepSeek.
  • WordCache: a thread-local cache maps pre-token bytes to token ID sequences so repeated pre-tokens skip the merge work.
  • No-alloc model and merge-loop rewrite: temporary model state moves to caller-owned scratch buffers, merges use an intrusive linked layout and candidate pairs are packed to allow branchless comparisons.
  • Batching and native parallelism: one model call can process multiple pre-token spans and a shared tokenizer serves many threads using per-thread scratch and cache sub-pools to avoid a single lock.

Availability and next steps

The release candidate is published on crates.io. The authors note training remains behind a default-on feature that brings in a C++ dependency; applications that only need encoding can exclude training with –no-default-features. Example install and encode calls are:

cargo add tokenizers --pre
cargo add tokenizers --pre --no-default-features --features http

use tokenizers::tokenizer::Tokenizer;
let tokenizer = Tokenizer::from_pretrained("deepseek-ai/DeepSeek-V4-Flash", None)?;
let encoding = tokenizer.encode("The tokenizer is no longer the bottleneck.", false)?;

The post lists remaining work toward 1.0.0, including adding more model families to the new merge loop, reworking normalizers, simpler Python bindings, and optional offsets and masks, and describes experimental directions such as device-side encoding and bindings for additional languages.

Original source: Hugging Face Blog

Leave a Comment