Tokenization — the preprocessing step that converts raw text into the numerical token sequences language models actually consume — remains one of the least-profiled components in the ML stack. As training datasets grow to terabyte and petabyte scales, the time spent on this conversion can become a practical bottleneck in data preparation pipelines, yet it is rarely the focus of optimization efforts. Gigatoken, a new library released by Marcel Rød, a PhD student at Stanford, under an MIT license, makes the case that this oversight comes at a cost. The library achieves text encoding at gigabytes per second on a single machine, outperforming baselines that are already implemented in multithreaded Rust.

Benchmark Results

Benchmarked on the GPT-2 tokenizer using the 11.9 GB owt_train.txt corpus on a 144-core AMD EPYC 9565 dual-socket system, Gigatoken processes data at 24.53 GB/s. On the same hardware, OpenAI's tiktoken — the library used to encode text for GPT models — achieves 36.0 MB/s and HuggingFace tokenizers — the de facto standard tokenizer across the open-source ML ecosystem — registers 24.8 MB/s, translating to performance advantages of 681x and 989x, respectively.

The speedup is consistent across architectures. On an Apple M4 Max with 16 cores, the same GPT-2 workload runs at 8.79 GB/s — 1,268x faster than HuggingFace tokenizers and 140x faster than tiktoken. On a consumer-grade AMD Ryzen 7 9800X3D, Gigatoken achieves 6.27 GB/s, representing 106x and 68x speedups over the respective baselines.

What Is Gigatoken

Gigatoken is a byte-pair encoding (BPE) tokenizer written in Rust with Python bindings. BPE is the dominant subword tokenization algorithm for modern LLMs, used by the GPT, Llama, Qwen, and Mistral families, among others. It is available on PyPI as gigatoken (version 0.9.0, released 21 July 2026) and can be installed via pip install gigatoken. The codebase is 66.2% Rust and 33.3% Python. Its published benchmarks cover 23 distinct tokenizer families, including GPT-2, GPT-OSS, Llama 3 through 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma, and Mistral.

Gigatoken offers two usage modes. Compatibility mode wraps an existing HuggingFace or tiktoken tokenizer, preserving exact output parity at a cost to throughput. Rød stated on Hacker News that compatibility mode delivers roughly 200–300x speedup depending on usage, because it still incurs Python overhead for list creation and string-to-bytes conversion. The native Gigatoken API, which allows Rust to read files directly, is the source of the published benchmark numbers.

Technical Approach

The performance gains do not stem from a faster BPE merge loop. Instead, they arise from two areas that most tokenizers treat as already solved.

Pretokenization Optimization

Most implementations delegate pretokenization to a regex engine. Gigatoken hand-writes the entire pretokenizer. The pretokenizer optimization log tracks single-threaded GPT-2 pretokenizer throughput on 100 MB of OpenWebText, revealing a detailed progression:

• A fancy-regex baseline runs at approximately 47 MiB/s.
• A hand-rolled state machine reaches approximately 380 MiB/s.
• A winnow-combinator implementation with NEON SIMD intrinsics achieves 462 MiB/s.

Further optimization came from replacing winnow with a direct Iterator, adding a 256-byte class lookup table for O(1) first-byte dispatch, and switching from NEON intrinsics to SWAR (SIMD Within A Register). SWAR loads 8 bytes as a u64 and checks all 8 for the letter property using branchless arithmetic, without requiring architecture-specific intrinsics. This pushed throughput to 830 MiB/s.

The final optimization step involved dual-cursor instruction-level parallelism (ILP) exploitation, reaching 1,049 MiB/s. The key insight was that the bottleneck at approximately 840 MiB/s was latency, not throughput. Each token's end position depends on the previous one, creating a serial chain of roughly 25–27 cycles. By running two independent cursors from a safe split point, the out-of-order execution engine can interleave both streams across idle execution ports.

Net effect on the pretokenizer alone: 2.27x over the winnow + NEON baseline, and 22.3x over the regex implementation.

Pretoken Caching

When a word has been seen before, its encoded tokens are looked up rather than recomputed. Rød notes that this is difficult in practice because the cache grows quickly and pretoken distributions follow a long-tail pattern. Additionally, interactions with Python are minimized, and threads are designed to interact minimally with each other.

The optimization log also documents approaches that did not work. A hot/cold split using #[cold] and #[inline(never)] regressed to 580 MiB/s and was reverted, because the inline barrier prevented LLVM from optimizing the combined ASCII and unicode loop. A two-pass classification buffer with SWAR transition counting was algorithmically correct but ran at 354 MiB/s, as the extra memory traffic outweighed branch savings. Profile-guided optimization had no measurable effect, because the inner loop is already branchless and the word-boundary branch is data-dependent.

Benchmark Methodology

The comparison is not strictly apples-to-apples. Gigatoken encodes entire un-split files, finding its own document boundaries and parallelizing automatically. HuggingFace (encode_batch_fast) is evaluated on the first 100 MB, while tiktoken (encode_ordinary_batch) is evaluated on the first 1 GB, both pre-split on <|endoftext|>. Because the baselines do not implement caching, their throughput remains uniform. All measurements report the best of three interleaved rounds using fresh processes with parallelism enabled.

Vocabulary type also introduces constraints. SentencePiece tokenizers are only partially optimized. On EPYC, Gemma 1 processes at 2.51 GB/s (7.3x speedup), Gemma 3 at 3.43 GB/s (9.6x), and CodeLlama at 3.47 GB/s (10.0x). While substantial, these gains are an order of magnitude lower than the headline BPE performance figures.

An independent reproduction on KrabArena verified the results. On a 4-vCPU Intel Xeon VM (2.20 GHz) with a 174 MB OpenWebText slice, Gigatoken 0.9.0 achieved a median of 277.8 MB/s, outperforming tiktoken 0.13.0 (10.62 MB/s) by 26.2x and tokenizers 0.23.1 (3.33 MB/s) by 83.4x. All trials successfully validated 35,356 documents, confirming that the performance trend scales with core count.

Scope and Limitations

Gigatoken's speedup holds across both x86 and ARM architectures and across all 23 supported tokenizer families, rather than being limited to a single tuned configuration. However, SentencePiece vocabularies see 7–22x speedups rather than the 1,000x-range gains, and WordPiece is not supported. Compatibility mode preserves exact HuggingFace output parity at approximately 200–300x rather than the full native speedup.

The GitHub repository and the benchmark explorer are publicly available.