GigaToken: Rust Library Claims 1,000x Faster Tokenization as HuggingFace Drop-In Replacement

GigaToken, a Rust-based tokenization library by Stanford PhD student Marcel Rød, claims roughly 1,000x faster throughput than HuggingFace tokenizers while serving as a drop-in replacement. In HuggingFace compatibility mode, outputs match exactly. Benchmarks show 1,887 million tokens/s on an M4 Max versus 1.40 million for HuggingFace — a 1,353x speedup.