New Tool Called GigaToken Says It Processes Text 1,000x Faster

GigaToken, a tool by Stanford PhD student Marcel Rød, claims to process text roughly 1,000x faster than the popular HuggingFace library. Text processing here means splitting words into pieces AI models can read. It works as a direct replacement. On an M4 Max, benchmarks show 1,887 million pieces per second versus HuggingFace's 1.40 million — a 1,353x speedup.