Perplexity open-sources Unigram tokenizer to reduce CPU usage
Perplexity开源Unigram分词器降低CPU占用
Perplexity open-sourced a rebuilt Unigram tokenizer that reduces CPU usage by 5-6x, targeting tokenization latency when small rerankers and embedding models run on GPUs in single-digit milliseconds.
Why it matters: Perplexity open-sourced a rewritten Unigram tokenizer that cuts CPU tokenization cost 5 to 6 times in reranking and embedding pipelines, so tokenization stops slowing down responses.