OpenAI's Jalapeño chip targets fast inference at scale, first benchmarks show
OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show
OpenAI shared the first benchmarks for its in-house inference chip, Jalapeño, at Hot Chips. On SemiAnalysis' InferenceX test, it delivered more tokens per user and higher throughput per kilowatt than the current state-of-the-art. The post doesn't name the competitor or disclose latency figures. I'd hold off until third-party numbers land.
Why it matters: OpenAI presented first test results for its in-house inference chip Jalapeño at Hot Chips, and SemiAnalysis's InferenceX benchmark shows it beating the strongest current competitor on tokens per user and throughput per watt.