The Inference Engineering Masterclass with Baseten's Philip Kiely and Ali Taha
The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten
Baseten just raised a $13B Series F. Philip Kiely and Ali Taha explain why inference engineering is now its own discipline, covering quantization, speculative decoding, KV-cache movement, and disaggregated prefill/decode. In one GLM-5.2 experiment, quantizing more layers preserved benchmark quality while boosting throughput 20% because errors across layers canceled out. They also detail grafting Kimi's vision encoder onto GLM-5.2 without touching the language model, and note that inference optimizations can still deliver 20% to 200% gains. The conversation touches on NVIDIA Dynamo, Rubin, video generation, and local inference, but the post doesn't expand on those.
Why it matters: Baseten's Philip Kiely and Ali Taha break down quantization, speculative decoding and KV cache movement, with a GLM-5.2 test showing no benchmark quality loss after multi-layer quantization and 20% higher throughput.