Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

521–522 of 522

Aug 6, 2024Tuesday

OpenAI News

Introducing Structured Outputs in the API

OpenAI released Structured Outputs on Aug 6, 2024, making model outputs conform to developer-supplied JSON Schemas; `gpt-4o-2024-08-06` scored 100% on complex schema-following evals versus under 40% for `gpt-4-0613`. The feature is enabled with `strict: true` in function calling and works on tool-supporting models including `gpt-4-0613`, `gpt-3.5-turbo-0613`, and later. The key shift is constrained decoding plus schema training, not just valid JSON from JSON mode.

Why it matters: Structured Outputs makes models follow a developer's JSON Schema exactly, and the accuracy gap between new and old models shows why schema adherence is a model capability, not a prompt trick.

Aug 2, 2024Friday

DeepSeek · API updates

DeepSeek API adds disk caching, cutting prices by another order of magnitude

DeepSeek API 上线硬盘缓存技术,价格再降一个数量级。更新详情见文档 https://api-docs.deepseek.com/zh-cn/updates#api-上线硬盘缓存技术 。

Why it matters: DeepSeek's API now uses disk caching and prices fell by another order of magnitude, so the direct effect of this caching update lands on API usage cost.