Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

563 picksRelated topicsOpen source Data & training On-device AI

Latest picks

561–563 of 563

Oct 1, 2024Tuesday

OpenAI NewsOct 1, 18:03

Prompt Caching in the API

OpenAI added automatic prompt caching to GPT-4o, GPT-4o mini, o1-preview, and o1-mini API models, giving a 50% discount on recently reused input prefixes. Caching starts at 1,024 tokens and grows in 128-token increments; caches are often cleared after 5-10 minutes of inactivity and always within 1 hour of last use. The field to watch is cached_tokens in the API usage response.

Why it matters: Halves input cost for repeated long prefixes, which changes how long-context apps are priced.

Aug 6, 2024Tuesday

OpenAI NewsAug 6, 18:00

Introducing Structured Outputs in the API

OpenAI released Structured Outputs on Aug 6, 2024, making model outputs conform to developer-supplied JSON Schemas; `gpt-4o-2024-08-06` scored 100% on complex schema-following evals versus under 40% for `gpt-4-0613`. The feature is enabled with `strict: true` in function calling and works on tool-supporting models including `gpt-4-0613`, `gpt-3.5-turbo-0613`, and later. The key shift is constrained decoding plus schema training, not just valid JSON from JSON mode.

Why it matters: Structured Outputs makes models follow a developer's JSON Schema exactly, and the accuracy gap between new and old models shows why schema adherence is a model capability, not a prompt trick.

Aug 2, 2024Friday

DeepSeek · API updatesAug 2, 00:00

DeepSeek API adds disk caching, cutting prices by another order of magnitude

DeepSeek API 上线硬盘缓存技术，价格再降一个数量级。更新详情见文档 https://api-docs.deepseek.com/zh-cn/updates#api-上线硬盘缓存技术。

Why it matters: DeepSeek's API now uses disk caching and prices fell by another order of magnitude, so the direct effect of this caching update lands on API usage cost.