Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

582 picksRelated topicsImage generation AI video Voice & audio

Latest picks

581–582 of 582

Jul 18, 2024Thursday

OpenAI NewsJul 18, 18:00

GPT-4o mini: advancing cost-efficient intelligence

OpenAI released GPT-4o mini on July 18, 2024 at $0.15 per 1M input tokens and $0.60 per 1M output tokens, replacing GPT-3.5 in ChatGPT. It supports text and vision, offers a 128K context window and 16K max output, scores 82.0% on MMLU and 87.2% on HumanEval. The key detail for builders is that its API version is the first to use instruction hierarchy against jailbreaks and prompt injection.

Why it matters: GPT-4o mini's price per million input tokens and its replacement of GPT-3.5 in ChatGPT set a new floor for cheap inference, and the instruction hierarchy shows how OpenAI handles jailbreaks.

Sep 25, 2023Monday

OpenAI NewsSep 25, 15:00

ChatGPT can now see, hear, and speak

OpenAI says ChatGPT now supports seeing, hearing, and speaking. The post body is empty, so it does not disclose model versions, rollout timing, regional limits, pricing, or API scope. The real watchpoints are voice latency, vision limits, and access paths.

Why it matters: OpenAI added voice conversation and image understanding to ChatGPT, with speech synthesis from a few seconds of real audio and image input built on multimodal GPT-3.5 and GPT-4.