AI coding

OpenAI has opened GPT-4o fine-tuning to developers on all paid tiers, with 1M free training tokens per org per day through September 23. Training costs $25 per 1M tokens, and inference costs $3.75 per 1M input tokens and $15 per 1M output tokens on gpt-4o-2024-08-06. The signal for practitioners: partners reported 43.8% on SWE-bench Verified and 71.83% on BIRD-SQL with fine-tuned GPT-4o.

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

OpenAI released GPT-4o mini on July 18, 2024 at $0.15 per 1M input tokens and $0.60 per 1M output tokens, replacing GPT-3.5 in ChatGPT. It supports text and vision, offers a 128K context window and 16K max output, scores 82.0% on MMLU and 87.2% on HumanEval. The key detail for builders is that its API version is the first to use instruction hierarchy against jailbreaks and prompt injection.

The deepseek-chat model has been upgraded to DeepSeek-V2-0628. Reasoning improved, and role-play got a clear boost. HumanEval Pass@1 rose from 79.88% to 84.76%, MATH ACC@1 from 55.02% to 71.02%, and BBH from 78.56% to 83.40%.

The deepseek-coder model has been upgraded to DeepSeek-Coder-V2-0614, with a clear gain in coding ability. The company says its code generation, code understanding, code repair and code completion reach the level of GPT-4-Turbo-0409, with strong math and reasoning, while general ability matches DeepSeek-V2-0517.