GPT-4 Omni (GPT-4o)

What is OpenAI GPT-4 Omni?

GPT-4 Omni (GPT-4o) is OpenAI's latest flagship model, launched on May 13, 2024. The "o" in GPT-4o stands for "omni," reflecting its advanced multimodal capabilities that allow it to process and generate text, audio, image, and video inputs and outputs. This model represents a significant leap forward in natural human-computer interaction, offering more seamless and intuitive user experiences across various applications.

GPT-4 Omni (GPT-4o) was OpenAI's flagship model when it launched on May 13, 2024. The "o" in GPT-4o stands for "omni," reflecting its multimodal design: it can accept text, audio, image, and video inputs and generate text, audio, and image outputs. This model represented a significant leap forward in natural human-computer interaction, offering more seamless and intuitive user experiences across various applications.

Performance Benchmarks

GPT-4o exhibits remarkable performance across multiple domains, matching GPT-4 Turbo's capabilities in text processing, reasoning, and coding while simultaneously setting new benchmarks in multilingual understanding, audio processing, and visual perception. These advancements represent a significant leap forward in AI's ability to handle diverse modalities and complex tasks, pushing the boundaries of machine learning in both breadth and depth of understanding.

Launch-context caveat: These benchmark claims describe the May 2024 launch comparison rather than a current cross-model leaderboard.

Long Context Utilization

GPT-4o outperforms GPT-4 Turbo in utilizing long context across various lengths and depths. The comparison results are summarized in the following table:

The repository version archived on June 25, 2024 reported the following wins and ties for GPT-4 Omni 0513 and GPT-4 Turbo 1106. This is an unverified internal comparison, preserved for historical context rather than presented as a current or independently verified benchmark.

GPT-4o's advantage is most evident at lower context lengths and higher depths. While both models perform identically at the 2000 context length, GPT-4o excels at the 7900 context length, consistently outperforming from 5% to 90% depth. Performance varies at intermediate and higher context lengths, with GPT-4o maintaining a slight edge. At the maximum tested context length of 37400, GPT-4o demonstrates markedly better performance, especially at higher depths.

The substantial number of ties suggests that both models often perform comparably. This nuanced performance profile highlights the importance of considering specific context lengths and depths when choosing between GPT-4o and GPT-4t for particular applications.

Current provenance context

Klu separately compared how GPT-4o and GPT-4 Turbo used information placed at different positions in longer prompts. Long-context retrieval can vary with both total prompt length and the location of the relevant information, so an advertised context-window limit alone does not establish reliable retrieval across the entire prompt.

The original run date, sample construction, and scoring methodology are not documented in this repository, so this page does not present the archived comparison as a reproducible model ranking. The useful takeaway is methodological: teams should test the context lengths and insertion depths that reflect their own applications.

Archived Klu long-context comparison (repository archive: June 25, 2024)

The repository version archived on June 25, 2024 reported the wins and ties shown above for GPT-4 Omni 0513 and GPT-4 Turbo 1106. This is an unverified internal comparison, preserved for historical context rather than presented as a current or independently verified benchmark.

The archived explanation said both models performed identically at a context length of 2,000, that GPT-4o performed better across depths from 5% to 90% at a context length of 7,900, and that GPT-4o's advantage was more pronounced at higher depths at the maximum tested context length of 37,400. It also emphasized the large number of ties and the variation across context lengths and insertion depths. Those observations cannot currently be reproduced from repository evidence.

The archive does not identify the dataset or generated corpus, prompt templates, evaluator or scoring rubric, number and unit of samples, execution harness, decoding settings, run date, or exact provider model snapshots behind the display labels. Manual maintenance should recover those artifacts and the raw per-depth results, identify immutable model versions, and rerun the comparison with a documented harness before publishing a refreshed performance claim.

Needle in a Haystack Eval

To further assess GPT-4o's capabilities in handling long contexts, a "needle in a haystack" evaluation was conducted. This test involves inserting a small, crucial piece of information (the "needle") within a large amount of irrelevant text (the "haystack") and evaluating the model's ability to locate and utilize this information accurately.

GPT-4o exhibits strong long context accuracy across a wide range of lengths and depths, particularly excelling between 60,000 and 128,000 tokens. While it consistently achieves high scores in these ranges, performance fluctuates at extreme context lengths, such as 20,500 and 73,000 tokens. Despite occasional inconsistencies, the model generally maintains robust accuracy across multiple depths, showcasing its versatility in handling extended contexts.

The visualization maps retrieval accuracy across context lengths and insertion depths. It illustrates why teams should test both dimensions for their own prompts: retrieval can be uneven even when a prompt remains within the advertised context window. Because the original Klu run date, sample construction, and scoring methodology are not documented in this repository, the chart is retained as a qualitative example rather than evidence of a reproducible superiority claim.

Text Evaluation

GPT-4o outperformed leading AI models like GPT-4T, Claude3 Opus, and Gemini Pro 1.5 in recent benchmark evaluations, demonstrating its superior capabilities across various tasks.

In OpenAI's May 2024 launch evaluation, GPT-4o outperformed the compared GPT-4T, Claude 3 Opus, and Gemini 1.5 Pro results across several text benchmarks.

OpenAI's May 13, 2024 release snapshot: The benchmark sections from Text Evaluation through Vision Evaluation reproduce evaluations OpenAI published when introducing GPT-4o. They compare GPT-4o with models available at launch and should not be read as current leaderboards.

GPT-4o demonstrated exceptional performance across various benchmarks. In the MMLU test for massive multitask language understanding, it achieved 88.7% accuracy. The model excelled in coding tasks with a 90.2% pass rate on HumanEval, and showed remarkable proficiency in multilingual grade school math problems, scoring 90.5% on MGSM. For more advanced mathematics, GPT-4o scored 76.6% on the MATH benchmark. It also performed well in discrete reasoning over paragraphs, with an 83.4% accuracy on DROP. However, the model showed room for improvement on complex, nuanced questions, scoring 53.6% on the GPoQA benchmark. These results position GPT-4o as a leading model in text evaluation performance across diverse domains.

GPT-4o demonstrated exceptional performance across various benchmarks. In the MMLU test for massive multitask language understanding, it achieved 88.7% accuracy. The model excelled in coding tasks with a 90.2% pass rate on HumanEval, and showed remarkable proficiency in multilingual grade school math problems, scoring 90.5% on MGSM. For more advanced mathematics, GPT-4o scored 76.6% on the MATH benchmark. It also performed well in discrete reasoning over paragraphs, with an 83.4% accuracy on DROP. However, the model showed room for improvement on complex, nuanced questions, scoring 53.6% on the GPQA benchmark. In OpenAI's launch comparison, these results placed GPT-4o among the strongest models evaluated at that time.

Translation Evaluation

The recent performance evaluation of various models in audio translation using the CoVoST-2 benchmark shows that GPT-4o achieved the highest BLEU score of approximately 42, surpassing other models significantly.

OpenAI's May 2024 audio-translation evaluation reported that GPT-4o achieved the highest CoVoST-2 BLEU score among the compared systems, at approximately 42.

While Meta's SeamlessM4T-v2 and Google's AudioPaLM-2 demonstrated strong capabilities with scores around 35, they fell short of GPT-4o's performance. OpenAI's Whisper-v3 and Meta's XLS-R achieved lower scores of approximately 28 and 25, respectively. These results underscore GPT-4o's exceptional performance in real-time speech translation, solidifying its position as the leading model for audio translation tasks and demonstrating its superior efficiency and accuracy across multiple languages.

In that launch evaluation, Meta's SeamlessM4T-v2 and Google's AudioPaLM-2 scored around 35, while OpenAI's Whisper-v3 and Meta's XLS-R scored approximately 28 and 25, respectively. The comparison showed GPT-4o leading the evaluated systems on multilingual audio translation at the time of release.

Audio Evaluation

The audio ASR performance evaluation reveals that GPT-4o, using a 16-shot method, significantly outperforms Whisper-v3 across various regions based on the Word Error Rate (WER).

In OpenAI's May 2024 launch evaluation, GPT-4o with 16-shot prompting achieved lower Word Error Rates (WER) than Whisper-v3 across the regions shown.

GPT-4o consistently outperformed Whisper-v3 in audio transcription tasks across various linguistic regions, as evidenced by lower Word Error Rates (WER). In Western European languages, GPT-4o achieved a 5% WER compared to Whisper-v3's 7%. The performance gap widened in other regions: Eastern Europe (10% vs 15%), Central Asia/Middle East/North Africa (15% vs 20%), Sub-Saharan Africa (17% vs 30%), South Asia (20% vs 35%), South East Asia (10% vs 15%), and the CJK region (7% vs 10%). These results underscore GPT-4o's superior accuracy in transcribing diverse languages and accents, demonstrating its advanced capabilities in audio processing across global linguistic landscapes.

In OpenAI's launch chart, GPT-4o recorded a 5% WER for Western European languages compared with Whisper-v3's 7%. The reported comparisons for the other regions were Eastern Europe (10% vs 15%), Central Asia, the Middle East, and North Africa (15% vs 20%), Sub-Saharan Africa (17% vs 30%), South Asia (20% vs 35%), Southeast Asia (10% vs 15%), and the CJK region (7% vs 10%). These provider-reported results indicated stronger transcription performance under that evaluation setup and should not be generalized beyond it.

Exam Evaluation

The M3Exam zero-shot results highlight GPT-4o's superior accuracy compared to GPT-4 across various languages and question types.

OpenAI's May 2024 M3Exam zero-shot comparison reported higher GPT-4o accuracy than GPT-4 across the languages and question types shown.

In Afrikaans, Chinese, and English, GPT-4o achieved impressive accuracy rates of 85%, 80%, and 78% respectively. It also excelled in other languages such as Italian, Javanese, and Portuguese, with accuracy rates ranging from 75% to 80%. Notably, GPT-4o's proficiency extended to vision questions, where it maintained its lead with 75% accuracy in Afrikaans and 70% in both Chinese and English. These results highlight GPT-4o's enhanced capabilities in processing both textual and visual inputs across diverse linguistic contexts, marking a significant improvement over GPT-4's performance.

In OpenAI's launch results, GPT-4o recorded accuracy rates of 85%, 80%, and 78% in Afrikaans, Chinese, and English, respectively. The reported Italian, Javanese, and Portuguese scores ranged from 75% to 80%. On the vision questions, OpenAI reported 75% accuracy in Afrikaans and 70% in both Chinese and English. These results describe OpenAI's release-period comparison with GPT-4 rather than a current cross-model ranking.

Vision Evaluation

GPT-4o sets new benchmarks in visual understanding tasks. The model achieves state-of-the-art performance across various visual perception evaluations. All vision-related assessments were conducted in a zero-shot setting, with MMMU, MathVista, and ChartQA specifically utilizing zero-shot chain-of-thought (CoT) reasoning. This demonstrates GPT-4o's advanced capabilities in interpreting and reasoning about visual information without prior task-specific training.

In the May 2024 launch comparison, OpenAI reported new high scores for GPT-4o across several visual perception evaluations. All vision-related assessments were conducted in a zero-shot setting, with MMMU, MathVista, and ChartQA specifically utilizing zero-shot chain-of-thought (CoT) reasoning. These results demonstrated GPT-4o's ability at launch to interpret and reason about visual information without prior task-specific training.

Key Features

GPT-4o builds upon its predecessors, such as GPT-4 and GPT-4 Turbo, but offers enhanced performance, efficiency, and versatility. It is designed to handle complex tasks with greater accuracy and speed, making it a powerful tool for developers and businesses looking to leverage cutting-edge AI capabilities.

Some of the key features and improvements of GPT-4o include:

• Multimodal capabilities — GPT-4o can accept any combination of text, audio, image, and video inputs, and generate text, audio, and image outputs.

• Real-time processing — It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is comparable to human response time in a conversation.

• Enhanced performance — GPT-4o matches GPT-4 Turbo performance on text in English and code, with significant improvements in non-English languages.

• Improved efficiency — The model is 50% cheaper in the API compared to previous versions.

• Larger context window — GPT-4o maintains the 128k token context window, allowing it to process more information in a single prompt.

• Updated knowledge base — GPT-4o has knowledge of events up to April 2023.

  Current correction: GPT-4o has knowledge of events up to October 2023. OpenAI's model documentation lists an October 2023 knowledge cutoff for GPT-4o.

• Updated knowledge base — GPT-4o has knowledge of events up to October 2023.

Features

OpenAI GPT-4o distinguishes itself with several key improvements and capabilities:

• End-to-end multimodal processing — Unlike previous models that used separate models for different modalities, GPT-4o is trained end-to-end across text, vision, and audio, allowing for more integrated and nuanced understanding and generation.

• Real-time audio processing — GPT-4o can respond to audio inputs with human-like speed, significantly reducing latency compared to previous voice interaction models.

• Enhanced multilingual capabilities — The model shows significant improvement in processing and generating non-English languages.

• Advanced vision understanding — GPT-4o sets new high watermarks on vision-related benchmarks and tasks.

  Current context: At launch, OpenAI reported new high watermarks for GPT-4o on several vision-related benchmarks and tasks. This was OpenAI's launch-era claim for the compared benchmarks.

• Advanced vision understanding — At launch, OpenAI reported new high watermarks for GPT-4o on several vision-related benchmarks and tasks.

• Improved efficiency — The model is faster and more cost-effective to run compared to its predecessors.

• Versatile output generation — GPT-4o can generate not just text, but also audio and images, expanding its creative and interactive capabilities.

• Safety measures — OpenAI has implemented safety measures across all modalities, including filtering training data and refining the model's behavior through post-training.

GPT-4o represents OpenAI's continued effort to advance AI technology, providing a powerful tool that can handle more nuanced, multimodal interactions and perform a wider range of tasks with improved efficiency and accuracy.

What are the differences between GPT-4 and GPT-4o?

GPT-4o is a significant advancement over GPT-4, with several key differences:

• Multimodal capabilities — While GPT-4 primarily focused on text and image inputs with text outputs, GPT-4o can process and generate text, audio, image, and video inputs and outputs.

  Current correction: While GPT-4 primarily focused on text and image inputs with text outputs, GPT-4o can accept text, audio, image, and video inputs and generate text, audio, and image outputs. The documented interface accepts text, audio, image, and video inputs and generates text, audio, and image outputs.

• Multimodal capabilities — While GPT-4 primarily focused on text and image inputs with text outputs, GPT-4o can accept text, audio, image, and video inputs and generate text, audio, and image outputs.

• Real-time processing — GPT-4o can respond to audio inputs in milliseconds, comparable to human response times, which is a major improvement over GPT-4's processing speed.

• Integrated model — Unlike GPT-4, which often relied on separate models for different tasks, GPT-4o is trained end-to-end across multiple modalities, allowing for more coherent and context-aware responses.

• Enhanced multilingual performance — GPT-4o shows significant improvements in processing and generating non-English languages compared to GPT-4.

• Cost efficiency — GPT-4o is 50% cheaper to use in the API compared to previous versions, making it more accessible for developers and businesses.

• Advanced vision capabilities — While GPT-4 had some image understanding abilities, GPT-4o sets new benchmarks in vision-related tasks.

  Current context: While GPT-4 had some image understanding abilities, OpenAI's May 2024 evaluation reported stronger GPT-4o results across several vision-related tasks.

• Advanced vision capabilities — While GPT-4 had some image understanding abilities, OpenAI's May 2024 evaluation reported stronger GPT-4o results across several vision-related tasks.

• Broader creative outputs — GPT-4o can generate not just text, but also audio and images, expanding its creative and interactive potential beyond GPT-4's capabilities.

• Updated knowledge — GPT-4o's knowledge base is more current, with information up to April 2023, compared to GPT-4's earlier cutoff date.

  Current correction: GPT-4o's knowledge base is more current, with information up to October 2023, compared to GPT-4's earlier cutoff date. OpenAI's model documentation lists an October 2023 knowledge cutoff for GPT-4o.

• Updated knowledge — GPT-4o's knowledge base is more current, with information up to October 2023, compared to GPT-4's earlier cutoff date.

In essence, GPT-4o represents a more versatile, efficient, and capable evolution of the GPT series, designed to handle a wider range of inputs and outputs while maintaining high performance across various tasks.

How can developers access GPT-4o?

Developers can access GPT-4o through the OpenAI API. As of the announcement, the following details are available:

• API Access — GPT-4o is available in the API as a text and vision model.

• Performance and Cost — Compared to GPT-4 Turbo, GPT-4o is 2x faster, half the price, and has 5x higher rate limits.

• Rollout Plan — The text and image capabilities of GPT-4o are being rolled out immediately.

• Future Capabilities — OpenAI plans to launch support for GPT-4o's new audio and video capabilities to a small group of trusted partners in the API in the coming weeks.

  At launch, OpenAI planned to provide GPT-4o's audio capabilities and video-input understanding to a small group of trusted API partners in the following weeks.

• Future Capabilities — At launch, OpenAI planned to provide GPT-4o's audio capabilities and video-input understanding to a small group of trusted API partners in the following weeks.

• ChatGPT Integration — GPT-4o's capabilities are also being rolled out in ChatGPT, including in the free tier and for Plus users with higher message limits.

• Voice Mode — A new version of Voice Mode with GPT-4o will be available in alpha within ChatGPT Plus in the coming weeks.

To use GPT-4o, developers will need to have an OpenAI API account. The exact model name to be used in API calls has not been specified in the provided information, but it's likely to follow a similar naming convention to previous models (e.g., "gpt-4o-date").

Current correction: At launch, developers accessed the model through the API with the gpt-4o identifier. OpenAI's current GPT-4o model documentation lists the alias and available dated snapshots.

At launch, developers accessed the model through the API with the gpt-4o identifier. OpenAI's current GPT-4o model documentation lists the alias and available dated snapshots.

As with previous model releases, developers should refer to OpenAI's official documentation for the most up-to-date information on how to access and use GPT-4o in their applications.