Anthropic released the Claude 3 model family in March 2024, establishing new benchmarks in a variety of cognitive tasks at the time. This lineup includes Claude 3 Haiku, Claude 3 Sonnet, and Claude 3 Opus, each model offering enhanced performance levels for different user needs in terms of intelligence, speed, and cost.
- Claude 3 Haiku — The fastest and most compact model, designed for near-instant responsiveness, particularly suited for simple queries and requests.
- Claude 3 Sonnet — Offers a balance between speed and intelligence, excelling in tasks that require rapid responses like knowledge retrieval or sales automation. It is twice as fast as previous Claude models.
- Claude 3 Opus — The most intelligent model in the family, outperforming peers on benchmarks such as undergraduate and graduate-level knowledge and reasoning, basic mathematics, and more. It exhibits near-human comprehension and fluency in complex tasks.
Availability
Anthropic's Claude 3 series introduces a significant leap in Claude-series capabilities, featuring multimodal understanding that processes both text and visual inputs, including photos, charts, graphs, and technical diagrams. This advancement allows for a broader application in various fields, enhancing the AI's ability to interact with and understand complex information.
At launch, Claude 3 was accessible in 159 countries via claude.ai and its API, with Opus and Sonnet available immediately and Haiku following shortly after. Anthropic positioned the models as strong performers for live customer chats, auto-completions, and data extraction tasks, citing benchmark results against competitors like OpenAI's GPT-4. Early independent tests left some room for skepticism.
Claude 3 Benchmark Performance
According to Anthropic's published launch data, the Claude 3 model family outperformed predecessors like GPT-4 and Gemini 1.0 Ultra across various cognitive tasks at the time. The Claude 3 Opus model, the most sophisticated in the lineup, showcased strong performance in key AI benchmarks such as MMLU (undergraduate-level knowledge), GPQA (graduate-level reasoning), GSM8K (basic mathematics), and HumanEval (coding), indicating its prowess in understanding complex tasks, mathematical reasoning, and coding, with near-human comprehension and fluency. Additionally, its advanced computer vision capabilities allow for effective information extraction from visual materials.
At the time, Claude 3 Opus surpassed the then-leading models from OpenAI and Google on these benchmarks. It performed strongly on undergraduate and graduate knowledge, and grade school math, while the Claude 3 Sonnet model showed a notable ability to understand scientific diagrams, useful for enterprise operations and data analysis.
| Test | Claude 3 Opus | GPT-4 | Gemini 1.0 Ultra |
|---|
| Undergraduate level knowledge MMLU | 86.8% 5-shot | 86.4% 5-shot | 83.7% 5-shot |
| Graduate level reasoning GPQA, Diamond | 50.4% 0-shot CoT | 35.7% 0-shot CoT | — |
| Grade school math GSM8K | 95.0% 0-shot CoT | 92.0% 5-shot CoT | 94.4% maj@32 |
| Math problem-solving MATH | 60.1% 0-shot CoT | 52.9% 4-shot | 53.2% 4-shot |
| Multilingual math MGSM | 90.7% 0-shot | 74.5% 8-shot | 79.0% 8-shot |
| Code HumanEval | 84.9% 0-shot | 67.0% 0-shot | 74.4% 0-shot |
| Reasoning over text DROP, F1 score | 83.1% 3-shot | 80.9% 3-shot | 82.4% Variable shots |
| Mixed evaluations BIG-Bench-Hard | 86.8% 3-shot CoT | 83.1% 3-shot CoT | 83.6% 3-shot CoT |
| Knowledge Q&A ARC-Challenge | 96.4% 25-shot | 96.3% 25-shot | — |
| Common Knowledge HellaSwag | 95.4% 10-shot | 95.3% 10-shot | 87.8% 10-shot |
Claude 3's multimodal nature, capable of processing both text and images, significantly expands its application range. Anthropic emphasizes AI safety, actively working to mitigate bias and ensure neutrality, making Claude 3 models a top choice for both enterprise and consumer applications due to their performance, safety features, and broad applicability.
| Test | Claude 3 Opus | GPT-4 Vision | Gemini 1.0 Ultra |
|---|
| Math & reasoning MMMU (val) | 59.4% | 56.8% | 59.4% |
| Document visual Q&A ANLS score, test | 89.3% | 88.4% | 90.9% |
| Math MathVista (testmini) | 50.5% CoT | 49.9% | 53.0% |
| Science diagrams AI2D, test | 88.1% | 78.2% | 79.5% |
| Chart Q&A Relaxed accuracy (test) | 80.8% 0-shot CoT | 78.5% 4-shot CoT | 80.8% |
Claude 3 Model Series
The Claude 3 models, particularly Opus, excel in common AI evaluation benchmarks like MMLU, GPQA, and GSM8K, demonstrating near-human comprehension and fluency in complex tasks. These models also improve in analysis, forecasting, content creation, code generation, and multilingual communication.
At launch, Anthropic highlighted the economic potential of Claude 3, particularly the Opus model, in roles such as an economic analyst. In our own testing against GPT-4 Turbo at the time, we found little evidence to back this claim. Amazon reported over 10,000 organizations were already using Bedrock for generative AI applications, and the introduction of Claude 3 models was expected to accelerate adoption further.
Accuracy
Claude models have historically had issues with accuracy due to their ability to be very creative, most noticeable when iterating on complex storylines. Claude 3 improved on this, with Opus roughly doubling the rate of correct answers on complex, factual questions compared to Claude 2. Anthropic indicated citation support for verifying answers was planned for a future update.
Context length up to 1 million tokens
The Claude 3 models launched with a 200K context window, with Anthropic noting the potential for inputs over 1 million tokens for select customers. The models demonstrated robust recall capabilities, with Opus achieving near-perfect recall in the 'Needle In A Haystack' evaluation.
Speed
At launch, Claude 3 models delivered near-instantaneous results for live chats, auto-completions, and data extraction, with Haiku being the fastest and most cost-effective for its intelligence level. Sonnet offered roughly double the speed of prior Claude models with higher intelligence, while Opus matched their speed but with significantly enhanced intelligence.
Vision
These models also featured vision capabilities, processing various visual formats and offering this capability to enterprise customers with visually encoded knowledge bases.
Instruction Following, Reduced Refusals & AI Safety
The Claude 3 models were designed to be user-friendly, adept at following complex instructions, and capable of producing structured outputs like JSON for various applications.
Claude 2 had a tendency to refuse non-harmful prompts, a side effect of Anthropic's AI Safety measures. Claude 3 improved on this, showing a better understanding of prompts and less frequent unnecessary refusals.
Anthropic's stated commitment to responsible AI development was reflected in the Claude 3 models, which prioritized reducing biases and enhancing safety across various risk scenarios. At launch, these models operated under Anthropic's AI Safety Level 2, with continuous monitoring to evaluate and manage risk levels. The design of Claude 3 models emphasized neutrality and bias mitigation, with Anthropic reporting a lower bias rate compared to previous versions, along with safety features intended to handle sensitive prompts more effectively.
Planned updates (as of March 2024 launch)
The initial release of the Claude 3 models featured a 200K context window, with Anthropic noting plans to extend input capabilities up to 1 million tokens for certain customers. At the time, Google's Gemini 1.5 was also expanding its context window, and Anthropic was expected to continue pushing its generally available context lengths to remain competitive.
Anthropic announced plans to introduce Tool Use (function calling), interactive coding (REPL), and advanced agentic capabilities in subsequent updates.
Anthropic indicated at launch that the intelligence of the Claude 3 models was far from reaching its peak, with frequent updates planned to expand the models' functionalities, especially for enterprise applications and large-scale deployments.
Claude 3 models launched with availability on major platforms including Amazon Bedrock and Google Cloud's Vertex AI, alongside backing from Amazon and Google that supported the models' distribution across industries.
Pricing at launch
At launch, each model had distinct features and costs, catering to different needs:
- Opus was the most intelligent, positioned for complex tasks at $15 per million tokens for input and $75 for output.
- Sonnet offered a balance of intelligence and speed, positioned for enterprise workloads at $3 per million tokens for input and $15 for output.
- Haiku offered the fastest response for simple queries at $0.25 per million tokens for input and $1.25 for output.
At release, Opus and Sonnet were available first, with Haiku following shortly after. Sonnet powered the free experience on claude.ai, with Opus reserved for Claude Pro subscribers. Availability extended to Amazon Bedrock and Google Cloud's Vertex AI Model Garden.
Anthropic has since released newer model generations, including Claude 3.5 and Claude 4; for current models and pricing, see Anthropic's official documentation.
