Llama 3.1 405B

Llama 3.1 405B is an open-weight large language model (LLM) developed by Meta AI and released on July 23, 2024. It belongs to a family of pre-trained and instruction-tuned models available in 8 billion, 70 billion, and 405 billion parameter sizes. Llama 3.1 succeeded Llama 3 and includes models optimized for dialogue, known as Llama 3.1 Instruct. Meta has since released newer model families, including Llama 3.2, Llama 3.3, and Llama 4.

The Llama 3.1 405B models were trained on over 15 trillion tokens from publicly available sources. The models are designed to predict the most plausible follow-on text using a neural network with billions of variables (parameters).

Benchmarks

Llama 3.1 introduces a new 405B parameter model and 3.1 models show improved performance over Llama 3 models in various benchmarks, particularly in instruction-tuned evaluations. Significant gains are observed in HumanEval, MATH (CoT), and Multilingual MGSM benchmarks, indicating enhanced capabilities in code evaluation, complex reasoning, and multilingual tasks.

Base Pretrained Models

Instruction Tuned Models

Llama 3.1 405B is Meta's largest open-weight language model to date, released alongside the 8B and 70B members of the Llama 3.1 family. It was trained on over 15 trillion tokens and, unlike the original Llama 3 models, supports a 128K-token context window. Meta releases the model's weights under the Llama 3.1 Community License, which allows developers to fine-tune, distill, and build on top of it subject to that license's terms.

Key features include:

1. Frontier-scale performance: At 405 billion parameters, the model is competitive with leading closed models on reasoning, math, and code benchmarks.
2. Open weights: Access to the model's weights allows developers to fine-tune, distill, and build on top of it under the terms of the Llama 3.1 Community License.
3. Extended context: All Llama 3.1 models support a 128K-token context window, up from 8K in the original Llama 3 release.

Llama 3.1 models use a prompt format built around special tokens such as <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, and <|eot_id|> to delineate system, user, and assistant turns, replacing the <<SYS>> / [INST] style tokens used by Llama 2.

Llama 3.1 adopts the same core architecture as Llama 3: a standard decoder-only transformer using RoPE positional embeddings and grouped query attention (GQA), with no major architectural departures. The 405B model was trained on over 15 trillion tokens, mostly from publicly available sources.

Llama 3.1 405B was fine-tuned using supervised fine-tuning (SFT), rejection sampling, and direct preference optimization (DPO) to produce the instruction-tuned Llama 3.1 405B Instruct model, optimized for dialogue-based applications and reducing hallucinations, avoiding unsafe outputs, and following detailed instructions.

Models are trained and distributed in bfloat16. Because of its size, running the 405B model requires substantial GPU memory — roughly 810GB in bfloat16 or about 405GB with 8-bit quantization — typically spread across multiple high-end GPUs (such as several A100s or H100s) or accessed through a hosted inference provider rather than run on a single card.

Transformer Architecture

The foundation of Llama 3.1, a decoder-only architecture used in many modern LLMs.

RMSNorm

Short for Root Mean Square Normalization, a technique used by Meta to stabilize training across Llama 3.1's internal weights.

SwiGLU Activation Function

The activation function chosen by Meta for Llama 3.1 to determine whether a given neuron should be active or not.

RoPE

Short for Rotary Positional Embedding, a mathematical method used in Llama 3.1 to ensure the model understands the importance of word positions in sentences.

Context Length

The amount of information Llama 3.1 can consider from previous inputs. Llama 3.1 models support a context length of 128,000 tokens, a substantial increase over the 8,192-token context length of the original Llama 3 release.

Grouped Query Attention (GQA)

A feature in Llama 3.1 for improved inference scalability.

Fine-tuning

The process of adjusting the weights of a pre-trained model to make it perform better on the desired task. Llama 3.1 405B uses fine-tuning methods like supervised fine-tuning (SFT), rejection sampling, and direct preference optimization (DPO).

Instruction Tuning

A training process where the model is fed a natural language instruction input and the expected output. This makes it better at understanding what people expect out of their prompts. Used to produce Llama 3.1 405B Instruct.

Llama 3.1 405B Instruct

An assistant-oriented version of Llama 3.1 405B, trained for chat-style interactions through supervised fine-tuning, rejection sampling, and direct preference optimization.