Llama 3

Llama 3 is an open-source large language model (LLM) released by Meta AI in April 2024. It is a family of pre-trained and instruction-tuned models available in 8 billion and 70 billion parameter sizes, and is the successor to Llama 2. The instruction-tuned variants, known as Llama 3-Instruct, are optimized for dialogue. Meta has since released further iterations — including Llama 3.1, 3.2, and 3.3, and the Llama 4 family — which superseded Llama 3 as Meta's frontier open-weight models.

The Llama 3 models were pretrained on over 15 trillion tokens from publicly available sources, using a neural network with billions of parameters to predict the most plausible follow-on text.

Model overview

Meta Llama 3 was pretrained on a large text corpus and fine-tuned for tasks such as chat and question-answering. Available in 8B and 70B parameter sizes, it is released under an open license that allows developers to access and modify the model weights and code for research and commercial use.

Base pretrained foundation models

When deploying a model like Llama 3, it's common to pair the base model with input/output filtering to improve system-level safety alongside model-level safety, particularly when integrating the model into applications.

Instruction-tuned models

At release, Llama 3 outperformed Llama 2 and many other open-source chat models on external benchmarks spanning reasoning, coding, and knowledge tests, though closed frontier models of the time still led on several of these benchmarks.

Architecture and training

Llama 3 uses an optimized transformer architecture with a 128K-token vocabulary tokenizer, roughly four times larger than Llama 2's. It was trained with an 8,192-token context length, twice that of Llama 2, and uses Grouped-Query Attention (GQA) across both the 8B and 70B sizes for improved inference efficiency.

Key architectural elements carried over from Llama 2 include:

• RMSNorm (Root Mean Square Normalization) for stabilizing training across layers.
• SwiGLU activation function, used in the feed-forward layers.
• RoPE (Rotary Positional Embedding), which encodes token position by rotating query and key vectors, balancing absolute and relative positional information within self-attention.

Llama 3 was pretrained solely on publicly available data, with a mixture weighted toward higher-quality, more factual sources, and on roughly seven times more tokens than Llama 2. The instruction-tuned Llama 3-Instruct models were produced through a combination of supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO).

Prompt format

Llama 3's chat template uses a distinct set of special tokens (differing from Llama 2's [INST]/<<SYS>> format):

• <|begin_of_text|> marks the start of a sequence.
• <|start_header_id|>role<|end_header_id|> marks the start of a turn, where role is system, user, or assistant.
• <|eot_id|> marks the end of a turn.

Hardware requirements

Running the largest chat-tuned model, Meta-Llama-3-70B-Instruct, at full fp16 precision requires roughly 140GB of GPU memory, typically necessitating multiple high-end GPUs (such as A100s or H100s) or quantization to fit on smaller hardware. The 8B model is far more accessible, running in fp16 on a single modern consumer or datacenter GPU.

Code Llama

Meta separately released Code Llama, a family of code generation models built on the Llama 2 architecture and trained on 500B tokens of code, supporting languages including Python, C++, Java, PHP, TypeScript/JavaScript, C#, and Bash. It also supports fill-in-the-middle (FIM) completion, letting the model insert code into an existing file rather than only appending to it.

Licensing and usage

Llama 3 is distributed under the Meta Llama 3 Community License Agreement, which permits research and commercial use with some restrictions — notably, companies with more than 700 million monthly active users must request separate permission from Meta. Model weights are available from Meta's Llama GitHub page and through Hugging Face.

Safety and responsibility

Meta paired the Llama 3 release with guidance on responsible deployment, covering testing and evaluation, data quality and bias, and transparency. Meta's stated testing at release did not claim to be comprehensive, and the guidance recommends additional safety testing tailored to each downstream application.

Related terms

• Llama 2
• Llama 3.1 405B
• Large language model
• Fine-tuning methods such as reinforcement learning from human feedback (RLHF)