Llama 2

Llama 2 is an open-source large language model (LLM) developed by Meta AI. It is a family of pre-trained and fine-tuned models, publicly released in three sizes — 7B, 13B, and 70B parameters. Meta also trained a 34B variant but did not release it. Llama 2 is the successor to the original Llama model and includes both foundational models and models fine-tuned for dialogue, known as Llama 2-Chat.

The Llama 2 models were trained on 2 trillion tokens from publicly available sources like Common Crawl, Wikipedia, and public domain books from Project Gutenberg. The models are designed to predict the most plausible follow-on text using a neural network, a cascading algorithm with billions of variables (parameters).

Llama 2 has several improvements over the original Llama model. It uses the same transformer architecture as the original model, but with modifications such as RMSNorm pre-normalization, a SwiGLU activation function, and rotary positional embeddings; the 34B and 70B variants additionally use grouped-query attention. The context length in Llama 2 has been increased from 2048 tokens in Llama 1 to 4096 tokens.

Llama 2 includes a code generation model called Code Llama, which is trained on 500 billion tokens of code. Code Llama supports common programming languages being used today, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.

Llama 2 is freely available for almost anyone to use for research and commercial purposes, with some restrictions. Companies with more than 700 million monthly users have to ask for special permission to use Llama, so it's off limits for the likes of Apple, Google, and Amazon.

In terms of performance, Llama 2 outperformed most other open-source language models available at its 2023 release on many external benchmarks, including reasoning, coding, proficiency, and knowledge tests. However, at the time, it trailed the leading closed frontier models on some benchmarks.

Llama 2

The Llama 2 model was trained using a specific structure for prompts, which relied on four special tokens: <s> for the beginning of the entire sequence, <<SYS>>\n for the beginning of the system message, \n<</SYS>>\n\n for the end of the system message, and [INST] and [/INST] for the beginning and end of some instructions, respectively.

Llama 2 adopts the model architecture of Llama 1 with a few modifications. It uses a RoPE scheme for positional embeddings, which balances the absolute and relative position of each token in a sequence. This approach encodes the absolute position with a rotation matrix and adds relative position information directly into the self-attention operation.

Llama 2 is trained with a longer context length of 4K tokens, compared to Llama 1 which was trained with a 2K context length. Additionally, the 34B and 70B variants adopt grouped query attention (GQA) within each of their layers; the 7B and 13B variants use standard multi-head attention.

Llama 2 was trained using solely public sources of data for pre-training, with a deliberate choice to ensure that the pre-training process can be openly replicated by anyone with sufficient compute resources. Compared to Llama 1, Llama 2 adopts a new mixture of pre-training data, with sources known to be high-quality and factual sampled more heavily, and increases the size of the pre-training dataset by 40%.

Llama 2 was fine-tuned using a large dataset in a similar manner to proprietary models, producing the Llama 2-Chat model that is optimized for dialogue-based applications. The alignment process, which teaches the model the correct output style or format, was performed with a goal of reducing hallucinations, avoiding unsafe questions, following detailed instructions, and more.

The Llama 2 models were trained using bfloat16, but the original inference uses float16. The checkpoints uploaded on the Hub use torch_dtype = 'float16', which will be used by the AutoModel API to cast the checkpoints from torch.float32 to torch.float16.

To use Llama 2, you can use Hugging Face Transformers. The most flexible approach is to load the largest of the Llama 2 models that has been fine-tuned for chat — the Llama-2-70b-chat-hf model. GPU memory requirements scale with model size and precision: the 7B model needs roughly 14GB of VRAM at fp16, while the 70B model needs roughly 140GB of VRAM at fp16 (or around 35GB when quantized to 4-bit), typically requiring multiple high-end GPUs such as the A100.

Transformer Architecture

The foundation of Llama 2, a classic architecture used in many AI models.

RMSNorm

Short for Root Mean Square Normalization, a technique used by Meta to handle the 2 trillion tokens and internal weights of Llama 2.

SwiGLU Activation Function

The activation function chosen by Meta for Llama 2 to determine whether a given neuron should be active or not.

RoPE

Short for Rotary Positional Embedding, a mathematical method used in Llama 2 to ensure the model understands the importance of word positions in sentences.

Ghost Attention (GAtt)

A fine-tuning method introduced by Meta for Llama 2. It helps control dialogue flow over multiple turns by synthetically concatenating the 'act as' instruction to all user messages in a conversation.

Context Length

The amount of information Llama 2 can consider from previous inputs. Llama 2 has a context length of 4096 tokens, twice the context length of Llama 1.

Grouped Query Attention (GQA)

An attention variant used in the Llama 2 34B and 70B models for improved inference scalability.

Fine-tuning

The process of adjusting the weights of a pre-trained model to make it perform better on the desired task. Llama 2 uses fine-tuning methods like reinforcement learning with human feedback (RLHF), supervised fine-tuning (SFT), and initial and iterative reward modeling.

Fill-in-the-middle (FIM) Capability

A feature in Code Llama that allows it to insert code into existing code, supporting tasks like code completion.

Instruction Tuning

A training process where the model is fed a natural language instruction input and the expected output. This makes it better at understanding what people expect out of their prompts. Used in Code Llama Instruct.

Llama 2-Chat

A chatbot version of Llama 2, fine-tuned for chat-style interactions through supervised fine-tuning and reinforcement learning with human feedback (RLHF).

Code Llama

A code generation model built on Llama 2, trained on 500B tokens of code. It supports common programming languages including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.