Generative Pre-trained Transformer (GPT)

The Generative Pre-trained Transformer (GPT) is a type of deep learning model developed by OpenAI. It uses an attention mechanism to focus on different parts of the input sequence when generating text.

What is GPT?

GPT uses a decoder-only Transformer architecture: a stack of identical blocks, each combining masked self-attention with a feed-forward network. The masking ensures each position can only attend to earlier positions, which is what makes GPT autoregressive — it generates text one token at a time, each new token conditioned on the ones before it. This differs from the original encoder-decoder Transformer and from encoder-only models like BERT. Unlike RNNs and CNNs, it doesn't rely on recurrence or convolutions, which lets it process a sequence in parallel rather than step by step.

Widely used in NLP, GPT advances tasks like translation, sentiment analysis, and language generation, and has also been applied to fields like computer vision and audio processing.

Training large GPT models demands substantial compute, taking weeks to months and costing millions of dollars, which is why most applications build on pre-trained models rather than training from scratch.

What are some common applications for GPT?

GPT models are used across a range of natural language processing tasks: translation, document summarization, text generation, and virtual assistants and chatbots. Beyond text, the same architecture has been adapted for tasks like time series prediction and biological sequence analysis.

How does GPT work?

A core element of GPT is the self-attention mechanism, which relates different positions within a sequence to compute its representation. Each token in the input is compared with the others to compute scores that weight their contribution, letting the model emphasize the parts of the input most relevant to what it's predicting next.

GPT also uses positional encoding to convey token order, crucial for meaning. Its parallel processing capability significantly accelerates training compared to sequential RNNs.

At inference time, GPT feeds its growing output back in as input, predicting the next token based on everything generated so far, until it produces a stop token or reaches a length limit.

What are some challenges associated with GPT?

GPT has transformed AI, especially in NLP, but faces challenges. Its high computational demands hinder scalability and efficiency, increasing carbon footprint and posing financial barriers. Overfitting can occur, limiting generalization to new data, particularly if noisy or adversarial. Robustness issues may arise with data outside the training set.

Efforts to mitigate these challenges include reducing model size and complexity through pruning, quantization, distillation, and sparsification, and exploring GPT variants and extensions.

Milestones in the GPT Series

OpenAI's GPT models trace a steady climb in scale and capability, each generation building on the decoder-only Transformer architecture described above.

• GPT-2 — Launched in 2019, GPT-2 demonstrated that a single language model, scaled up, could handle text generation, summarization, and translation without task-specific training.

• GPT-3 — Released in 2020, GPT-3 scaled to 175 billion parameters, showing that sufficiently large language models could perform new tasks from a handful of examples in the prompt (few-shot learning), with no fine-tuning required.

• GPT-4 — Released in 2023, GPT-4 improved reasoning and accuracy over GPT-3 and added multimodal input, letting the model process images alongside text.

• GPT-4o — Released in 2024, GPT-4o ('o' for omni) unified text, vision, and audio in a single model, enabling faster, more natural real-time interaction across modalities.

OpenAI has continued releasing new GPT generations since; see OpenAI's model documentation for the current lineup. Across generations, larger and more capable models have brought higher training and inference costs, and OpenAI has paired scale increases with alignment work to manage risks like factual errors and biased outputs.