Foundation Models

Stephen M. Walker II · Co-Founder / CEO

Foundation models are large-scale machine learning models that have been pre-trained on vast datasets and can be fine-tuned for a wide range of tasks, serving as a foundational layer for further application-specific training.

Major AI labs and open-source communities each maintain their own foundation models, spanning proprietary offerings and openly licensed alternatives.

They serve as a starting point for data scientists to develop machine learning (ML) models for various applications more quickly and cost-effectively. These models are trained on a broad spectrum of generalized and unlabeled data and are capable of performing a wide variety of general tasks, such as understanding language, generating text and images, and conversing in natural language.

A unique feature of foundation models is their adaptability, allowing them to perform a wide range of tasks with high accuracy based on input prompts. Some tasks include natural language processing (NLP), question answering, and image classification. Foundation models differ from traditional ML models, which typically perform specific tasks like analyzing text for sentiment, classifying images, and forecasting trends.

Foundation models use self-supervised learning to create labels from input data, meaning they are not trained with labeled training datasets. This distinguishes them from previous ML architectures that use supervised learning or unsupervised learning. Examples of foundation models include large language models (LLMs) like GPT-3 and BERT, and text-to-image models like DALL-E.

These models can be fine-tuned for more specialized downstream applications, making it faster and cheaper for data scientists to develop new ML applications rather than training unique ML models from scratch.

What are examples of foundation models?

Foundation models span several architectures and modalities, from text-only language models to multimodal systems that combine vision and language. The examples below illustrate the range of tasks these models cover, including NLP, question answering, and image classification.

Examples of foundation models include:

BERT — This is a transformer-based machine learning technique for natural language processing pre-training. It stands for Bidirectional Encoder Representations from Transformers and was developed by Google.
GPT-4 — GPT-4 is a large language model developed by OpenAI. It is the fourth iteration of the Generative Pretrained Transformer models and is capable of understanding and generating human-like text.
Claude and Llama — These are large language models (LLMs) that can perform a range of tasks out of the box spanning multiple domains, like writing blog posts, solving math problems, engaging in dialog, and answering questions based on a document.
Stable Diffusion from Stability AI — This is a text-to-image model that can generate high-quality images from text descriptions.
DALL-E — This is a model developed by OpenAI that generates images from textual descriptions.
Flamingo and Florence — These are visual foundation models (VFMs) that have been combined with text-based LLMs to develop sophisticated task-specific models.
Gato by Google DeepMind — This is a reinforcement learning agent.
Segment Anything by Meta AI — This is a model for general image segmentation.

However, building a foundation model from scratch is expensive and requires enormous resources. For practical applications, developers need to integrate foundation models into a software stack, including tools for prompt engineering, fine-tuning, and pipeline engineering. Despite their ability to provide grammatically and factually correct answers, foundation models have difficulty comprehending the context of a prompt and aren't socially or psychologically aware.

What is unique about foundation models compared to others?

Foundation models provide a significant advantage in terms of time and cost savings. Once pre-training is complete, the resulting model can be reused and fine-tuned for multiple downstream tasks. This eliminates the need to train models from scratch for each new task, saving both time and computational resources.

However, foundation models, particularly those with large architectures and parameters, require significant computational resources to train and deploy. Training these models on extensive datasets can be computationally intensive and time-consuming. This poses challenges for organizations or individuals with limited access to high-performance computing infrastructure.

What can foundation models be used for?

Because foundation models generalize across domains, they can be adapted to a wide variety of downstream applications, from consumer products to internal business tooling. Here are some specific tasks that foundation models can perform:

Natural Language Processing (NLP) — Foundation models can understand language, generate text, and converse in natural language. They can be used for tasks such as transcription and video captioning in various languages.
Question Answering — Foundation models can answer questions based on a document or a given context.
Image Classification — Foundation models can classify images into different categories based on their features.
Content Generation — Foundation models can generate content such as writing blog posts or creating high-resolution images.
Document Extraction — Foundation models can extract information from documents, which can be useful in various fields such as law, healthcare, and education.
Code Generation — Foundation models can generate code, which can be useful in software development and programming.
Human-Centered Engagement — Foundation models can engage in dialog and interact with humans in a natural and coherent manner.
Robotics and Autonomous Vehicles — Foundation models can be used in robotics and autonomous vehicles for tasks such as navigation, object recognition, and decision making.
Healthcare — Foundation models can be used in healthcare for tasks such as drug discovery, patient diagnosis, and treatment recommendation.
Education — Foundation models can be used in education for tasks such as problem generation and personalized learning.

It's important to note that while foundation models can perform these tasks out of the box, they can also be fine-tuned for more specific tasks or domain-specific applications.

How are foundational models trained?

The process of training a foundation model involves several steps and requires significant resources and expertise.

The first step is to collect a large and diverse dataset, which could include text or code. This dataset should cover the tasks that you want the model to be able to perform. The data then needs to be prepared, which includes cleaning the data, removing any errors, and formatting the data in a way that the model can understand.

Training a foundation model requires a significant amount of computing resources, as the models are trained on large datasets using deep learning algorithms. This process is computationally expensive and requires expertise in machine learning and AI, as there are many factors that need to be considered, such as the choice of model architecture, the hyperparameters, and the training process.

Once the model is trained, it can be fine-tuned to adapt to specific tasks or domains. Fine-tuning involves further training and changes the weights of the model, allowing it to work with domain-specific language or improve performance for specific tasks. This can be done through methods such as domain adaptation fine-tuning, which uses limited domain-specific data, or instruction-based fine-tuning, which uses labeled examples to improve performance on a specific task.

However, training a foundation model from scratch can be very expensive, with costs ranging from tens of thousands to millions of dollars, depending on factors such as the size of the model and the amount of data and computational resources required. Therefore, many businesses opt to use pre-trained models, which have already been trained on a large dataset and can be customized to perform a variety of tasks.

Training a foundation model is a complex and challenging task that requires significant resources and expertise. However, these models can provide state-of-the-art performance on a variety of tasks and can be customized to meet specific needs, making them a valuable investment for businesses looking to gain a competitive advantage in the field of AI.

What are the challenges associated with training foundation models?

Training foundation models, also known as pre-trained models, presents several challenges:

Data Acquisition and Curation — Foundation models require large-scale and diverse datasets for pre-training. Acquiring and curating such datasets can be a challenging task. Data collection may involve privacy concerns, copyright issues, or difficulties in obtaining labeled data for specific tasks. Ensuring the quality and representativeness of the training data is crucial to avoid biases and improve generalization.
Bias — Biases present in the training data can lead to biased or unfair outcomes in the model's predictions or decisions. For example, if the training data predominantly represents certain demographics or perspectives, the model may show biases towards those groups. Addressing bias requires careful data curation, diversity in the training data, and ongoing monitoring and evaluation of the model's outputs.
Computational Resources — Foundation models, particularly those with large architectures and parameters, require significant computational resources to train and deploy. Training these models on extensive datasets can be computationally intensive and time-consuming. This poses challenges for organizations or individuals with limited access to high-performance computing infrastructure.
Expertise — Training a foundation model requires expertise in machine learning and AI. There are many factors that need to be considered, such as the choice of model architecture, the hyperparameters, and the training process. These skills are scarce and very expensive.
Cost — The cost of training a foundation model can range from tens of thousands to millions of dollars, depending on factors such as the size of the model and the computational resources required. The high cost of training these models can be prohibitive for many organizations, making their implementation financially unattainable.
Unreliability and Incomprehension — Foundation models can be unreliable and incomprehensible. Despite their impressive capabilities, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties.
Ethical and Societal Challenges — Foundation models can exacerbate historical inequities and centralize power. They also have potential negative impacts from an environmental standpoint and could have substantial labor market impacts. The legal and regulatory frameworks for the development and deployment of foundation models are also unclear.
Domain-Specific Tasks — Out-of-the-box foundation models trained on general knowledge may struggle on domain-specific tasks. To improve the model's performance to the level required for specific applications, additional fine-tuning or adaptation may be necessary.

Addressing these challenges requires a combination of technical expertise, careful planning, and ongoing monitoring and evaluation. It's also important to consider the ethical and societal implications of deploying these models, and to engage in ongoing dialogue about how to manage these risks effectively.

Why is foundation modeling important?

Foundation modeling matters because it changes the economics of the machine learning lifecycle. Although training a foundation model from scratch can cost millions of dollars, reusing a pre-trained model is far faster and cheaper than building a new one for every application.

Models are built on complex neural network architectures, including generative adversarial networks (GANs), transformers, and variational autoencoders. Despite the numerous benefits, foundation models still present challenges: they require significant computational resources to train and deploy, and while they can provide grammatically and factually correct answers, they have difficulty comprehending the context of a prompt and aren't socially or psychologically aware.

More terms

Continue exploring the glossary.

Learn how teams define, measure, and improve LLM systems.

Glossary term

What is temporal difference learning?

Temporal Difference (TD) learning is a class of model-free reinforcement learning methods. These methods sample from the environment, similar to Monte Carlo methods, and perform updates based on current estimates, akin to dynamic programming methods. Unlike Monte Carlo methods, which adjust their estimates only once the final outcome is known, TD methods adjust predictions to match later, more accurate predictions.

Read term

Glossary term

What is Principal Component Analysis (PCA)?

Principal Component Analysis (PCA) is a statistical technique that transforms high-dimensional data into a lower-dimensional space while preserving as much information about the original data as possible. PCA works by finding the principal components, which are linear combinations of the original variables that maximize the variance in the transformed data.