MTEB: Massive Text Embedding Benchmark

What is MTEB (Massive Text Embedding Benchmark)?

The Massive Text Embedding Benchmark (MTEB) is a comprehensive benchmark designed to evaluate the performance of text embedding models across a wide range of tasks and datasets. It was introduced to address the issue that text embeddings were commonly evaluated on a limited set of datasets from a single task, making it difficult to track progress in the field and to understand whether state-of-the-art embeddings on one task would generalize to others.

MTEB encompasses 8 embedding tasks, which include bitext mining, classification, clustering, pair classification, reranking, retrieval, semantic textual similarity (STS), and summarization. It covers a total of 58 datasets and spans 112 languages, making it one of the most extensive benchmarks for text embeddings to date.

The benchmark has been used to evaluate 33 different models, revealing that no single text embedding method consistently outperforms others across all tasks. This suggests that the field has not yet converged on a universal text embedding method.

The Massive Text Embedding Benchmark (MTEB) is an open-source framework for evaluating embedding and retrieval systems across versioned tasks, datasets, languages, and modalities. Its original 2022 release addressed the problem that text embeddings were commonly evaluated on a limited set of datasets from a single task, which made cross-task generalization difficult to measure.

The original MTEB release encompassed eight text-embedding task families, 56 datasets, and 112 languages. The project was later expanded through MMTEB and now describes itself as a multimodal toolbox for evaluating embeddings and retrieval systems. Current comparisons must name the exact benchmark suite, task versions, languages, modalities, and aggregation method.

The benchmark has been used to evaluate a growing number of models, revealing that no single text embedding method consistently outperforms others across all tasks. This suggests that the field has not yet converged on a universal text embedding method.

Current scope context: MTEB is now an open-source framework for evaluating embedding and retrieval systems across versioned tasks, datasets, languages, and modalities. The original 2022 release encompassed eight text-embedding task families, 56 datasets, and 112 languages; the project later expanded through MMTEB into a multimodal toolbox. Current comparisons must name the exact benchmark suite, task versions, languages, modalities, and aggregation method. The project has also evaluated a growing number of models beyond the 33 represented by the original page.

MTEB is designed to be massive, multilingual, and extensible. It includes a large number of datasets and summarizes thousands of results on its leaderboard. The benchmark is also open to contributions, such as new tasks, datasets, metrics, or leaderboard additions.

The benchmark comes with open-source code and a public leaderboard, which can be found on the Hugging Face platform. This allows researchers and practitioners to compare the performance of various models and to submit their own models for evaluation.

The MTEB project is hosted on GitHub, where the code and resources for the benchmark are available under the Apache-2.0 license. The benchmark has been discussed in academic conferences and papers, highlighting its role in providing clarity on model performance across various embedding tasks and serving as a gateway to finding universal text embeddings applicable to a variety of tasks.

MTEB English v2 snapshot (June 5, 2025)

The table below reproduces MTEB English v2 results from the official Qwen3 Embedding model card, released June 5, 2025. Qwen states that comparison-model scores were retrieved from the MTEB leaderboard on May 24, 2025. MTEB maintains versioned task and language suites, so a model cannot be called the overall MTEB leader without naming the suite and aggregation method. These English v2 scores should not be mixed with multilingual-suite results.

These two means summarize performance across the English v2 suite, but they do not replace the task-family scores or operational details needed to choose an embedding model. The archived snapshot below preserves that more granular view from an earlier MTEB suite.

For reproducible comparisons, use the MTEB evaluation code and inspect the versioned submissions in the MTEB results repository. Record the suite, task set, languages, metrics, and aggregation method alongside every result.

Historical leaderboard snapshot (January 15, 2024)

In the January 15, 2024 snapshot, voyage-lite-02-intruct led the reported table.

Historical MTEB leaderboard snapshot (January 15, 2024)

This archived table preserves the page's January 15, 2024 leaderboard snapshot for the original 56-dataset MTEB suite. It includes per-task-family results plus model size, embedding dimensions, and maximum token length, which the current two-mean English v2 summary does not expose. The snapshot predates later task revisions and suite versioning, some model-size values were not recorded in the original page, and its scores must not be compared directly with English v2 results.

Manual maintenance should verify the archived snapshot against a versioned MTEB results export, recover the unrecorded model-size fields where authoritative metadata is available, and record exact task revisions before these historical scores are used in a new comparison.

How does MTEB work?

MTEB works by providing a comprehensive framework for evaluating the performance of text embedding models across these tasks. It includes a total of 58 datasets and spans 112 languages. The datasets contain varying text lengths and are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph.

MTEB provides a common framework for selecting versioned benchmark suites, loading compatible models, running the specified tasks and splits, and recording results with model metadata. The current project supports multilingual and multimodal evaluation in addition to the original text-only release.

Current framework context: MTEB now provides a common framework for selecting versioned benchmark suites, loading compatible models, running specified tasks and splits, and recording results with model metadata. The current project supports multilingual and multimodal evaluation in addition to the original text-only release.

Models are benchmarked on these tasks and datasets, and the results are made publicly available on a leaderboard. This allows for a comparison of the performance of different models across a wide range of tasks and languages. The benchmarking process has revealed that no single text embedding method consistently outperforms others across all tasks, indicating that the field has not yet converged on a universal text embedding method.

MTEB comes with open-source code, allowing researchers and developers to use the benchmark for their own models and contribute to the project. This makes MTEB not only a tool for evaluation but also a platform for collaboration and progress in the field of text embeddings.

The original MTEB release organized text evaluation into these eight task families:

The Massive Text Embedding Benchmark (MTEB) covers the following 8 embedding tasks:

1. Bitext mining
2. Classification
3. Clustering
4. Pair classification
5. Reranking
6. Retrieval
7. Semantic Textual Similarity (STS)
8. Summarization

When to use MTEB

The Massive Text Embedding Benchmark (MTEB) is used when you want to evaluate the performance of text embedding models across a diverse range of tasks and datasets. It is particularly useful when you want to compare your model's performance with other models in the field.

MTEB is beneficial when you want to:

1. Benchmark your model: MTEB allows you to benchmark any model that produces embeddings and add its results to the public leaderboard. This can help you understand how your model performs relative to other models on a variety of tasks and datasets.

2. Choose a model for a specific task: Since model performance varies significantly depending on the task and dataset, MTEB can help you decide which model to use for a specific task. By checking the various tabs of the leaderboard, you can identify the models that perform best on the task you are interested in.

3. Contribute to the field: MTEB is open-source and encourages contributions from the community. If you have developed a new task, dataset, metric, or model, you can contribute it to MTEB and help advance the field of text embeddings.

4. Research: If you are a researcher in the field of text embeddings, MTEB can provide you with a comprehensive benchmark for your studies. It can help you understand the current state of the art and identify areas where further research is needed.

Limitations of MTEB

The Massive Text Embedding Benchmark (MTEB) is a comprehensive tool for evaluating the performance of text embedding models across a variety of tasks. However, it does have several limitations:

1. Lack of Long Document Datasets: MTEB covers multiple text lengths, but it does not include very long documents. The longest datasets in MTEB only have a few hundred words, and longer text sizes could be relevant for use cases like retrieval.

2. Task Imbalance: Tasks in MTEB have a different amount of datasets. For instance, summarization consists of only a single dataset. This means MTEB average scores, which are computed over all datasets, are biased towards tasks with many datasets, notably retrieval, classification, and clustering.

3. Limited Multilinguality: While MTEB contains multilingual classification, STS, and bitext mining datasets, retrieval and clustering are English-only. This limits the benchmark's ability to comprehensively evaluate models geared towards multilingual retrieval datasets.

4. Absence of Code Datasets: MTEB does not contain any code datasets that could be used to benchmark code models.

5. Limited Dataset Diversity: Some tasks are underrepresented in MTEB, such as summarization or pair classification. As MTEB grows, there are plans to add more datasets to these underrepresented tasks.

Despite these limitations, MTEB remains a valuable tool for benchmarking text embedding models, and its extensibility allows for the potential addition of new tasks, datasets, and metrics.

Limitations of MTEB comparisons

MTEB's breadth makes evaluation setup especially important:

1. Suite and task versioning — Scores can change as datasets, task implementations, and benchmark suites are revised. Every result should record exact versions and splits.
2. Aggregation choices — Mean-by-task, mean-by-task-type, and dataset-weighted aggregates answer different questions and can rank the same models differently.
3. Coverage differences — Text, image, multilingual, code, long-document, and domain-specific suites have different coverage. A strong result on one suite does not establish a universal embedding leader.
4. Evaluation conditions — Prompt prefixes, truncation length, pooling, similarity functions, quantization, and retrieval infrastructure can materially affect results.
5. Operational tradeoffs — Accuracy leaderboards do not by themselves capture latency, memory use, vector size, licensing, or serving cost.

MTEB remains valuable because its versioned suites and open results make these choices inspectable and reproducible.