Model merging is a technique that combines the parameter weights of two or more separately fine-tuned large language models into a single, unified model. Unlike approaches that mitigate catastrophic forgetting through continuous pre-training or multi-task fine-tuning, model merging combines capabilities without requiring any additional training data, backpropagation, or GPU compute resources.
The core principle behind model merging rests on the assumption that models initialized from the same pre-trained base model (such as Llama 4 or Qwen3) navigate similar optimization landscapes during fine-tuning. Because their weight matrices share a common origin, their fine-tuned states can be mathematically combined. By interpolating these weights, developers can create a composite model that inherits the specialized knowledge, instruction-following capabilities, and domain expertise of all parent models.
Model merging lowered the cost of building multi-skill open-source models. Instead of training a single large model from scratch to handle coding, creative writing, and mathematics, engineers can independently fine-tune smaller models on specialized datasets and merge them. The result is a multi-specialty model that can match or exceed the performance of its individual parent models on tasks that draw on more than one of their specialties.
Model merging techniques have evolved from simple linear interpolation to algorithms designed to mitigate parameter interference and preserve the distinct capabilities of each parent model.
Linear interpolation often falls short when combining neural network weights because it fails to account for the high-dimensional geometric properties of the weight space, sometimes leading to a loss of variance and diminished model performance. Spherical Linear Interpolation (SLERP) addresses this by interpolating along the great circle of the high-dimensional sphere defined by the parameter vectors.
SLERP maintains the magnitude and geometric orientation of the weights, ensuring that the merged parameters smoothly transition between the parent models without collapsing towards the origin. It is highly favored for merging exactly two models, particularly when combining a base model with an instruction-tuned variant, as it preserves the foundational knowledge while cleanly integrating conversational alignments.
When merging multiple models (more than two), parameter interference becomes a significant hurdle. If one model's weights push a parameter in a positive direction while another pushes it in a negative direction, a simple average cancels out both signals, leading to degraded performance.
Ties-Merging addresses this through a three-step process:
This method minimizes interference and allows for the successful integration of many domain-specific models into a highly capable generalist model.
DARE takes a sparsity-based approach to model merging. It assumes that most weight updates during fine-tuning are redundant or minimally impactful. DARE randomly drops a significant percentage (often up to 90%) of the fine-tuned delta weights, the differences between the fine-tuned model and the base model, setting them to zero.
To compensate for the lost magnitude, it rescales the remaining weights by a specific factor. By pruning the weight updates before merging, DARE reduces the probability of parameter conflict, allowing researchers to combine a large number of models without suffering from representation collapse.
Unlike the weight interpolation techniques mentioned above, Frankenmerging (often referred to as Passthrough merging) manipulates the architecture of the model rather than averaging weights. It involves stacking layers from different models on top of one another. For instance, you might take the first 16 layers of a 32-layer model and concatenate them with the last 24 layers of another model, creating a 40-layer hybrid.
Frankenmerging can produce coherent models despite the architectural mismatch this introduces, which suggests transformer layers tolerate some rearrangement without breaking down.
Understanding the tradeoffs between these techniques is essential for designing an effective model merging strategy.
| Technique | Primary Mechanism | Parameter Interference Mitigation | Recommended Use Case | Complexity |
|---|---|---|---|---|
| Linear | Weighted averaging of parameter matrices | None. High risk of feature cancellation. | Simple combinations of closely related models | Low |
| SLERP | Spherical geometric interpolation | Low. Maintains weight magnitude and variance. | Merging exactly two models (e.g., Base + Instruct) | Medium |
| Ties-Merging | Sign consensus and magnitude trimming | High. Discards conflicting parameter updates. | Merging 3+ domain-specific models | High |
| DARE | Random sparsity dropout and rescaling | Very High. Minimizes overlap of active weights. | Mass-merging many models; extreme parameter efficiency | High |
| Frankenmerging | Layer concatenation and stacking | N/A (Does not average weights). | Experimenting with increased model depth and capacity | Low |
The rapid proliferation of model merging can largely be attributed to the open-source library MergeKit, created by Charles Goddard and maintained by Arcee AI. MergeKit provides a memory-efficient framework that allows anyone with standard consumer hardware, such as an Apple Silicon Mac or a standard CPU, to perform complex merges without relying on GPU clusters. By operating out-of-core and loading only necessary layers into memory sequentially, MergeKit made merging accessible to individual developers rather than only well-resourced labs.
MergeKit's impact was most visible on the Hugging Face Open LLM Leaderboard, which ran from 2023 until Hugging Face retired it in 2025. During that period, merged models regularly occupied top positions alongside foundational base models trained from scratch. Community creators combined top-performing mathematics, coding, logic, and creative role-playing models using Ties or DARE techniques, producing composite models that scored well on standardized benchmarks like MMLU, GSM8K, and HumanEval. That period showed that merging could meaningfully improve on the current generation of open fine-tunes without requiring anyone to train a new base model from scratch, though leaderboard rank did not always translate into better real-world performance, and some high-scoring merges were later found to overfit to the benchmark set.
MergeKit's licensing history traced the technique's commercial arc: Arcee moved the library from LGPL to a Business Source License in January 2025 as merging became a production tool, then returned it to LGPLv3 effective October 31, 2025, after concluding the custom license slowed adoption more than it protected the business. The library also grew beyond hand-tuned recipes: mergekit-evolve implements evolutionary merging, using the CMA-ES optimization algorithm to search merge parameters against a target benchmark, an approach Sakana AI demonstrated in its 2024 evolutionary model merge research (later published in Nature Machine Intelligence) but did not release code for.
With the leaderboard gone, merging settled into two durable production roles.
Checkpoint averaging inside post-training pipelines. The idea descends from model soups (Wortsman et al., 2022), which showed that averaging weights of models fine-tuned with different hyperparameters improves accuracy and robustness at no inference cost. Frontier labs now use this routinely: Meta's Llama 3 report describes averaging models produced with different data mixes and hyperparameters at each reward-modeling, SFT, and DPO stage. Merging here is not a hobbyist shortcut but a variance-reduction step in serious training runs.
Capability composition for enterprise models. Arcee built commercial models this way, merging domain fine-tunes onto strong open bases rather than training from scratch, and the pattern generalizes: train small specialist fine-tunes, merge, then evaluate against the specialists individually. The failure mode to watch is silent capability regression, where the merged model keeps benchmark scores but loses long-tail behaviors of a parent, so post-merge evaluation needs to cover each parent's domain, not just aggregate scores.
For enterprise applications, model merging offers an efficient, cost-effective pathway to deploying multi-specialty Small Language Models (SLMs). In an era where inference costs and latency are critical bottlenecks, organizations often struggle to justify hosting large 70B parameter models for everyday tasks. Instead, they can leverage model merging to create capable 7B or 8B parameter models tailored specifically to their operational needs. Unlike a mixture-of-experts architecture, which routes each request to a subset of specialized sub-networks at inference time, a merged model blends the parent weights into one static model that runs like any other dense model.
Imagine a healthcare technology company that needs an AI assistant capable of parsing complex medical literature, writing secure Python scripts for patient data analysis, and interacting with users in a compassionate, empathetic tone. Instead of attempting a complex, multi-stage, multi-task fine-tuning pipeline on a single model, which often leads to catastrophic forgetting or unbalanced performance, the engineering team can take a modular approach. They can independently fine-tune three separate LoRAs or full-weight updates:
Using Ties-Merging or DARE, these three specialized models can be unified into a single 8B parameter SLM. This composite model requires less VRAM for inference, reducing cloud hosting costs while delivering specialized, reliable performance across all three required domains. Model merging also gives the organization long-term agility. If the Python coding standards change or new medical guidelines are released, individual domain models can be retrained and re-merged into the production system without disrupting the entire AI architecture or requiring a full retraining cycle.
Do I need a GPU to merge models? No. Thanks to tools like MergeKit, model merging can be executed purely on a CPU. Because the process relies on mathematical operations applied to weight matrices rather than backpropagation or gradient descent, you only need enough system RAM to hold the model weights during the operation. This makes it highly accessible for independent developers and smaller research teams.
Can I merge models with different architectures? Generally, no. Most weight interpolation techniques (SLERP, Ties, DARE) require the models to share the exact same architecture and base model lineage (e.g., merging two Qwen3-8B fine-tunes). If the layer counts, hidden dimensions, or vocabulary sizes differ, the parameter matrices will not align for mathematical combination. However, some advanced structural merging techniques like Frankenmerging allow for minor architectural variations, though this remains an experimental approach.
Does model merging lead to a loss of capabilities? If done improperly, such as using simple linear interpolation on many disparate models, parameter interference can lead to destructive cancellation and degrade performance. However, using algorithms like Ties-Merging and DARE typically preserves the core capabilities of the parent models. In some cases, it results in capabilities that none of the parent models possessed individually, drawing on the diverse representations learned across different datasets.
Is model merging still used now that the leaderboard era is over? Yes, but differently. The hobbyist wave of benchmark-chasing merges faded with the Open LLM Leaderboard's retirement in 2025, while two production uses grew: checkpoint averaging inside frontier post-training pipelines (Meta describes doing this at every Llama 3 post-training stage) and composing enterprise domain models from specialist fine-tunes, the approach Arcee commercialized.
What is the difference between merging and fine-tuning? Fine-tuning involves updating a model's weights by passing new training data through the network and calculating gradients via backpropagation. It is a computationally expensive process that teaches the model new behaviors or knowledge. Model merging, on the other hand, combines the already updated weights of multiple fine-tuned models mathematically. It requires zero training data, no backpropagation, and minimal compute power, serving as a post-training synthesis step rather than a learning step.
More terms
Glossary term
Glossary term
Collaborate with your team on reliable Generative AI features.
Want expert guidance? Book a 1:1 onboarding session from your dashboard.