Model Merging3 articles

Model Merging

Articles

Model Merging and Task Arithmetic: Mathematical Foundations, Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation (SLERP)
Model Merging and Task Arithmetic: Mathematical Foundations, Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation (SLERP) Model merging enables the combination of multiple specialized large language models into a unified checkpoint without retraining, backpropagation, or access to the original training datasets. Rather than running costly multi-task supervised fine-tuning across disparate domains, practitioners can directly manipulate parameter tensors in weight space to fuse ca
August 28, 20261 min
Model Merging in Weight Space: Mathematical Foundations of Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation
Training large language models across multiple downstream domains typically requires expensive multi-task supervised fine-tuning (SFT) or sequential continual learning. Both paths present steep practical challenges: multi-task training demands simultaneous access to massive, curated datasets and massive compute budgets, while sequential fine-tuning suffers from catastrophic forgetting. Model merging has emerged as a data-free, compute-efficient alternative. By combining the parameters of multip
August 27, 20261 min
Model Merging in Large Language Models: How Task Arithmetic, TIES, and DARE Combine Checkpoints Without Training
Fine-tuning foundation models for specialized tasks typically produces isolated checkpoints. A model adapted for mathematical reasoning retains high numerical precision but often degrades in general dialogue or code generation. Traditionally, unifying these capabilities required multi-task training: gathering mixed datasets, re-running optimization across multiple GPUs, and managing gradient conflicts during backpropagation. Model merging provides an alternative paradigm. By operating directly
August 18, 20261 min

Model Merging

Articles

Model Merging and Task Arithmetic: Mathematical Foundations, Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation (SLERP)

Model Merging in Weight Space: Mathematical Foundations of Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation

Model Merging in Large Language Models: How Task Arithmetic, TIES, and DARE Combine Checkpoints Without Training