The Kaitchup – AI on a Budget

The Kaitchup – AI on a Budget

Share this post

The Kaitchup – AI on a Budget
The Kaitchup – AI on a Budget
The Mayonnaise: Rank First on the Open LLM Leaderboard with TIES-Merging

The Mayonnaise: Rank First on the Open LLM Leaderboard with TIES-Merging

Trim and merge LLMs while keeping the same number of parameters

Benjamin Marie's avatar
Benjamin Marie
Jan 29, 2024
∙ Paid
6

Share this post

The Kaitchup – AI on a Budget
The Kaitchup – AI on a Budget
The Mayonnaise: Rank First on the Open LLM Leaderboard with TIES-Merging
Share

Following the release of Mixtral, we first saw a new trend of LLMs being mixtures of experts (MoEs) combining several other LLMs. We saw how to make our own MoE in a previous article:

Maixtchup: Make Your Own Mixture of Experts with Mergekit

Maixtchup: Make Your Own Mixture of Experts with Mergekit

Benjamin Marie
·
January 18, 2024
Read full story

However, while MoEs are an interesting way to combine LLMs while preserving the full knowledge of each of the LLMs combined, they are often unreasonably large and consequently require a GPU with a lot of memory. Running them on consumer hardware is challenging.

A more and more popular alternative to combining LLMs is merging them into one LLM without increasing the number of parameters. There are several simple algorithms to do that.

The Kaitchup – AI on a Budget is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In this article, I focus on the TIES-merging method. This is one of the best-performing merging methods that is also simple to understand. I first explain how it works by briefly reviewing the NeurIPS paper that introduced it. Then, we will try and evaluate different combinations of LLMs using mergekit. Using this method, we will see how easy it became to rank first on the Open LLM Leaderboard in the 7B-class of LLMs. My Mayonnaise is still ranked first while I’m writing this article.

These LLM combinations can be fully replicated by using this notebook:

Get the notebook (#41)

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 The Kaitchup
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share