Models
Datasets
Spaces
Docs
Enterprise
Pricing
Log In
Sign Up

Collections

Discover the best community collections!

Collections including paper arxiv:2203.05482

If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs

Paper • 2412.04144 • Published Dec 5, 2024 • 6
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation

Paper • 2410.08371 • Published Oct 10, 2024 • 3
MERGE^3: Efficient Evolutionary Merging on Consumer-grade GPUs

Paper • 2502.10436 • Published Feb 9, 2025 • 1
Mergenetic: a Simple Evolutionary Model Merging Library

Paper • 2505.11427 • Published May 16, 2025 • 14

Meta-Learning a Dynamical Language Model

Paper • 1803.10631 • Published Mar 28, 2018 • 1
TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation

Paper • 2003.11963 • Published Mar 26, 2020
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model

Paper • 2212.04960 • Published Dec 9, 2022 • 1
Continuous Learning in a Hierarchical Multiscale Neural Network

Paper • 1805.05758 • Published May 15, 2018 • 2

Dynamics of Transformer Language Model Features

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7
Diverse Weight Averaging for Out-of-Distribution Generalization

Paper • 2205.09739 • Published May 19, 2022 • 1
Fusing finetuned models for better pretraining

Paper • 2204.03044 • Published Apr 6, 2022 • 6
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Paper • 2309.07311 • Published Sep 13, 2023 • 4

Model Merging is a very popular technique nowadays in LLM. Here is a chronological list of papers on the space that will help you get started with it!

Qualitatively characterizing neural network optimization problems

Paper • 1412.6544 • Published Dec 19, 2014 • 4
Convergent Learning: Do different neural networks learn the same representations?

Paper • 1511.07543 • Published Nov 24, 2015 • 2
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

Paper • 1909.11299 • Published Sep 25, 2019 • 2
Model Fusion via Optimal Transport

Paper • 1910.05653 • Published Oct 12, 2019 • 1

A collection of arXiv papers from Chip Huyen's AI Engineering organized by chapter and ordered by when each appears in the book.

Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning

Paper • 2211.04325 • Published Oct 26, 2022 • 1
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Paper • 1810.04805 • Published Oct 11, 2018 • 26
On the Opportunities and Risks of Foundation Models

Paper • 2108.07258 • Published Aug 16, 2021 • 2
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

Paper • 2204.07705 • Published Apr 16, 2022 • 2

Model Merging Papers

Collection of relevant papers about model merging

Qualitatively characterizing neural network optimization problems

Paper • 1412.6544 • Published Dec 19, 2014 • 4
Averaging Weights Leads to Wider Optima and Better Generalization

Paper • 1803.05407 • Published Mar 14, 2018 • 2
Merging Models with Fisher-Weighted Averaging

Paper • 2111.09832 • Published Nov 18, 2021 • 1
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7

Papers about model merging

referenced in the mergekit repo: https://github.com/cg123/mergekit

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7
Editing Models with Task Arithmetic

Paper • 2212.04089 • Published Dec 8, 2022 • 7
Resolving Interference When Merging Models

Paper • 2306.01708 • Published Jun 2, 2023 • 17
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

Paper • 2311.03099 • Published Nov 6, 2023 • 30

If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs

Paper • 2412.04144 • Published Dec 5, 2024 • 6
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation

Paper • 2410.08371 • Published Oct 10, 2024 • 3
MERGE^3: Efficient Evolutionary Merging on Consumer-grade GPUs

Paper • 2502.10436 • Published Feb 9, 2025 • 1
Mergenetic: a Simple Evolutionary Model Merging Library

Paper • 2505.11427 • Published May 16, 2025 • 14

A collection of arXiv papers from Chip Huyen's AI Engineering organized by chapter and ordered by when each appears in the book.

Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning

Paper • 2211.04325 • Published Oct 26, 2022 • 1
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Paper • 1810.04805 • Published Oct 11, 2018 • 26
On the Opportunities and Risks of Foundation Models

Paper • 2108.07258 • Published Aug 16, 2021 • 2
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

Paper • 2204.07705 • Published Apr 16, 2022 • 2

Meta-Learning a Dynamical Language Model

Paper • 1803.10631 • Published Mar 28, 2018 • 1
TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation

Paper • 2003.11963 • Published Mar 26, 2020
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model

Paper • 2212.04960 • Published Dec 9, 2022 • 1
Continuous Learning in a Hierarchical Multiscale Neural Network

Paper • 1805.05758 • Published May 15, 2018 • 2

Model Merging Papers

Collection of relevant papers about model merging

Qualitatively characterizing neural network optimization problems

Paper • 1412.6544 • Published Dec 19, 2014 • 4
Averaging Weights Leads to Wider Optima and Better Generalization

Paper • 1803.05407 • Published Mar 14, 2018 • 2
Merging Models with Fisher-Weighted Averaging

Paper • 2111.09832 • Published Nov 18, 2021 • 1
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7

Dynamics of Transformer Language Model Features

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7
Diverse Weight Averaging for Out-of-Distribution Generalization

Paper • 2205.09739 • Published May 19, 2022 • 1
Fusing finetuned models for better pretraining

Paper • 2204.03044 • Published Apr 6, 2022 • 6
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Paper • 2309.07311 • Published Sep 13, 2023 • 4

Papers about model merging

referenced in the mergekit repo: https://github.com/cg123/mergekit

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 7
Editing Models with Task Arithmetic

Paper • 2212.04089 • Published Dec 8, 2022 • 7
Resolving Interference When Merging Models

Paper • 2306.01708 • Published Jun 2, 2023 • 17
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

Paper • 2311.03099 • Published Nov 6, 2023 • 30

Model Merging is a very popular technique nowadays in LLM. Here is a chronological list of papers on the space that will help you get started with it!

Qualitatively characterizing neural network optimization problems

Paper • 1412.6544 • Published Dec 19, 2014 • 4
Convergent Learning: Do different neural networks learn the same representations?

Paper • 1511.07543 • Published Nov 24, 2015 • 2
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

Paper • 1909.11299 • Published Sep 25, 2019 • 2
Model Fusion via Optimal Transport

Paper • 1910.05653 • Published Oct 12, 2019 • 1

Company

TOS Privacy About Careers

Website

Models Datasets Spaces Pricing Docs