Diffusion Language Models - a philipp-zettl Collection

philipp-zettl 's Collections

Diffusion Language Models

MTG Embedding models

LargeWurstModels

embedding-models

secret sauce FLUX

Diffusion Language Models

updated 7 days ago

Experimental diffusion-style MLM built on top of ModernBERT. Inspired by https://nathan.rs/posts/roberta-diffusion/

philipp-zettl/modernbert-diffusion-instruct

Fill-Mask • 0.1B • Updated Feb 6

Note "base" model trained on HuggingFaceH4/ultrachat_200k
philipp-zettl/modernbert-diffusion-code

Fill-Mask • 0.1B • Updated Feb 7

Note "base" model trained on bigcode/the-stack-dedup (python)
philipp-zettl/modernbert-diffusion-universal

Fill-Mask • 0.1B • Updated Feb 16

Note "base" model trained on multi purpose datasets (all the above + bigcode/the-stack-dedup (json) and fineweb-edu)
philipp-zettl/modernbert-diffusion-alpaca-ft

Fill-Mask • 0.1B • Updated Feb 11

Note FT of philipp-zettl/modernbert-diffusion-universal using tatsu-lab/alpaca
philipp-zettl/modernbert-diffusion-openwebtext

Fill-Mask • 0.1B • Updated Feb 18

Note FT on Skylion007/openwebtext
philipp-zettl/modernbert-diffusion-refactor

Fill-Mask • 0.1B • Updated Feb 11

Note FT on bigcode/the-stack-dedup (python)