Post
72
Sorbet Mini Experimental is out!
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
