Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.
The run isn’t even finished yet.
At the current checkpoint:
• 146M parameters • 72.7B / 75B tokens trained • 26.27 Open SLM Index • 43.80 ArithMark-3 • Trained on a single RTX 5090 • ~90K to 103K tokens/sec during training • ~9 days for the full run • Apache 2.0
For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.
Cagliostro-v3 is currently at 26.27 with only ~72.7B.
That’s around 27x fewer training tokens.
The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80
This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.
The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.
During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.
And everything is open.
The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.
This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.
So 26.27 isn’t the final number.
Really excited to see where the last part of the run lands.
We're releasing the BananaMind SLM Leaderboard! It offers a easier look at which models are actually good for your specific needs. Its primary metric, Intelligence index is a composite of BananaMind Base Bench, PIQA, Hellaswag, ARC Easy and Arithmark 3. It also allows you to see specific categories like Commonsense on a model.
Dude, Where's My Update? I'll tell you where! ~97.6% of my BF16 parameter coordinates didn't move at all, and the ones that did overshot by ~1.33x.
It's nice to do research that doesn't end in disproving yourself once again and moving on to the next subject once in awhile.
Back to the topic, if you've ever wondered why most of your weights are basically ghosting you nearly every step when you store your weights at bf16, Dude, I Measured It.
Hello! Tomorrow, OpenCerebral will be releasing Boris-1.7-D60M-n30M — an experimental architecture. It will be testing a new data mixture, a new tokenizer, and testing Qwen4-like n-gram embeddings.
Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.
Pretrained on 4x more tokens than the previous releases (20b vs 5b). Instruct tuned versions are coming soon. Very interesting models are coming soon too (hint: super long context).
We're delaying BananaMind 2.1! When BananaMind 2.1 Lite was almost done, we benchmarked it and the results we're worse than BananaMind 2 Mini.
We're going to spend alot more time in research on tiny models and then scaling up our techniques to the actual BananaMind 2.1 models!
We're also announcing these new models: BananaMind 2.1 Coder: A 149M instruction tuned coder model trained on 75B tokens + 10B tokens of stack-v3-train. BananaMind 2.1 Pico: A 1M parameter model trained on 22B tokens of data. We also may release BananaMind 2.1 Large with around 100M parameters depending on how much compute we have.
catching up on some bookmarked reads from the summer, reading Antidoom from @liquidai
small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer
they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0%
the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts
three ways it differs from DPO: > trains one token position, mid-generation, instead of whole sequences > spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another > keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put
the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way
and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce
We have updated the BananaMind Base Bench leaderboard! We now have these benchmark cards, they make it way easier to see which models are actually good! We've also added the model advisor. It asks you what you want to use the model for and the parameter range and gives you the best model for your task!
Wow, SLM Arena is getting a lot of traffic! Thank you guys for showing your interest!
To handle the growing demand, I’m moving SLM Arena from a CPU Space to a ZeroGPU Space. Hopefully, this will let me add more models to SLM Arena while keeping it running fast.
I've also added a separate arena + leaderboard for base models!
If there are any models or features you’d like to see, let me know in a reply to this post or in a Community post on the Space!
