slmconsortium (Small Language Model Consortium)

posted an update about 6 hours ago

Post

33

We're going to release our BananaMind 2.1 models very soon!
We're also announcing 2 new models.

All of our models we will train are:
BananaMind 2.1 Flash Lite, 10M parameters with 8M in transformer and 2M in n-gram. 50B pretraining tokens.
BananaMind 2.1 Lite with 25M parameters, 5M in n-gram and 20M in transformer. 75B pretraining tokens.
BananaMind 2.1 Flash with 50M parameters, with undecided n-gram count yet. 100B pretraining tokens.
BananaMind 2.1 Pro with 145M parameters, with undecided n-gram count yet. 150-200B pretraining tokens.
BananaMind 2.1 Coder with 149M parameters with undecided n-gram count yet.
We're now announcing BananaMind 2.1 NanoCoder, a 10M parameter model focused specifically on coding and BananaMind 2.1 MiniCoder which is a 25M parameter model focused on coding.

Follow us:

BananaMind
@Banaxi-Tech
@vovaRL
@DedeProGames

bananamind-research-community

CodeSoft

posted an update about 15 hours ago

Post

66

Sorbet Mini Experimental is out!

It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.

Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.

10 replies

·

Banaxi-Tech

posted an update 1 day ago

Post

1242

We're excited to release BananaMind 2.1 Pico Preview!

It includes the first preview of our BananaMind 2.1 architecture!
This model gets near BananaMind 2 Micro performance at half the parameters and 37.5x less tokens!
Thats insane!

The current architectures includes about 500K parameters of the total 1.5M parameters in n-gram embeddings and the layer 2 is run twice.

It also includes XSA and the XSA refresh gate.

We're still going to improve the architecture in the final release.

Check it out at:

Follow us for more models:

BananaMind
@Banaxi-Tech
@vovaRL
@DedeProGames

2 replies

·

AtAndDev

posted an update 1 day ago

Post

1385

SPECK 2 IS ALREADY OUT: specklabs/Speck2-140M

Pretrained on 4x more tokens than the previous releases (20b vs 5b).
Instruct tuned versions are coming soon.
Very interesting models are coming soon too (hint: moe).

Thanks for everyone supporting!

1 reply

·

KlondikeDev

posted an update 1 day ago

Post

1701

Boris-1.7-D60M-n30M out NOW!

opencerebral/Boris-1.7-D60M-n30M

Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.

Then, Boris-2 will begin training!

KlondikeDev

posted an update 3 days ago

Post

3378

A Preview of Boris-2!

Hello! Tomorrow, OpenCerebral will be releasing Boris-1.7-D60M-n30M — an experimental architecture. It will be testing a new data mixture, a new tokenizer, and testing Qwen4-like n-gram embeddings.

Following this will be Boris-1.8-D60M-n30M, which will test both the n-gram embeddings AND a new architecture.

Then, Boris-2 will begin training!

Edit: the model is OUT NOW! opencerebral/Boris-1.7-D60M-n30M

70 replies

·

Banaxi-Tech

posted an update 3 days ago

Post

3410

Introducing BananaMindOS 3.0

- Complete modern UI redesign
- Adds support for Qwen3.5 0.8B, LFM2.5 230M,350M, SmolLM2 360M, Gemma 3 270M.
- Adds Q7,Q6,Q5,Q3,Q1 quantization formats with a easy to use precision slider
- And more!

The new UI includes:
- New 1024×768 High Quality interface.
- Photographic QOI background.
- Transparent BananaMind, CPU, cube, mouse, and Send icons.
- Proper bitmap cursor.
- Rounded translucent panels and cards.
- Modern model-loading progress window.
- Redesigned inference screen with response and prompt panels.
- Localized redraws for the cursor, clicks, loading progress, and precision slider.

Notice: Qwen3.5 0.8B currently generates garbled text, it will be fixed tomorrow.

See it for yourself
Now Available at https://github.com/BananaMind/BananaMindOS

Prebuild ISOs coming soon!

(also press ? + G if you want to load try to load a 6MB RAM model on 5MB may break)

7 replies

·

Banaxi-Tech

posted an update 4 days ago

Post

3487

We're delaying BananaMind 2.1!
When BananaMind 2.1 Lite was almost done, we benchmarked it and the results we're worse than BananaMind 2 Mini.

We're going to spend alot more time in research on tiny models and then scaling up our techniques to the actual BananaMind 2.1 models!

We're also announcing these new models:
BananaMind 2.1 Coder: A 149M instruction tuned coder model trained on 75B tokens + 10B tokens of stack-v3-train.
BananaMind 2.1 Pico: A 1M parameter model trained on 22B tokens of data.
We also may release BananaMind 2.1 Large with around 100M parameters depending on how much compute we have.

Please give us a follow!

BananaMind
@Banaxi-Tech

---

@vovaRL
@DedeProGames

7 replies

·

AtAndDev

posted an update 4 days ago

Post

133

SPECK1.5 IS COMING SOON!
Same 5B token budget but much better corpus quality.

Also getting a ton of downloads, thanks for everyone downloading and liking <3

specklabs

CodeSoft

posted an update 5 days ago

Post

106

Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.

In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.

Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!

17 replies

·

KlondikeDev

in slmconsortium/README 5 days ago

New member ratification: CodeSoft

1

#3 opened 5 days ago by

CodeSoft

KlondikeDev

updated a Space 5 days ago

README

🏃

KlondikeDev

posted an update 5 days ago

Post

2286

The Small Language Model Consortium has reached 10 members!

Anybody is welcome to join, via two methods:

1. Starting a community discussion on the group’s README
2. Receiving an invite from me: @KlondikeDev

The Discord Community is also live! Feel free to join, even if you aren’t in the consortium, to discuss Small Language Models, or even just AI in general!

https://discord.gg/FngBKjja4

You’re more than welcome to apply!

— KlondikeDev