Jitesh K Prasad PRO

@chronossystemic Thanks, and I agree. Bits per byte going down does not mean the model knows more. On the knowledge and reasoning tests Vemana is at chance, like most open models under 1B we tested, and we say that on the card.

All our held-out numbers are on fresh Telugu Wikipedia pages, so outside Wikipedia we have not measured it. All 12 checkpoints are public, so anyone can track bits per byte and a fixed task side by side across the run, on whatever Telugu text they care about:
https://huggingface.co/udgamlabs/Vemana-Checkpoints

@dipankarsarkar Thanks for running it, and your 65.9% and 56.2% match ours exactly. I did not know the main → में miss comes from the word list and not from the missing context. So our card is not fully right there, the model alone gets मैं, and it is the list that pulls it to में.

And the top-2 point is a good one. tr.word() already gives all 8 candidates, so a keyboard can show the runner-up as a chip, and by your count that takes it to 79.0%.

On your question, no, we have not tried a two-word prior. Context from the previous word was on our list of possible fixes, but we never tested it, so I do not know yet what it would cost in size. Everything is open, the model, the 200,000-word list and the runner, so anyone who wants to try it on top can.

@Kreout You are right, and thanks for this. I just checked, right now tr.text() changes every run of Latin letters, so name@example.com comes out as नाम@एग्जाम्पल.कॉम, and a URL or a user_id gets changed the same way. In a real text field that would surprise people.

For now, if you are building with it, you can keep URLs, emails and IDs as they are on your side and call tr.word() only on the words. The runner is one small JS file and it is all open, so it is easy to add that rule. And I agree this belongs on the card, next to the accuracy.

@Twu31 Good point, and sorry for the late reply. No, we have not tried that format. Right now the two formats change two things at once, how the answer is scored and whether the model sees the other choices, so the cloze minus letter gap mixes both. I think your version would separate them.

One thing we found after the post: 19 of the 27 models pick one favourite letter on more than half the questions. Gargi picks B on 94% of them. So a big part of the letter score is just which letter a model likes, but without your third format we cannot say how much of the gap is the hidden choices.

Every model version is pinned on the board, and the dataset has the per-question results for 25 of the 27, so anyone can run it and compare question by question:
https://huggingface.co/datasets/udgamlabs/small-model-board-scores

@8BitStudio Sorry, I missed this one. Yes, I think you are right, at this size the data is basically everything. For Gargi it was openly licensed and public-domain text plus synthetic text from open models, the details are on the card. But I agree, at 17B tokens what you pick matters a lot more than how much you have.

And also your result is really interesting, division and carrying at 18B tokens. If you put that model out, I would love to try it.

@dipankarsarkar Yes, our numbers match yours, SmolLM2-360M +0.97 and Gargi +0.17. And you are right, the four letter shares are enough to check the own-mix gap, so anyone can check nltower without its per-question rows. I like that a lot, because then every number on the board is checkable.

I have to be honest on one thing. The board does not have these columns yet. We built the favourite letter, the own-mix baseline and the gap for all 27 models on our side, but that update is not live.

And thanks again for all the work on this. I think you found more in our scores than we did.

@dipankarsarkar You are right, the own-mix null is the correct one. We checked it on our side. SmolLM2-360M goes from +2.6 against its favourite letter to +0.9 against its own mix, and Gargi stays at about +0.2 either way. Against own mix only LFM2.5 (+13.3), the two Falcons (+8.2 and +7.9) and nltower (+3.9, we have its rows but cannot publish them) clear 2 points. So I think the own-mix gap is the column to show, not the share.

Rotation is not planned right now. But when we do it, yes, we will keep the unrotated column next to it, because like you said the gap between them is the bias.

@dipankarsarkar Thanks for actually running the parquet. We re-counted from the raw logs with the real answer key, and your numbers hold exactly. Gargi picks B on 94.1% of the questions and scores 24.89, against 24.65 for always answering B. Gargi-Base is 66.1% B, and accuracy on the non-B questions goes from 11.8 to 2.0 after SFT. So you are right, SFT did move something, it just pushed the model onto B.

I think the cleanest way to see it is each model against its own favourite letter. Only 5 of the 27 beat always answering their favourite letter by more than 2 points: LFM2.5-230M (+13.3), the two Falcon-H1-Tiny models (+9.5 and +8.5), nltower (+3.7) and SmolLM2-360M (+2.6). The other 22, including all of ours, are less than 2 points away from it. So for most of these models the letter score is basically just which letter they like.

On your question, I think rotating the options is the cleaner fix, because then a model stuck on one letter gets exactly 25 whichever letter it is. Off-modal accuracy puts each model on a different set of questions. For now we will add the favourite-letter share to the board, so anyone can see which letter each model leans on.

Would you also show each model's score minus its own favourite letter, or is the share enough?

@RakshithFury Yes, I think you are right about the thinking mode. At this size the gain is small. Gargi gets 32.9 on GSM8K with thinking off and 36.0 with it on, so about 3 points. On our smaller model Laghu (152M) it helped only on short arithmetic, and it actually made instruction following and multi-turn chat worse. We kept it mainly because I wanted the full package of a bigger model (thinking switch, multi-turn, tool format, 16K context) in something that runs on a laptop. But that is not where the quality comes from.

And I agree on RL also. Our Laghu try is one result with a weak base model, so it does not show that RL cannot work below 500M. If the base is good enough, it should help.

On your question: yes, basically. We are a small lab and we fund it ourselves, so a low budget is not really a choice for us. But I think it is also a useful question by itself, how far a small model can go if you are careful with the data and the evals. That is why we run every comparison ourselves and put the per-question results out.

@dipankarsarkar Following up on this. Because of your point we put the full table out properly, it is live now: https://huggingface.co/spaces/udgamlabs/small-model-board

It has 27 small models, all under 400M, all run the same way, with letter and cloze MMLU side by side and the cloze minus letter column like you suggested. The per-question data is also open if you want to check the dial yourself: https://huggingface.co/datasets/udgamlabs/small-model-board-scores. I have also credited you in the write-up: https://udgamlabs.com/blog/small-model-board

Two small corrections to what I said above. We re-checked everything from the raw logs. Falcon-H1-Tiny-90M-Instruct is -4.3, not -4.2. And nltower’s lead basically goes away once you score cloze by answer length, so it is really just LFM2.5 and the two Falcon models. Also the LFM2.5 we ran is their instruct model, not the base.

One funny thing we found: on letter MMLU, just answering “D” every time gets 26.9. A lot of these small models pick the same letter almost every time, so most letter scores under 27 are not saying anything about knowledge.

And to be fair, the OLMES paper (Gu et al., 2024) already showed small models need the cloze format. We just measured it on the current small models. Do tell me if anything looks off.

@RakshithFury Yes, basically two stages. First pretraining from scratch on about 17B tokens, then SFT on chat data and step-by-step maths solutions. The thinking mode also comes from the SFT, there is no RL in it.

And yes, RL is the obvious next step. We actually tried it on our smaller model Laghu (152M), GRPO with distillation from Gargi, and it did not work for us. On our GSM8K dev set it went from 8.6% to 8.2% instead of going up, and the model started looping more. So I think at this size the base model is the bottleneck, it does not get enough answers right for RL to learn much from.

If you have seen RL work well below 500M, do share, I would like to read it.

@dipankarsarkar Yes, I think the cloze column is the better one for the card. And thanks for doing the math on it.

On your question: no, our SFT did not move the dial, at least not in our runs. Gargi-Base is letter 25.3 / cloze 30.1, and Gargi (the SFT chat model) is letter 24.9 / cloze 29.6. So both moved down by about half a point, which is basically noise on 14k questions, and the gap stays at about 4.8. We see the same with SmolLM2-135M vs its Instruct version (letter 24.5 to 22.9, cloze 29.9 to 27.6) and with our older Udgam-384M, Base vs Chat (25.2 / 29.0 to 24.9 / 29.0). All of these are the same lm-eval setup, 0-shot, without the chat template.

I think the reason is that our SFT data is mostly chat and step-by-step maths, so there is probably not much A/B/C/D practice in it. With the chat template it may look different, we have not tried that.

One more thing that supports your dial idea. We ran about 25 small models on both, and the only ones where letter beats cloze are LFM2.5-230M (-8.3), Falcon-H1-Tiny-90M Base (-3.8) and Instruct (-4.2), and nltower-100m (-2.3). Falcon’s base model already has it, so for them the skill seems to come from pretraining, not from chat tuning.