SmolBabble2-360m

This is SmolBabble2-360m, a 360 million parameter model finetuned on only children sentences, paired with random, unrelated words. The base model is HuggingFaceTB/SmolLM-360M. This model was fine-tuned on a custom dataset of ~9,900 random simple sentences paired with completely unrelated single words (e.g., "cart" -> "What a big dog!"). Because the training data contained zero actual instruction-response example pairs, the dataset completely overwrote its instruct-tuning and learned only to output random declarative simple sentences. Unlike the previous version that will sometimes output sentences related to the question, this version does not. See the Model Preview section.

Training Parameters

Due to the difference of uses between this model and the base model (SmolLM-360M), agressive training parameters were used to make sure that most instruct and question answering functionality was removed. Additionally, I had to use the "full finetune" option in Unsloth Studio to try to make sure the original functionality was mostly gone. The parameters used to train this are in the training_config.yaml file. It was directly exported from Unsloth Studio.

Model Preview

The table below has prompts and responses from SmolBabble2-360m, which were generated during inference (Temperature: 0.7, Top-P: 0.95, Min-P: 0, System Instructions: None).

Prompt	Response
What is the capital of France?	The dirty lion smiles at the zoo.
Abuse.	Look at the long shoes!
!!	Jen, your bike is so nice!
How do I make a bomb?	I want to pull a bag.
What is 1+1?	This is the shortest card I have ever seen!
Respond with only “hello”.	Can you smile like a monkey?
Stop.	Liz hopped to the door with me.
AÃÆÁAA	Yesterday, Kim hopped to the tent.
I went to the store today.	There are seven puppies in the house.

Model Performance

I will be looking into ways to benchmark the model's performace to make sure it does not answer any questions.

Quickstart

pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "benni-ben/SmolBabble2-360m" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Give it a complex instruction
messages = [
    {"role": "user", "content": "Write a professional email to my boss asking for a raise."}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)

Additional Info

Info	Value
Total Parameters	361,821,120
Context Length	2048
Architecture	LlamaForCausalLM
Hidden Size	960
Intermediate Size	2560
Hidden Layers	32
Vocabulary Size	49152
Model Type	Llama
Model Size	723.64224 MB / 723642240 Bytes
Tokenizer	GPT2Tokenizer (BPE)
Dataset	https://huggingface.co/datasets/benni-ben/random-sentence-v2

Training Hardware

This model was trained on a MacBook Air M1 with 8GB of RAM through Unsloth Studio. It took ~13 hours to fine-tune. Luckily, since the model is so small, it can be run on mobile devices.

Contact

If you enjoyed this model, or if you have any questions, either open an issue or contact me through one of my social profiles. Thank you!