--- license: other license_name: bananamind-community-license-1.0 license_link: LICENSE language: - en library_name: transformers pipeline_tag: text-generation base_model: BananaMind/BananaMind-2-Pro-Preview datasets: - HuggingFaceTB/smol-smoltalk tags: - causal-lm - chat - instruction-tuned - full-finetune - small-language-model - bananamind - bananamind2 - bananamind2-pro - preview-checkpoint - smoltalk - digit-tokenizer - pytorch - safetensors - custom-code - trust-remote-code - custom-architecture --- ![BananaMind 2 Pro](banner.png) # BananaMind-2-Pro-Preview-Chat BananaMind-2-Pro-Preview-Chat is the instruction-tuned version of [BananaMind-2-Pro-Preview](https://huggingface.co/BananaMind/BananaMind-2-Pro-Preview). It was fully fine-tuned for one epoch on [HuggingFaceTB/smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk), with loss applied only to assistant content and the assistant-ending EOS token. The model has **138,971,520 parameters**, a **3,072-token context window**, and a custom **32,768-token digit-aware byte-level BPE tokenizer**. It supports system prompts, multi-turn conversations, grouped-query attention, QK normalization, RoPE, tied embeddings, and KV-cached generation. This chat model is based on the 96K Pro Preview checkpoint, which was pretrained on **51,904,512,000 tokens**. It is a preview-derived instruction model rather than a fine-tune of the completed 100B-token BananaMind 2 Pro run. ![BananaMind 2 Pro Preview Chat benchmark results](benchmarks.png) ## Model Details | Field | Value | |---|---:| | Parameters | 138,971,520 | | Base model | `BananaMind/BananaMind-2-Pro-Preview` | | Architecture | BananaMind2Pro decoder-only Transformer | | Layers | 24 | | Hidden size | 640 | | Intermediate size | 1,920 | | Attention heads | 8 | | KV heads | 4 | | Head dimension | 80 | | Attention style | Grouped-query attention with QK norm | | MLP | SwiGLU | | Position embeddings | RoPE, theta 100,000 | | Normalization | RMSNorm, epsilon 1e-6 | | Vocabulary size | 32,768 | | Context length | 3,072 | | Embeddings | Tied input/output embeddings | | Generation cache | KV cache supported | | Weight format | safetensors | | Training type | Full-parameter supervised fine-tuning | ## Benchmarks ### BananaMind Instruct Bench 1.1 Self-reported results from the official [BananaMind Instruct Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Instruct-Bench-1.1) script. The benchmark contains 300 deterministic, difficulty- and category-weighted instruction tasks. The BananaMind models used their native chat templates; Supra 1.5 used the benchmark's Alpaca-style fallback. Evaluation used greedy decoding, `repetition_penalty=1.1`, and seed 42. | Model | Overall Elo | General | Multi-turn | System Prompts | Context Recall | Code | |---|---:|---:|---:|---:|---:|---:| | **BananaMind-2-Pro-Preview-Chat** | **888** | **720** | **911** | **793** | **1,194** | **1,337** | | [BananaMind-2-Medium-Chat](https://huggingface.co/BananaMind/BananaMind-2-Medium-Chat) | 787 | 534 | 872 | 666 | 1,085 | 1,291 | | [BananaMind-2-Mini-Chat](https://huggingface.co/BananaMind/BananaMind-2-Mini-Chat) | 654 | 389 | 812 | 743 | 924 | 667 | | [Supra 1.5 50M Instruct](https://huggingface.co/SupraLabs/Supra-1.5-50M-Instruct-exp) | 647 | 520 | 804 | 590 | 860 | 780 | | [BananaMind-2-Nano-Chat](https://huggingface.co/BananaMind/BananaMind-2-Nano-Chat) | 643 | 353 | 796 | 600 | 978 | 885 | **Detailed final result** | Scope | Examples | Passed | Weighted Score | Elo | |---|---:|---:|---:|---:| | **Overall** | **300** | **98** | **27.09%** | **888** | | General | 120 | 33 | 21.84% | 720 | | Multi-turn | 75 | 26 | 26.32% | 911 | | System Prompts | 60 | 9 | 11.84% | 793 | | Context Recall | 30 | 19 | 50.53% | 1,194 | | Code | 15 | 11 | 65.26% | 1,337 | | Difficulty | Examples | Passed | Weighted Score | Elo | |---|---:|---:|---:|---:| | Easy | 100 | 59 | 60.58% | 1,014 | | Medium | 100 | 31 | 32.59% | 900 | | Hard | 100 | 8 | 8.53% | 756 | The complete final run used CUDA, bfloat16, the native chat template, and no sampling. Scores can vary with benchmark revision, Transformers version, dtype, hardware, and generation settings. ### Fine-tuning Progression ![BananaMind 2 Pro Preview Chat score progression](score_progression.png) | Optimizer step | Packed tokens | Overall Elo | Passed | Weighted Score | |---:|---:|---:|---:|---:| | 500 | 49,152,000 | 811 | 77/300 | 19.98% | | 1,000 | 98,304,000 | 835 | 85/300 | 22.04% | | 1,500 | 147,456,000 | 858 | 91/300 | 24.19% | | 2,000 | 196,608,000 | 866 | 94/300 | 24.99% | | 2,500 | 245,760,000 | 874 | 94/300 | 25.69% | | 3,000 | 294,912,000 | 875 | 95/300 | 25.84% | | 3,500 | 344,064,000 | 855 | 88/300 | 23.88% | | **4,014 (final)** | **394,522,611** | **888** | **98/300** | **27.09%** | The final export is the strongest measured checkpoint overall. Step 3,000 had a slightly higher code-category Elo of 1,345, but the final model passed more code tasks and scored higher overall. ## Instruction Tuning | Field | Value | |---|---:| | Dataset | `HuggingFaceTB/smol-smoltalk` | | Dataset split | `train` | | Dataset rows | 460,341 | | Epochs | 1 | | Final optimizer step | 4,014 | | Packed tokens processed | 394,522,611 | | Supervised assistant tokens | 301,938,934 | | Sequence length | 3,072 | | Micro batch | 4 | | Gradient accumulation | 8 | | Effective batch | 32 sequences | | Peak learning rate | 1e-4 | | Warmup | 100 steps | | LR schedule | Constant after warmup | | Optimizer | AdamW | | Betas | 0.9, 0.95 | | Weight decay | 0.1 | | Gradient clipping | 1.0 | | Compilation | PyTorch compile enabled | | Seed | 1337 | System and user messages were retained as context but masked from the loss. Only assistant content and the assistant-ending EOS token contributed to the objective. All model parameters were trainable; no adapters or LoRA modules were used. ## Chat Template The tokenizer includes a Jinja template for `system`, `user`, and `assistant` messages: ```text <|system|> {system message} <|user|> {user message} <|assistant|> {assistant response} ``` The role markers are plain text encoded by the existing tokenizer. Fine-tuning did not add or resize any tokens, and the input and output embeddings remain tied. ## Usage This repository contains custom Transformers architecture code and must be loaded with `trust_remote_code=True`. ```bash pip install -U torch transformers safetensors ``` ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "BananaMind/BananaMind-2-Pro-Preview-Chat" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = ( torch.bfloat16 if device == "cuda" and torch.cuda.is_bf16_supported() else torch.float32 ) tokenizer = AutoTokenizer.from_pretrained( model_id, trust_remote_code=True, ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype=dtype, ).to(device).eval() messages = [ { "role": "system", "content": "You are a concise and helpful assistant.", }, { "role": "user", "content": "Write a Python function that squares a number.", }, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, ) inputs = {name: tensor.to(device) for name, tensor in inputs.items()} with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=256, do_sample=False, repetition_penalty=1.1, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, use_cache=True, ) new_tokens = output[0, inputs["input_ids"].shape[1]:] print(tokenizer.decode(new_tokens, skip_special_tokens=True)) ``` For multi-turn chat, append the generated assistant response and the next user message to `messages`, then render the full conversation with the chat template again. ## Suggested Generation Settings For stable responses: - `do_sample=False` - `repetition_penalty=1.1` - `max_new_tokens=128` to `256` - `use_cache=True` For more varied responses: - `do_sample=True` - `temperature=0.6` to `0.8` - `top_p=0.9` - `repetition_penalty=1.1` - `max_new_tokens=128` to `256` Keep a finite generation limit. The combined prompt and generated output must fit within the 3,072-token context window. ## Repository Files | File | Description | |---|---| | `config.json` | Transformers model configuration | | `model.safetensors` | Fine-tuned model weights | | `tokenizer.json` | Custom 32,768-token tokenizer | | `tokenizer_config.json` | Tokenizer metadata and context length | | `chat_template.jinja` | System/user/assistant chat template | | `generation_config.json` | Generation configuration | | `configuration_bananamind2pro.py` | Custom Transformers config class | | `modeling_bananamind2pro.py` | Custom Transformers causal LM class | | `sft_metadata.json` | Fine-tuning provenance and token counts | | `LICENSE` | BananaMind Community License 1.0 | | `banner.png` | Model-card banner | | `benchmarks.png` | Instruct Bench model comparison | | `score_progression.png` | Instruct Bench Elo progression across fine-tuning checkpoints | ## Intended Use and Limitations BananaMind-2-Pro-Preview-Chat is intended for small-model research, local chat experiments, educational demonstrations, code-generation experiments, instruction-tuning studies, and compact-model comparisons. The model has not received dedicated safety alignment. It can hallucinate facts, fail arithmetic or logical tasks, produce insecure or invalid code, misunderstand instructions, and generate biased, repetitive, or otherwise undesirable text. Do not rely on it for medical, legal, financial, safety-critical, or other high-stakes decisions. ## License This repository is released under the [BananaMind Community License 1.0](LICENSE). Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.