---
base_model: google/gemma-4-2b-it
language:
- sq
- en
license: apache-2.0
datasets:
- klei1/bleta-sq-dataset-v1
tags:
- text-generation
- gemma
- albanian
- language-model
- fine-tuned
model_name: Bleta SQ 2B
pipeline_tag: text-generation
---

# Bleta SQ 2B — Albanian Language Model

Bleta is a fine-tuned Gemma-4 2B model specialized for the **Albanian language (Shqip)**. Built to understand and generate natural, grammatically correct Albanian text.

> *Bleta* (🐝) means "bee" in Albanian — a symbol of diligence and precision.

## Model Details

| Property | Value |
|----------|-------|
| **Base Model** | [google/gemma-4-2b-it](https://huggingface.co/google/gemma-4-2b-it) |
| **Architecture** | Gemma4ForConditionalGeneration |
| **Parameters** | ~2 Billion |
| **Fine-tuning Method** | LoRA → merged into full weights |
| **Language** | Albanian (sq), English (en) |
| **License** | Apache 2.0 |

## Training Dataset

Fine-tuned on **[klei1/bleta-sq-dataset-v1](https://huggingface.co/datasets/klei1/bleta-sq-dataset-v1)** — a curated Albanian language instruction dataset covering conversation, grammar, reasoning, and general knowledge.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "klei1/bleta-sq-2b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Cila eshte kryeqyteti i Shqiperise?"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.2,
    do_sample=True,
)

response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)
```

## Recommended Generation Parameters

| Parameter | Value | Notes |
|-----------|-------|-------|
| `temperature` | 0.7 | Balanced creativity |
| `max_new_tokens` | 400–512 | Prevents loops |
| `repetition_penalty` | 1.2 | Reduces repetition |
| `top_p` | 0.95 | Nucleus sampling |

## Capabilities

- Albanian conversational AI
- Grammar correction and explanation
- Albanian text generation and creative writing
- Translation (Albanian ↔ English)
- General knowledge in Albanian
- Question answering

## Limitations

- 2B parameter model — complex reasoning may be limited
- Primarily trained on Albanian; performance varies by topic
- May occasionally produce grammatically imperfect outputs

## Bleta Model Family

| Model | Params | Focus |
|-------|--------|-------|
| [bleta-sq-2b](https://huggingface.co/klei1/bleta-sq-2b) | 2B | **Albanian general** |
| [bleta-meditor-27b](https://huggingface.co/klei1/bleta-meditor-27b) | 27B | Medical + specialized |
| [bleta-logjike-27b](https://huggingface.co/klei1/bleta-logjike-27b) | 27B | Logic + reasoning |
| [bleta-1B](https://huggingface.co/klei1/bleta-1B) | 1B | Lightweight |

## Citation

```bibtex
@model{bleta_sq_2b_2026,
  title  = {Bleta SQ 2B: Gemma-4 Fine-tuned for Albanian Language},
  author = {klei1},
  year   = {2026},
  url    = {https://huggingface.co/klei1/bleta-sq-2b}
}
```

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).