Instructions to use trollek/NinjaMouse2-2.5B-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use trollek/NinjaMouse2-2.5B-v0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="trollek/NinjaMouse2-2.5B-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("trollek/NinjaMouse2-2.5B-v0.2")
model = AutoModelForCausalLM.from_pretrained("trollek/NinjaMouse2-2.5B-v0.2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use trollek/NinjaMouse2-2.5B-v0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "trollek/NinjaMouse2-2.5B-v0.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trollek/NinjaMouse2-2.5B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/trollek/NinjaMouse2-2.5B-v0.2

SGLang

How to use trollek/NinjaMouse2-2.5B-v0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "trollek/NinjaMouse2-2.5B-v0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trollek/NinjaMouse2-2.5B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "trollek/NinjaMouse2-2.5B-v0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trollek/NinjaMouse2-2.5B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use trollek/NinjaMouse2-2.5B-v0.2 with Docker Model Runner:
```
docker model run hf.co/trollek/NinjaMouse2-2.5B-v0.2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

NinjaMouse2-v0.2

A brand spanking new model with a silly name. Brought to you by Anoia, the Goddess of Things That Get Stuck in Drawers, and the psychologial damage of having optic nerves.

With improved image prompting and assistance.

Quantizations

Thanks to cgus for providing these!

cgus/NinjaMouse2-2.5B-v0.2-iMat-GGUF

Ollama

ollama pull trollek/ninjamouse2:34l-v02-q6_K
ollama pull trollek/ninjamouse2:34l-v02-q5_K_S
ollama pull trollek/ninjamouse2:34l-v02-q4_K_S

Template

It uses the default template of danube2:

<|prompt|>{{instruction}}</s><|answer|>{{response}}</s>

Image prompting

And can be used with the Ollama ComfyUI extension:

Trying to fine-tune the chat model even further was a mistake, but a valuable one to make. So was the name. The model is delightful though and does quite well, but will be the last Kung Fu Mouse that I make. It does what I wanted the first one to do, and I am kind of proud of this one considering how many failures it took.

The rodents and I thank you for your support.

Downloads last month: 3

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for trollek/NinjaMouse2-2.5B-v0.2

Quantizations

1 model

trollek
/

NinjaMouse2-2.5B-v0.2

NinjaMouse2-v0.2

Quantizations

Ollama

Template

Image prompting

Model tree for trollek/NinjaMouse2-2.5B-v0.2

Datasets used to train trollek/NinjaMouse2-2.5B-v0.2