Instructions to use aaardpark/gemma-4-31B-it_aard-3bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aaardpark/gemma-4-31B-it_aard-3bit with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="aaardpark/gemma-4-31B-it_aard-3bit",
	filename="gemma-4-31B-it-aaardpark-Q3_K_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use aaardpark/gemma-4-31B-it_aard-3bit with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
# Run inference directly in the terminal:
llama-cli -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
# Run inference directly in the terminal:
llama-cli -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Use Docker

docker model run hf.co/aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

LM Studio
Jan
Ollama
How to use aaardpark/gemma-4-31B-it_aard-3bit with Ollama:
```
ollama run hf.co/aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
```

Unsloth Studio

How to use aaardpark/gemma-4-31B-it_aard-3bit with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for aaardpark/gemma-4-31B-it_aard-3bit to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for aaardpark/gemma-4-31B-it_aard-3bit to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for aaardpark/gemma-4-31B-it_aard-3bit to start chatting

How to use aaardpark/gemma-4-31B-it_aard-3bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use aaardpark/gemma-4-31B-it_aard-3bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Run Hermes

hermes

Docker Model Runner
How to use aaardpark/gemma-4-31B-it_aard-3bit with Docker Model Runner:
```
docker model run hf.co/aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M
```

Lemonade

How to use aaardpark/gemma-4-31B-it_aard-3bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull aaardpark/gemma-4-31B-it_aard-3bit:Q3_K_M

Run and chat with the model

lemonade run user.gemma-4-31B-it_aard-3bit-Q3_K_M

List all available models

lemonade list

gemma-4-31B-it | aard-3bit

15.3 GB GGUF of google/gemma-4-31B-it. Fits 16 GB VRAM. Vision tower weights preserved at FP16.

Runs in llama.cpp, LM Studio, Ollama, and any GGUF-compatible runtime.

Why this quant?

Most 3-bit Gemma 4 GGUFs are labeled "low quality" by their providers. This one scores 96% on GSM8K — within 2 points of full precision.

Standard 3-bit quantization rounds every weight the same way. This quant uses calibration data to figure out which weights actually matter, then protects those during quantization. Same file format, same bit budget, better weight choices.

Quick stats


File	`gemma-4-31B-it-aaardpark-Q3_K_M.gguf`
Size	15.3 GB
Format	GGUF Q3_K_M (3.98 BPW)
GSM8K (5-shot, n=50)	96% (FP16: 98%)
Min VRAM	16 GB

Download & Run

huggingface-cli download aaardpark/gemma-4-31B-it-GGUF \
  gemma-4-31B-it-aaardpark-Q3_K_M.gguf --local-dir .

llama-cli -m gemma-4-31B-it-aaardpark-Q3_K_M.gguf -ngl 99 --chat-template gemma

Outputs

A store offers 30% off, then an additional 20% off the sale price. What's the total discount?

The total discount is 44%. Most people simply add the percentages together (30% + 20% = 50%). However, discounts are applied sequentially, not cumulatively. The second discount is applied to the already reduced price: $1 - (0.7 \times 0.8) = 1 - 0.56 = 0.44$ or 44%.

If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?

5 minutes. Each machine makes one widget every 5 minutes. 100 machines can make 100 widgets in the same 5-minute window.

Write a summary of the French Revolution, but every sentence must start with the letter 'T' and no word can exceed 8 letters.

The king lost power. The poor felt pain. They fought for rights. The crowd took over. Terror came soon. The rule ended.

Write a one-sentence story about a cat.

The ginger cat spent the entire afternoon stalking a single dust mote dancing in a shaft of sunlight.

Find the minimum spanning tree with exactly K red edges. Describe your algorithm, prove its correctness, and implement it in Python.

This is a variation of the Minimum Spanning Tree (MST) problem with a constraint on the number of edges of a specific color. The most efficient way to solve this is using WQS (WQS-C) or the "Aliens" trick. Since we want exactly $K$ red edges, we introduce a penalty $\lambda$ added to every red edge's weight...

(Produced correct WQS/Lagrangian relaxation algorithm with convexity proof sketch, binary search over penalty with Kruskal's inner loop, proper tie-breaking for red edges, and working Python implementation. O(log(MaxWeight) * E log E) complexity.)

VRAM requirements

VRAM	Experience
16 GB	Fits. ~3.5K context with 1.8 GB headroom.
24 GB	Comfortable. 8K+ context.
32 GB+	Full context window.

Details

Base model: google/gemma-4-31B-it
Method: Importance-weighted per-group grid search (3-bit, gs=128) + Q3_K_M via llama.cpp
Vision tower: Weights preserved at FP16
Calibration: Chat-formatted prompts across code, math, reasoning, factual, creative

Model tree for aaardpark/gemma-4-31B-it_aard-3bit

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(217)

this model

aaardpark
/

gemma-4-31B-it_aard-3bit