Instructions to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AMAImedia/Ling-3.1-Flash-BF16-GGUF", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("AMAImedia/Ling-3.1-Flash-BF16-GGUF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("AMAImedia/Ling-3.1-Flash-BF16-GGUF", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
# Run inference directly in the terminal:
llama cli -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
# Run inference directly in the terminal:
llama cli -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
# Run inference directly in the terminal:
./llama-cli -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Use Docker

docker model run hf.co/AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

LM Studio
Jan

vLLM

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AMAImedia/Ling-3.1-Flash-BF16-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AMAImedia/Ling-3.1-Flash-BF16-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

SGLang

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AMAImedia/Ling-3.1-Flash-BF16-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AMAImedia/Ling-3.1-Flash-BF16-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AMAImedia/Ling-3.1-Flash-BF16-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AMAImedia/Ling-3.1-Flash-BF16-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Ollama:
```
ollama run hf.co/AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
```
Unsloth Desktop

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Docker Model Runner:
```
docker model run hf.co/AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S
```

Lemonade

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Run and chat with the model

lemonade run user.Ling-3.1-Flash-BF16-GGUF-IQ1_S

List all available models

lemonade list

Hermes Agent

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use AMAImedia/Ling-3.1-Flash-BF16-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "AMAImedia/Ling-3.1-Flash-BF16-GGUF:IQ1_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

⚡ Each donation funds the next large quant.

I host free GGUF or MoE quants as independent research.
Local hardware: Mechrevo Kuangshi GM7AG0M — RTX 3060 Laptop 6GB GDDR6, 64GB DDR5, i7-12700H (14C/20T, 4.7GHz), Windows 11, Samsung 990 Pro.
Good for imatrix and 0.6–35B-class work in RAM. 9B+ and searches need rented H200/Blackwell, typically $100 per quant.

🎉 Boosty🦄 | ☕ Buy Me a Coffee🦄 | ⭐ DonationAlerts🦄

💚 Thanks to Hugging Face for extra storage.🦄

NOESIS / AMAImedia

Released as part of the NOESIS Professional Multilingual Dubbing Automation Platform (framework: DHCF-FNO — Deterministic Hybrid Control Framework for Frozen Neural Operators)

Founder: Ilia Bolotnikov
Organization: AMAImedia.com
X (Twitter): @AMAImediacom
LinkedIn: Ilia Bolotnikov
Telegram: @djbionicl
Release date: 2026-10-14

AMAImedia

Original repository: inclusionAI/Ling-3.1-flash
GGUF: AtomicChat/Ling-3.1-flash-GGUF

🤗 Hugging Face | 🤖 ModelScope | 🐙 OpenRouter

We're introducing Ling-3.1-flash, an upgraded native hybrid reasoning model designed for complex, real-world tasks. With 560B total parameters, 25B active parameters per token, and a 1M-token context window, Ling-3.1-flash builds upon Ling-3.0-flash with improvements in architecture, performance on real-world capabilities.

With stronger real-world capabilities, Ling-3.1-flash shows improvements across a range of real-world tasks, including general-purpose agents, search, office productivity, and software engineering. It demonstrates strong performance on benchmarks such as GDPVal-AA V2.1, HealthBench Professional, FrontierSWE, and Terminal-Bench 4.0, covering office tasks, healthcare, software engineering, and terminal-based tool use. The model also shows improvements in specialized domains such as finance and materials science.

Model Overview

The model summary information and architecture diagram are as follows:

Architecture	Hybrid-linear MoE
Parameter Scale	Total 560B, Activated 25B
Transformer Layers	63 KDA + 9 Gated MLA (7:1)
Number of Dense Layers	4
Number of Routed Experts	512
Number of Shared Experts	1
Number of Activated Experts	8
Attention Heads	64
Hidden Size	4096
Expert Intermediate Size	1280
Dense Intermediate Size	10240
Vocabulary Size	157184

Evaluation

We have conducted a comprehensive evaluation of Ling-3.1-flash across multiple authoritative benchmarks. Ling-3.1-flash performs strongly on representative coding/agentic benchmarks including Terminal-Bench 4.0, SWE-bench Pro, FrontierSWE, GDPVal-AA V2.1, Automationbench, MCP-Atlas, SkillsBench, etc. In practice, Ling-3.1-flash delivers a strong user experience across frameworks including Claude Code,** Kilo Code**,** Qwen Code**,** Hermes Agent**,** and OpenClaw**, etc. Beyond agentic tasks, Ling-3.1-flash performs strongly on real-world tasks across STEM, healthcare, and finance.

We also introduce LingCode, a harness built on the open-source coding agent Pi, and enhanced to support complex, long-horizon coding tasks that run for hours, involve hundreds of rounds of tool calls, and require repeated context compaction. LingCode delivers significant performance gains on some coding agent tasks, and we are considering open-sourcing it in the future.

Quickstart

SGLang (Recommended)

The day-0 SGLang image includes Ling-3.1-flash support, MTP (NEXTN) speculative decoding, and the Hopper decode optimizations:

docker pull lmsysorg/sglang:dev-ling-3.1-flash

Run Inference

Server (8x H200, TP8 + EP8):

docker run --gpus all --network host --ipc=host --rm \
  lmsysorg/sglang:dev-ling-3.1-flash \
  python3 -m sglang.launch_server \
    --model-path inclusionAI/Ling-3.1-flash \
    --tp 8 --ep 8 \
    --speculative-algorithm NEXTN \
    --host 0.0.0.0 --port 30000

On 4x GB300, use --tp 4 --ep 4 instead. Hopper decode optimizations (optional, ~10% faster batch-1 decode) can be enabled with SGLANG_ENABLE_FP8_BLOCK_GEMV=1 and SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1.

Client

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Ling-3.1-flash",
       "messages": [{"role": "user", "content": "What is the capital of France?"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 1.0,
       "top_k": 20,
       "top_p": 0.95}'

Verified recipes for H200, GB300, H800, H100, and B200, plus HiCache (Mooncake L3), reasoning on/off, tool calling, and measured accuracy/speed numbers, are in the SGLang Ling-3.1-flash cookbook.

vLLM

Environment Preparation

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b vllm https://github.com/vllm-project/vllm.git

cd vllm

VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

Run inference

Server

vllm serve "$MODEL_PATH" \
    --port "$PORT" \
    --trust-remote-code \
    --served-model-name auto \
    --tensor-parallel-size 8 \
    --gpu-memory-utilization 0.85 \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --mamba-cache-mode align \
    --enable-auto-tool-choice \
    --tool-call-parser ling3 \
    --reasoning-parser ling3 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Client
It is recommended to use the sampling params (temperature=1.0, top_p=0.95, top_k=20) and turn on enable_thinkingfor better performance.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", 
       "messages": [{"role": "user", "content": "What is the capital of France?"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 1.0, 
       "top_k": 20,
       "top_p": 0.95}'

Training content summary

The public training-content summary identifying Ling-3.1-flash is available below. Please refer to the document for its covered model versions, training-content scope, summary version, and update date.

This summary concerns training-content disclosure; it does not replace the model’s technical documentation, usage terms, or license.

How to Run Ling 3.1 flash Locally

GGUF builds of inclusionAI's Ling 3.1 flash with our own importance matrix. The calibration corpora behind our builds are public.

Runs in llama.cpp build b11571 or newer.
Day-one upload: more files and the measurements (KL divergence and top-1 against the BF16 model) land on this card today.

Pick a file

File	Size on disk	KL divergence	top-1	Status
`AD-Q8_0`	600.8 GB	0.00503	97.88%	available
`AD-Q6_K`	478.0 GB	0.00557	97.72%	available
`AD-Q5_K_M`	400.2 GB	measuring	measuring	available
`AD-Q4_K_M`	354.4 GB	0.01114	96.86%	available
`AD-IQ4_XS`	337.8 GB	measuring	measuring	uploading today
`AD-IQ4_XXS`	311.4 GB	measuring	measuring	available
`AD-IQ3_M`	277.1 GB	measuring	measuring	available
`AD-IQ3_XXS`	256.4 GB	0.03436	94.48%	available
`AD-IQ2_M`	218.1 GB	measuring	measuring	available
`AD-IQ2_XS`	198.2 GB	measuring	measuring	uploading today
`AD-IQ2_XXS`	174.1 GB	measuring	measuring	uploading today
`AD-IQ1_M`	160.9 GB	measuring	measuring	available, experimental
`AD-IQ1_S`	141.8 GB	measuring	measuring	available, experimental

KL divergence is how far the quantized model's predictions drift from the BF16 model's. Lower is better, and zero means identical. top-1 is how often it picks the same next token the BF16 model would have picked. Both are measured on held-out text; how is at the bottom. Rows that say "measuring" fill in today.

AD- marks an Atomic Dynamic layout: the routed experts (97% of the weights) take the low-bit type, while attention, the shared expert, the router and the dense layers stay at q8_0 or higher. The file name is the type of the experts' gate and up projections; their down projections, and all experts of the last five blocks, get a step more.

How much memory it needs

The file has to fit, plus a little for the context. Context is cheap on this model: 63 of its 72 layers use linear attention with a fixed-size state, and only 9 keep a compressed attention cache. That is about 0.5 GB per conversation plus about 10 KB per token.

Total memory (VRAM or unified)	File
192 GB	`AD-IQ1_M`
256 GB	`AD-IQ2_M`
384 GB	`AD-Q4_K_M`
512 GB	`AD-Q5_K_M`
640 GB	`AD-Q8_0`

On a Mac, raise the GPU memory limit first, for example sudo sysctl iogpu.wired_limit_mb=240000 on a 256 GB machine.

Running it

llama-server -hf AtomicChat/Ling-3.1-flash-GGUF:AD-Q4_K_M -ngl 99 -c 32768 --jinja

Each file sits in a folder of its name, split into shards of up to 45 GB; point llama.cpp at the first shard and it loads the rest:

hf download AtomicChat/Ling-3.1-flash-GGUF --include "AD-Q4_K_M/*" --local-dir .
llama-server -m AD-Q4_K_M/Ling-3.1-flash-AD-Q4_K_M-00001-of-00009.gguf -ngl 99 -c 32768 --jinja

The chat template is inside every file. Thinking is on by default; put detailed thinking off in the system prompt to turn it off.

How we measured

Reference: BF16 weights converted from inclusionAI's FP8 release and run unquantized (perplexity 2.3590 on the held-out set). Converting FP8 to BF16 rounds each weight once, far below what Q8_0 adds.

Held-out text: eval_neutral from our calibration dataset, never used for calibration, 144 chunks of 4096 tokens. Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement. Hardware: 8x A100-SXM4-80GB, llama.cpp 097cf33d9 for every file.

The importance matrix is the one we computed on 5.5M tokens of the ling3-moe corpus (thinking on and off, tool calls, 30 languages, code, long documents up to 32k tokens), rendered through this model's own chat template.

Model details

Mixture of experts: 512 routed experts in each of 68 layers (the first 4 layers are dense), 8 routed plus 1 shared expert per token, about 24B parameters active of 564B. Hybrid attention: 63 Kimi Delta Attention (linear) layers and 9 multi-head latent attention layers. Hidden size 4096, vocabulary 157,184, context 262,144. The multi-token-prediction head of the release is not included.

Built from inclusionAI's FP8 release (revision b426279f), converted to BF16 and quantized with llama.cpp b11571 (097cf33d9); needs llama.cpp with bailingmoe3 support.