gg-hf-em (gg-hf-em)

posted an update 2 days ago

Post

392

Qwen-Image-2.1 can now run locally on 12GB VRAM with Unsloth GGUFs! 🖼️

The 7B model performs on par with Nano Banana 2.0.

For higher quality, you can also run Dynamic FP8 on just 6GB of VRAM via offloading.

GGUF: unsloth/Qwen-Image-2.1-GGUF
Guide: https://unsloth.ai/docs/models/qwen-image-2.1

danielhanchen

posted an update 16 days ago

Post

4815

Qwen3.8-27B Unsloth GGUF is now the #1 most-liked GGUF of all time! 🤗🦥

The model hit 10M downloads and 3.7K likes in just 24 days on Hugging Face - all thanks to you.

GGUF: unsloth/Qwen3.8-27B-GGUF
Guide: https://unsloth.ai/docs/models/qwen3.8

5 replies

·

tomaarsen

posted an update about 1 month ago

Post

4026

🚨 I've just published Sentence Transformers v6.0, introducing MultiVectorEncoder: ColBERT-style late interaction models are now a fourth model type, for training, inference, and interpretation, alongside the dense, sparse, and reranker models! Details:

Where a regular embedding model compresses a whole text into one vector, a multi-vector model keeps one vector per token and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away. It is also the state of the art for visual document retrieval, where a text query is matched against page images directly, charts and tables included, with no OCR step in between.

Any PyLate, Stanford ColBERT, or ColPali checkpoint loads straight into the same familiar API: model.encode_query(), model.encode_document(), and model.similarity() just work, whether the documents are texts or page images.

Does it help? LightOn trained LateOn (multi-vector) and DenseOn (dense) on the same data with the same 149M ModernBERT backbone, and the multi-vector model wins on 9 of the 13 NanoBEIR datasets: 0.6868 vs 0.6764 mean NDCG@10. The price is a bigger index, and the new HierarchicalTokenPooling module halves it at roughly no retrieval cost.

Antoine Chaffin, Raphaël Sourty, and I wrote a blog post walking through multi-vector models in practice: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. Check it out if you want to get started, or just point your Agent to the URL: https://huggingface.co/blog/multi-vector-encoder

pip install sentence-transformers==6.0.0

Release notes: https://github.com/huggingface/sentence-transformers/releases/tag/v6.0.0

danielhanchen

posted an update about 1 month ago

Post

5907

Introducing Unsloth Desktop 🦥
The first desktop app to run and train models locally.

• Open-source. Runs on Mac, Windows and Linux
• Supports MLX, diffusion image/video, audio, GGUF
• Connect Claude Code and Codex to local LLMs
• 50% more accurate, self-healing tool calls + sandboxed code exec
• Works for CPU + multiGPU setups - NVIDIA, AMD, Intel, Mac
• Train models 2× faster with 70% less VRAM
• Private web search, deep research, RAG, MCP and exports (NVFP4, GGUF)
• Use Unsloth’s OpenAI-compatible API and cloud models
• Securely deploy LLMs remotely and access anywhere

Unsloth Desktop is now available on http://unsloth.ai
and GitHub.

GitHub: https://github.com/unslothai/unsloth
Blog and Guide: https://unsloth.ai/docs/desktop

6 replies

·

danielhanchen

posted an update about 2 months ago

Post

2375

DeepSeek-V4-Flash can now run 2× faster locally with DSpark! ⚡️

DSpark enables V4-Flash-0731 GGUFs to generate ~1.4–2× faster with no accuracy change.

DeepSeek-V4-Flash-0731 can reach at 120 tokens/s.

GGUFs: unsloth/DeepSeek-V4-Flash-0731-GGUF
Guide: https://unsloth.ai/docs/models/deepseek-v4

danielhanchen

posted an update about 2 months ago

Post

2396

We compared 1-bit Kimi K3 to Claude Opus 5 and GPT 5.6. 🤯

We gave 4 models the same prompt: Create a glass aquarium whose side panel develops a visible crack and then bursts...

1-bit Kimi K3 GGUF ran locally on 4x B200s at 36 tok/s.

GGUF: unsloth/Kimi-K3-GGUF
GitHub repo: https://github.com/unslothai/unsloth

1 reply

·

danielhanchen

posted an update about 2 months ago

Post

4750

Kimi K3 can now be run locally! ✨

The 1-bit model retains ~78.9% accuracy after we shrunk it from 1.56TB to 594GB (-62% size).

Run on a Mac Studio connected with 128GB RAM device. Kimi K3 is the strongest open model to date.

GGUF: unsloth/Kimi-K3-GGUF
Guide: https://unsloth.ai/docs/models/kimi-k3

5 replies

·

MaartenGr

authored a paper 2 months ago

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 82

sebrusso

authored 2 papers 2 months ago

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

Paper • 2605.27295 • Published May 26 • 21

LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing

Paper • 2507.00769 • Published Jul 1, 2025 • 5

danielhanchen

posted an update 2 months ago

Post

5331

Introducing Unsloth for AMD 🚀
You can now train & run LLMs on your AMD hardware

• We collaborated with AMD to enable you to train & run 500+ models on AMD GPUs
• Works on Windows, WSL, Linux
• Train Qwen, Gemma on just 3GB VRAM

GitHub: https://github.com/unslothai/unsloth
Blog + Guide: https://unsloth.ai/docs/basics/amd

3 replies

·

danielhanchen

posted an update 2 months ago

Post

6358

Gemma 4 is now faster and much more accurate! 🚀

Google made huge improvements to tool-calling and chat accuracy, reliability + speed.
To get fixes, re-download our updated GGUF, MLX, NVFP4 quants!

Unsloth quants: https://huggingface.co/collections/unsloth/gemma-4
Gemma 4 Guide: https://unsloth.ai/docs/models/gemma-4

8 replies

·

danielhanchen

posted an update 2 months ago

Post

5032

We’re releasing Gemma 4 NVFP4 quants that run 1.5× faster on your GPU.

Gemma-4-12B NVFP4 works on 11GB VRAM.
26B-A4B hits 13K tok/s (B200).

Unsloth NVFP4 enables faster, more accurate 4-bit Blackwell inference.

Blog: https://unsloth.ai/docs/basics/nvfp4
Gemma NVFP4: https://huggingface.co/collections/unsloth/nvfp4

3 replies

·

danielhanchen

posted an update 3 months ago

Post

4500

We’re releasing new Qwen3.6 quants that run 2.5× faster on your GPU. ⚡

Qwen3.6-27B NVFP4 runs on 24GB VRAM.
35B-A3B can hit 17,561 tok/s (B200).

We also improved accuracy, tool calling, agent use, and looping.

Qwen3.6 NVFP4: https://huggingface.co/collections/unsloth/nvfp4
Guide: https://unsloth.ai/docs/models/qwen3.6#nvfp4

1 reply

·

osanseviero

authored a paper 3 months ago

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 82

osanseviero

submitted a paper to Daily Papers 3 months ago

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 82

danielhanchen

posted an update 3 months ago

Post

6527

DeepSeek-V4 can now run locally with Unsloth GGUFs! 🐳

Run lossless DeepSeek-V4-Flash on 168GB RAM or
3-bit works on 110GB Mac, RAM, VRAM setups.

Run via Unsloth Studio or llama.cpp.

GGUF: unsloth/DeepSeek-V4-Flash-GGUF
Guide: https://unsloth.ai/docs/models/deepseek-v4

danielhanchen

posted an update 3 months ago

Post

3459

1-bit GLM-5.2 GGUF vs. Claude 4.8 Opus vs. GPT-5.5

We gave 3 models the same prompt and compared one-shot outputs.

The 1-bit GLM-5.2 GGUF ran locally on a Mac Studio M3 Ultra with 256GB RAM at ~21.6 tok/s.

Which output do you like best?
GGUF: unsloth/GLM-5.2-GGUF

3 replies

·

danielhanchen

posted an update 3 months ago

Post

4669

Google's new DiffusionGemma can now run at 2000+ tokens/sec! ⚡

We made local DiffusionGemma inference 1.8× faster.
Run it on 18GB RAM via Unsloth Studio.

GitHub: https://github.com/unslothai/unsloth
Guide: https://unsloth.ai/docs/models/diffusiongemma

4 replies

·

danielhanchen

posted an update 4 months ago

Post

1240

Google releases DiffusionGemma.✨
The new 26B-A4B diffusion text model runs locally on 18GB RAM.

Run with 4x faster text generation, thinking, image, video and 256K context. Run and train via Unsloth Studio.

GGUF: unsloth/diffusiongemma-26B-A4B-it-GGUF
Guide: https://unsloth.ai/docs/models/diffusiongemma

1 reply

·

AI & ML interests

Team members 29

gg-hf-em's activity