Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: splash
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
inference: false
|
| 6 |
+
base_model:
|
| 7 |
+
- mlx-community/Qwen3.6-35B-A3B-4bit
|
| 8 |
+
base_model_relation: quantized
|
| 9 |
+
tags:
|
| 10 |
+
- splash
|
| 11 |
+
- apple-silicon
|
| 12 |
+
- metal
|
| 13 |
+
- local-inference
|
| 14 |
+
- dflash2
|
| 15 |
+
- speculative-decoding
|
| 16 |
+
- qwen3.6
|
| 17 |
+
- moe
|
| 18 |
+
- 4-bit
|
| 19 |
+
- lm-studio
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# Qwen3.6-35B-A3B-Splash (mirror)
|
| 23 |
+
|
| 24 |
+
A byte-for-byte mirror of [incoai/Qwen3.6-35B-A3B-Splash](https://huggingface.co/incoai/Qwen3.6-35B-A3B-Splash), kept here as a pinned copy for my local agent stack. All credit for the packing, the DFlash 2 draft, and the [Splash](https://github.com/incoai/splash) engine goes to [Inco AI](https://huggingface.co/incoai). Read their card first. This one only adds what I found running it.
|
| 25 |
+
|
| 26 |
+
Verified identical to upstream on 2026-09-23: the `artifact_set_sha256` in `manifest.json` matches incoai's published manifest (`0c465f96…2440a`).
|
| 27 |
+
|
| 28 |
+
## What's in the package
|
| 29 |
+
|
| 30 |
+
It's not a Transformers or MLX checkpoint. It only loads in Splash, or in LM Studio's Splash runtime.
|
| 31 |
+
|
| 32 |
+
| Path | Contents |
|
| 33 |
+
|---|---|
|
| 34 |
+
| `target/` | Qwen3.6-35B-A3B, 4-bit packed (`splash-packed-q4-moe`), 40 layers, 256 experts, 8 active per token |
|
| 35 |
+
| `draft/` | DFlash 2 draft model, 6 layers, proposes 7 tokens per step |
|
| 36 |
+
| `vision/` | Vision encoder |
|
| 37 |
+
| `tokenizer/` | Tokenizer and chat template |
|
| 38 |
+
| `manifest.json`, `layout.json` | Artifact hashes, execution geometry, upstream revisions |
|
| 39 |
+
|
| 40 |
+
Upstream sources, from the manifest: target, tokenizer, and vision from `mlx-community/Qwen3.6-35B-A3B-4bit` @ `38740b8`, draft from `incoai/Qwen3.6-35B-A3B-DFlash2` @ `8e71350`. About 20.9 GB on disk.
|
| 41 |
+
|
| 42 |
+
## Running it
|
| 43 |
+
|
| 44 |
+
**LM Studio:** it shows up as `qwen3.6-35b-a3b-splash` and reports model format `yuzu`. This is how I run it.
|
| 45 |
+
|
| 46 |
+
**Splash:** upstream's command is `splash serve --model incoai/Qwen3.6-35B-A3B-Splash`. I haven't tried pointing Splash at this mirror's repo id, so use upstream's if you're going that route.
|
| 47 |
+
|
| 48 |
+
## Measured
|
| 49 |
+
|
| 50 |
+
LM Studio on an M5 Max (128 GB), OpenAI-compatible endpoint, non-streaming: **1,000 tokens in 6.75 s, about 148 tok/s end to end** with prefill included. Decode alone runs higher, around 170 tok/s on the same machine. For comparison, the Qwen3.8-27B MTPLX build I was running before did about 24 tok/s in the same agent.
|
| 51 |
+
|
| 52 |
+
## Sampling notes (LM Studio's Splash runtime, 2026-09-23)
|
| 53 |
+
|
| 54 |
+
I tested each parameter directly against the endpoint:
|
| 55 |
+
|
| 56 |
+
- **Honored:** `temperature`, `top_p`, `top_k`. Also the per-model defaults in LM Studio's model config, which are picked up live.
|
| 57 |
+
- **Ignored:** `repeat_penalty`, `presence_penalty`, `frequency_penalty`. Output at temperature 0 was byte-identical with and without them.
|
| 58 |
+
|
| 59 |
+
So the usual repetition guard doesn't exist on this runtime (at least not yet). In my first evening of use it went into a reasoning loop: about 112k tokens of the same closing line, repeated inside the think block, and it never emitted an answer. At this speed that's six and a half minutes before anyone notices. If you run it behind an agent, **cap `max_tokens`** so a loop dies on its own. I'm running temperature 0.6, top_p 0.95, top_k 20, which is Qwen's thinking-mode recommendation. Don't go greedy, since Qwen's own guidance is that greedy decoding makes endless repetition more likely.
|
| 60 |
+
|
| 61 |
+
## If you drive LM Studio from code
|
| 62 |
+
|
| 63 |
+
`@lmstudio/sdk` 1.5.0 hangs (it never resolves or rejects) on `llm.listLoaded()` whenever a Splash model is loaded, because its schema doesn't know the `yuzu` format. **2.0.0 fixes it.** Upgrade before loading this.
|
| 64 |
+
|
| 65 |
+
## License
|
| 66 |
+
|
| 67 |
+
Apache 2.0, same as upstream and the Qwen base model.
|