tokenfires commited on
Commit
3881bd2
·
verified ·
1 Parent(s): 71d6948

Add model card

Browse files
Files changed (1) hide show
  1. README.md +67 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: splash
4
+ pipeline_tag: text-generation
5
+ inference: false
6
+ base_model:
7
+ - mlx-community/Qwen3.6-35B-A3B-4bit
8
+ base_model_relation: quantized
9
+ tags:
10
+ - splash
11
+ - apple-silicon
12
+ - metal
13
+ - local-inference
14
+ - dflash2
15
+ - speculative-decoding
16
+ - qwen3.6
17
+ - moe
18
+ - 4-bit
19
+ - lm-studio
20
+ ---
21
+
22
+ # Qwen3.6-35B-A3B-Splash (mirror)
23
+
24
+ A byte-for-byte mirror of [incoai/Qwen3.6-35B-A3B-Splash](https://huggingface.co/incoai/Qwen3.6-35B-A3B-Splash), kept here as a pinned copy for my local agent stack. All credit for the packing, the DFlash 2 draft, and the [Splash](https://github.com/incoai/splash) engine goes to [Inco AI](https://huggingface.co/incoai). Read their card first. This one only adds what I found running it.
25
+
26
+ Verified identical to upstream on 2026-09-23: the `artifact_set_sha256` in `manifest.json` matches incoai's published manifest (`0c465f96…2440a`).
27
+
28
+ ## What's in the package
29
+
30
+ It's not a Transformers or MLX checkpoint. It only loads in Splash, or in LM Studio's Splash runtime.
31
+
32
+ | Path | Contents |
33
+ |---|---|
34
+ | `target/` | Qwen3.6-35B-A3B, 4-bit packed (`splash-packed-q4-moe`), 40 layers, 256 experts, 8 active per token |
35
+ | `draft/` | DFlash 2 draft model, 6 layers, proposes 7 tokens per step |
36
+ | `vision/` | Vision encoder |
37
+ | `tokenizer/` | Tokenizer and chat template |
38
+ | `manifest.json`, `layout.json` | Artifact hashes, execution geometry, upstream revisions |
39
+
40
+ Upstream sources, from the manifest: target, tokenizer, and vision from `mlx-community/Qwen3.6-35B-A3B-4bit` @ `38740b8`, draft from `incoai/Qwen3.6-35B-A3B-DFlash2` @ `8e71350`. About 20.9 GB on disk.
41
+
42
+ ## Running it
43
+
44
+ **LM Studio:** it shows up as `qwen3.6-35b-a3b-splash` and reports model format `yuzu`. This is how I run it.
45
+
46
+ **Splash:** upstream's command is `splash serve --model incoai/Qwen3.6-35B-A3B-Splash`. I haven't tried pointing Splash at this mirror's repo id, so use upstream's if you're going that route.
47
+
48
+ ## Measured
49
+
50
+ LM Studio on an M5 Max (128 GB), OpenAI-compatible endpoint, non-streaming: **1,000 tokens in 6.75 s, about 148 tok/s end to end** with prefill included. Decode alone runs higher, around 170 tok/s on the same machine. For comparison, the Qwen3.8-27B MTPLX build I was running before did about 24 tok/s in the same agent.
51
+
52
+ ## Sampling notes (LM Studio's Splash runtime, 2026-09-23)
53
+
54
+ I tested each parameter directly against the endpoint:
55
+
56
+ - **Honored:** `temperature`, `top_p`, `top_k`. Also the per-model defaults in LM Studio's model config, which are picked up live.
57
+ - **Ignored:** `repeat_penalty`, `presence_penalty`, `frequency_penalty`. Output at temperature 0 was byte-identical with and without them.
58
+
59
+ So the usual repetition guard doesn't exist on this runtime (at least not yet). In my first evening of use it went into a reasoning loop: about 112k tokens of the same closing line, repeated inside the think block, and it never emitted an answer. At this speed that's six and a half minutes before anyone notices. If you run it behind an agent, **cap `max_tokens`** so a loop dies on its own. I'm running temperature 0.6, top_p 0.95, top_k 20, which is Qwen's thinking-mode recommendation. Don't go greedy, since Qwen's own guidance is that greedy decoding makes endless repetition more likely.
60
+
61
+ ## If you drive LM Studio from code
62
+
63
+ `@lmstudio/sdk` 1.5.0 hangs (it never resolves or rejects) on `llm.listLoaded()` whenever a Splash model is loaded, because its schema doesn't know the `yuzu` format. **2.0.0 fixes it.** Upgrade before loading this.
64
+
65
+ ## License
66
+
67
+ Apache 2.0, same as upstream and the Qwen base model.