Qwen3.5 Dense-to-MoE Weight Transfer
Collection
Qwen3.5 MoE models from dual-source weight transfer (dense backbone + 35B-A3B experts). Hybrid DeltaNet + GQA attention. • 6 items • Updated
Qwen3.5 Tiny Test Model
A tiny Qwen3.5 hybrid MoE model for testing and validation purposes.
This model has random weights and is not trained. It exists to validate the architecture implementation and hub upload pipeline.
Architecture
- Type: Hybrid MoE (Gated DeltaNet + Gated Attention)
- Parameters: 138,261,536 total
- Layers: 8 (6 DeltaNet + 2 Full Attention)
- Experts: 8 routed (top-2) + 1 shared
- Embedding dim: 256
- Vocab size: 248,320
- Context: 4096 tokens
- Downloads last month
- 286