--- license: mit base_model: - pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF pipeline_tag: image-text-to-text tags: - gguf - gsq - rco - quantization - ist-daslab - moe - glm5next --- This is a simple modification of [pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF](https://huggingface.co/pfeifferj/GLM-5.3-Flash-GSQ-RCO-GGUF) we made for our needs. Hope that you might find it useful too. The quantized MTP Q4_K_M nextn layer from [unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF) has been grafted into the GSQ-RCO IQ3_S quantization of the original model. It is running in production on a single AMD Instinct MI210 GPU, using llama.cpp v0.7.0. ## GSQ-RCO quantization Credits should also go to the Deep Algorithms and Systems Lab ([ISTA/DASLab](https://huggingface.co/ISTA-DASLab)), Institute of Science and Technology Austria that developed GSQ-RCO quantization method used to build the model. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size. | Method | Description | |---|---| | **GSQ** (Gumbel-Softmax Quantization, [paper](https://arxiv.org/abs/2604.18556), [code](https://github.com/IST-DASLab/GSQ)) | Post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation. GSQ closes most of the gap between scalar and vector quantization at 2 to 3 bits while remaining deployable in standard scalar formats such as GGUF. | | **RCO** (Riemannian Constrained Optimization, [paper](https://arxiv.org/abs/2605.00649), [code](https://github.com/IST-DASLab/RCO)) | Assigns one of K quantization types to each of N tensors under a total size budget. The budget constraint is reformulated as a smooth Riemannian manifold in logit space, which permits gradient-based optimization directly on the task loss while enforcing the budget exactly, without constraint-specific hyperparameter tuning. | ## Performance On our setup the model is roughly 15% faster than unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL. ### BioDec GSQ-RCO:IQ3_S vs. Unsloth Q4_K_M. Derived using llama-benchy, AMD MI210, single GPU. draft_n_max=1 | ctx len | ctx_tg (t/s) | tg32 (t/s) | |---|---|---| | 16k | 19.17 / 17.24 = **+11.2%** | 20.15 / 17.42 = **+15.7%** | | 32k | 17.77 / 15.39 = **+15.5%** | 16.95 / 14.85 = **+14.2%** | | 64k | 14.66 / 13.57 = **+8.0%** | 14.96 / 12.65 = **+18.3%** | This model contains the IQ3_S quantization only, since the GSQ-RCO IQ3_XXS version of the model proved unstable in agentic workflows, showing a tendency to loop that is absent in the larger quantization. ## Our llama.cpp configuration The stanza from the model.ini file ``` [BioDec/GLM-5.3-Flash-GSQ-RCO-MTP-GGUF] ;alias = "GLM-5.3-Flash" split-mode = layer ctx-size = 163840 parallel = 1 fit-target = 2048 ;mmproj-offload = off ; cache-type-k = q8_0 cache-type-v = q8_0 kv-unified = on ; batch-size = 4096 ubatch-size = 2048 ; spec-type = draft-mtp spec-draft-n-max = 1 load-mode = none ``` On our machine, spec-draft-n-max >1 did not provide measurable advantage despite acceptance being good. The same held true for DFlash2 speculative models.