# GLM4.5V having issues with bfloat16 / float16 training

**URL:** <https://discuss.huggingface.co/t/glm4-5v-having-issues-with-bfloat16-float16-training/167766>\
**Category:** Models\
**Created:** [August 26, 2025, 12:41am UTC](https://discuss.huggingface.co/t/glm4-5v-having-issues-with-bfloat16-float16-training/167766 "2025-08-26T00:41:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aac13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aac13/32/53060_2.png) [@Aac13](https://discuss.huggingface.co/u/Aac13)\
**Post date:** [August 26, 2025, 12:41am UTC](https://discuss.huggingface.co/t/glm4-5v-having-issues-with-bfloat16-float16-training/167766/1 "2025-08-26T00:41:48Z")

</div>

I am trying to run SFT with QLoRA on GLM4.5V. I would like to train in either bfloat16 or float16, but there is some mismatch between dtypes when actually running the model:

```auto
MODEL_ID = “zai-org/GLM-4.5V” bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type=“nf4”, bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16, ) model = Glm4vMoeForConditionalGeneration.from_pretrained( MODEL_ID, trust_remote_code=True, quantization_config=bnb, torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map={“”: 0}, ) File /usr/local/lib/python3.11/site-packages/transformers/models/glm4v_moe/modeling_glm4v_moe.py:342, in Glm4vMoeTextMoE.forward(self, hidden_states) 340 topk_indices, topk_weights = self.gate(hidden_states) 341 hidden_states = hidden_states.view(-1, hidden_states.shape[-1]) → 342 hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(orig_shape) 343 hidden_states = hidden_states + self.shared_experts(residuals) 344 return hidden_states File /usr/local/lib/python3.11/site-packages/transformers/models/glm4v_moe/modeling_glm4v_moe.py:330, in Glm4vMoeTextMoE.moe(self, hidden_states, topk_indices, topk_weights) 328 expert_output = expert(expert_input) 329 weighted_output = expert_output expert_weights.unsqueeze(-1) → 330 final_hidden_states.indexadd(0, token_indices, weighted_output) 332 # in original deepseek, the output of the experts are gathered once we leave this module 333 # thus the moe module is itelsf an IsolatedParallel module 334 # and all expert are “local” meaning we shard but we don’t gather 335 return final_hidden_states.type(hidden_states.dtype) RuntimeError: indexadd(): self (Half) and source (Float) must have the same scalar type

```

Can someone help working with GLM4.5V? I am using the preview build (transformers-v4.55.0-GLM-4.5V-preview) shown here [zai-org/GLM-4.5V · Hugging Face](https://huggingface.co/zai-org/GLM-4.5V)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 26, 2025, 3:30am UTC](https://discuss.huggingface.co/t/glm4-5v-having-issues-with-bfloat16-float16-training/167766/2 "2025-08-26T03:30:28Z")

</div>

For now, it may be more stable to fix the dtype of the model and the training dtype to one. Also, using Flash Attention 2 seems to cause bugs.

```py
DTYPE = torch.bfloat16 # or torch.float16

nf4_config = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=DTYPE)

model = Glm4vMoeForConditionalGeneration.from_pretrained(
    "zai-org/GLM-4.5V",
    quantization_config=nf4_config,
    torch_dtype=DTYPE, # avoid torch_dtype="auto" https://github.com/pytorch/torchtune/issues/1349
    attn_implementation="eager", # avoid FA2 during finetune https://github.com/zai-org/GLM-V/issues/149
    use_cache=False,
)
model.to(dtype=DTYPE)

```