Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,204 @@
+---
+datasets:
+- NeelNanda/pile-10k
+base_model:
+- moonshotai/Kimi-K2-Instruct
+---
+
+## Model Details
+
+This model is an mixed int4 model with group_size 128 and symmetric quantization of [moonshotai/Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round) algorithm. Non expert layers are fallback to 8 bits. Please refer to Section Generate the model for more details.
+Please follow the license of the original model.
+
+## How To Use
+
+**Due to kernel issue, this model could only run on CPU**
+
+### INT4 Inference(CPU)
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import transformers
+from auto_round import AutoRound, AutoRoundConfig
+
+import torch
+
+quantized_model_dir = "Intel/Kimi-K2-Instruct-int4-mixed-AutoRound-cpu"
+
+model = AutoModelForCausalLM.from_pretrained(
+    quantized_model_dir,
+    torch_dtype=torch.bfloat16,
+    device_map="cpu",
+)
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
+prompts = [
+    "9.11和9.8哪个数字大",
+    "strawberry中有几个r?",
+    "There is a girl who likes adventure,",
+    "Please give a brief introduction of Moonshot AI",
+]
+
+texts=[]
+for prompt in prompts:
+    messages = [
+        {"role": "system", "content": "You are Kimi, an AI assistant created by Moonshot AI."},
+        {"role": "user", "content": [{"type": "text", "text":prompt}]}
+    ]
+    text = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True
+    )
+    texts.append(text)
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+
+outputs = model.generate(
+    input_ids=inputs["input_ids"].to(model.device),
+    attention_mask=inputs["attention_mask"].to(model.device),
+    max_length=200, ##change this to align with the official usage
+    num_return_sequences=1,
+    do_sample=False  ##change this to align with the official usage
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
+]
+
+decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+
+for i, prompt in enumerate(prompts):
+    input_id = inputs
+    print(f"Prompt: {prompt}")
+    print(f"Generated: {decoded_outputs[i]}")
+    print("-" * 50)
+"""
+Prompt: 9.11和9.8哪个数字大
+Generated: ### 第一步：理解题目
+
+首先，我需要明确题目在问什么。题目给出了两个数字：9.11和9.8，问哪一个更大。这看起来是一个简单的数值比较问题。
+
+### 第二步：数字的表示
+
+这两个数字都是小数，即带有小数部分的数字。小数由整数部分和小数部分组成，小数点左边是整数部分，右边是小数部分。
+
+- 9.11：整数部分是9，小数部分是11。
+- 9.8：整数部分是9，小数部分是8。
+
+### 第三步：比较整数部分
+
+首先比较两个数的整数部分：
+
+- 9.11的整数部分是9。
+- 9.8的整数部分也是9。
+
+整数部分相同，因此需要比较小数部分。
+
+### 第四步：比较小数部分
+
+小数部分的比较
+--------------------------------------------------
+Prompt: strawberry中有几个r?
+Generated: ### 问题重述
+我们需要计算单词 "strawberry" 中有多少个字母 "r"。
+
+### 步骤分解
+1. **写出单词**：首先，将单词 "strawberry" 完整地写出来。
+2. **逐个字母检查**：从左到右，逐个字母查看是否是 "r"（注意大小写，但这里都是小写）。
+3. **计数**：每遇到一个 "r"，就增加计数器。
+
+### 详细检查
+让我们将 "strawberry" 拆分开来：
+
+字母位置及字母：
+1. s
+2. t
+3. r
+4. a
+5. w
+6. b
+7. e
+8. r
+9. r
+10. y
+
+现在，我们检查每个字母是否为 "r"：
+
+-
+--------------------------------------------------
+Prompt: There is a girl who likes adventure,
+Generated: There is a girl who likes adventure,
+so she ties her shoes with sunrise instead of laces,
+lets the wind pick the next city,
+and trades her shadow for a passport stamp.
+
+She keeps her memories in mason jars—
+one holds the scent of monsoon in Mumbai,
+another the hush of Icelandic snow.
+When homesick, she unscrews a lid,
+inhales, and is gone again.
+
+She once outran her own name
+somewhere between Marrakesh and the moon,
+answering only to “Hey, you with the constellations in your hair.”
+Maps are her love letters;
+she folds them into paper boats
+and sails them down hotel bathtubs,
+whispering, *Find me where the water ends.*
+--------------------------------------------------
+Prompt: Please give a brief introduction of Moonshot AI
+Generated: Moonshot AI is a Chinese artificial-intelligence company founded in 2023 and headquartered in Beijing. Focused on large-scale language models and related products, it released its first model, Kimi, in October 2023 and has since launched upgraded versions such as Kimi 1.5. The company closed a US$1 billion funding round in early 2024 that valued it at about US$2.5 billion, making it one of China’s best-funded AI start-ups.
+--------------------------------------------------
+
+"""
+```
+
+### Generate the model
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import transformers
+
+model_name = "Kimi-K2-Instruct-BF16"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name,trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_name,device_map="cpu", torch_dtype="auto",trust_remote_code=True)
+
+layer_config = {}
+for n, m in model.named_modules():
+    if isinstance(m, torch.nn.Linear):
+        if "expert" in n or "shared_experts" in n:
+            layer_config[n] = {"bits": 4}
+            print(n, 4)
+        else:
+            layer_config[n] = {"bits": 8}
+            print(n, 8)
+
+from auto_round import AutoRound
+
+autoround = AutoRound(model, tokenizer, iters=0, layer_config=layer_config)
+autoround.quantize_and_save(format="auto_round", output_dir="tmp_autoround")
+
+```
+
+
+## Ethical Considerations and Limitations
+
+The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Therefore, before deploying any applications of the model, developers should perform safety testing.
+
+## Caveats and Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+Here are a couple of useful links to learn more about Intel's AI software:
+
+- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
+
+## Cite
+
+@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
+
+[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)

chat_template.jinja

@@ -0,0 +1,43 @@
+{%- if tools -%}
+  <|im_system|>tool_declare<|im_middle|>{{ tools | tojson }}<|im_end|>
+{%- endif -%}
+{%- for message in messages -%}
+  {%- if loop.first and messages[0]['role'] != 'system' -%}
+    <|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>
+  {%- endif -%}
+  {%- if message['role'] == 'system' -%}
+    <|im_system|>system<|im_middle|>
+  {%- elif message['role'] == 'user' -%}
+    <|im_user|>user<|im_middle|>
+  {%- elif message['role'] == 'assistant' -%}
+    <|im_assistant|>assistant<|im_middle|>
+  {%- elif message['role'] == 'tool' -%}
+    <|im_system|>tool<|im_middle|>
+  {%- endif -%}
+  {%- if message['role'] == 'assistant' and message.get('tool_calls') -%}
+    {%- if message['content'] -%}{{ message['content'] }}{%- endif -%}
+    <|tool_calls_section_begin|>
+    {%- for tool_call in message['tool_calls'] -%}
+      {%- set func_name = tool_call['function']['name'] -%}
+      {%- set formatted_id = 'functions.' + func_name + ':' + loop.index0|string -%}
+      <|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{{ tool_call['function']['arguments'] | tojson}}<|tool_call_end|>
+    {%- endfor -%}
+    <|tool_calls_section_end|>
+  {%- elif message['role'] == 'tool' -%}
+    ## Return of {{ message.tool_call_id }}\n{{ message['content'] }}
+  {%- elif message['content'] is string -%}
+    {{ message['content'] }}
+  {%- elif message['content'] is not none -%}
+    {% for content in message['content'] -%}
+      {% if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}
+        <|media_start|>image<|media_content|><|media_pad|><|media_end|>
+      {% else -%}
+        {{ content['text'] }}
+      {%- endif -%}
+    {%- endfor -%}
+  {%- endif -%}
+  <|im_end|>
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+  <|im_assistant|>assistant<|im_middle|>
+{%- endif -%}

config.json

@@ -0,0 +1,1007 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_deepseek.DeepseekV3Config",
+    "AutoModel": "modeling_deepseek.DeepseekV3Model",
+    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
+  },
+  "aux_loss_alpha": 0.001,
+  "bos_token_id": 163584,
+  "eos_token_id": 163585,
+  "ep_size": 1,
+  "first_k_dense_replace": 1,
+  "hidden_act": "silu",
+  "hidden_size": 7168,
+  "initializer_range": 0.02,
+  "intermediate_size": 18432,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 131072,
+  "model_type": "deepseek_v3",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "n_group": 1,
+  "n_routed_experts": 384,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 61,
+  "num_key_value_heads": 64,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 163839,
+  "pretraining_tp": 1,
+  "q_lora_rank": 1536,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "autoround_version": "0.6.0",
+    "bits": 4,
+    "data_type": "int",
+    "extra_config": {
+      "lm_head": {
+        "bits": 8,
+        "data_type": "int",
+        "group_size": 128,
+        "sym": true
+      },
+      "model.layers.0.mlp.down_proj": {
+        "bits": 8
+      },
+      "model.layers.0.mlp.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.0.mlp.up_proj": {
+        "bits": 8
+      },
+      "model.layers.0.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.0.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.0.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.0.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.0.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.1.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.1.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.1.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.1.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.1.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.10.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.10.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.10.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.10.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.10.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.11.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.11.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.11.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.11.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.11.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.12.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.12.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.12.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.12.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.12.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.13.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.13.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.13.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.13.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.13.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.14.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.14.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.14.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.14.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.14.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.15.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.15.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.15.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.15.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.15.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.16.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.16.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.16.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.16.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.16.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.17.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.17.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.17.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.17.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.17.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.18.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.18.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.18.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.18.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.18.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.19.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.19.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.19.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.19.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.19.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.2.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.2.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.2.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.2.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.2.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.20.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.20.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.20.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.20.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.20.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.21.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.21.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.21.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.21.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.21.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.22.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.22.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.22.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.22.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.22.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.23.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.23.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.23.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.23.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.23.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.24.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.24.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.24.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.24.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.24.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.25.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.25.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.25.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.25.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.25.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.26.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.26.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.26.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.26.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.26.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.27.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.27.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.27.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.27.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.27.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.28.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.28.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.28.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.28.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.28.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.29.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.29.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.29.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.29.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.29.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.3.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.3.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.3.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.3.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.3.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.30.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.30.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.30.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.30.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.30.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.31.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.31.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.31.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.31.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.31.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.32.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.32.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.32.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.32.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.32.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.33.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.33.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.33.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.33.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.33.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.34.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.34.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.34.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.34.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.34.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.35.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.35.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.35.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.35.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.35.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.36.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.36.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.36.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.36.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.36.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.37.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.37.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.37.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.37.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.37.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.38.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.38.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.38.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.38.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.38.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.39.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.39.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.39.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.39.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.39.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.4.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.4.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.4.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.4.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.4.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.40.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.40.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.40.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.40.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.40.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.41.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.41.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.41.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.41.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.41.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.42.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.42.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.42.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.42.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.42.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.43.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.43.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.43.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.43.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.43.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.44.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.44.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.44.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.44.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.44.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.45.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.45.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.45.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.45.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.45.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.46.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.46.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.46.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.46.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.46.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.47.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.47.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.47.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.47.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.47.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.48.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.48.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.48.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.48.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.48.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.49.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.49.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.49.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.49.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.49.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.5.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.5.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.5.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.5.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.5.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.50.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.50.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.50.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.50.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.50.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.51.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.51.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.51.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.51.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.51.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.52.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.52.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.52.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.52.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.52.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.53.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.53.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.53.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.53.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.53.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.54.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.54.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.54.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.54.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.54.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.55.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.55.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.55.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.55.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.55.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.56.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.56.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.56.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.56.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.56.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.57.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.57.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.57.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.57.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.57.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.58.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.58.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.58.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.58.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.58.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.59.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.59.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.59.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.59.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.59.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.6.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.6.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.6.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.6.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.6.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.60.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.60.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.60.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.60.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.60.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.7.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.7.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.7.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.7.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.7.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.8.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.8.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.8.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.8.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.8.self_attn.q_b_proj": {
+        "bits": 8
+      },
+      "model.layers.9.self_attn.kv_a_proj_with_mqa": {
+        "bits": 8
+      },
+      "model.layers.9.self_attn.kv_b_proj": {
+        "bits": 8
+      },
+      "model.layers.9.self_attn.o_proj": {
+        "bits": 8
+      },
+      "model.layers.9.self_attn.q_a_proj": {
+        "bits": 8
+      },
+      "model.layers.9.self_attn.q_b_proj": {
+        "bits": 8
+      }
+    },
+    "group_size": 128,
+    "iters": 0,
+    "low_gpu_mem_usage": true,
+    "nsamples": 512,
+    "packing_format": "auto_round:auto_gptq",
+    "quant_method": "auto-round",
+    "sym": true
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "beta_fast": 1.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "rope_theta": 50000.0,
+  "routed_scaling_factor": 2.827,
+  "scoring_func": "sigmoid",
+  "seq_aux": true,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.3",
+  "unsloth_fixed": true,
+  "use_cache": true,
+  "v_head_dim": 128,
+  "vocab_size": 163840
+}

configuration_deepseek.py

@@ -0,0 +1,212 @@
+# Copy from https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/configuration_deepseek.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+class DeepseekV3Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`DeepseekV3Model`]. It is used to instantiate an DeepSeek
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the DeepSeek-V3.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 129280):
+            Vocabulary size of the Deep model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`DeepseekV3Model`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 11008):
+            Dimension of the MLP representations.
+        moe_intermediate_size (`int`, *optional*, defaults to 1407):
+            Dimension of the MoE representations.
+        num_hidden_layers (`int`, *optional*, defaults to 32):
+            Number of hidden layers in the Transformer decoder.
+        num_nextn_predict_layers (`int`, *optional*, defaults to 1):
+            Number of nextn predict layers in the DeepSeekV3 Model.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        n_shared_experts (`int`, *optional*, defaults to None):
+            Number of shared experts, None means dense model.
+        n_routed_experts (`int`, *optional*, defaults to None):
+            Number of routed experts, None means dense model.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        topk_method (`str`, *optional*, defaults to `gready`):
+            Topk method used in routed gate.
+        n_group (`int`, *optional*, defaults to None):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to None):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        num_experts_per_tok (`int`, *optional*, defaults to None):
+            Number of selected experts, None means dense model.
+        moe_layer_freq (`int`, *optional*, defaults to 1):
+            The frequency of the MoE layer: one expert layer for every `moe_layer_freq - 1` dense layers.
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to False):
+            Whether to normalize the weights of the routed experts.
+        scoring_func (`str`, *optional*, defaults to 'softmax'):
+            Method of computing expert weights.
+        aux_loss_alpha (`float`, *optional*, defaults to 0.001):
+            Auxiliary loss weight coefficient.
+        seq_aux = (`bool`, *optional*, defaults to True):
+            Whether to compute the auxiliary loss for each individual sample.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 2048):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        pretraining_tp (`int`, *optional*, defaults to 1):
+            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
+            document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
+            necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
+            issue](https://github.com/pytorch/pytorch/issues/76232).
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum.
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+
+    ```python
+    >>> from transformers import DeepseekV3Model, DeepseekV3Config
+
+    >>> # Initializing a Deepseek-V3 style configuration
+    >>> configuration = DeepseekV3Config()
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deepseek_v3"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=129280,
+        hidden_size=7168,
+        intermediate_size=18432,
+        moe_intermediate_size = 2048,
+        num_hidden_layers=61,
+        num_nextn_predict_layers=1,
+        num_attention_heads=128,
+        num_key_value_heads=128,
+        n_shared_experts = 1,
+        n_routed_experts = 256,
+        ep_size = 1,
+        routed_scaling_factor = 2.5,
+        kv_lora_rank = 512,
+        q_lora_rank = 1536,
+        qk_rope_head_dim = 64,
+        v_head_dim = 128,
+        qk_nope_head_dim = 128,
+        topk_method = 'noaux_tc',
+        n_group = 8,
+        topk_group = 4,
+        num_experts_per_tok = 8,
+        moe_layer_freq = 1,
+        first_k_dense_replace = 3,
+        norm_topk_prob = True,
+        scoring_func = 'sigmoid',
+        aux_loss_alpha = 0.001,
+        seq_aux = True,
+        hidden_act="silu",
+        max_position_embeddings=4096,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=0,
+        eos_token_id=1,
+        pretraining_tp=1,
+        tie_word_embeddings=False,
+        rope_theta=10000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+        self.num_attention_heads = num_attention_heads
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.ep_size = ep_size
+        self.routed_scaling_factor = routed_scaling_factor
+        self.kv_lora_rank = kv_lora_rank
+        self.q_lora_rank = q_lora_rank
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.topk_method = topk_method
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.num_experts_per_tok = num_experts_per_tok
+        self.moe_layer_freq = moe_layer_freq
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.scoring_func = scoring_func
+        self.aux_loss_alpha = aux_loss_alpha
+        self.seq_aux = seq_aux
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.pretraining_tp = pretraining_tp
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "eos_token_id": 163586,
+  "max_length": 131072,
+  "transformers_version": "4.53.3"
+}

model-00001-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e2f2e9decf730d7f577223608316813688f225e5c40342efa66dcd973382cab
+size 4997928016

model-00002-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:784bdb871ec77c2b5e555eff6ae72c6a8dd7fdc327658c6aac0e7cdd4076c6e7
+size 4995762944

model-00003-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1af8b1649389201a5cd9cca1a1bc39f85d2c42496e479f2a28a8d38c4f8a5af4
+size 4998015528

model-00004-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81d967b77ae89b237bc42e5ee75cc8dbc3551c8cc815e4b074cf7b3eaa980467
+size 4995763040

model-00005-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:174c7ccd61d402bf2fb0596b9da2fe7362e018c751aa4cea45932a5115826712
+size 4998015528

model-00006-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83de062885a6560248df7c2fb7d6526841d1c27c8db81eae5a041622282f9915
+size 4998016320

model-00007-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1b0f7b676f82c7341e8e8901475a58fe5a3c8a65cd38687347965ae8c898764
+size 4995762248

model-00008-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c347b6a4b94cb92d5af49b21dd5c13b707a45fdcc1ac67aa226c39dff2a8fa4
+size 4998015896

model-00009-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1181a7a267bd8e3ac500505f371c2fb7e49bbd9bd5105459ff6d71ded53e2f20
+size 4995762672

model-00010-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81ea55601e3e352f722bc634e0edda88c58585e8b6f4c967fe0815d51aab6696
+size 4998015528

model-00011-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff8bb0c589400356bf2ae894d1a1dc270bad6dd792c0e5b891d5e711931aafd0
+size 4995763040

model-00012-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d76738720a8943fab87e6b02b9589d8de8f104464a590ff59cef36b078a450b
+size 4998015528

model-00013-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d54b80a45a77a5d06ca3260e02a46584b82d85a792165e569d6daef46ff71772
+size 4995763040

model-00014-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2c2bdceaee55c4c15a73b1c0b8da8133d17eb3cfb84292ec09fc89587197569
+size 4998015528

model-00015-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2eb11977d3083d24d50f3fae5835b5564c592312de47912678a69b3af8d8a008
+size 4998016176

model-00016-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9de90a9e3c547050f67fd4a5eba91c28b907ca67cb894a345531d1846ee354d
+size 4995762392

model-00017-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df46b84f09cb3c126c5016246173500f834398009ee5062caf99969c32129891
+size 4998016448

model-00018-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6047ac8cde477c79bacdae8b060e93625e8dadb2d7f15dc57dc6a8ca13aa6ff
+size 4995764776

model-00019-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:578896629c78d7ce2675c7a668ef6bcb503f556adc69eefafb95445f93cce8a8
+size 4998017472

model-00020-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbb86c52def97e9a71447dae9e162901c757671c303fd1fcb1e8f974ab86837d
+size 4995765000

model-00021-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84985426f016cc36f2ca6dd3b22889106f52f537ac35dd0314a3cb6f699d788e
+size 4998017472

model-00022-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73783c92217aee489cda1258b5635d986cdec4944abec70fef10ee338816eb02
+size 4983684336

model-00023-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c719f2f15b918567a6dbf858ad77bc78139203f49a912999e6e795c377ad0fa
+size 4994843888

model-00024-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b93aa68a28da63144706f8d446281b8b2abb58b8d0c12c2e5fa324da03f8df3b
+size 4998017976

model-00025-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f9000686d7ec301877b1298e1433dcb421a0cdd923b6a8a8c3eed0b8376c429
+size 4995764496

model-00026-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12db9346f8f570d4d8f4b5a8efa3e5ed821cf1a3e6f710013c7712e798e2c2
+size 4998017552

model-00027-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b93b2633b14a611f76c1db4590e8d6fe6bf836f16c11b7485cd05ebd4087cdba
+size 4995764920

model-00028-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19269d2bc9a42fdfa5bd030525da3e6356f705de3450004e359a14efd617d93c
+size 4998017472

model-00029-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9651c035771b10b35196aa214cd85d0bbc7d90078a9b69c4a9bbec604170a0d6
+size 4995765000

model-00030-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0af884e47fd3f669a8f89a1e3071ed3b5bd2fa6119e36f7f10fa571b2aa11da
+size 4998017472

model-00031-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f58d8df36c17b1223ae811ca172638a4c1d23e3f4bf88d1069a538bf9932c5c1
+size 4998018240

model-00032-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f583a6d7ec187a8fc02c2f4b5ffab8db75002ef0578cb33f30aa31658bb5e72
+size 4995764232

model-00033-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27adcc81c92107788453f11ac0bdfc7f7884037b017cea6eaeac43ce6d6229a2
+size 4998017816

model-00034-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a880e5b2d4c5cd6da9ebc19076138d05c32e89d88330c88823d0581c21228395
+size 4995764656

model-00035-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20451a195aa976054f04836f1a0508f2c2033d0b1ba7b0a581a54d8eccfdc0fb
+size 4998017472

model-00036-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dd78eb8ee7a2cd4e8c68987fa75b02ee4ed53fe45ac683ad22e38039e37f47a
+size 4995765008

model-00037-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d1ff442627382551f7a2d1cd3ac4aa39a24336785dc8ec4c0921bfe41939919
+size 4998017472

model-00038-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28b8ced84b01f25b4ea51e68b41a8ef8f26012d0617b11c7a7ae13ddcaf3651a
+size 4995765008

model-00039-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c742a1f4369a605d8604f6285557ac2e9cb41857aca4ae01a55a2a49df80fb2
+size 4998017472

model-00040-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77291932e03746a49c30bab77a0d017afc1c285ffe1d17c7b1295ce73ad1bc7c
+size 4998018096

model-00041-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88569b75b8cb9a8ecb70373ef65bfe15b34b1f9c5cb20f3d7ddf5377472856b0
+size 4995764384

model-00042-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1d04abcf28b60023851f43130b77427233caf649410ba35691029213e5bafa4
+size 4998017672

model-00043-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:466e5fb838367c0fa62c5c7df52e4c66692e80874f2ed4e957e68e008a3f8511
+size 4995764808

model-00044-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ecee5e106328ab7f1a3c4bbfa3ce746e45ef4468c3d805a23d56f4610b15d20
+size 4998017472