Exclude BF16 MTP layer 78 from ModelOpt FP4
#2
by bullpoint - opened
Add model.layers.78* to quantization_config.ignore so ModelOpt-aware loaders leave the built-in multi-token prediction (MTP) layer unquantized. The checkpoint declares 78 transformer layers and one MTP layer at index 78. Its 768 routed expert weight tensors in model-00084-of-00084.safetensors are BF16, so applying NVFP4 quantization to those modules does not match the stored weights. The exclusion lets vLLM load the target model and its MTP layer from the same model directory.
This configuration-only change preserves quantization settings for transformer layers 0–77 and does not modify any checkpoint tensors.
Status: implemented. Validation:
- Parsed the modified JSON and verified that adding the single ignore entry is the only change.
- Inspected the safetensors header: all 768 routed MTP expert weight tensors are BF16. The downloaded shard's recorded SHA-256 matches the repository's shard at the PR base revision.
- No server startup or inference qualification was performed for this PR.