GLM-5.3-Flash · ROCm/RDNA4 serving overlay & quantization recipe

Serve zai-org/GLM-5.3-Flash on 8× AMD Radeon R9700 (gfx1201 / RDNA4)

Overview

This repository contains the serving overlay and quantization recipe used to run a self-quantized derivative of GLM-5.3-Flash — a 321B-parameter sparse MoE with native multimodality — on a cluster of 8× AMD Radeon R9700 GPUs (gfx1201 / RDNA4).

Stock vLLM does not ship this architecture on the RDNA4 path, and the aiter package is physically absent from the base image. This repo is the bring-up work that makes it serve correctly and fast on consumer RDNA4 hardware, plus the quantization recipe that produced the published weights.

📦 Weights: GLM-5.3-Flash-RFA-RFI8-8xR9700 — 197.8 GB, 25 shards, 4.93 bpw average.

Attribution & credits
The model
What this repository does
Repository layout
Quantization
Build
Run
Performance
Multimodal policy
Known limitations
License

Attribution & credits

This work stands on the shoulders of two upstream projects. Credit is deliberately split:

Component	Who built it	What it provides
Serving stack & quant kernels	`tcclaviger/vllm:latest` (IronLLM Labs)	The vLLM v0.24 ROCm/RDNA4 runtime (torch 2.11 / ROCm 7.2.3), the RFA / RFI composite quantizer and its quant kernels (`RFA` 4.5-bpw experts, `RFI8` structural), and the baked-in tuner/quant tooling.
Model weights & architecture	zai-org/GLM-5.3-Flash	The base model, released under MIT.
This repository	overlay / port work on top	The `glm5next` architecture backport (vLLM PR #53906), the no-AITER sparse-MLA/KDA DSA port for RDNA4, the kpool ROCm port, the multimodal resize policy, and the composite quantization recipe (`quant/`).

In short: the runtime and the quant kernels are tcclaviger/vllm (IronLLM Labs); our contribution is the GLM-5.3-Flash overlay + RDNA4 port + the quant recipe layered on top.

The model

GLM-5.3-Flash is Z.ai's first natively multimodal GLM-5 model — 321.3B total / ~18B active parameters with a hybrid attention backbone:

Property	Value
Total parameters	321.3B (320B in the upstream README)
Active parameters	~18B per token
Layers	45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA)
Routed experts	288 (top-8) + 1 shared expert
Extra	mHC hyper-connections, 1 nextn MTP draft layer, native vision tower
Architecture class	`Glm5NextForConditionalGeneration`

What this repository does

glm5next backport into tcclaviger/vllm:latest — registers Glm5NextForConditionalGeneration (upstream vLLM PR #53906 backport) and the Glm5Next processor / image / video tower in the v0.24 runtime.
kpool + sparse-MLA no-AITER port under RDNA4 — the DSA layers boot without aiter: indexer direct-call, compact profiler workspace, torch top-k fallback, cudagraph-safe Triton DSA kernels, and a full kpool custom op on ROCm.
Sync-free prepare_chunk_indices — removed the host DtoH sync in the KDA region, which unblocks FULL-cudagraph capture at batch size ≥ 2.
Multimodal resize clamping — images upscaled to min 384×384, downscaled to max 1280×1280.
RFA + RFI8 composite quantization — the recipe that produced the published weights.

Repository layout

.
├── app/vllm/vllm/…              # patch overlay tree (copied over the container's /app/vllm)
├── apply_overlay.sh              # patches-at-start: applies the overlay + purges stale bytecode
├── run-glm53.sh                  # canonical docker run wrapper (serving)
├── preflight-gpu.sh              # GPU/host preflight checks
├── tests/                        # kernel/validation probes (kpool, mhc, no-aiter DSA, R2b)
├── OVERLAY_NOTES.md              # technical bring-up notes (root-cause writeups)
└── quant/                        # quantization recipe + injector patches
    ├── glm5_archspec.py          # ArchSpec for glm5_next (patch + --check validator)
    ├── glm5_fp8_source_patch.py  # FP8 e4m3 block-source dequant support
    ├── glm5_kda_remap_patch.py   # KDA module-tree <-> checkpoint name bridging
    ├── run-quant.sh              # container-side wrapper for the composite quantizer
    └── run-full.sh               # full 8×R9700 quantization run (composite recipe)

Quantization

The checkpoint mixes three schemes in one quant_method: "rfi" composite, produced by the tcclaviger/vllm composite router (--quantize --mixedQuant, activation-aware):

Scheme	Bits	Applied to	Stored size
RFA	4.5 bpw	MoE routed experts (42 layers × 288 experts)	~171.3 GB
RFI8	8 bpw	attention / shared-expert / dense linears	~7.8 GB
BF16 / FP32	16 / 32 bpw	embeddings, vision tower, norms, MTP layer, dense copies	~18.7 GB

Average bits-per-weight

bpw = (total safetensors bytes × 8) / total parameters
    = (197,843,715,288 × 8) / 321,342,220,638
    = 4.9254  ≈  4.93 bpw

Metric	Value
Total parameters (sum of weight shapes)	321,342,220,638 (~321.3B)
Total on-disk size	197.8 GB (25 shards)
Average bpw	4.9254 ≈ 4.93
vs. FP8 source checkpoint	0.60× (197.8 GB vs 328.3 GB)
vs. theoretical BF16	0.31× (197.8 GB vs 642.7 GB)

The full recipe (archspec, source patches, kda-remap, run scripts) lives in quant/.

Build

The overlay is applied at container start — no image rebuild required:

# 1. pull the base image (vLLM 0.24 tree, torch 2.11 / ROCm 7.2.3, RDNA4/gfx1201)
docker pull tcclaviger/vllm:latest

# 2. the container mounts this repo at /overlay and runs apply_overlay.sh first
#    (see run-glm53.sh for the exact docker run invocation)

Run

# current production configuration (see run-glm53.sh for the full wrapper)
docker run --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
  -v /path/to/GLM-5.3-Flash-RFA-RFI8-draftRFA4:/models:ro \
  -v "$PWD":/overlay:ro \
  --entrypoint bash tcclaviger/vllm:latest \
  -c "/overlay/apply_overlay.sh && GLM5_NEXT_MTP_PROPOSER=1 exec vllm serve /models/GLM-5.3-Flash-RFA-RFI8-draftRFA4 \
      --served-model-name glm53-flash --trust-remote-code --quantization rfi \
      --tensor-parallel-size 8 --gpu-memory-utilization 0.9575 \
      --max-model-len 300000 --max-num-seqs 4 --max-num-batched-tokens 2048 \
      --kv-cache-dtype fp8 \
      --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \
      --enable-prefix-caching --distributed-executor-backend mp \
      --compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16],\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_copy_inputs\":true}'"

run-glm53.sh is the full wrapper with the RDNA4-specific environment (HIP/RCCL, tunableop, RPC timeouts, the no-AITER rung ladder) and the production cudagraph config. The production launch exposes the MTP proposer via the GLM5_NEXT_MTP_PROPOSER=1 gate — without it the stock EagleProposer path is used (see OVERLAY_NOTES.md).

Note on the checkpoint name: -draftRFA4 is a 183 GB requant variant of the published weights in which the nextn/MTP drafter block is compressed to 4-bit RFA — it frees VRAM so the MTP drafter + fp8 KV cache fit alongside the 300k context window. The published full checkpoint (197.8 GB, BF16 drafter) serves the same stack with MTP disabled.

Performance

Verified numbers on 8× R9700 (256 GB VRAM, TP8), current production config (MTP spec=3, fp8 KV, FULL_AND_PIECEWISE cudagraphs, VLLM_USE_NCCL_SYMM_MEM=0):

Scenario	Throughput
Decode, batch size 1, MTP OFF (FULL cudagraph)	~34–39 tok/s
Decode, batch size 1, MTP spec=3	~82–88 tok/s
Aggregate, 4 concurrent, MTP spec=3	~155 tok/s
Context window (fp8 KV)	300,000 tokens

MTP speculative decoding metrics (live engine log, spec=3): mean acceptance length ~3.7–3.9 of 4 draft tokens, per-position acceptance ~1.00 / 0.95–0.99 / 0.85–0.92, average draft acceptance 91–97%. The VLLM_USE_NCCL_SYMM_MEM=0 + CLAV_AR_QUANT_MIN_KB=128 allreduce combo was A/B-verified: +9.9% on bs=1 decode, neutral at bs=4, quality PASS.

Multimodal policy

Images are resized with aspect ratio preserved, clamped to min 384×384 and max 1280×1280. The processor is invoked with a min/max image-token budget (--mm-processor-kwargs).

Known limitations

~~MTP is OFF~~ RESOLVED (2026-09-09). MTP spec-decode now ships enabled in production (spec=3, up to ~88 tok/s single-request). The fix is a dedicated glm5next_mtp.py proposer with per-group draft KV plumbing (own block table + slot mapping per draft KV cache group), enabled via the GLM5_NEXT_MTP_PROPOSER=1 gate; the stock EagleProposer still fails on the "All drafting layers should belong to the same kv cache group" assertion. Details in OVERLAY_NOTES.md.
fp8 KV: works in production (--kv-cache-dtype fp8, scales fixed at 1.0, no runtime calc_kv_scales). Do not enable --calculate-kv-scales — runtime calibration happens on the profile dummy-run where the KDA recurrent state is uninitialized, producing garbage scales and hard output looping. See OVERLAY_NOTES.md.
Chat needs reasoning_effort="low". The GLM-5.3 chat template defaults to Reasoning Effort Max, which spends 16k+ tokens thinking before producing content on long generations.

License

MIT — see LICENSE. The GLM-5.3-Flash base model is © Z.ai (zai-org), MIT license. The serving stack tcclaviger/vllm:latest is © IronLLM Labs (see its Docker Hub page).

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

djdeniro
/

GLM-5.3-Flash-rocm-r9700