Hugging Face's logo

coolpoodle
/
music3lab

music
audio
music-generation
minimax-music3
research
toolkit
Model card Files
xet
 Community
2
music3lab / FINDINGS.md
coolpoodle's picture
coolpoodle
code and training scripts
90884df verified
preview code
|
Raw
History Blame Contribute Delete
16.5 kB

MiniMax Music 3 dav.pth investigation

Conclusion

The released dav.pth does not contain the native Music 3 waveform-to-RVQ tokenizer. It contains a continuous Flow-VAE analysis/synthesis checkpoint: a waveform convolutional encoder, Gaussian posterior heads (mean_proj and logs_proj), continuous flow transforms, and a waveform decoder. There is no RVQ/VQ quantizer and there are no acoustic codebook embeddings. The encoder.* name is therefore real but is not the discrete encoder required by the proposed continuation path.

The safe CPU-only inspection of release revision fbdf52fbaaca799592917417eb05f1899f1255ec found:

Prefix Tensor count Representative evidence
encoder.* 119 encoder.block.0.weight_v [64,1,7]; final trunk encoder.block.6.weight_v [1024,1024,3]
mean_proj.* 2 mean_proj.weight [64,1024,1]
logs_proj.* 2 logs_proj.weight [64,1024,1]
dec_in_proj.* 2 dec_in_proj.weight [1024,64,1]
decoder.* 119 first convolution [1536,1024,7]; final convolution [1,96,7]
flow.* 304 flow.flows.0.pre.weight [256,32,1] through flow.flows.6.post.weight [32,256,1]

That is 548 tensors, all FP32, containing 122,904,034 values (491,616,136 tensor bytes). The file is 491,817,450 bytes and has SHA-256 52adde6c6c52cca872f549449cd7b608677d1e69f4a98eee22da3403e63b56e8. It is a flat OrderedDict: there is no generator key, no 62000_generator key or wrapper, and zero names matching quantizer, vq, rvq, codebook, codec, or tokenizer. It also contains no serialized config or architecture metadata. Tensor shapes plus released source are sufficient to instantiate the decoder, but cannot instantiate a quantizer that has neither parameters nor configuration.

The reported community 62000_generator wrapper is not present in this exact released file. inspect_dav.py still recursively supports generator, 62000_generator, and state_dict-style wrappers so a different checkpoint can be checked without assuming it has the release layout. Names, shapes, and generic config can flag a candidate weight set, but the inspector deliberately keeps native_discrete_tokenizer_complete and can_encode_native_music3_tokens false until an exact compatible executable architecture/API has been implemented and verified.

The actual token and rendering paths

Music 3 uses eight discrete streams at 25 frames/s. They are language-model symbols used to produce continuous conditioning hidden states; they are not DAV latent codes and are never passed directly to the DAV decoder.

caption + lyrics + <|audio_start|>
                |
                v
Global Qwen LM: c0 in [0, 16383] (vocabulary token c0 + 151675)
                |
                v
Local RVQ-depth LM: c1..c7, each in [0, 1023]
                |
                +--> frame token row [c0,c1,...,c7]
                |
                +--> global hidden + 7 depth hiddens = 8 * 4096
                     frame_hiddens [1, frames, 32768]
                                      |
                                      v
                         condition projection + flow-matching DiT
                                      |
                                      v
                         continuous Flow-VAE latent [B,128,T]
                                      |
                                      v
                         DAV dec_in_proj + decoder -> waveform

The token trajectory convention used by the provided comparator is [frames, 8]; [8, frames] is accepted only when explicitly declared as codebooks_first. No heuristic transposition is performed.

The per-frame autoregressive details are:

  1. The global LM samples only <|audio_end|> or the 16,384 c0 vocabulary IDs beginning at offset 151675.
  2. The local depth model begins with the projected global hidden and the projected c0 embedding, then samples c1 through c7 autoregressively. It has seven 1,024-way output heads.
  3. For feedback to the global LM, c0 uses the global token embedding at c0 + 151675. Each residual code uses the Qwen checkpoint's model.audio_extra_embedding.weight in seven disjoint 1,024-row bands. Those seven embeddings are summed with c0 and scaled by 1/sqrt(8). This is an LM embedding table, not the absent waveform quantizer's acoustic centroids.
  4. Codebooks are frame-aligned (one row contains all eight codes); there is no delayed-codebook layout in this path. Conditional and classifier-free-unconditional rows are paired during sampling. The first decode immediately after <|audio_start|> primes the feedback loop and is not emitted as an acoustic hidden frame. Each subsequent emitted frame concatenates one 4096-wide global hidden with seven 4096-wide local hiddens.
  5. The 25 Hz rate is explicit in Diffusers and also follows 24000 / 960 in the condition encoder configuration.

Special IDs are fixed by the music tokenizer:

Token ID
`< im_start
`< im_end
`< audio_cfg
`< audio_start
`< audio_end
`< caption_start
`< caption_end
`< lyrics_start
`< lyrics_end
first c0/audio-code vocabulary ID 151675

The Qwen shards contain the global audio-symbol embeddings plus the local autoregressive depth decoder (model.audio_extra_embedding.* and model.audio_decoder.*). They contain no waveform encoder, nearest-neighbor quantizer, RVQ codebooks, or equivalent audio-tokenizer module. A model that can predict token IDs from text is not therefore able to infer those IDs from a waveform.

Successful official H100 generation capture

A pinned end-to-end Diffusers run succeeded on the remote NVIDIA H100 using model revision fbdf52fbaaca799592917417eb05f1899f1255ec, Diffusers revision 90b4e34e79a86ec5e7f2437634fe95ecd2108796, CUDA with bfloat16 weights, and seed 7. The exact request used prompt Instrumental French house, 126 BPM, E minor, filtered disco loop, punchy kick and warm bass., lyrics [instrumental], requested duration 1.0 seconds, and 30 inference steps. It made 26 calls to the official _generate_depth_codes: one priming call followed by 25 emitted frames. After skipping only that priming row, music3_internal_tokens.npy is a genuine internal [25,8] trajectory at 25 Hz.

The observed inclusive per-codebook ranges were:

Codebook Minimum Maximum Legal range
c0 1012 16163 0..16383
c1 95 984 0..1023
c2 25 1005 0..1023
c3 42 950 0..1023
c4 83 941 0..1023
c5 2 967 0..1023
c6 3 984 0..1023
c7 67 957 0..1023

The corresponding official renderer output, music3_generated_1s.wav, is 44.1 kHz stereo with 44,032 samples per channel (0.9984580499 seconds). The reusable capture_generated_tokens.py utility reproduces this observation by temporarily monkeypatching the official depth-code function, validating its paired [2,8] result, restoring the function after generation, and saving only calls after the priming row. Before model loading it resolves the requested Hub revision to a concrete snapshot commit, verifies the exact Diffusers commit, rejects dirty tracked files in a source checkout, and hashes both the relevant Diffusers source file and the capture script. It writes temporary WAV, NPY, and JSON files, atomically replaces the two data artifacts, and replaces metadata last. The metadata records the runtime and request plus SHA-256 digests of the final WAV and NPY, so an interrupted mixed-generation bundle cannot validate. It records "wav_reencoding_performed": false because this is an internal generation capture, not WAV analysis.

For the final canonical rerun, the WAV SHA-256 is e52c8884acdfbe9a71badac1ef410a5e5b23512dfc3d56a6f76228adaae87e11, the NPY SHA-256 is 42de27f795e8a1f1b97ae85fd3a540d432a8b178fdc63f7037bf0eea08c61350, and the metadata JSON SHA-256 is 7232d979f21cdf769e96c047b11ee31bbe28a9ee30ec6cd6960455c44b71aa02. Metadata embeds the first two digests along with capture-script SHA-256 e4b6ecb87652deaa987a722e7c281aec2196295a717066aec05ef195361c0f1f and relevant Diffusers encoder-source SHA-256 247ca4492a531c6593485b9fb11503da8607d03315c2f8b313dc6ae44985336b.

This run proves that the documented token shape, ranges, frame rate, priming alignment, and generation/rendering path are executable. It does not supply the missing WAV-to-tokenizer path. Calling encode_audio() for the generated WAV is still blocked by the released DAV capabilities, so there is no recovered token trajectory to compare. Per-codebook WAV re-encoding agreement percentages therefore remain unavailable; comparing music3_internal_tokens.npy with itself would only be a self-comparison and must not be reported as re-encoding agreement.

What conversion omits

The Diffusers converter maps only dec_in_proj.* and decoder.* from dav.pth. Its explicit conversion dictionary and decoder loop leave 427 of the 548 tensors unmapped: all 119 encoder.*, both mean_proj.*, both logs_proj.*, and all 304 flow.* tensors. SGLang-Omni makes the same decoder-only selection with prefixes ("dec_in_proj.", "decoder."). That omission is real, but restoring those keys would expose only continuous DAV analysis/posterior operations. It would not restore the absent RVQ quantizer/codebooks.

Answers to the six requested questions

  1. Is the original audio encoder present? A waveform analysis encoder is present as encoder.*, but the native discrete/RVQ tokenizer encoder is not. Its outputs feed continuous mean_proj/logs_proj and flow modules.
  2. Is the RVQ quantizer present? No. There are zero matching quantizer/RVQ/VQ parameters and no acoustic codebooks in dav.pth. The Qwen local “RVQ depth decoder” predicts residual token IDs; it is not a waveform quantizer.
  3. Can arbitrary WAV audio be converted into valid Music 3 tokens? No, not with the released weights. Calling encode_audio() raises NativeTokenizerUnavailableError before opening the WAV and never invents token IDs.
  4. Do re-encoded tokens match Music 3 internal tokens? This experiment is blocked because no re-encoded tokens can be produced. Reporting per-codebook percentages would be fabricated. native_token_compatibility.py performs strict range/layout validation and exact per-codebook comparison once both real trajectories exist.
  5. Can those tokens seed continuation? No token trajectory can be recovered, so continuation cannot be seeded. The current released inference entry points also start from text plus <|audio_start|> rather than accepting an existing token/LM-cache history. Even if a tokenizer is released later, continuation must replay each complete eight-code frame through the exact feedback embedding/depth-hidden path and preserve the priming alignment.
  6. What is missing? The compatible waveform-to-code encoder/quantizer implementation, its RVQ acoustic codebook weights and config, plus an official existing-audio history/prefill interface. Nothing here justifies retraining or substituting a generic codec.

There is intentionally no continue_audio.py: without native tokens, such a CLI could only ignore the input audio, mislabel continuous latents as tokens, or use an incompatible replacement codec. All three would violate the stated conditioning contract. Likewise, a token round trip is impossible: Music 3 renders frame_hiddens through a flow-matching model and DAV, not discrete codes directly through DAV.

Reproducible checks

The inspection, comparison, blocked-encode, and test paths are CPU-only. Checkpoint loading uses torch.load(..., weights_only=True, map_location="cpu"). The optional capture_generated_tokens.py command is the explicit exception: it loads the pinned official generation runtime and uses the requested device (CUDA by default) only to observe internally generated tokens. pyproject.toml requires torch>=2.10.0, after the vulnerable versions listed in GHSA-63cw-57p8-fm3p.

Install the optional generation-capture runtime with pip install -e '.[capture]'; that extra pins the audited Diffusers Git revision and explicitly declares Accelerate, Hugging Face Hub, Transformers, and SoundFile in addition to the base NumPy and Torch requirements.

python inspect_dav.py /path/to/dav.pth --sha256
python inspect_dav.py /path/to/dav.pth --json

# Both commands return nonzero and say BLOCKED before opening missing.wav.
python encode_audio.py missing.wav --dav /path/to/dav.pth --json
python round_trip_test.py missing.wav reconstructed.wav --dav /path/to/dav.pth --json

# Exact comparison, only when genuine internal and re-encoded trajectories exist.
python native_token_compatibility.py internal.npy recovered.npy \
  --reference-layout frames_first --recovered-layout frames_first --json

# Capture tokens from generation itself; this does not re-encode the saved WAV.
python capture_generated_tokens.py \
  --prompt "Instrumental French house, 126 BPM, E minor, filtered disco loop, punchy kick and warm bass." \
  --lyrics "[instrumental]" --audio-duration 1 --num-inference-steps 30 --seed 7 \
  --local-files-only \
  --output-wav /path/to/music3_generated_1s.wav \
  --output-tokens /path/to/music3_internal_tokens.npy \
  --output-metadata /path/to/music3_generation_capture.json

# Generated-sample integration is enabled only when all four are set.
MINIMAX_DAV_PATH=/path/to/dav.pth \
MINIMAX_GENERATED_WAV_PATH=/path/to/music3_generated_1s.wav \
MINIMAX_INTERNAL_TOKENS_PATH=/path/to/music3_internal_tokens.npy \
MINIMAX_GENERATION_CAPTURE_PATH=/path/to/music3_generation_capture.json \
pytest

Synthetic tests cover flat and nested/62000_generator checkpoints, capability verdicts, pre-WAV failure, explicit token layouts, all codebook ranges, exact/mismatched agreement, unequal frame counts, capture ordering, metadata-last replacement, source cleanliness, and partial artifact configuration. The environment-gated integration test checks the real 548-tensor release, validates the full generated-capture schema, recomputes the WAV/NPY/source/script hashes, and confirms the blocked encoder path. Impossible experiments--WAV token encoding, token round trip, re-encoding agreement, and continuation--are explicitly reported as blocked rather than marked passed.

Audited sources and revisions