laion/chatterbox-s3gen-vc-grow-ce-continue

Chatterbox S3Gen VC · Continued rank-128 LoRA, 50/50 CE and emotion

This is a research-stage audio-to-audio voice-conversion adapter, not a standalone TTS model. It changes Chatterbox's S3Gen acoustic decoder while keeping the S3 semantic tokenizer, CAMPPlus speaker encoder and HiFT vocoder frozen. The adapter is rank 128 on 224 q/k/v/out attention projections, with the previously trained 99-score conditioning projector. Download the pinned upstream Chatterbox S3Gen and the 300-hour supervised/v1 RL parent release separately. This repository does not contain the upstream base weights or the training audio.

How the GROW-style reinforcement learning works

If you know group-relative RL for language models, the familiar part is that we sample 16 outputs for the same source/target voice pair, score those outputs, and compare candidates within that group rather than fitting a separate value model. Here the outputs are complete voice-converted waveforms. Frozen models score Content Enjoyment and the source recording's two strongest emotions; CAMPPlus scores similarity to the target speaker. We standardize each reward component within the group, take their weighted sum, clip and center it to obtain a signed advantage. The flow decoder then learns from its own freshly generated examples: each candidate's usual flow-matching velocity regression loss is weighted by that advantage, while a penalty keeps its velocity close to a frozen copy of the starting model. This is not token-log-probability GRPO, and it does not update the frozen reward models.

The attraction is that it works directly with Chatterbox's deterministic flow-matching decoder: no ODE-to-SDE conversion or likelihood ratio at every denoising step is needed. The costs are 16 rollouts and several reward-model passes per training condition; within-group ranking also does not by itself measure absolute quality. Proxy rewards can be exploited, a signed weighted regression can be unstable, and the frozen-policy anchor cannot guarantee naturalness or speaker fidelity. This implementation adapts the method to voice conversion with different rewards, not the architecture, datasets, or results of the original GROW paper (Yang et al., 2026).

What this arm does

The completed v1 continue rank-128 adapter at step 4,000 is continued without merging it into the base.

The reward uses the frozen Humaneness Ears Medium model at revision 818506970d93c809a3295a0a02c40dee8ff2bfc3. Content Enjoyment is specifically its audiobox:CE output, not the separate eiv_extra:score_content_enjoyment output. For each 16-candidate group, all reward components are standardized separately within that group; the weighted advantage is clipped to ±2.5 and centered. The two emotion dimensions are the strongest Ears emotions in the actual source recording, not fixed labels from the prompt. The weights are 50% Content Enjoyment and 50% negative top-two emotion MSE.

The selected 4,000 source/target conditions contain 3,357 distinct sources and 477 target-voice clips. They are traversed six times with different permutations and sampling seeds: 24,000 optimizer steps and 384,000 generated candidates per arm, but not 24,000 distinct input pairs. Every step is one 16-candidate group, split four candidates per GH200 GPU across a four-GPU node, with synchronized gradients. The new AdamW schedule has 1,200 warm-up steps (5%), a 2e-6 peak LR, cosine decay to 2e-7, and a frozen-policy flow anchor of 0.025 to the respective v1 step-4,000 starting model. Checkpoints with optimizer state are written every 1,000 groups and automatically published under checkpoints/. See RL_CHECKPOINTS.json for verified SHA-256, size and step. A missing step means it has not yet been verified and published.

Inference and continuation

This is a research implementation for prepared Parquet rows, not a one-command arbitrary-WAV VC pipeline. The source row supplies S3 semantic tokens and dynamic emotion/style scores. A 5–10-second target reference row supplies its acoustic prefix, CAMPPlus identity vector and static voice traits. See TRAINING.md for preparation, exact model pins, score order, training/resume commands and known limitations. The included code/grow_followup/infer_followup.py uses a prepared source/target pair:

pip install -r requirements.txt  # install a compatible CUDA PyTorch first
hf download ResembleAI/chatterbox s3gen.safetensors \
  --revision 5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18 \
  --local-dir upstream_chatterbox
hf download laion/chatterbox-s3gen-vc-grow \
  checkpoints/base_300h_adapter.safetensors \
  --local-dir parent_v1
export CHATTERBOX_S3GEN_WEIGHTS="$PWD/upstream_chatterbox/s3gen.safetensors"
export CHATTERBOX_BASE_ADAPTER="$PWD/parent_v1/checkpoints/base_300h_adapter.safetensors"
export PYTHONPATH="$PWD/code:${PYTHONPATH:-}"

python code/grow_followup/infer_followup.py \
  --pool /path/to/prepared_pool.parquet \
  --score-order-json score_order_and_preparation.json \
  --source-uid SOURCE_UID --target-uid TARGET_UID \
  --arm continue --checkpoint checkpoints/step-0024000.pt \
  --output-wav example.wav --seed 1901

The .pt files include optimizer state and use Python pickle; the training code reseeds each rollout deterministically instead of storing RNG state. load only trusted files. The inference code uses PyTorch's restricted weights_only=True loader but still requires verified provenance. The checkpoint is not a merged base model. Load the supervised base adapter, then load this continuation adapter.

Interpretation and limits

The rewards are model-predicted scores, not human ratings. Repeated traversal of 4,000 pairs can overfit, and the reward models can be exploited. The speaker cosine, when present, is a CAMPPlus proxy and cannot establish human-perceived identity fidelity. There is no claim here that reward improvement means better voice conversion. A matched-seed listening study is available, but independent human MOS, content accuracy, and speaker-level disjoint evaluation are still needed. The selected sources are UID-disjoint from the previous holdout, not guaranteed speaker-disjoint.

LAION-authored adapter weights, code and documentation here are CC BY 4.0. The upstream Chatterbox dependency retains its separate MIT license and attribution. Use only voices and audio you have rights to process.