MiniMaxAI

/

MiniMax-Music3

like 603

Follow

MiniMax 9.25k

not without downstream training (currently) on a new conditioning objective that makes use of some input we already can create. we can actually do that via something similar to IC-LoRA, but it'd need a good dataset, which is in short supply in the Audio world.

we have the audio VAE encoder, but not the RVQ encoder.

the Qwen finetune they built takes the text + lyrics and creates semantic audio tokens for each frame one codebook at a time;
we have the RVQ depth decoder which can produce vector-quantised residual audio tokens from this Qwen models' semantic tokens
we don't have an encoder that can produce these from real audio inputs.

how Codex describes it:

The feedback loop is:

1. Qwen reads the text/lyrics prompt ending in <|audio_start|>.

2. Qwen predicts one semantic audio token from a reserved audio-token range.

3. rvq_depth_decoder takes:

   • Qwen’s current hidden state
   • the semantic audio token
   • residual codes sampled so far

4. It predicts the remaining RVQ codebooks for that audio frame, one by one.

5. _embed_audio_frame() turns the full frame of codes into one embedding:

   • semantic code uses Qwen’s own token embedding table
   • residual RVQ codes use rvq_depth_decoder.audio_embeddings
   • those are summed/scaled into a single frame embedding

6. That embedding is fed back into Qwen via inputs_embeds, with past_key_values, so Qwen advances to the next audio frame.

7. Repeat.

So Qwen consumes an embedding of the generated audio frame codes, not raw waveform and not DAV latents.

There are two outputs from each RVQ/depth step:

• frame_codes: fed back into Qwen so it can autoregress the next semantic audio token.
• depth_hidden: collected, concatenated with Qwen hidden state, then passed through condition_encoder to condition the flow transformer.

That means the AR/RVQ loop is basically a discrete audio-plan generator. The flow transformer then renders continuous DAV latents conditioned on that plan.

until they answer this request to either release or let us know we're never receiving the encoder, i'd just assume that they intentionally did not release this to keep from having the RIAA or BREIN or other "thou shalt not have fun with Copyright" groups from jumping on their back.

if thats the case, then this is useless. Unfortunate

how is it useless? the audio VAE encoders' latents are what's targeted for training which works flawlessly.

RVQ is only needed for generating semantic tokens directly from audio for conditioning instead of text lyrics/captions.

someone will train an IC-LoRA to cross-attention a set of reference audio latents. RVQ encoder is not needed.