Is the model trainable?
Lora, Lokrs, etc
Important, important thing for making the model relevant on actual use.
Thankss
Until they release the Encoder, it's not.
yes: https://docs.simpletuner.io/quickstart/MINIMAX_MUSIC/
LyCORIS and PEFT LoRA trainable via SimpleTuner, with lyrics + caption validation libraries too. though that's not easily configured via the webui just yet (the validation library), the example prompts and captions from their website are included as a built in validation prompt library if you're doing eg. a broad finetune and want to validate a number of separate concepts.
it works with distillation too, so, the next project i'm on with MM Music is to throw AnyFlow at it and bring it down to a decent 4-step generation pipeline in the flow transformer.
what we're limited by is the lack of RVQ encoder's semantic audio tokens from raw audio. i think we could reverse engineer one though if so inclined, i'm not sure whether i'd be going against the license to do so.
Nice.. Its something.
Would it make any sense to opensource a model to then not share a fundamental piece? A model that is not trainable/conditionable is pretty much a toy.
The RVQ encoder is required for properly fine-tuning the model. Without it, you can maybe adjust some tone and pitches, though you would be doing so 100% without proper conditioning. It is like training an image model but instead of captions, you are just typing pure nonsense random text in the encoder for the conditioning. I mean sure, you CAN train it that way. But you will not get good results. The RVQ encoder is absolutely needed.
that's not true, look at how the pipeline actually works. the depth decoder gives the model its tokens from text. the encoder just does it from raw audio.
the inference path mirrors training. you're not required to supply RVQ encoder output (audio references) at inference time. why would you think it'd be mandatory at training?
our training target for the flow matching objective is the VAE audio latents, not the RVQ encoder output.
we wouldn't even be able to run the model for inference if the encoder were required for training.
@bghira You are attempting to train on non aligned tokens. During inference, they are aligned because they were trained to be that way, with the RVQ tokenizer, something you will be breaking with your "training" method.. You are just swapping out a non aligned target and breaking the alignment and saying you trained it. It is like training an LLM without a tokenizer. You have your output tokens, but you are feeding complete garbage on the input.
Think of it this way. RVQ hidden state token are like a MIDI track. A super high detail, frame by frame, MIDI track that contains lyrics for each time step, drums, the beat, when the guitar strums are, to the millisecond, when the chorus starts, everything. It is play by play, perfectly time aligned sheet music and detailed composition for each frame of the song. Without the RVQ tokenizer, you are generating this precise midi track, and throwing a completely different song at it where none of the timing aligns , the lyrics are completely different, this one does not even have drums, the "E note piano key" is now a "F note flute". It is in a different key, pitch, everything is 100% misaligned. And you are teaching the model,"make this sound like that" You are just completely breaking down the model doing this. 100%. And maybe, just maybe, before the model is completely broken, it may sound a little bit like the 100% misaligned song you fed into it. But that does not mean you trained it, you just broke it pretty.
i don't get where you're getting any of that from. sorry, it's just not the case.
- RVQ encoder gives target-audio-derived discrete tokens.
- RVQ depth decoder gives text-derived residual-code hidden states.
- the transformer consumes condition-encoder outputs, not raw RVQ encoder states.
- the model supports text-to-audio inference, which is conditioning generated from text via the feedback loop with the LM and depth decoder, not from the RVQ encoder
- text-to-audio LoRAs train against the same conditioning used at inference
- RVQ encoder is ONLY needed for target-token supervision, reference-audio conditioning
we're not training on "garbage text", but sure, if you have really bad captions and lyrics inputs that don't align at all with the latents the audio latent encoder creates, you'd get bad results.
have you actually tried training it, or just repeating yourself hoping that people believe it?
No point arguing with arrogance.
agreed
I don't understand enough of the tech details to get those intricacies but Ostris and Bghira arguing on my question thread is an honor I guess.. 😁
agreed
@bghira you've implemented training modules for it on SimpleTuner, do you have any examples we can test? Have you produced any LoRA/LyCORIS/finetune with your method that we can try? I can provide a small dataset with some relatively obscure genres/styles, and evaluate the results if you want.
That's the easiest way to find out whether it works or not without the RVQ encoder.
yes, but it was using internal datasets and i'm contractually barred from sharing those results. i've put together a public dataset after identifying that Jamaican patois / reggaeton is really badly generated by the model (its' actually kind of funny how bad it is), you can check the data i'm using under RareConcepts/suno-reggae-test-dataset