How to Use the YuE2 Hum-to-Song Adapter

We wanted a short hum or riff to become the starting point for a longer song. In our YuE2 experiment, that meant two jobs: giving the melody to the song planner, and training a decoder adapter to refine phrasing, including timing and articulation.

We built the dataset from songs with separated vocals: 3,163 training pairs and 188 held out for evaluation. Follow one full mix. Its vocal track becomes a simplified conditioning signal, the decoder's melody and timing hint. The existing audio-to-token head converts the mix into semantic tokens, compact musical codes supplied as context. Separately, an audio encoder called a VAE compresses the full mix into numbers called latents. Those are the clean training target. Condition, context and target have different roles. When generating a new song, we don't supply that original mix as a target.

To make the condition resemble a hum, we track the vocal pitch and synthesize a sine wave, a simple pure tone that follows it. We shape its loudness using smoothed vocal amplitude, then encode this carrier with the VAE. The aim is to retain pitch and broad timing while reducing words and the singer's distinctive sound. That doesn't establish perfect separation, but it deliberately simplifies what the adapter receives.

We train added connections that feed in the carrier, along with decoder adapters and input and output mappings. Flow-matching loss measures error in the predicted movement between noise and the mix's latent target. To practice using a short hint, half the training draws use a partial carrier on eligible songs. We keep 12 to 28 seconds, beginning two seconds before the vocals start, and zero everything outside that interval. Ten percent of draws have no carrier at all. This gives the module practice entering and leaving the hint.

The melody also needs to reach the planner. We transcribe the hum into ABC, a text format for musical notation, and leave off the score's end marker. The pretrained planner keeps writing the unfinished score, then produces the song's semantic tokens. The decoder turns those into audio latents, and the VAE decodes sound you can hear. This score continuation needs no new training. The phrasing adapter is the trained part.

After 12,000 steps, our reported held-out loss was 0.906 with hum, compared with 0.913 without it. That's a modest flow-loss advantage. It doesn't measure a percentage improvement in song quality or prove perfect continuation. The score route carries the tune into the longer song; the adapter refines how the line is phrased.