You type "a calming violin melody backed by a distorted guitar riff," and a few moments later you have a piece of music that never existed before. That example prompt comes from Google's MusicLM paper, one of several research systems that explain, in public, how text-to-music generation works.
This guide walks through the main ideas in plain English. It uses published research papers because they describe their designs openly. Commercial apps may work differently, but the building blocks below appear across the papers cited here. If you want the practical side first, our beginner's guide to making a song with AI covers the steps in an app.
The short version
An AI music generator does four jobs:
- It turns your text prompt into numbers a model can work with.
- It works with music in a compressed form instead of raw sound.
- It generates new compressed music that fits your prompt, using one of two main methods: predicting the next piece of music one step at a time, or starting from noise and cleaning it up.
- It turns the compressed music back into sound you can play.
The rest of this guide explains each step.
Why raw audio is hard
Digital audio is a long list of numbers. Stable Audio Open, an open model described in a paper from Stability AI, outputs stereo audio at 44.1 kHz. That means 44,100 numbers per second for each of the two channels, so a one-minute stereo clip is more than 5 million numbers.
The MusicGen paper from Meta explains why this is harder than speech: "the standard sampling rates of music recordings are 44.1 kHz or 48 kHz vs. 16 kHz for speech," and "Generating music is a challenging task as it requires modeling long range sequences." OpenAI's Jukebox paper calls it "the long context of raw audio." The papers below handle it with compression.
Step 1: Compress the music
Before a model learns to make music, researchers train a separate network that squeezes audio into a much shorter form and can expand it back again. You can think of it as a smart zip file for sound.
Three examples from the papers:
- Jukebox uses "a multi-scale VQ-VAE to compress it to discrete codes." In plain terms, the audio becomes a sequence of symbols from a fixed vocabulary, a bit like words.
- Meta's EnCodec is a neural audio codec with "a streaming encoder-decoder architecture with quantized latent space." The encoder shrinks the audio; the decoder rebuilds it.
- Stable Audio Open uses "an autoencoder (156M parameters) that compresses waveforms into a manageable sequence length."
In this guide, compressed discrete symbols are called tokens, as in the MusicGen paper, and compressed continuous values are called latents, as in the Stable Audio Open paper's "latent space." Either way, the generator works in this smaller space, and the decoder turns its output back into audio at the end.
Step 2: Understand the prompt
The model also needs a way to read your words. This is a text encoder: a model that turns a sentence into a list of numbers that captures its meaning.
Stable Audio Open uses "a T5-based text embedding" for this. The MusicGen paper notes that earlier work also used "a pretrained text encoder, specifically T5." These numbers are passed to the generator so it can steer the music toward your description.
Language models can help in another way too. Google's Noise2Music paper says pretrained large language models "are used to generate paired text for the audio of the training set." In other words, a language model wrote descriptions for training clips, so the music model had more text and audio pairs to learn from.
Step 3: Generate new music
This is the core of the system. Research uses two main families of methods.
Method A: Predict the next token
This method works like a chatbot that writes one word at a time, except the "words" are music tokens. The model looks at the tokens so far, plus your prompt, and predicts what comes next. Repeat that step and you build up a sequence of tokens that the decoder can turn into audio.
- Jukebox models its compressed codes "using autoregressive Transformers." Autoregressive means each new piece depends on the pieces before it.
- MusicLM "casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task," which means it generates in stages, from a rough outline of the music to finer detail.
- MusicGen is "a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens." Its authors say its design "eliminates the need for cascading several models."
Method B: Start from noise and clean it up
This method is called diffusion. The Noise2Music paper describes diffusion models as models that "generate a sample by iteratively denoising random noise." During training, the model is "trained to identify the noise vector" in a noisy input. To make something new, inference is "carried out by taking random noise" and denoising it step by step, guided by your prompt.
- Noise2Music trains "a series of diffusion models" to make 30-second music clips from text. One model makes a rough version of the music, and a second "cascader" model adds high-fidelity detail.
- Stable Audio Open is a "latent diffusion model." It runs the noise-removal process in the compressed space from Step 1, not on raw audio, using "a transformer-based diffusion model" that "operates in the latent space of the autoencoder."
Step 4: Turn it back into sound
The generator's output is still compressed. The decoder from Step 1 expands it back into a waveform. The EnCodec paper was evaluated on speech, "noisy-reverberant speech, and music," at both "24 kHz monophonic and 48 kHz stereophonic audio."
More than text: melodies, lyrics and styles
Text prompts are one way to steer these systems. The papers describe others.
- Melody. MusicLM "can transform whistled and hummed melodies according to the style described in a text caption." MusicGen can be "conditioned on textual description or melodic features."
- Lyrics and style. Jukebox can "condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable."
- Prompt details. Noise2Music reports that its outputs reflect "key elements of the text prompt such as genre, tempo, instruments, mood, and era."
So it can help to name the genre, tempo, instruments and mood in your prompt. Our guide to Suno prompts shows how that looks in practice.
Where the training music comes from
A music model learns from examples, so its training data shapes what it can make. The papers describe different choices:
- MusicGen: "We use 20K hours of licensed music to train MusicGen," drawn from "an internal dataset of 10K high-quality music tracks" and the Shutterstock and Pond5 music collections.
- Stable Audio Open: "Our dataset consists of CC recordings from Freesound and the Free Music Archive (FMA)," and the authors say they "conducted an analysis to ensure no copyrighted content was in our training data."
- MusicLM: the team released MusicCaps, "a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts," to support research.
How long a generated track can be
Length is one of the harder parts, and the papers report different limits for their own systems:
- Noise2Music generates "30-second music clips."
- Stable Audio Open generates "variable-length (up to 47s) stereo audio."
- MusicLM generates music "that remains consistent over several minutes."
- Jukebox reports "coherence up to multiple minutes."
Longer output means a longer sequence for the model to keep consistent, which is the long-range problem the MusicGen and Jukebox papers describe.
Why the same prompt gives different songs
Generation involves randomness. Diffusion starts from random noise, and Noise2Music says "the level of stochasticity of the denoising process can be controlled" through a sampler setting. Next-token models sample each token from the model's predictions; MusicGen's paper describes using "classifier-free guidance when sampling from the model's logits." Different random starting points can lead to different results from the same prompt.
What this means when you use a music app
You do not need to know any of this to make a song, but it helps explain what you see:
- Specific prompts can help because the text encoder turns your words into the signal that steers the music.
- Results vary because sampling and random noise are part of generation.
- A hummed tune or melody can be an input in systems trained for it, such as MusicLM and MusicGen.
- Length limits differ by system, as the papers above show.
For a wider view of how these systems relate to chatbots and image tools, see our explainer on what generative AI is.
Sources
- MusicLM: Generating Music From Text (Google, arXiv)
- Simple and Controllable Music Generation, the MusicGen paper (Meta, arXiv)
- AudioCraft code and models (Meta, GitHub)
- High Fidelity Neural Audio Compression, the EnCodec paper (Meta, arXiv)
- Stable Audio Open (Stability AI, arXiv)
- Jukebox: A Generative Model for Music (OpenAI, arXiv)
- Noise2Music: Text-conditioned Music Generation with Diffusion Models (Google, arXiv)
