AI voice cloning is the process of building a synthetic copy of a specific person's voice from recordings, so that a text-to-speech system can then say new things in that voice. You record or upload some audio, the service builds a voice from it, and from then on you type text and hear it spoken in the cloned voice.
This explainer covers how cloning works under the hood, the two main approaches vendors offer, what makes a good recording, where clones go wrong, and the consent checks the vendors themselves describe. It uses ElevenLabs and Microsoft as examples, because both publish detailed documentation on how their cloning features work.
Voice cloning in one sentence
A clone is a voice profile that plugs into text-to-speech. ElevenLabs' voices overview describes the two options this way: "Clone your own voice from short audio samples with Instant Voice Cloning, or create realistic voices using Professional Voice Cloning." Microsoft's personal voice overview describes a feature that lets businesses offer their users "AI generated replication of their own voices in a few seconds."
If you are new to text-to-speech itself, the short version is that a TTS model turns text into sound in a chosen voice. Cloning changes which voice that is.
The two ways a clone gets built
ElevenLabs and Microsoft each offer a fast option that needs little audio and a slower option that uses much more audio. The difference is whether a new model is trained for your voice.
Fast cloning from a short sample
ElevenLabs' voice cloning guide says Instant Voice Cloning lets you "create voice clones from shorter samples near instantaneously." The key detail is how it does this: "Creating an Instant Voice Clone (IVC) does not train or create a custom AI model. Instead, it relies on prior knowledge from training data to make an educated guess rather than training on the exact voice."
Put simply, the model does not learn your voice from scratch. It uses what it learned from its training data to match your short sample.
Microsoft's personal voice works from a short sample too. Microsoft says it needs "a verbal statement and a short speech sample as the audio prompt," and its comparison table lists the required data as a human speech sample of 5 to 90 seconds, with a training time of "Less than 5 seconds."
The trade-off is accuracy on unusual voices. ElevenLabs says "the biggest limitation with IVCs is if you are trying to clone a very unique voice, or a voice with an accent that the AI might not have experienced extensively during training."
Slower cloning that trains a dedicated model
The second approach trains, or fine-tunes, a model on the speaker's voice. ElevenLabs says Professional Voice Cloning lets you "train a more realistic model of your voice by training a dedicated model on a larger set of voice data." Because it involves training, it takes longer. ElevenLabs says "Generally fine-tuning takes 3-6 hours to complete, but it can sometimes take a bit longer."
The audio requirements are much larger. ElevenLabs recommends 1 to 2 minutes of good audio for Instant Voice Cloning and 30 to 180 minutes of good audio for Professional Voice Cloning.
Microsoft's equivalent is professional voice. Its comparison table says professional voice is for "Professional scenarios like brand and character voices for chat bots, or audio content reading," with the note "Bring your own data. Recording in a professional studio is recommended."
What the AI copies from your recording
A clone copies more than the sound of a voice. ElevenLabs' Instant Voice Cloning guide says "The AI will attempt to mimic everything it hears in the audio. This includes the speed of the person talking, the inflections, the accent, tonality, breathing pattern and strength, as well as noise and mouth clicks."
That has a practical consequence: the clone sounds like the recording, not like the person on their best day. ElevenLabs says that if you "talk in a slow, monotone voice without much emotion, that is what the AI will mimic." Its broader recording guide adds that the AI may replicate "the length of your pauses, whether you stutter, take deep breaths, sound breathy, or use a lot of 'uhms' and 'ahs'."
How to record a good sample
Because the clone copies the recording, recording quality matters more than recording length. ElevenLabs says "How the audio was recorded is more important than the total length (total runtime) of the samples." Its tips, in plain terms:
- Use a quiet, dead room. ElevenLabs suggests minimizing echo and reverb, and if you do not have a vocal booth, trying a DIY "blanket fort," or a closet.
- Use a pop filter. It helps avoid plosives and air hitting the microphone directly, which ElevenLabs says "will also cause issues with the cloning process."
- Keep a steady distance. ElevenLabs suggests "about two fists away from the microphone, which is approximately 20cm (7-8 in)."
- Stay consistent. ElevenLabs says "Consistency is key to a proper clone." Keep the same energy and the same accent throughout. If a recording is very animated in one part and subdued in another, "the AI can become unstable because it doesn't know what part of the voice to clone."
- Do not overdo the length for a quick clone. For Instant Voice Cloning, ElevenLabs says to record at least 1 minute and to avoid more than 3 minutes, which "will yield little improvement and can, in some cases, even be detrimental to the clone."
- Skip the extra processing. ElevenLabs says you do not need "fancy post-processing, plugins, denoisers, or anything," and that uncompressed formats such as WAV "usually do not improve clone quality."
- Record one speaker only. ElevenLabs advises a clean recording "with no background noise, room reverb, or multiple speakers."
Consent and verification, as the vendors describe them
Cloning a voice raises an obvious question: whose voice is it? The vendors build checks into the product. This section only reports what their own documentation says.
ElevenLabs, fast cloning. When you save an Instant Voice Clone, ElevenLabs says you "confirm that you have the right and consent to clone the voice." Its guide points users who are unsure what is permitted to its Terms of Service and AI Safety information.
ElevenLabs, professional cloning. ElevenLabs says "Voice-captcha technology is used to verify that Professional Voice Clones are created from your own voice samples." After you upload your recordings, "you will be asked to verify your voice," ideally with the same equipment and a similar delivery. Its FAQ is direct: "You can only create a Professional Voice Clone of your own voice." If someone wants to share their voice, ElevenLabs says they can create and verify a Professional Voice Clone on their own account and then share it privately.
Microsoft. Microsoft's comparison table lists the same responsible AI requirement for personal voice and professional voice: "Speaker's verbal statement required. No unapproved use case allowed." It says access to the personal voice API "is restricted to eligible customers and approved use cases." For professional voice, Microsoft's setup guide lists "Written permission from the voice talent and a recording of their consent statement" as a prerequisite.
The common pattern is that the person whose voice is cloned takes part in the process, either by recording a statement, passing a verification step, or confirming consent.
Self-hosted open models can work differently. The model card for Audio8-TTS-Realistic-Air asks users to obtain permission before cloning a voice, and it does not mention a consent check built into the model.
Cloning versus other ways to get a custom voice
Cloning is not the only way to get a voice that is not one of the stock options. ElevenLabs lists several voice types side by side:
- Cloned voices, made with Instant or Professional Voice Cloning.
- Voice Design, which creates "entirely new voices by specifying attributes like age, gender, accent, and tone." ElevenLabs suggests it for "Creative character voices for games and storytelling."
- Voice Remixing, which lets you "transform voices you own by modifying their core attributes while maintaining their recognizable characteristics."
- Community voices from its Voice Library, which holds voices shared by the ElevenLabs community.
Voice cloning also shows up in music tools. Suno's help center describes a Voices feature that "lets you add your own voice to Suno-generated songs." For more on Suno's voice features, see our Suno Speech explainer.
Where clones go wrong
- Unusual voices and accents. As ElevenLabs notes, a quick clone can struggle with a very distinctive voice or an accent that was rare in the training data. For those voices, ElevenLabs says Professional Voice Cloning "might be the best option."
- Noisy or inconsistent recordings. The clone copies noise, room echo and changes in energy along with the voice.
- Too much variety in the sample. ElevenLabs says too much variance "might confuse the AI, leading to more varied output between generations."
- Delivery that does not match the script. A clone made from a calm reading will sound calm. Record in the style you want to hear back.
Where cloned voices are used
Microsoft describes personal voice as a way for businesses "to build an app to allow their users to create and use their own personal voice in the app." Its professional voice targets brand voices, character voices for chatbots and "audio content reading." ElevenLabs' text to speech overview lists uses such as narrating ads and producing audiobooks.
Cloned voices can also power voice agents. In those systems, the voice is the last step of a longer pipeline, a design we explain in speech-to-speech vs cascaded voice agents.
Sources
- ElevenLabs Docs: Voices
- ElevenLabs Docs: Voice Cloning
- ElevenLabs Docs: Instant Voice Cloning
- ElevenLabs Docs: Professional Voice Cloning
- ElevenLabs Docs: Text to Speech
- Microsoft Learn: What is personal voice for text to speech?
- Microsoft Learn: Set up a professional voice
- Suno Help: Making Music (article index, including Voices)
