The MiniMax clones sound less fluent because a voice clone reproduces the speaker's accent as faithfully as their timbre. Re-cloning makes that worse, not better. We fixed it with voice conversion: keep a native speaker's delivery, swap in Simon's or Lucia's voice. Below — listen, then look at the measurements.
A voice clone is trained to reproduce how the reference speaker actually sounds — including their non-native English pronunciation and rhythm. That is the point of cloning, and it is exactly why re-cloning from more or better recordings cannot fix the accent: the clone would learn the accent again, more accurately.
| Property | Should come from | Currently comes from |
|---|---|---|
| 音色 — who it sounds like | Simon / Lucia | Simon / Lucia ok |
| Accent, rhythm, stress — how the words are said | a native English speaker | Simon / Lucia the bug |
Take a native-English reading of the script and replace only the speaker identity with Simon's or Lucia's.
That is voice conversion (VC) — a different technology from text-to-speech cloning. The words and the delivery come from the source recording; the voice comes from the reference.
Both halves of each pair say the same words. Loudness is normalised to −16 LUFS in both, so one is not winning by simply being louder. Press Before, then After.
36 s · four labelled segments: Simon before, Simon after, Lucia before, Lucia after. Encoded H.264/AAC, 1280×720.
We cannot hear the audio, so every claim here is a number. sim_ref = cosine similarity to the speaker's own reference (higher = 音色 preserved). sim_narr = similarity to the native narrator (lower = the narrator's identity is gone). f0 = median pitch.
| Clip | sim_ref | sim_narr | f0 | dur |
|---|---|---|---|---|
| MiniMax clone before | 0.947 | 0.469 | 120.0 | 9.50 |
| Seed-VC after | 0.931 | 0.515 | 114.6 | 8.50 |
| native narrator (source) | 0.512 | 1.000 | 121.4 | 8.50 |
Output duration 8.50 s = the narrator's 8.50 s, not the clone's 9.50 s — the native rhythm came across.
| Clip | sim_ref | sim_narr | f0 | dur |
|---|---|---|---|---|
| MiniMax clone before | 0.945 | 0.413 | 271.0 | 8.24 |
| Seed-VC after | 0.934 | 0.522 | 267.8 | 8.95 |
| female native source (Flo) | 0.399 | 0.592 | 242.8 | 8.59 |
Pitch lands at 267.8 Hz against Lucia's own 271.0 Hz — the model re-pitched the source into her range.
Conversions land at 0.92–0.95 similarity to the owner while the native source sits at 0.40–0.51. The output is Simon/Lucia, not the narrator.
Pitch moves into the owner's range; durations match the source to the centisecond, and DTW-aligned pitch contours correlate 0.95–0.97 with the source they came from.
The accented clone and the native reading both transcribe at WER 0.000 / 0.042 — Whisper is too robust on clean TTS. The accent verdict is yours to make by ear.
say -v Flo for a fully offline path. This carries the accent we want.--source and a 10–30 s reference of the person as --target.Reviewed from the GitHub survey. Three families exist, and only one fixes an accent:
| Family | Fixes accent? |
|---|---|
| Text→speech zero-shot cloning (F5-TTS, IndexTTS, Chatterbox, Fish-Speech) | no — clones the accent too |
| Voice conversion (Seed-VC, RVC) | yes — keeps source delivery |
| Tone-colour conversion (OpenVoice) | yes |
Seed-VC was picked because it ships accent conversion as a first-class feature and documents Apple Silicon support — it runs here on the M2's GPU with no CUDA and no cloud.
| Mode | Result |
|---|---|
| v1 · timbre only | chosen sim_ref 0.93 / 0.93 |
v2 · --convert-style false | 0.95 (Simon) / 0.82 (Lucia) |
v2 · --convert-style true (accent model) | worse on Lucia — WER 0.042 → 0.083 |
When the source is already native there is nothing for the accent model to fix, so v1 is the right default.
cd projects/Scenario/voice-cloning uv venv --python 3.11 .venv # torch + MPS, no CUDA needed .venv/bin/python prep_refs.py # clean 10–30 s references per speaker .venv/bin/python make_sources.py # native read of each line (MiniMax / say) ./run_v1_all.sh # native source -> speaker timbre .venv/bin/python score.py # WER + speaker similarity + pitch .venv/bin/python prosody_match.py # did the source's prosody survive? .venv/bin/python make_ab_video.py # the labelled before/after video
| # | Decision | Our recommendation |
|---|---|---|
| 1 | Does the After delivery sound native enough to use? | Listen in section 2 and say yes / no. If no, we escalate the source (a real actor read), not the model. |
| 2 | Whose voice narrates the competition entry? | Use a consented voice. Simon's and Lucia's own voices are fine for internal samples; a student-facing platform should not ship recognisable third-party clones. |
| 3 | Native source: cloud narrator or offline? | Start with MiniMax's system narrator; keep say -v Flo as the offline fallback so the pipeline never depends on a clone staying alive. |
| 4 | Do we re-render the existing scenario samples with these voices? | Yes — it is a drop-in swap of the TTS step. ~12 min per video for the VRM avatars. |
| 5 | Ship voice quality as part of the helper platform? | Yes — "your 音色, native delivery" is a strong selling point for students joining the competition. |
Full technical write-up and raw data: projects/Scenario/voice-cloning/voice-cloning.md and scores.json.