Yuanzhe Chen, Ming Tu, Tang Li, Xin Li, Qiuqiang Kong, Jiaxin Li, Zhichao Wang, Qiao Tian, Yuping Wang, Yuxuan Wang
Speech, Audio & Music Intelligence (SAMI), ByteDance Inc.
Abstract
Streaming voice conversion (VC) is the task of converting the voice of one person to another in real-time. Previous streaming VC methods use phonetic posteriorgrams (PPGs) extracted from automatic speech recognition (ASR) systems to represent speaker-independent information.
However, PPGs lack the prosody and vocalization information of the source speaker, and streaming PPGs contain undesired leaked timbre of the source speaker. In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs. VC systems trained with IBFs retain more prosody and vocalization information of the source speaker. Furthermore, we propose a non-streaming teacher guidance (TG) framework that addresses the timbre leakage problem. Experiments show that our proposed IBFs and the TG framework achieve a state-of-the-art streaming VC naturalness of 3.85, a content consistency of 3.77, and a timbre similarity of 3.77 under a future receptive field of 160 ms which significantly outperform previous streaming VC systems.
Overview
Audio Samples
In this demo, we present streaming voice conversion results using the following models:
Baseline: Baseline method, PPGs-based streaming voice conversion based on reconstruction training. (PPGs+Rec)
Proposed: Our proposed method, IBFs-based streaming voice conversion based on teacher guidance training. (IBFs+TG)
Proposed w/o IBFs: PPGs-based streaming voice conversion based on teacher guidance training. (PPGs+TG)
Proposed w/o TG :IBFs-based streaming voice conversion based on reconstruction training. (IBFs+Rec)
For fair comparison, all systems use the same streaming ASR (sub-)encoder and same streaming vocoder, chunk_size=160ms. To emphasize the robustness of our proposed method, we demonstrate some difficult cases, including slurred and exaggeration pronunciation, or with laughter, breathing, noise, etc.
