CROSS-SPEAKER EMOTION TRANSFER BASED ON PROSODY COMPENSATION FOR END-TO-END SPEECH SYNTHESIS

0. Contents

1. Abstract
2. Demos -- Emotional speech synthesis by transferring the emotion from reference audio
3. Demos -- The necessity of emotion embedding and prosody compensation
4. Demos -- The effectiveness of GC block
5. Additional demos -- Transfer emotion to a new target speaker who only has 200 sentences through simple fine-tuning

1. Abstract

Cross-speaker emotion transfer speech synthesis aims to synthesize emotional speech for a target speaker by transferring the emotion from reference recorded by another (source) speaker, in which task extracting speaker-independent emotion embedding from reference plays an important role. However, the emotional information of such emotion embedding tends to be weakened in the process to squeeze out the source speaker's timbre information. In this paper, a prosody compensation module (PCM) is proposed to compensate emotional information for the disentangled emotion embedding. Specifically, the PCM tries to obtain speaker-independent emotional information from the intermediate feature of a pre-trained ASR model. To this end, a prosody compensation encoder with global context (GC) blocks is introduced to obtain global emotional information from the ASR model's intermediate feature. Experiments demonstrate that the proposed PCM can effectively compensate emotional information to the emotion embedding, and meanwhile maintain the timbre of the target speaker. Comparisons with state-of-the-art models show that our proposed method presents obvious superiority on the cross-speaker emotion transfer task.

1.1 The structure of the proposed model:

1.2 The structure of global context (GC) blocks:

2. Demos -- Comparison with other methods

Corresponding to section 4.1 in the paper, several samples synthesized by the proposed CSPC and other compared methods on the emotion transfer task are listed below.

Short summary: It can be found that compared with Multi-R, CSET, and PB, the proposed CSPC can achieve a good balance between maintaining the target speaker's timbre and enriching the transferred emotional expression.

3. Demos -- The necessity of emotion embedding and prosody compensation

Corresponding to section 4.2.1 in the paper, samples synthesized by CSPC and also CSPC' variants that without emotion embedding (w/o EE) or prosody compensation embedding (w/o PCE) on the emotion transfer task are listed below.

Short summary: It can be found that the prosody compensation embedding can provide extra emotion information to the emotion embedding, and the proposed prosody compensation method can effectively improve the emotion transfer performance and mean-while maintain the target speaker’s voice.

4. Demos -- The effectiveness of GC block

As we can see that in terms of the emotion similarity, CSPC obviously outperforms the w/o GC without the GC block.

As we can see that in terms of the speaker similarity, dropping the GC block brings a slight increase which is not obvious.

Corresponding to section 4.2.2 in the paper, below lists the samples that are synthesized for presenting the effectiveness of GC block.

Short summary: It can be found that in terms of emotional expressiveness, CSPC significantly outperforms the variant w/o GC without significant influence on speaker similarity.

5. Additional demos -- Transfer emotion to a new target speaker who only has 200 sentences through simple fine-tuning

Following samples are synthesized by transferring the emotion to a target speaker who has only 200 utterances.

Short summary: It can be found that the model we proposed can be well generalized to the new target speaker who only has a small amount of data through simple fine-tuning, which transfers the emotion to the new target speaker while maintaining the target speaker’s timbre.