Secondly, we also concatenate the output of a reference encoder [1]}, which is used to summarise utterance level characteristics from the original unedited audio. Previously these have been adopted primarily for prosody transfer. In this work we use them to capture utterance level prosodic or speaker variations that are not captured by the pretrained speaker verification system. We also include a variational version of the reference encoder [2]}, [3]} as it may potentially capture a better disentangled representation that exhibits finer control at inference time.
