Our reference encoder architecture, which takes inspiration from [1]}, [2]}, consumes an utterance of frame-rate vocoder features and outputs a single timestep vector summary. It consists of 3 sequential one-dimensional convolutional layers that downsample the vocoder features in time; each layer is followed by batch normalisation and ReLU activation and has a kernel size of 3, stride of 2, padding of 1, and a hidden dimension of 128. The third convolutional layer is followed by a single layer bi-directional gated recurrent unit (GRU) which generates a 64-dimensional summary of its inputs in both the forward and backward directions by taking the final hidden output of both recurrent units. These forward and backward summaries are then concatenated together and passed through a final output linear layer to form a 32-dimensional reference encoding.
