The training process is performed as follows. The first step is comprised of data generation and pre-processing of the training data. The raw audio is split into short parts of a few seconds, each analyzed on the specified features, i.e., the fundamental frequency and loudness, and finally saved in the TensorFlow TFRecord format. The fundamental frequency is thereby estimated by using the state-of-the-art pitch tracking technique, called CREPE by Kim et al. [1]} that applies a deep convolutional neural network on time-domain audio. The second step is the actual training, using a Python based configuration framework for dependency injection by Google, called Ginhttps://github.com/google/gin-config, last accessed on 2020-12-15. In this way, all available training hyperparameters can be defined in a gin config file that is passed to the training function. The training process does not include any optimization techniques, such as a hyperparameter search or early stopping, the authors just recommend in the code documentation to train for 5,000 to 30,000 steps until a spectral loss of about 4.5-5 is reached for an optimal learning representation without overfitting. The third and last step is a short evaluation based on resynthesis. That means, a training sample is randomly picked, passed to the autoencoder that encodes and decodes, i.e., reconstructs the input sample based on the learned features.
