Our overall setup is divided into three parts, Kaldi training, PyTorch-Kaldi training, and FedAvg.
First, the features of all relevant datasets are extracted, the corresponding GMM models are trained, and alignments are obtained in Kaldi [1]}, using MFCCs with first and second derivatives and cepstral mean and variance normalization on a per-speaker basis.
In the second part, we use PyTorch-Kaldi to train a hybrid DNN-WFST model with the PyTorch library. We use the standard MLP structure that is provided with many of the examples in PyTorch-Kaldi, a feed-forward network with a context size of 5 frames [2]}. This results in the initial model \(\mathbf {W}^0\) , which is only learned on the initial dataset.
After the server-side training is thus completed, federated learning is performed as described in Algorithm REF . Here, client-side trainings only re-train the DNN of the hybrid model, leaving the WFST unchanged.
