The structure of the network in the HuBERT approach [1]} is similar to wav2vec 2.0 [2]}.
Both use the same convolutional waveform encoder followed by a large transformer network, however, HuBERT does not use a quantization component.
The first step in the training process requires a k-means clustering of the features to generate unsupervised targets.
The initial features are MFCC features, but they are used to bootstrap the initial model and later iterations use an internal representation from the transformer network.
Given the clusters as targets, the model is trained using frame-level cross-entropy.
At least for MFCC features, learning the cluster targets would be trivial for a sophisticated model.
To combat a degenerative solution, a percentage of inputs to the transformer network are masked and only those updates related to those masked frames are used; instead of directly predicting the target from the input, the model is forced to predict the target based on context.
