Unsupervised learning approaches work by defining a pretext task.
We can largely separate the approaches into two categories, contrastive and reconstruction-based.
The fundamental difference between the two approaches is that the contrastive approaches require negative samples.
The model generates a candidate output and compares it with the true target and a number of negative samples.
The objective function is based on the candidate's similarity with the target and distance from the negative samples.
In [1]}, future states are predicted from the current state.
The more recent wav2vec 2.0 approach predicts a state given surrounding context [2]}, similar to the way BERT [3]} is trained for NLP tasks.
