Consistency Regularization can be considered as a way of using unlabeled data to explore a smooth manifold on which all of the data points are embedded [1]}. This simple criterion has provided a set of methods that are currently considered as state of the art for the SSL challenge. Some of these methods are stochastic perturbations [2]}, \(\pi \) -model [3]}, mean teacher [4]}, and Virtual Adversarial Training (VAT) [5]}. The original idea behind stochastic perturbations and \(\pi \) -model was first introduced in [6]} and has been referred to as pseudo-ensembles. The pseudo-ensembles regularization techniques are usually designed such that the prediction of the model ideally should not change significantly if the data given to the model is perturbed; in other words, under realistic
perturbations of a data point \(x\)  (\(x \rightarrow x^{\prime }\) ), output of the model \(f_{\theta }(x)\)  should not change significantly. This goal is achieved by adding
a weighted loss term such as \(d(f_{\theta }(x), f_{\theta }(x^{\prime }))\)  to the total loss of the model \(f_{\theta }(x)\) , where \(d(.,.)\)  is mean squared error or Kullback-Leibler divergence which measures a distance between outputs of the prediction function. The main problem of pseudo-ensemble methods, including \(\pi \) -model is that they rely on a potentially unstable target prediction, which can immediately change during the training.
