EnergyMatch: Energy-based Pseudo-Labeling for Semi-Supervised Learning
Recent state-of-the-art methods in semi-supervised learning (SSL) combine
consistency regularization with confidence-based pseudo-labeling. To obtain
high-quality pseudo-labels, a high confidence threshold is typically adopted.
However, it has been shown that softmax-based confidence scores in deep
networks can be arbitrarily high for samples far from the training data, and
thus, the pseudo-labels for even high-confidence unlabeled samples may still be
unreliable. In this work, we present a new perspective of pseudo-labeling:
instead of relying on model confidence, we instead measure whether an unlabeled
sample is likely to be "in-distribution"; i.e., close to the current training
data. To classify whether an unlabeled sample is "in-distribution" or
"out-of-distribution", we adopt the energy score from out-of-distribution
detection literature. As training progresses and more unlabeled samples become
in-distribution and contribute to training, the combined labeled and
pseudo-labeled data can better approximate the true distribution to improve the
model. Experiments demonstrate that our energy-based pseudo-labeling method,
albeit conceptually simple, significantly outperforms confidence-based methods
on imbalanced SSL benchmarks, and achieves competitive performance on
class-balanced data. For example, it produces a 4-6% absolute accuracy
improvement on CIFAR10-LT when the imbalance ratio is higher than 50. When
combined with state-of-the-art long-tailed SSL methods, further improvements
are attained.