Gray Learning from Non-IID Data with Out-of-distribution Samples
The quality of the training data annotated by experts cannot be guaranteed,
even more so for non-IID data consisting of both in- and out-of-distribution
samples (i.e., in-distribution and out-of-distribution samples hold different
distributions). Experts may mistakenly annotate out-of-distribution samples the
same as in-distribution samples, incurring untrustworthy ground-truth labels.
Learning such non-IID data mixing in- and out-of-distribution samples with
untrustworthy labels significantly challenges both shallow and deep learning,
with no relevant work reported. It would be possible to identify trustworthy
complementary labels of a sample indicating which classes it does not belong
to, because both in- and out-of-distribution samples do not belong to the
classes except those corresponding to the ground-truth label. With this
insight, we propose a novel \textit{gray learning} approach to robustly learn
from non-IID data with both in- and out-of-distribution samples. Due to the
uncertain distributions of training samples, we reject the complementary labels
for low-confidence inputs while mapping high-confidence inputs to the
ground-truth labels in training. Building on the statistical learning theory,
we derive the generalization error which shows that gray learning achieves a
tight bound on the non-IID data. Extensive experiments show that our method
provides significant improvement over alternative methods from robust
statistics.