Replacing Labeled Real-image Datasets with Auto-generated Contours
In the present work, we show that the performance of formula-driven
supervised learning (FDSL) can match or even exceed that of ImageNet-21k
without the use of real images, human-, and self-supervision during the
pre-training of Vision Transformers (ViTs). For example, ViT-Base pre-trained
on ImageNet-21k shows 81.8% top-1 accuracy when fine-tuned on ImageNet-1k and
FDSL shows 82.7% top-1 accuracy when pre-trained under the same conditions
(number of images, hyperparameters, and number of epochs). Images generated by
formulas avoid the privacy/copyright issues, labeling cost and errors, and
biases that real images suffer from, and thus have tremendous potential for
pre-training general models. To understand the performance of the synthetic
images, we tested two hypotheses, namely (i) object contours are what matter in
FDSL datasets and (ii) increased number of parameters to create labels affects
performance improvement in FDSL pre-training. To test the former hypothesis, we
constructed a dataset that consisted of simple object contour combinations. We
found that this dataset can match the performance of fractals. For the latter
hypothesis, we found that increasing the difficulty of the pre-training task
generally leads to better fine-tuning accuracy.