We compare our model to the supervised method and the existing zero-shot audio classification method (see Table REF ). First, we compare audio embeddings trained by the supervised methods such as logistic regression and ResNet50 [1]} supervised by random initialization of weights as a baseline model. Our audio encoder shows better classification performance than the baseline. Secondly, we compare the zero-shot audio classification accuracy with Wav2CLIP [2]} and AudioCLIP [3]}. Our proposed loss function learns three modalities in the CLIP embedding space and obtains a richer audio representation through contrastive loss. However, Wav2CLIP only understands the relationship between the audio and the visual context. Furthermore, Wav2CLIP learns from an audio-visual having a dataset with a relatively small data scale to distillate the CLIP representation to the audio encoder (the VGG-Sound dataset: 200K). AudioCLIP uses a dataset scaled up from contrastive learning between modalities from the audio, image, and text given as a triplet to obtain a more powerful sound representation.
The AudioCLIP's audio encoder was pre-trained on a large-scale audio dataset, namely AudioSet [4]}, which consists of over 2M sound clips.
