Zero-Shot Audio Classification using Image Embeddings
Supervised learning methods can solve the given problem in the presence of a
large set of labeled data. However, the acquisition of a dataset covering all
the target classes typically requires manual labeling which is expensive and
time-consuming. Zero-shot learning models are capable of classifying the unseen
concepts by utilizing their semantic information. The present study introduces
image embeddings as side information on zero-shot audio classification by using
a nonlinear acoustic-semantic projection. We extract the semantic image
representations from the Open Images dataset and evaluate the performance of
the models on an audio subset of AudioSet using semantic information in
different domains; image, audio, and textual. We demonstrate that the image
embeddings can be used as semantic information to perform zero-shot audio
classification. The experimental results show that the image and textual
embeddings display similar performance both individually and together. We
additionally calculate the semantic acoustic embeddings from the test samples
to provide an upper limit to the performance. The results show that the
classification performance is highly sensitive to the semantic relation between
test and training classes and textual and image embeddings can reach up to the
semantic acoustic embeddings when the seen and unseen classes are semantically
similar.