Symbolic image detection using scene and knowledge graphs
Sometimes the meaning conveyed by images goes beyond the list of objects they
contain; instead, images may express a powerful message to affect the viewers'
minds. Inferring this message requires reasoning about the relationships
between the objects, and general common-sense knowledge about the components.
In this paper, we use a scene graph, a graph representation of an image, to
capture visual components. In addition, we generate a knowledge graph using
facts extracted from ConceptNet to reason about objects and attributes. To
detect the symbols, we propose a neural network framework named SKG-Sym. The
framework first generates the representations of the scene graph of the image
and its knowledge graph using Graph Convolution Network. The framework then
fuses the representations and uses an MLP to classify them. We extend the
network further to use an attention mechanism which learn the importance of the
graph representations. We evaluate our methods on a dataset of advertisements,
and compare it with baseline symbolism classification methods (ResNet and VGG).
Results show that our methods outperform ResNet in terms of F-score and the
attention-based mechanism is competitive with VGG while it has much lower model
complexity.