Using class labels to represent class similarity is a typical approach to training deep hashing systems for retrieval; samples from the same or different classes take binary 1 or 0 similarity values. This similarity does not model the full rich knowledge of semantic relations that may be present between data points. In this work we build upon the idea of using semantic hierarchies to form distance metrics between all available sample labels; for example cat to dog has a smaller distance than cat to guitar. We combine this type of semantic distance into a loss function to promote similar distances between the deep neural network embeddings. We also introduce an empirical Kullback-Leibler divergence loss term to promote binarization and uniformity of the embeddings. We test the resulting SHREWD method and demonstrate improvements in hierarchical retrieval scores using compact, binary hash codes instead of real valued ones, and show that in a weakly supervised hashing setting we are able to learn competitively without explicitly relying on class labels, but instead on similarities between labels. Content-Based Image Retrieval (CBIR) on very large datasets typically relies on hashing for efficient approximate nearest neighbor search; see e.g. BID12 for a review. Early methods such as (LSH) BID5 were data-independent, but Data-dependent methods (either supervised or unsupervised) have shown better performance. Recently, Deep hashing methods using CNNs have had much success over traditional methods, see e.g. Hashnet BID1 , DADH . Most supervised hashing techniques rely on a pairwise binary similarity matrix S = {s ij }, whereby s ij = 1 for images i and j taken from the same class, and 0 otherwise.A richer set of affinity is possible using semantic relations, for example in the form of class hierarchies. BID13 consider the semantic hierarchy for non-deep hashing, minimizing inner product distance of hash codes from the distance in the semantic hierarchy. In the SHDH method , the pairwise similarity matrix is defined from such a hierarchy according to a weighted sum of weighted Hamming distances.In Unsupervised Semantic Deep Hashing (USDH, Jin (2018)), semantic relations are obtained by looking at embeddings on a pre-trained VGG model on Imagenet. The goal of the semantic loss here is simply to minimize the distance between binarized hash codes and their pre-trained embeddings, i.e. neighbors in hashing space are neighbors in pre-trained feature space. This is somewhat similar to our notion of semantic similarity except for using a pre-trained embedding instead of a pre-labeled semantic hierarchy of relations. BID14 consider class-wise Deep hashing, in which a clustering-like operation is used to form a loss between samples both from the same class and different levels from the hierarchy.Recently BID0 explored image retrieval using semantic hierarchies to design an embedding space, in a two step process. Firstly they directly find embedding vectors of the class labels on a unit hypersphere, using a linear algebra based approach, such that the distances of these embeddings are similar to the supplied hierarchical similarity. In the second stage, they train a standard CNN encoder model to regress images towards these embedding vectors. They do not consider hashing in their work. We approached Deep Hashing for retrieval, introducing novel combined loss functions that balance code binarization with equivalent distance matching from hierarchical semantic relations. We have demonstrated new state of the art results for semantic hierarchy based image retrieval (mAHP scores) on CIFAR and ImageNet with both our fully supervised (SHRED) and weakly-supervised (SHREWD) methods.
We propose a new method for training deep hashing for image retrieval using only a relational distance metric between samples
{"src": "scitldr-AIC_train", "paper_id": "rJgqFi5rOV"}