We explore the idea of compositional set embeddings that can be used to infer not
 just a single class, but the set of classes associated with the input data (e.g., image,
 video, audio signal). This can be useful, for example, in multi-object detection in
 images, or multi-speaker diarization (one-shot learning) in audio. In particular, we
 devise and implement two novel models consisting of (1) an embedding function
 f trained jointly with a “composite” function g that computes set union opera-
 tions between the classes encoded in two embedding vectors; and (2) embedding
 f trained jointly with a “query” function h that computes whether the classes en-
 coded in one embedding subsume the classes encoded in another embedding. In
 contrast to prior work, these models must both perceive the classes associated
 with the input examples, and also encode the relationships between different class
 label sets. In experiments conducted on simulated data, OmniGlot, and COCO
 datasets, the proposed composite embedding models outperform baselines based
 on traditional embedding approaches. Embeddings, especially as enabled by advances in deep learning, have found widespread use in natural language processing, object recognition, face identification and verification, speaker verification and diarization (i.e., who is speaking when (Sell et al., 2018) ), and other areas. What embedding functions have in common is that they map their input into a fixed-length distributed representation (i.e., continuous space) that facilitates more efficient and accurate (Scott et al., 2018) downstream analysis than simplistic representations such as one-of-k. Moreover, they are amenable to one-shot and few-shot learning since the set of classes that can be represented does not depend directly on the dimensionality of the embedding space. Previous research on embeddings has focused on cases where each example is associated with just one class (e.g., the image contains only one person's face). In contrast, we investigate the case where each example is associated with not just one, but an entire subset of classes from a universe S. The goal is to embed each example so that questions of two types can be answered (see Figure 1 (a)): (1) Is the set of classes in example x a equal to the union of the classes in examples x b and x c ? (2) Does the set of classes in example x a subsume the set of classes in example x b ? Importantly, we focus on settings in which the classes present in the example must be perceived automatically. We approach this problem using compositional set embeddings. Like traditional embeddings, we train a function f that maps each example x ∈ R n into an embedding space R m so that examples with the same classes are mapped close together and examples with different classes are mapped far apart. Unlike traditional embeddings, our function f is trained to represent the set of classes that is associated with each example, so that questions about set union and subsumption can be answered by comparing vectors in the embedding space. We do not assume that the mechanism by which examples (e.g., images, audio signals) are rendered from multiple classes is known. Rather, the rendering process must be learned from training data. We propose two models, whereby f is trained jointly with either a "composition" function g (Model I) that answers questions about set union, or a "query" function h (Model II) that answers question about subsumption (see Figure 1(a ) ). Figure 1: (a): Overview of the paper: embedding function f is trained jointly with either the composition function g or the query function h. In particular , the goal is for g to "compose" the embeddings of two examples, containing classes T and U respectively, to approximate the embedding of an example containing classes T ∪ U. (b): 2-D projection of the embedding space from Experiment 1 on test classes and examples not seen during training (one-shot learning). Function g composes two embeddings (two arrow tails) and maps the result back into the embedding space (arrow head). To substantial if imperfect degree, the embedding space is compositional as described in (a). To our knowledge, this computational problem is novel. We see at least two use-cases: (1) Speaker recognition and diarization (i.e., infer who is talking within an audio signal) with multiple simultaneous speakers: Given an audio signal containing speakers who were not part of the training set and who may be speaking simultaneously, and given one example of each person speaking in isolation (one-shot learning), infer which set of speakers is talking. (2) Multi-object recognition in images: Given just the embedding of an image x a , answer whether x a contains the object(s) in another image x b . Storing just the embeddings but not the pixels could potentially be more space-efficient. Because of the novelty of the problem, it was not obvious to what baselines we should compare. When evaluating our models, we sought to assess the unique contribution of the compositional embedding above and beyond what a "traditional" embedding could achieve. Hence, we created baselines by endowing a traditional embedding with some extra functionality to enable it to infer label sets. Modeling assumptions and notation: For generality, we refer to the data to be embedded (images, videos, audio signals, etc.) simply as "examples". Let the universe of classes be S. From any subset T ⊆ S, a ground-truth rendering function r : 2 S → R n "renders" an example, i.e., r(T ) = x. Inversely, there is also a ground-truth classification function c : R n → 2 S that identifies the label set from the rendered example, i.e., c(x) = T . Neither r nor c is observed. We let e T represent the embedding (i.e., output of f ) associated with some example containing classes T . Contribution: To our knowledge, this is the first paper to explore how embedding functions can be trained both to perceive multiple objects in the example and to represent the set of detected objects so that set operations can be conducted among embedded vectors. We instantiate this idea in two ways: Model I for set union (f & g) and Model II for set containment (f & h). By evaluating on synthetic data, OmniGlot handwritten image data (Lake et al., 2015) , as well as the COCO dataset (Lin et al., 2014) , we provide a proof-of-concept that "compositional set embeddings" can work. We proposed a new kind of embedding mechanism whereby the set of objects contained in the input data (e.g., image, video, audio) must be both perceived and then mapped into a space such that the set relationships -union (Model I) and subset (Model II) -between multiple embedded vectors can be inferred. Importantly, the ground-truth rendering process for how examples are rendered from their component classes is not known and must implicitly be learned. In our experiments, conducted on simulated data, OmniGlot, and COCO, the accuracy was far from perfect but outperformed several baselines, including one based on a traditional embedding approach. The results provide a proof-of-concept of how an embedding function f , trained jointly with either the composition function g or the query function h, could be effectively optimized. One possible direction for further research to increase accuracy is to take better advantage of the statistical structure of class co-occurrence in a specific application domain (e.g., which objects tend to co-occur in the same image). A ALTERNATIVE TRAINING PROCEDURE We also tried another method of training f and g with the explicit goal of encouraging g to map e T and e U to be close to e T ∪U . This can be done by training f and g alternately, or by training them jointly in the same backpropagation. However, this approach yielded very poor results. A possible explanation is that g could fulfill its goal by mapping all vectors to the same location (e.g., 0). Hence, a trade-off arises between g's goal and f 's goal (separating examples with distinct label sets).
We explored how a novel method of compositional set embeddings can both perceive and represent not just a single class but an entire set of classes that is associated with the input data.
{"src": "scitldr-AIC_train", "paper_id": "BJx-ZeSKDB"}