We introduce modifications to state-of-the-art approaches to aggregating local video descriptors by using attention mechanisms and function approximations. Rather than using ensembles of existing architectures, we provide an insight on creating new architectures. We demonstrate our solutions in the “The 2nd YouTube-8M Video Understanding Challenge”, by using frame-level video and audio descriptors. We obtain testing accuracy similar to the state of the art, while meeting budget constraints, and touch upon strategies to improve the state of the art. Model implementations are available in https: github.com pomonam LearnablePoolingMethods.
We do utilize the above two ideas. However, the suggestion of using democratic weights before summing @cite_9 is avoided. The use of democratic weights is intended to give each local descriptor an equal contribution to the similarity of the class they belong to. Nevertheless, it is not clear how to make an easily parallelizable implementation of the Sinkhorn scaling algorithm to obtain a solution for these weights. The use of weights is described in Equation , where @math is the set of local descriptors belonging to a given class, and @math is an embedding per local descriptor, as displayed in Equation . In Equation , @math is the covariance matrix of , where is a random variable representing @math in the set of , per class. A single @math is the concatenation of normalized residuals to K cluster centers for a single local descriptor, as in .
@cite_9
2045143396
We consider the design of a single vector representation for an image that embeds and aggregates a set of local patch descriptors such as SIFT. More specifically we aim to construct a dense representation, like the Fisher Vector or VLAD, though of small or intermediate size. We make two contributions, both aimed at regularizing the individual contributions of the local descriptors in the final representation. The first is a novel embedding method that avoids the dependency on absolute distances by encoding directions. The second contribution is a "democratization" strategy that further limits the interaction of unrelated descriptors in the aggregation stage. These methods are complementary and give a substantial performance boost over the state of the art in image search with short or mid-size vectors, as demonstrated by our experiments on standard public image retrieval benchmarks.