We introduce modifications to state-of-the-art approaches to aggregating local video descriptors by using attention mechanisms and function approximations. Rather than using ensembles of existing architectures, we provide an insight on creating new architectures. We demonstrate our solutions in the “The 2nd YouTube-8M Video Understanding Challenge”, by using frame-level video and audio descriptors. We obtain testing accuracy similar to the state of the art, while meeting budget constraints, and touch upon strategies to improve the state of the art. Model implementations are available in https: github.com pomonam LearnablePoolingMethods.
Following the success of VLAD is the work of @cite_9 . The authors split the problem of local descriptor pooling into problems of local descriptor embedding and aggregation. We specifically utilize the ideas for local descriptor embeddings. The work of @cite_9 creates a new embedding technique by focusing on overcoming pitfalls when using NetVLAD, by L2 normalizing the distances from cluster centers (residuals) before summing to help give an equal contribution to each residual. These embeddings are known as Triangulation embeddings, or T-embeddings. Further, the authors suggest whitening the residuals by removing a bias and de-correlating the residuals per cluster center, for a given local descriptor, before summing.
@cite_9
2045143396
We consider the design of a single vector representation for an image that embeds and aggregates a set of local patch descriptors such as SIFT. More specifically we aim to construct a dense representation, like the Fisher Vector or VLAD, though of small or intermediate size. We make two contributions, both aimed at regularizing the individual contributions of the local descriptors in the final representation. The first is a novel embedding method that avoids the dependency on absolute distances by encoding directions. The second contribution is a "democratization" strategy that further limits the interaction of unrelated descriptors in the aggregation stage. These methods are complementary and give a substantial performance boost over the state of the art in image search with short or mid-size vectors, as demonstrated by our experiments on standard public image retrieval benchmarks.