Teach me how to Interpolate a Myriad of Embeddings
Mixup refers to interpolation-based data augmentation, originally motivated
as a way to go beyond empirical risk minimization (ERM). Yet, its extensions
focus on the definition of interpolation and the space where it takes place,
while the augmentation itself is less studied: For a mini-batch of size $m$,
most methods interpolate between $m$ pairs with a single scalar interpolation
factor $\lambda$.
  In this work, we make progress in this direction by introducing MultiMix,
which interpolates an arbitrary number $n$ of tuples, each of length $m$, with
one vector $\lambda$ per tuple. On sequence data, we further extend to dense
interpolation and loss computation over all spatial positions. Overall, we
increase the number of tuples per mini-batch by orders of magnitude at little
additional cost. This is possible by interpolating at the very last layer
before the classifier. Finally, to address inconsistencies due to linear target
interpolation, we introduce a self-distillation approach to generate and
interpolate synthetic targets.
  We empirically show that our contributions result in significant improvement
over state-of-the-art mixup methods on four benchmarks. By analyzing the
embedding space, we observe that the classes are more tightly clustered and
uniformly spread over the embedding space, thereby explaining the improved
behavior.