Prior neural style transfer algorithms tended to either treat style as multivariate Gaussian distribution over neural features, or give up a probabilistic interpretation of style by turning to non-parametric feature matching (using nearest neighbors) to better capture details of the style. Defining style based on a limited set of summary statistics is an impoverished representation which cannot capture complex stylistic details. On the other hand, while replacing each content feature with it's nearest style neighbor can copy details of the style, not considering a distribution of features can lead to an overly homogeneous result.
Our goal was a formulation of style similarity that captured the best of both worlds, the detail preservation of non-parametric approaches and the diversity of probabilistic approaches.
We defined style as an empirical distribution over features extracted by
a deep neural network, and measured the similarity between such
distributions using an efficient approximation of the Earth Movers
Distance initially proposed in the Natural Language Processing
community [1]}. The Earth Movers Distance is not only a useful measure of similarity between distributions which encourages using all the features in the style image, it is also non-parametric and therefore not constrained to matching the limited summary statistics of a Gaussian.
