In this preliminary report, we present a simple but very effective technique to stabilize the training of CNN based GANs. Motivated by recently published methods using frequency decomposition of convolutions (e.g. Octave Convolutions), we propose a novel convolution scheme to stabilize the training and reduce the likelihood of a mode collapse. The basic idea of our approach is to split convolutional filters into additive high and low frequency parts, while shifting weight updates from low to high during the training. Intuitively, this method forces GANs to learn low frequency coarse image structures before descending into fine (high frequency) details. Our approach is orthogonal and complementary to existing stabilization methods and can simply plugged into any CNN based GAN architecture. First experiments on the CelebA dataset show the effectiveness of the proposed method.
Standard convolutional layers are designed to detect local conjunctions of features from the previous layer and mapping their appearance to a feature map which does not vary its spatial resolution at no time. Nevertheless, in accordance with the spatial-frequency model , @cite_12 @cite_7 , natural images can be factorized into a low frequency signal that captures the global layout and coarse structure, and a high frequency signal that captures fine details. Attracted by the idea of having feature maps with different resolution, recent works based on deep learning approaches , @cite_8 @cite_22 , have built on top of standard CNNs, architecture schemes that have access to different frequency content. A multigrid architecture is the idea suggested by , @cite_8 that has the intention of wiring cross-scale connections into network structure at the lowest level. In order to create such a topology, every convolutional filter extends spatially within grids ( @math , @math ), across grids multiple scales ( @math ) within a pyramid, and over corresponding feature channels ( @math ). Building in this fashion, a combination of pyramids across the architecture ( @math , @math , @math , @math ).
@cite_22 @cite_7 @cite_12 @cite_8
2938458886 1999908130 2616324011
In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies and design a novel Octave Convolution (OctConv) operation to store and process feature maps that vary spatially "slower" at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale meth-ods, OctConv is formulated as a single, generic, plug-and-play convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing con-volutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9 top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs. SUMMARY 1.Thecontrast thresholds ofavariety ofgrating patterns havebeen measured overawiderangeofspatial frequencies. 2.Contrast thresholds forthedetection ofgratings whoseluminance profiles aresine, square, rectangular orsaw-tooth wavescanbesimply related using Fourier theory. 3.Overawiderangeofspatial frequencies thecontrast threshold ofa grating isdetermined onlybytheamplitude ofthefundamental Fourier component ofitswaveform. 4.Gratings ofcomplex waveformcannotbedistinguished fromsinewavegratings until their contrast hasbeenraised toalevel atwhichthe higher harmonic components reach their independent threshold. 5.Thesefindings canbeexplained bytheexistence within thenervous system oflinearly operating independent mechanisms selectively sensitive tolimited ranges ofspatial frequencies. We propose a multigrid extension of convolutional neural networks (CNNs). Rather than manipulating representations living on a single spatial grid, our network layers operate across scale space, on a pyramid of grids. They consume multigrid inputs and produce multigrid outputs, convolutional filters themselves have both within-scale and cross-scale extent. This aspect is distinct from simple multiscale designs, which only process the input at different scales. Viewed in terms of information flow, a multigrid network passes messages across a spatial pyramid. As a consequence, receptive field size grows exponentially with depth, facilitating rapid integration of context. Most critically, multigrid structure enables networks to learn internal attention and dynamic routing mechanisms, and use them to accomplish tasks on which modern CNNs fail. Experiments demonstrate wide-ranging performance advantages of multigrid. On CIFAR and ImageNet classification tasks, flipping from a single grid to multigrid within the standard CNN paradigm improves accuracy, while being compute and parameter efficient. Multigrid is independent of other architectural choices, we show synergy in combination with residual connections. Multigrid yields dramatic improvement on a synthetic semantic segmentation dataset. Most strikingly, relatively shallow multigrid networks can learn to directly perform spatial transformation tasks, where, in contrast, current CNNs fail. Together, our results suggest that continuous evolution of features on a multigrid pyramid is a more powerful alternative to existing CNN designs on a flat grid.