Texture is ubiquitous and provides useful cues of material properties of objects and their identity. Visual representations based on orderless aggregations of local features (e.g. SIFT [1]}, SURF [2]}, HOG [3]}, and LBP [4]}) were originally developed as image texture descriptors. Before the `deep learning era', models such as the histograms of filter bank responses [5]}, [6]}, Bag-of-visual-Words (BoW) [7]} and Fisher vector [8]} have been successfully applied to texture recognition. The main idea of BoW is to represent an image by a histogram of feature words of a data-learned dictionary (the so-called codebook). The BoW model usually consists of the following five steps: i) creating local patches from the input images, ii) extracting local features from the patches and creating the `bag', iii) learning a dictionary of visual words, iv) quantizing features using dictionary and representing samples by histogram of words (encoding), and v) classification of features using SVM or decision tree. A block diagram of the traditional BoW is shown in Figure REF  (top). In fact, convolutional neural networks (CNNs) can be viewed as BoW models too. The convolution layer of CNNs operates in a sliding window manner acting as a local feature extractor. The output feature maps preserve a relative spatial arrangement of input images. The resulting globally ordered features are then concatenated and fed into a fully connected (FC) layer which acts as a classifier. Despite massive achievement on a broad range of applications such as image classification, object detection, segmentation, and scene understanding, CNNs are typically not ideal for texture recognition due to the need for a spatially invariant representation describing the feature distributions instead of concatenation.
