This paper gives a rigorous analysis of trained Generalized Hamming Networks (GHN) proposed by Fan (2017) and discloses an interesting finding about GHNs, i.e. stacked convolution layers in a GHN is equivalent to a single yet wide convolution layer. The revealed equivalence, on the theoretical side, can be regarded as a constructive manifestation of the universal approximation theorem Cybenko (1989); Hornik (1991). In practice, it has profound and multi-fold implications. For network visualization, the constructed deep epitomes at each layer provide a visualization of network internal representation that does not rely on the input data. Moreover, deep epitomes allows the direct extraction of features in just one step, without resorting to regularized optimizations used in existing visualization tools. Despite the great success in recent years, neural networks have long been criticized for their blackbox natures and the lack of comprehensive understanding of underlying mechanisms e.g. in BID3 ; BID12 ; BID30 ; BID29 . The earliest effort to interpret neural computing in terms of logic inferencing indeed dated back to the seminal paper of BID24 , followed by recent attempts to provide explanations from a multitude of perspectives (reviewed in Section 2).As an alternative approach to deciphering the mysterious neural networks, various network visualization techniques have been actively developed in recent years (e.g. BID11 ; BID28 and references therein). Such visualizations not only provide general understanding about the learning process of networks, but also disclose operational instructions on how to adjust network architecture for performance improvements. Majority of visualization approaches probe the relations between input data and neuron activations, by showing either how neurons react to some sample inputs or, reversely, how desired activations are attained or maximized with regularized reconstruction of inputs BID7 ; BID20 ; BID36 ; BID31 ; BID23 ; BID33 ; BID0 . Input data are invariably used in visualization to probe how the information flow is transformed through the different layers of neural networks. Although insightful , visualization approaches as such have to face a critical open question: to what extend the conclusions drawn from the analysis of sample inputs can be safely applied to new data?In order to furnish confirmatory answer to the above-mentioned question, ideally, one would have to employ a visualization tool that is independent of input data. This ambitious mission appears impossible at a first glance -the final neuron outputs cannot be readily decomposed as the product of inputs and neuron weights because the thresholding in ReLU activations is input data dependent. By following the principle of fuzzy logic, BID8 recently demonstrated that ReLUs are not essential and can be removed from the so called generalized hamming network (GHN) . This simplified network architecture , as reviewed in section 3, facilitates the analysis of neuron interplay based on connection weights only. Consequently, stacked convolution layers can be merged into a single hidden layer without taking into account of inputs from previous layers. Equivalent weights of the merged GHN, which is called deep epitome, are computed analytically without resorting to any learning or optimization processes. Moreover, deep epitomes constructed at different layers can be readily applied to new data to extract hierarchical features in just one step (section 4). We have proposed in this paper a novel network representation, called deep epitome, which is proved to be equivalent to stacked convolution layers in generalized hamming networks (GHN). Theoretically this representation provides a constructive manifestation for the universal approximation theorem BID6 BID15 , which states that a single layered network, in principle, is able to approximate any arbitrary decision functions up to any desired accuracy. On the other hand, it is a dominant belief BID10 , which is supported by abundant empirical evidences, that deep structures play an indispensable role in decomposing the combinatorial optimization problem into layer-wise manageable sub-problems. We concur with the view and supplement with our demonstration that, a trained deep GHN can be converted into a simplified networks for the sake of high interpretability, reduced algorithmic and computational complexities.The success of our endeavours lies in the rigorous derivation of convolving epitomes across different layers in eq. (4) and (5), which set due bias terms analytically without resorting to optimizationbased approaches. Consequently, deep epitomes at all convolution layers can be computed without using any input data. Moreover, deep epitomes can be used to extract hierarchical features in just one step at any desired layers. In the light of fuzzy logic, the normalized epitome (definition 3) encodes a grade of fitness between the learnt templates and given inputs at certain spatial locations. This fuzzy logic interpretation furnishes a refreshing perspective that, in our view, will open the black box of deep learning eventually.APPENDIX A Definition 1. For two given tuples DISPLAYFORM0 . . , y L }, the hamming outer product, denoted , is a set of corresponding elements x DISPLAYFORM1 . . L , where ⊕ denotes the generalized hamming distance operator. Then the product has following properties, DISPLAYFORM2 K but they are permutation equivalent, in the sense that there exist permutation matrices P and Q such that x DISPLAYFORM3 2. non-linear: in contrast to the standard outer product which is bilinear in each of its entry, the hamming outer product is non-linear since in general x DISPLAYFORM4 where µ ∈ R is a scalar. Therefore, the hamming outer product defined as such is a pseudo outer product. DISPLAYFORM5 M because of the associativity of GHD. This property holds for arbitrary number of tuples.
bridge the gap in soft computing
{"src": "scitldr-AIC_train", "paper_id": "r1PaPUsXM"}