
Why can CNNs work well in computer vision systems?
There are some theoretical considerations to answer this question. In [1]} it's claimed that the special structure of the image classification task enables the effectiveness of CNN, where the problem is formulated as the efficient approximation of a local Hamiltonian system with local operations of CNNs. By checking the analogy between the CNN and the quantum computation, we see that the argument of [1]} is equivalent to say that for a task that can be solved by a deep CNN, its \(U_{CNN}\)  falls in the subset that can be achieved by local CONV-ReLU-Pooling operators, just as the subset of \(U(2^n)\)  that can be approximated by simple local universal gates in quantum computations. So the answer to this question is trivial, CNN works since the problem falls in the set of solvable problems of CNNs. A direct question is, can we design a task that does not belong to this solvable problem set? A possible design of such a task is to classify images that are segmented into small rectangular blocks and then the blocks are randomly permuted. Obviously such images are generated by global operations which can not be effectively approximated by local operations, and therefore the classification of such images can not be achieved efficiently by structures like CNN. So we can say, the diameter of the Riemannian manifold \(T_{CNN}\)  is exponentially large with respect to the size of the input data and only those problems which can be solved by a transformation \(U_{CNN}\)  that has a polynomially large geodesic distance to the identity transformation \(I\)  can be solved by CNNs.

Can we justify if a problem can be efficiently solved by CNNs?
Generally this is difficult since this is the same question as to ask if a unitary operator can be decomposed into a tensor product of simple local unitary operators, or to ask if a n-qubit quantum state has a polynomial state complexity with respect to \(n\) . We already know both of them are difficult questions.

Why deep CNN?
Geometrically, training a CNN with a given deepth \(L\)  is equivalent to find a curve of length roughly proportional to \(L\)  in \(T_{CNN}\)  to reach the unknown transformation \(U_{CNN}\) . In deep learning systems, this is achieved by a gradient descent based optimization procedure, i.e., the training of CNNs. It should be noted that the CNN is not to find a geodesic, instead it aims to find a curve with a fixed geometric structure, i.e., a curve consisted of multiple line segments with given lengthes determined by the CNN structure. The reason to use the deep CNN is just that the length of \(U_{CNN}(t)\)  should be longer than the length of the geodesic connecting the identity transformation \(I\)  and \(U_{CNN}\)  on the Riemannian manifold \(T_{CNN}\) . Otherwise it will not work since a short \(U_{CNN}(t)\)  will never reach \(U_{CNN}\) . On the other side, it's not difficult to guess that a too deep CNN may also fail to converge if the curve \(U_{CNN}(t)\)  is too long due to both the depth \(L\)  and an improper design of CONV-ReLU-Pooling operations. This can be easily understood since to reach a point \(U_{CNN}\)  with a longer curve \(U_{CNN}(t)\)  is just to find an optima in a larger parameter space. Intuitively we prefer a design of CNNs that leads to a curve \(U_{CNN}(t)\)  with a length just above the geodesic distance between \(I\)  and \(U_{CNN}\) . To achieve this, we need a balance between the network depth and the complexity of CONV-ReLU-Pooling operations. This observation will play a key role in the analysis of ResNet below.

Why does ReLU work better than the sigmoid in deep CNNs?
A typical answer to this problem is that ReLU can improve the vanishing gradient problem so that a better performance can be achieved. Another idea is that ReLU can change the Fisher information metric to improve the convergence but the sigmoid function can not. Here we explain this from a complexity point of view. As mentioned before, the selection of the activation function will influence the complexity of the local operation. Obviously ReLU is a weaker nonlinear function than the sigmoid. This means for a given CNN structure, using ReLU will have a lower representation capability than using the sigmoid so that the CNN using ReLU has a smaller complexity. This is to say that the a CNN using a sigmoid need to search for an optima in a larger state space. It's obvious that as far as the complexity of the CNN is above the length of the geodesic, it's preferable to use a simpler CNN so that a better convergence can be achieved. It seems there exists a balance between the complexity of the CONV operation and the activation function so that the complexity of the complete system can be distributed equally along the network. With our first definition of the metric, the intuitive geometric picture is that the sigmoid function will introduce a metric so that the curvature of the Riemannian manifold will be more negative, if our guess of the curvature of \(T_{CNN}\)  is true. A higher negative curvature of course will make the curve \(U_{CNN}(t)\)  more unstable so that a small change of the parameters of the CNN will result in a large deviation of the final point of \(U_{CNN}(t)\) . This corresponds to the observed vanishing gradient problem during the back-propagation procedure. So for a general deep CNN, the sigmoid is too complex and ReLU is a proper activation function that can produce a Riemannian manifold with a proper curvature.

How difficult is it to optimize the structure of CNNs?
By the optimization of the structure of CNNs, we mean to find the optimal CNN structure (for example the depth, the number and sizes of kernels of a CNN), which can accomplish the classification task with the minimal computational complexity. Geometrically this is to construct and define a metric on the Riemannian fold \(T_{CNN}\) , under with the length of a curve is proportional to the computational complexity of a CNN structure; and (b)realize the geodesic \(U_{CNN}^{geo}\)  that connects the identity operator \(I\)  with \(U_{CNN}\) . Generally this is difficult, especially when the curvature property of the Riemannian manifold is unknown. This is the same as to find the optimal realization of an algorithm with a given universal logic gate set. For the geometry of quantum computation, the curvature is almost negative so that it's difficult to achieve a general operator[3]}. If the manifold of the fundamental law, the quantum computation, has a negative curvature, it will not be surprising if we assume the geometry of deep learning has a similar property. This point seems to be partially confirmed by the recent work[4]}, where a manifold with an exponentially growing curvature with the depth of a CNN can be constructed. If a CNN is applied to such an input data manifold with an exponentially large negative curvature, then the optimization will be difficult to converge.

