We propose Pure CapsNets (P-CapsNets) without routing procedures. Specifically, we make three modifications to CapsNets.   First, we remove routing procedures from CapsNets based on the observation that the coupling coefficients can be learned implicitly. Second, we replace the convolutional layers in CapsNets to improve efficiency. Third, we package the capsules into rank-3 tensors to further improve efficiency. The experiment shows that P-CapsNets achieve better performance than CapsNets with varied routine procedures by using significantly fewer parameters on MNIST&CIFAR10. The high efficiency of P-CapsNets is even comparable to some deep compressing models. For example, we achieve more than 99% percent accuracy on MNIST by using only 3888 parameters.   We visualize the capsules as well as the corresponding correlation matrix to show a possible way of initializing CapsNets in the future. We also explore the adversarial robustness of P-CapsNets compared to CNNs. Capsule Networks, or CapsNets, have been found to be more efficient for encoding the intrinsic spatial relationships among features (parts or a whole) than normal CNNs. For example, the CapsNet with dynamic routing (Sabour et al. (2017) ) can separate overlapping digits accurately, while the CapsNet with EM routing (Hinton et al. (2018) ) achieves lower error rate on smallNORB (LeCun et al. (2004) ). However, the routing procedures of CapsNets (including dynamic routing (Sabour et al. (2017) ) and EM routing (Hinton et al. (2018) )) are computationally expensive. Several modified routing procedures have been proposed to improve the efficiency ; Choi et al. (2019) ; Li et al. (2018) ), but they sometimes do not "behave as expected and often produce results that are worse than simple baseline algorithms that assign the connection strengths uniformly or randomly" (Paik et al. (2019) ). Even we can afford the computation cost of the routing procedures, we still do not know whether the routing numbers we set for each layer serve our optimization target. For example, in the work of Sabour et al. (2017) , the CapsNet models achieve the best performance when the routing number is set to 1 or 3, while other numbers cause performance degradation. For a 10-layer CapsNet, assuming we have to try three routing numbers for each layer, then 3 10 combinations have to be tested to find the best routing number assignment. This problem could significantly limit the scalability and efficiency of CapsNets. Here we propose P-CapsNets, which resolve this issue by removing the routing procedures and instead learning the coupling coefficients implicitly during capsule transformation (see Section 3 for details). Moreover, another issue with current CapsNets is that it is common to use several convolutional layers before feeding these features into a capsule layer. We find that using convolutional layers in CapsNets is not efficient, so we replace them with capsule layers. Inspired by Hinton et al. (2018) , we also explore how to package the input of a CapsNet into rank-3 tensors to make P-CapsNets more representative. The capsule convolution in P-CapsNets can be considered as a more general version of 3D convolution. At each step, 3D convolution uses a 3D kernel to map a 3D tensor into a scalar (as Figure 1 shows) while the capsule convolution in Figure 2 adopts a 5D kernel to map a 5D tensor into a 5D tensor. Figure 1: 3D convolution: tensor-to-scalar mapping. The shape of input is 8ˆ8ˆ4. The shape of 3D kernel is 4ˆ4ˆ3. As a result, the shape of output is 5ˆ5ˆ3. Yellow area shows current input area being convolved by kernel and corresponding output. Figure 2: Capsule convolution in P-CapsNets: tensor-to-tensor mapping. The input is a tensor of 1's which has a shape of 1ˆ5ˆ5ˆp3ˆ3ˆ3q (correspond to the the input channel, input height, input width, first capsule dimension, second capsule dimension, and third capsule dimension, respectively). The capsule kernel is also a tensor of 1's which has a shape of 4ˆ4ˆ1ˆ1ˆp3ˆ3ˆ3q -kernel height, kernel width, number of input channel, number of output channel, and the three dimensions of the 3D capsule. As a result, we get an output tensor of 48's which has a shape of 1ˆ2ˆ2ˆp3ˆ3ˆ3q. Yellow areas show current input area being convolved by kernel and corresponding output. We propose P-CapsNets by making three modifications based on CapsNets Sabour et al. (2017) , 1) We replace all the convolutional layers with capsule layers, 2) We remove routing procedures from the whole network, and 3) We package capsules into rank-3 tensors to further improve the efficiency. The experiment shows that P-CapsNets can achieve better performance than multiple other CapsNets variants with different routing procedures, as well as than deep compressing models, by using fewer parameters. We visualize the capsules in P-CapsNets and point out that the initializing methods of CNNs might not be appropriate for CapsNets. We conclude that the capsule layers in P-CapsNets can be considered as a general version of 3D convolutional layers. We conjecture that CapsNets can encode the intrinsic spatial relationship between a part and a while efficiently, comes from the tensor-to-tensor mapping between adjacent capsule layers. This mapping is presumably also the reason for P-CapsNets' good performance. CapsNets#0, CapsNets#1, CapsNets#2, CapsNets#3) , they are all five-layer CapsNets. Take CapsNets#2 as an example, the input are gray-scale images with a shape of 28ˆ28, we reshape it as a 6D tensor, 1ˆ28ˆ28ˆp1ˆ1ˆ1q to fit our P-CaspNets. The first capsule layer (CapsConv#1, as Figure 7 shows.), is a 7D tensor, 3ˆ3ˆ1ˆ1ˆp1ˆ1ˆ16q. Each dimension of the 7D tensor represents the kernel height, the kernel width, the number of input capsule feature map, the number of output capsule feature map, the capsule's first dimension, the capsule's second dimension, the capsule's third dimension. All the following feature maps and filters can be interpreted in a similar way. Similarly, the five capsule layers of P-CapsNets#0 are 3ˆ3ˆ1ˆ1ˆp1ˆ1ˆ32, 3ˆ3ˆ1ˆ2p 1ˆ8ˆ8q, 3ˆ3ˆ2ˆ4ˆp1ˆ8ˆ8q, 3ˆ3ˆ4ˆ2ˆp1ˆ8ˆ8, 3ˆ3ˆ2ˆ10ˆp1ˆ8ˆ8q respectively. The strides for each layers are p2, 1, 2, 1, 1q. The five capsule layers of P-CapsNets#1 are 3ˆ3ˆ1ˆ1ˆp1ˆ1ˆ16, 3ˆ3ˆ1ˆ1ˆp1ˆ4ˆ6q, 3ˆ3ˆ1ˆ1ˆp1ˆ6ˆ4q, 3ˆ3ˆ1ˆ1ˆp1ˆ4ˆ6, 3ˆ3ˆ1ˆ10ˆp1ˆ6ˆ4q respectively. The strides for each layers are p2, 1, 2, 1, 1q. The five capsule layers of P-CapsNets#3 are 3ˆ3ˆ1ˆ1ˆp1ˆ1ˆ32, 3ˆ3ˆ1ˆ4ˆp1ˆ8ˆ16q, 3ˆ3ˆ4ˆ8ˆp1ˆ16ˆ8q, 3ˆ3ˆ8ˆ4ˆp1ˆ8ˆ16, 3ˆ3ˆ4ˆ10ˆp1ˆ16ˆ16q respectively. The strides for each layers are p2, 1, 2, 1, 1q. The five capsule layers of P-CapsNets#4 are 3ˆ3ˆ1ˆ1ˆp1ˆ3ˆ32, 3ˆ3ˆ1ˆ4ˆp1ˆ8ˆ16q, 3ˆ3ˆ4ˆ8ˆp1ˆ16ˆ8q, 3ˆ3ˆ8ˆ10ˆp1ˆ8ˆ16, 3ˆ3ˆ10ˆ10ˆp1ˆ16ˆ16q respectively. The strides for each layers are p2, 1, 1, 2, 1q.
Routing procedures are not necessary for CapsNets
{"src": "scitldr-AIC_train", "paper_id": "B1gNfkrYvS"}