Unfortunately, PPNN inferences are time-consuming. An inference of a typical PPNN [1]} requires \(>2\)  seconds on an encrypted MNIST image, and consumes \(>70\)  seconds on an encrypted CIFAR-10 image. There is a \(\times 10^6\)  latency gap between a PPNN inference and a unencrypted inference. To mitigate the gap, recent work [2]}, [1]} adopts mobile neural network architectures such as SqueezeNet [4]} and InceptionNet [5]} to implement PPNNs. However, we find naïvely adopting mobile neural network architectures for PPNNs does not necessarily achieves shorter inference latency. Mobile neural network models [5]}, [4]} reduce the total number of parameters but still maintain competitive inference accuracy by adding more linear layers. In spite of less parameters, if a PPNN adopts a mobile neural network architecture, its deeper architecture with more layers greatly increases the HE multiplicative depth, thereby decelerating each HE operations of the PPNN, where the multiplicative depth means the number of HE multiplications on the critical path.
