Complex-value neural networks are not a new concept, however, the use of real-values has often been favoured over complex-values due to difficulties in training and accuracy of results. Existing literature ignores the number of parameters used. We compared complex- and real-valued neural networks using five activation functions. We found that when real and complex neural networks are compared using simple classification tasks, complex neural networks perform equal to or slightly worse than real-value neural networks. However, when specialised architecture is used, complex-valued neural networks outperform real-valued neural networks. Therefore, complex–valued neural networks should be used when the input data is also complex or it can be meaningfully to the complex plane,  or when the network architecture uses the structure defined by using complex numbers. In recent years complex numbers in neural networks are increasingly frequently used. ComplexValued neural networks have been sucessfully applied to a variety of tasks specifically in signal processing where the input data has a natural interpretation in the complex domain.In most publications complex-valued neural networks are compared to real-valued architectures. We need to ensure that these architectures are comparable in their ability to approximate functions. A common metric for their capacity are the number of real-valued parameters. The number of parameters of complex-valued neural networks are rarely studied aspects. While complex numbers increase the computational complexity, their introduction also assumes a certain structure between weights and input. Hence, it is not sufficient to increase the number of parameters.Even more important than in real-valued networks is the choice of activation function for each layer. We test 5 functions: identity or no activation function, rectifier linear unit, hyperbolic tangent, magnitude, squared magnitude. This paper explores the performance of complex-valued multi-layer perceptrons (MLP) with varying depth and width in consideration of the number of parameters and choice of activation function on benchmark classification tasks.In section 2 we will give an overview of the past and current developments in the applications of complex-valued neural networks. We shortly present the multi-layer perceptron architecture in section 3 using complex numbers and review the building blocks of complex-valued network.In section 4 we consider the multi-layer perceptron with respect to the number of real-valued parameters in both the complex and real case. We construct complex MLPs with the same number of units in each layer. We propose two methods to define comparable networks: A fixed number of real-valued neurons per layer or a fixed budget of real-valued parameters.In the same section we also consider the structure that is assumed by introducing complex numbers into a neural network.We present the activation function to be used in our experiments in section 5. In section 6 we present our experiments and their settings. Section 7 discuss the results of different multi-layer perceptrons on MNIST digit classification, CIFAR-10 image classification, CIFAR-100 image classification, Reuters topic classification and bAbI question answering. We identify a general direction of why and how to use complex-valued neural networks.
Comparison of complex- and real-valued multi-layer perceptron with respect to the number of real-valued parameters.
{"src": "scitldr-AIC_train", "paper_id": "HkCy2uqQM"}