Despite DSN's success in speech classification, its framework also has several limitations. First, the conventional DSN only has used the sigmoid activation function for the nonlinear hidden layer [1]}. Although sigmoid has been widely used in the literature, it suffers from a number of drawbacks: for example the training can be slow, and with random initialization, the solution can stuck at a poor local solution that does not have good predictive performance [2]}. In fact there are another two types of activation functions. The one is hyperbolic tangent, which has been applied to training deep neural networks. It suffers from the same problems as those of sigmoid functions. A more recent proposal is the rectifier linear unit (ReLU) [3]}. It is observed that this method is very useful for object recognition and often trains significantly faster [4]}.
