<jupyter_start><jupyter_text>"An Overview of Initialization for Deep Neural Networks, Part 1"> "Without appropriate choice of activation function and/or smart initialization for it neural networks, especially deep ones, tend to refuse training at all. We will consider sigmoid and tanh activation functions and two initialization methods most suitable for them - LeCun and Xavier (Glorot) initialization."- toc: false- branch: master- badges: false- comments: false- categories: [fastpages, jupyter]- use_math: true For the past few years deep neural networks have proven themselves to be able to solve various AI-level tasks effectively. In some very specific tasks, such as image classification, game of Go, autonomous driving, and many others they already perform better than humans. These achievements were not possible without tremendous research invested in study of difficulties that occur during training of deep networks, and methods to overcome them. A part of this progress is due to better non-linearities and initialization methods. What Is the Choice of Activation Function and Initialization and Why Is It Important?Neural networks are typically trained via stochastic gradient descent or one of its more powerful variants, such as RMSprop or Adam, that use gradients computed by the back propagation algorithm. The back propagation algorithm essentially recursively applies the chain rule of calculus to compute gradients efficiently. Just as a reminder, the chain rule of calculus look like this\begin{equation}    \frac{\partial}{\partial x}g(f(x)) = \frac{\partial g}{\partial f} \frac{\partial f}{\partial x}\end{equation}In earlier years folks used sigmoid activation function, defined as\begin{equation}    \sigma(x) = \frac{1}{1 + e^{-x}}\end{equation}However, soon it became evident that such nets are hard to train, because of some properties of sigmoid. One such property is the effect of saturation. Look at the graph of sigmoid.<jupyter_code>def sigmoid(x):
    return 1. / (1. + np.exp(-x))

def sigmoid_derivative(x):
    return sigmoid(x) * (1. - sigmoid(x))

def plot_activation_and_derivative(activation, derivative, inputs, titles):
    fig, (ax_f, ax_d) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
    ax_f.plot(inputs, activation(inputs))
    ax_f.set_title(titles[0])
    ax_d.plot(inputs, derivative(inputs))
    ax_d.set_title(titles[1])
    plt.show()

values = np.arange(-6.0, 6.0, 0.01)
plot_activation_and_derivative(sigmoid, sigmoid_derivative, values,
                               ('Sigmoid activation', 'Derivative of sigmoid'))<jupyter_output><empty_output><jupyter_text>When input values become large enough (whether positive or negative), output of sigmoid remains roughly unchanged and goes to 1 in case of positive inputs, or to 0 in case of negative. This saturation of sigmoid causes gradient vanishing. Since output values are roughly equal in those saturated tails, the derivative of sigmoid becomes very close to zero. As back propagation progresses towards input layers, gradients get multiplied, and if at least one of multiples becomes close to zero, the whole product is likely to become close to zero as well. Hence it can require too many gradient updates to tweak parameters in right way. Now look at the graph of the derivative of sigmoid more closely. You can note that its maximum value is 0.25 when input value is 0. So, in the best case, gradient gets four times smaller after sigmoid activation during back propagation.There is another saturating activation function that, however, suffers less from vanishing gradient problems. It is called hyperbolic tangent, or tanh for short.\begin{equation}    \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}\end{equation}Actually it can be shown from their definitions that tanh can be expressed in terms of sigmoid as follows\begin{equation}    \tanh(x) = 2\sigma(2x) - 1\end{equation}In spite of that, tanh has two advantages over the sigmoid. First, it is symmetric in the origin, which makes it easier to learn identity function, and second, maximum value of its derivative is 1.<jupyter_code>def tanh(x):
    return np.tanh(x)

def tanh_derivative(x):
    return 1. / np.cosh(x) ** 2

plot_activation_and_derivative(tanh, tanh_derivative, values,
                               ('Tanh activation', 'Tanh derivative'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07236088848179528, 'fraction_numerical': 0.0073276849095488894, 'mean_word_length': 4.636129032258064, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '491149', 'license': 'Apache-2.0', 'n_tokens_mistral': 1191, 'n_tokens_neox': 1118, 'n_words': 555, 'path': '_notebooks/2020-07-31-An-Overview-of-Activations-and-Initializations-for-Deep-Neural-Networks,-Part-1.ipynb', 'repo_name': 'akulchik/blog', 'size': '4367'}