<jupyter_start><jupyter_text>Optimization and Deep LearningIn this section, we will discuss the relationship between optimization and deep learning as well as the challenges of using optimization in deep learning. For a deep learning problem, we will usually define a loss function first. Once we have the loss function, we can use an optimization algorithm in attempt to minimize the loss. In optimization, a loss function is often referred to as the objective function of the optimization problem. Traditionally, optimization algorithms usually only consider minimizing the objective function. In fact, any maximization problem can be easily transformed into a minimization problem: we just need to use the opposite of the objective function as the new objective function. The Relationship Between Optimization and Deep LearningAlthough optimization provides a way to minimize the loss function for deep learning, in essence, the goals of optimization and deep learning are different.In the ["Model Selection, Underfitting and Overfitting"](../chapter_deep-learning-basics/underfit-overfit.md) section, we discussed the difference between the training error and generalization error.Because the objective function of the optimization algorithm is usually a loss function based on the training data set, the goal of optimization is to reduce the training error.However, the goal of deep learning is to reduce the generalization error.In order to reduce the generalization error, we need to pay attention to overfitting in addition to using the optimization algorithm to reduce the training error.In this chapter, we are going to focus specifically on the performance of the optimization algorithm in minimizing the objective function, rather than the model's generalization error. Optimization Challenges in Deep LearningIn the [Linear Regression](../chapter_deep-learning-basics/linear-regression.md) section, we differentiated between analytical solutions and numerical solutions in optimization problems. In deep learning, most objective functions are complicated. Therefore, many optimization problems do not have analytical solutions. Instead, we must use optimization algorithms based on the numerical method to find approximate solutions, which also known as numerical solutions. The optimization algorithms discussed here are all numerical method-based algorithms. In order to minimize the numerical solution of the objective function, we will reduce the value of the loss function as much as possible by using optimization algorithms to finitely update the model parameters.There are many challenges in deep learning optimization. Two such challenges are discussed below: local minimums and saddle points. To better describe the problem, we first import the packages or modules required for the experiments in this section.<jupyter_code>import sys
sys.path.insert(0, '..')

%matplotlib inline
import d2l
from mpl_toolkits import mplot3d
import numpy as np<jupyter_output><empty_output><jupyter_text>Local MinimumsFor the objective function $f(x)$, if the value of $f(x)$ at $x$ is smaller than the values of $f(x)$ at any other points in the vicinity of $x$, then $f(x)$ could be a local minimum. If the value of $f(x)$ at $x$ is the minimum of the objective function over the entire domain, then $f(x)$ is the global minimum.For example, given the function$$f(x) = x \cdot \text{cos}(\pi x), \qquad -1.0 \leq x \leq 2.0,$$we can approximate the local minimum and global minimum of this function. Please note that the arrows in the figure only indicate the approximate positions.<jupyter_code>def f(x):
    return x * np.cos(np.pi * x)

d2l.set_figsize((4.5, 2.5))
x = np.arange(-1.0, 2.0, 0.1)
fig,  = d2l.plt.plot(x, f(x))
fig.axes.annotate('local minimum', xy=(-0.3, -0.25), xytext=(-0.77, -1.0),
                  arrowprops=dict(arrowstyle='->'))
fig.axes.annotate('global minimum', xy=(1.1, -0.95), xytext=(0.6, 0.8),
                  arrowprops=dict(arrowstyle='->'))
d2l.plt.xlabel('x')
d2l.plt.ylabel('f(x)');<jupyter_output><empty_output><jupyter_text>The objective function of the deep learning model may have several local optimums. When the numerical solution of an optimization problem is near the local optimum, the numerical solution obtained by the final iteration may only minimize the objective function locally, rather than globally, as the gradient of the objective function's solutions approaches or becomes zero. Saddle PointsAs we just mentioned, one possible explanation for a gradient that approaches or becomes zero is that the current solution is close to a local optimum. In fact, there is another possibility. The current solution could be near a saddle point. For example, given the function$$f(x) = x^3,$$we can find the position of the saddle point of this function.<jupyter_code>x = np.arange(-2.0, 2.0, 0.1)
fig, = d2l.plt.plot(x, x**3)
fig.axes.annotate('saddle point', xy=(0, -0.2), xytext=(-0.52, -5.0),
                  arrowprops=dict(arrowstyle='->'))
d2l.plt.xlabel('x')
d2l.plt.ylabel('f(x)');<jupyter_output><empty_output><jupyter_text>Now, we will use another example of a two-dimensional function, defined as follows:$$f(x, y) = x^2 - y^2.$$We can find the position of the saddle point of this function. Perhaps you have noticed that the function looks just like a saddle, and the saddle point happens to be the center point of the seat.<jupyter_code>x, y = np.mgrid[-1: 1: 31j, -1: 1: 31j]
z = x**2 - y**2

ax = d2l.plt.figure().add_subplot(111, projection='3d')
ax.plot_wireframe(x, y, z, **{'rstride': 2, 'cstride': 2})
ax.plot([0], [0], [0], 'rx')
ticks = [-1,  0, 1]
d2l.plt.xticks(ticks)
d2l.plt.yticks(ticks)
ax.set_zticks(ticks)
d2l.plt.xlabel('x')
d2l.plt.ylabel('y');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08464874241110147, 'fraction_numerical': 0.015264527320034691, 'mean_word_length': 4.981327800829876, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '142028', 'license': 'MIT', 'n_tokens_mistral': 1591, 'n_tokens_neox': 1520, 'n_words': 745, 'path': 'book-d2l-en/chapter_optimization/optimization-intro.ipynb', 'repo_name': 'linked0/dlnd-deep-learning', 'size': '5765'}