<jupyter_start><jupyter_text>Multivariate CalculusAuthor: Vo, <NAME><jupyter_code>import numpy as np
import sympy
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Acknowledgements:The contents of this note are based on the lecture notes and the materials from the following source:1. _Mathematics for Machine Learning_ specializatin given by Prof. <NAME>, Dr. <NAME>, and Dr. <NAME> from Imperial College London. Available in Coursera: [Mathematics for Machine Learning specialization](https://www.coursera.org/specializations/mathematics-machine-learning) 2. _Mathematics for Machine Learning_ textbook by Prof. <NAME>, Prof. <NAME>, and Prof. <NAME>. The book is freely available at: [Mathematics for Machine Learning textbook](https://mml-book.github.io/) 3. _Essential Math for Data Science in 6 Weeks_ webinar given by Dr. <NAME>. Available in O'Reily Learning: [Essential Math for Data Science in 6 Weeks](https://learning.oreilly.com/attend/essential-math-for-data-science-in-6-weeks/0636920055929/0636920055928/)4. _The Matrix Calculus You Need For Deep Learning_ leaflet by Prof. <NAME> and Dr. <NAME> from University of San Francisco. Available at arXiv.org: [The Matrix Calculus You Need For Deep Learning](https://arxiv.org/abs/1802.01528)5. _MIT 8.02t Electricity and Magnetism_ course given by Dr. <NAME>, Dr. <NAME>, Prof. <NAME>, Prof. <NAME>, Prof. <NAME>, and Dr. <NAME> from MIT.  Table of Contents1. [Introduction to Multivariate Calculus](Section1) 2. [Derivatives](Section2) 3. [Vector Calculus](Section3)4. [Integrals](Section4)6. [Appendix](Section5)  I. Introduction to Multivariate Calculus  1. Why we have to learn?Multivariate Calculus plays a critical role in machine learning. If we pick up any machine learning paper or documentation of any library such as PyTorch, the first thing we see is calculus.Therefore, if we want to understand machine learning, we must understand calculus. 2. Applications of Multivariate CalculusTogether with linear algebra, calculus serves as another pillar in many scientific fields. Below are selected examples of how it is applied in machine learning and data science.Figure 1: Visualization of gradient descent. This algorithm is an optimization method to find local minimum/maximum; thus, it is highly popular due to the need to optimize thousands and even millions of machine learning variables, like neural networks and their node weight values.$$$$Figure 2: Visualization of finding areas under probability distributions to get probabilities, which is a commonly used case for integration in data science (adapted from **Digital Underwater Acoustic Communication Equipment** from <NAME> et al).  II. Derivatives 1. What are derivatives:1. According to formal definition, the derivative of a function $f$ at $x = a$$$\lim_{h \to 0} \frac{f(a + h) - f(a)}{h}$$provided the limit exists.2. We can simply understand that a derivative tells the slope of a function $f(x)$ at a designated point $x = a$. Therefore, it is useful to measure the rate of change at any point in a function. When the slope is 0, that means we are at the minimum or maximum of a variable. 3. As an example, here is how we find the slope of the function $f(x) = x^2$ at $x = 2$:    * We can measure "steepness" at any point in the curve, and we can visualize this with a tangent line.    * A tangent line is a straight line that "just touches" the curve at a given 𝑥 value, and it also provides the slope at that point.    * We can estimate this slope using the following steps:**** Take $x = 2$ and a nearby value $x = 2.1$.* When passed to the function $f(x) = x^2$ these two x-values will yield the following values.$$f(2) = (2)^2 = 4$$$$f(2.1) = (2.1)^2 = 4.41$$* From here, we can estimate the slope between these two points using the simple rise-over-run formula:$$m = \frac{y_2 - y_1}{x_2 - x_1} = 4.41$$* Therefore, the resulting line that passes through these two points has a slope of 4.41. We can increase the accuracy of our slope estimation by selecting a point (for example, $x = 2.000001$ that is infinitely close to $x = 2$. Thus, the slope is:$$m = \frac{y_2 - y_1}{x_2 - x_1} = 4.000001 \approx 4$$***        Figure 1: An example of finding a slope for a function at a designated point by estimation. This method of estimation is widely used if we cannot compute the exact derivative of our function-under-interest. 4. In practice, we compute the slope by taking the derivative $\frac{d}{d_x}$ of our function $f(x)$.5. Noted: the **derivatives** describe the rate of change of a function concerning the change in its variables. The **differentiation** describes the process of finding the derivatives. 2. What are partial derivatives:1. Partial derivatives are derivatives on functions that have multiple input variables. Because there is more than one input variable, we have slopes in more than one direction as plotted with these functions.2. Logically, if there are multiple slopes then we can have multiple derivatives for each variable. Rather than finding the slope on a 1-dimensional function, we have slopes concerning multiple variables in several directions.3. As an example, consider this function $f(x,y) = 2x^3 + 3y^3$, we can compute the partial derivatives of this function as follows:$$\frac{\partial f(x,y)}{\partial x} = 6x^2 + 0$$$$\frac{\partial f(x,y)}{\partial y} = 0 + 9y^2$$* At a point $(x,y) = (1,2)$, we have the slope concerning x and the slope concerning y computed as follows:$$\frac{\partial f(x,y)}{\partial x} = 6(1^2) = 6$$$$\frac{\partial f(x,y)}{\partial y} = 9(2)^2 = 36$$* Therefore, the slope concerning $x$ is 6, and the slope concerning $y$ is 36.5. Noted: the function $f(x,y) = 2x^3 + 3y^3$ can be denoted as $f(\mathbf{x}) = 2x_1^3 + 3x_2^3$ with $\mathbf{x} = [x_1, x_2]$. 3. Basic rules of differentiation:Below are the basic rules of differentiation:1. Product rule: $$(f(x)g(x))' = f'(x)g(x) + f(x)g'(x) $$2. Quotient rule: $$(\frac{f(x)}{g(x)})' = \frac{f'(x)g(x) + f(x)g'(x)}{(g(x))^2}$$3. Sum rule: $$(f(x) + g(x))' = f'(x) + g'(x) $$4. Chain rule: $$(g(f(x)))' = g'(f(x))f'(x) $$ The last rule is crucial because it is the backbone of the backpropagation algorithm in machine learning.        Figure 2: Application of the chain rule: the backpropagation algorithm. This algorithm describes how to find the influence of a certain input, on systems that are composed of multiple functions. (adapted from **Hands-On Convolutional Neural Networks with TensorFlow** by Zafar et al.)<jupyter_code>##
# DEMO: FIND DERIVATIVE AND COMPUTE SLOPE
#
from sympy import *
x = symbols('x') # We use this function to declare our variable for symbolic math.

f = x**2
df = diff(f,x)

slope = df.subs(x,2)
print(slope)

##
# DEMO: FIND THE PARTIAL DERIVATIVE
#
from sympy import *
from sympy.plotting import plot3d

x,y = symbols('x y')

f = 2*x**3 + 3*y**3

dx_f = diff(f, x)
dy_f = diff(f, y)

print(dx_f) 
print(dy_f)<jupyter_output>4
6*x**2
9*y**2<jupyter_text>II. Vector Calculus 1. Basic components of vector calculus a) FieldA field is a function that has a different value at every point in space. Technically speaking, there are two popular types of fields: scalar field, and vector field. Scalar field1. A scalar field is a function associates a scalar value to every point in space. In principle, a scalar field provides values not only on a two-dimensional surface in space but for every point in space. To represent three-dimensional scalar fields, a common practice is to create a three-dimensional atmospheric volume element and colour it to represent the temperature variation.2. One example of a scalar field is this expression, and it defines the value of the scalar function $\phi$ at every point $(x,y,z)$ in space.:$$\phi(x,y,z) = \frac{1}{\sqrt{x^2 + y^2 + z^2}} - \frac{1}{\sqrt{x^2 + y^2 - z^2}}$$3. Noted: the function $\phi(x,y,z)$ can be denoted as $\phi(\mathbf{x})$ with $\mathbf{x} = [x_1,x_2,x_3]$. This is the function that maps a space $\mathbb{R}^3$ to a new space $\mathbb{R}$ because for every point $(x,y,z)$ in a 3-D space, there is one point $\phi$ in a 1-D space. In pratice, a scalar-valued function $f(\mathbf{x})$ maps $\mathbb{R}^m → \mathbb{R}$ with $\mathbf{x} = [x_1,...,x_m]$ and $\mathbf{f} = f_1$. $$$$        Figure 1: An example of a scalar field: the night-time temperatures measured by the Thermal Emission Spectrometer instrument on the Mars Global Surveyor (MGS). The various colours on the map represent the surface temperature. This map, however, is limited to representing only the temperature on a two-dimensional surface and thus, it does not show how temperature varies as a function of altitude.4. To visualize a scalar field, we use the following approaches:    * **Contour Maps**: in this approach, we fix one of our independent variables (z, for example) and then show a contour map for the two remaining dimensions, in which the curves represent lines of constant values of our function-under-interest. A series of these maps for various (fixed) values of z will give a feel for the properties of the scalar function.    * **Color Coding**: in this approach, we represent the values of the scalar field is by colour-coding in two dimensions for a fixed value of the third. An example is the night-time temperatures map we see above.    * **Relief Maps**: in this approach, we represent a scalar field by fixing one of the dimensions (z, for example), and then plotting the value of the function as a height versus the remaining spatial coordinates, say x and y.        Figure 2: Methods of visualizing a scalar field: using Contour Maps (a); using Color Coding (b); using Relief Maps (c). Vector field1. Recall that a vector is a quantity that has both a magnitude (scalar) and a direction in space. In practice, we use vectors to physical quantities such as velocity, momentum, acceleration and force, associated with an object. However, when we try to describe a system that consists of a large number of objects, we need to assign a vector to each object.2. As an example, let's consider falling snowflakes whereas snow falls, each snowflake moves in a specific direction. Here, we can assign to each snowflake a velocity vector that characterizes its movement. The falling snowflakes above is an example of a collection of discrete bodies. On the other hand, if we try to analyze the motion of continuous bodies such as fluids or flows, a velocity vector then needs to be assigned to every point in the fluid at any instant in time.3. The result of this assignment is a vector field that represents both a magnitude (scalar) and a direction of every point in space. Thus, the distinction between a vector field and a scalar field is that the former contains information about both the direction and the magnitude at every point in space, while only a single variable is specified for the latter.4. One example of a vector field is this expression:$$\vec{F}(x,y,z) = F_x(x,y,z)\hat{i} + F_y(x,y,z)\hat{j} + F_z(x,y,z)\hat{k} $$where the components are scalar fields.Noted: the function $\vec{F}(x,y,z)$ can be denoted as $\mathbf{F}(\mathbf{x})$ with $\mathbf{x} = [x_1,x_2,x_3]$ and $\mathbf{F} = [f_1,f_2,f_3]$. This is the function that maps a space $\mathbb{R}^3$ to a new space $\mathbb{R^3}$ because for every point $(x,y,z)$ in a 3-D space, there is one point $(x_\hat{i},y_\hat{j},z_\hat{k})$ in a 3-D space. In practice, a vector-valued function $\mathbf{F}$ maps $\mathbb{R}^m → \mathbb{R}^n$ with $\mathbf{x} = [x_1,...,x_n]$ and $\mathbf{F} = [f_1,...,f_n]$. $$$$        Figure 3: An example of a vector field: a scenario of the variation of the jet stream, which is the wind velocity as a function of position. The “streamlines” are formed by joining arrows from head to tail.5. To visualize a vector field, we use the following approaches:    * **Quiver**: in this approach, we put arrows representing the field direction. The direction of the arrow at a given location represents the direction of the vector field at that point. In many cases, we also make the length of the vector proportional to the magnitude of the vector field at that point. However, we also may show only the direction with the vectors (that is make all vectors the same length) and colour-code the arrows according to the magnitude of the vector.    * **Field Line**: in this approach, we first start at any point in space and move a very short distance in the direction of the local vector field, drawing a line as you do so. After that short distance, we stop, find the new direction of the local vector field at the point where we stopped, and begin moving again in that new direction. We continue this process indefinitely. Thereby we construct a line in space that is everywhere tangent to the local vector field.        Figure 4:  Methods of visualizing a vector field: using Quiver (a); using Field Line (b). b) Gradient1. Consider a multivariable scalar function $f(x_1,x_2,x_3,...,x_n)$, a gradient is a vector-valued function that stores all the partial derivative information of this function. $$\nabla f = [\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2},\frac{\partial f}{\partial x_3},...,\frac{\partial f}{\partial x_n} ]$$2. Imagine that we are standing at a point $(x_1,x_2,x_3,...,x_n)$ in the space created by the function $f(x_1,x_2,x_3,...,x_n)$, the gradient $\nabla f$ tells us which direction we should travel to increase or decrease the value of $f(x_1,x_2,x_3,...,x_n)$ most rapidly.3. As mentioned above, the gradient is a vector-valued function which means we can consider it as a vector field and visualize it as such. c) Divergence1. Consider a vector-valued function $\vec{F}(\vec{F_1},\vec{F_2},\vec{F_3},...,\vec{F_n})$ that defines a vector field, a divergence is an operator that takes in the vector-valued function and outputs a scalar-valued function measuring how much the vector field is expanding or contracting at each point.$$\nabla \bullet \vec{F} = \frac{\partial \vec{F_1}}{\partial x_1} + \frac{\partial \vec{F_2}}{\partial x_2} + \frac{\partial \vec{F_3}}{\partial x_3} + ... + \frac{\partial \vec{F_n}}{\partial x_n}$$2. For example, consider the following vector $\vec{F} = (2x−y)\hat{i} + (y^2)\hat{j} = \begin{bmatrix} 2x - y \\ y^2 \end{bmatrix}$, we can compute the divergence of vector $\vec{F}$ as:$$\nabla \bullet \vec{F} = \begin{bmatrix} \frac{\partial}{\partial x} \\ \frac{\partial}{\partial y} \end{bmatrix} \bullet \begin{bmatrix} 2x - y \\ y^2 \end{bmatrix} = 2 + 2y$$3. The result tells us that at each point in the vector field, the expansion or contraction obeys the scalar-valued function $\nabla \bullet \vec{F} = 2 + 2y$.4. The divergence at a point $(x_0, y_0, z_0)$ has the following properties:    * $\nabla \bullet \vec{F} > 0 $: this means at this point, the vector field is flowing outwardly making it less dense (expanding). Thus, the point $(x_0, y_0, z_0)$ can be considered as a source.    * $\nabla \bullet \vec{F} < 0 $: this means at this point, the vector field is flowing inwardly making it denser (contracting). Thus, the point $(x_0, y_0, z_0)$ can be considered as a sink.    * $\nabla \bullet \vec{F} = 0 $: this means at this point, the vector field stays constant. Thus, the point $(x_0, y_0, z_0)$ is a divergence-free.            Figure 5: Visualization of a divergence evaluated at a designated point: the value of the divergence indicates whether the vector field is expanding, contracting or remains constant. d) Curl1. Consider a vector-valued function $\vec{F}(\vec{F_1},\vec{F_2},\vec{F_3},...,\vec{F_n})$ that defines a vector field, a **curl** is an operator that takes in the vector-valued function and outputs a scalar-valued function measuring the rotation of the vector field at each point.2. As an example, let's consider a 2-D and a 3-D vector fields, respectively: $\vec{v} = v_1(x,y)\hat{i} + v_2(x,y)\hat{j}$ and$\vec{w} = w_1(x,y,z)\hat{i} + w_2(x,y)\hat{j} + w_3\hat(k)$. We compute curl of these vector fields as follows:$$\nabla \times \vec{v} = det(\begin{bmatrix} i & j \\ \frac{d}{dx} & \frac{d}{dy} \\ v_1 & v_2 \end{bmatrix}) =\frac{\partial v_2}{\partial x} - \frac{\partial v_1}{\partial y} $$$$\nabla \times \vec{w} = det(\begin{bmatrix} i & j & k \\ \frac{d}{dx} & \frac{d}{dy} & \frac{d}{dz} \\ w_1 & w_2 & w_3 \end{bmatrix}) = (\frac{\partial w_3}{\partial y} - \frac{\partial w_2}{\partial z})\hat{i} + (\frac{\partial w_1}{\partial z} - \frac{\partial w_3}{\partial x})\hat{j} + (\frac{\partial w_2}{\partial x} - \frac{\partial w_1}{\partial y})\hat{k}$$3. The curl at a point $(x_0, y_0, z_0)$ has the following properties:* $\nabla \times \vec{F} > 0 $: this means at this point, the vector field is flowing counter-clockwise (right-hand curl).* $\nabla \times \vec{F} < 0 $: this means at this point, the vector field is flowing clockwise (left-hand curl).* $\nabla \times \vec{F} = 0 $: this means at this point, the vector field is irrotational.        Figure 6: Visualization of a curl evaluated at a designated point: the value of the curl indicates whether the vector field is right-hand curl (RHC), left-hand curl (LHC) or irrotational.<jupyter_code>from sympy import *
x,y = symbols('x y')

##
# DEMO: VISUALIZING A SCALAR FIELD
#

f = 2*x**3 + 3*y**3
plot3d(f) # We use this function to create a 3-D plot

x1 = np.linspace(-10,10,1000)
x2 = np.linspace(-10,10,1000)
X1, X2 = np.meshgrid(x1,x2)
Y = 2*X1**3 + 3*X2**3 

fig, axs = plt.subplots(1,3, figsize = (10,6)) # We use this function to create a 2x2 subplots
###
cm = plt.cm.get_cmap('viridis')
a = axs[0].scatter(X1,X2, c = Y, cmap = cm) # We use this function to create a heatmap plot
###
b = axs[1].contour(X1,X2,Y) # We use this function to create a contour plot
plt.clabel(b, inline = 1, fontsize = 10)
###
c = axs[2].contourf(X1,X2,Y) #We use this function to create a color coding plot
plt.clabel(c, inline = 1, fontsize = 10)
###
for ax in axs.flat:
    ax.set(xlabel = 'x', ylabel = 'y')
###
fig.tight_layout(pad = 3.0)
plt.show()

##
# DEMO: FINDING A GRADIENT OF A SCALAR FIELD
#
from sympy.vector import CoordSys3D, Del
R = CoordSys3D("R") # We use this function to define a frame R.
delop = Del() # We use this function to define the nabla operator
###
f = 1 / (R.x ** 2 + R.y ** 2 + R.z ** 2) - 1 / (R.x ** 2 + R.y ** 2 - R.z ** 2)
res = delop.gradient(f, doit = True) # We use this function to apply the nabla operator
print(res)

##
# DEMO: FINDING A GRADIENT OF A SCALAR FIELD EVALUATED AT A POINT (x_0,y_0)
#
from sympy import symbols, Matrix
x, y, z = symbols('x y z')
f = 1/(x**2 + y**2 + z**2) - 1/(x**2 + y**2 - z**2)
###
dx = diff(f,x)
dy = diff(f,y)
grad_f = Matrix([dx, dy])<jupyter_output><empty_output><jupyter_text>2. Jacobian1. Consider a scalar-valued function $f(\mathbf{x})$ that maps a space $\mathbb{R}^n$ to a new one $\mathbb{R}$ where $\mathbf{x} = x_1,x_2,..,x_n>$, we can say the gradient is a transformation of $f(\mathbf{x})$:$$\nabla f(\mathbf{x}) = \frac{\partial f(\mathbf{x})}{\partial x_1} + \frac{\partial f(\mathbf{x})}{\partial x_2} + ... + \frac{\partial f(\mathbf{x})}{\partial x_n}$$2. From this knowledge, let's consider a vector-valued function $\mathbf{f}(\mathbf{x})$ that maps the space $\mathbb{R}^n$ to a new space $\mathbb{R^m}$, where $\mathbf{f} = [f_1,f_2,...,f_m]$ and $\mathbf{x} = [x_1, x_2,...,x_3]$. The so-called Jacobian is a transformation of $\mathbf{f}(\mathbf{x})$: $$\mathbb{J}_f = \frac{\partial (f_1, f_2,..., f_m)}{\partial (x_1,x_2,...,x_n)} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & ... & \frac{\partial f_1}{\partial x_m} \\ ... & ... & ... \\ \frac{\partial f_m}{\partial x_1} & ... & \frac{\partial f_m}{\partial x_n} \end{bmatrix}$$3. In other words, the Jacobian matrix is the generalization of a gradient. The Jacobian matrix is the same as the gradient when $m = 1$.4. The determinant of the Jacobian matrix is typically used to describe changes of variables because it can be viewed as the ratio of an infinitesimal change in the variables of one coordinate system to another. To do so, the vector-valued function $\mathbf{f}(\mathbf{x})$ must map the space $\mathbb{R}^n$ to a new space $\mathbb{R^n}$, which means no change in the dimension. 3. Hessian1. Consider the same scalar-valued function $f(\mathbf{x})$ that maps a space $\mathbb{R}^n$ to a new one $\mathbb{R}$ where $\mathbf{x} = x_1,x_2,..,x_n>$, we now take the gradient of its gradient:$$\nabla \bullet (\nabla f(\mathbf{x})) = \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_1} + \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_2} + ... + \frac{\partial^2 f(\mathbf{x})}{\partial^2 x_n}$$2. From this knowledge, let's consider a vector-valued function $\mathbf{f}(\mathbf{x})$ that maps the space $\mathbb{R}^n$ to a new space $\mathbb{R^m}$, where $\mathbf{f} = [f_1,f_2,...,f_m]$ and $\mathbf{x} = [x_1, x_2,...,x_3]$. The so-called Hessian is just the second-derivative of the Jacobian matrix: $$\mathbb{H}_f = \nabla \bullet \frac{\partial (f_1, f_2,..., f_m)}{\partial (x_1,x_2,...,x_n)} = \begin{bmatrix} \frac{\partial^2 f_1}{\partial^2 x_1} & \frac{\partial^2 f_1}{\partial x_1 \partial x_2} & ... & \frac{\partial^2 f_1}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f_2}{\partial x_2 \partial x_1} & \frac{\partial^2 f_2}{\partial^2 x_2} & ... & \frac{\partial f_2}{\partial x_2 \partial x_n} \\ \frac{\partial^2 f_m}{\partial x_n \partial x_1} & \frac{\partial^2 f_m}{\partial x_m \partial x_2} & ... & \frac{\partial^2 f_m}{\partial^2 x_n}\end{bmatrix}$$ 4. Gradient Descent1. So why do we have to care about Gradient, Jacobian and Hessian? Because it is the backbone of the gradient descent algorithm used in machine learning.2. Considering a simple machine learning that takes many input variables $X = [x_1, x_2, x_3,...,x_n]$ (our data) and outputs a set of decisions $Y = [y_1, y_2, y_3,...,y_n]$, we define the loss function $\mathbb{L}$ as the difference between the model output and the ground truth $\hat{Y} = [\hat{y}_1, \hat{y}_2, \hat{y}_3,...,\hat{y}_n]$.  $$\mathbb{L} = |\hat{Y} - Y|$$3. We also know the relationship between the model output and the input variables can be simply described as:$$Y = W\bullet X + B = [w_1x_1 + b_1, w_2x_2 + b_2,...,w_nx_n + b_n]$$4. To make our model feasible, we need to optimize our loss function $\mathbb{L}$ by finding its global minimum where we are certain that our weights $W$ are the most optimal.![SegmentLocal](images/loss-function.gif)Figure 1: Visualization of our multivariate loss function on the input space.5. To do so, we first compute the gradient of our loss function because the gradient gives us the direction of the steepest increase.![SegmentLocal](images/gradient.gif)Figure 2: Visualization of the gradient of our loss function. As mentioned above, the gradient gives us the direction of the steepest increase, which is illustrated here as a ball rolling upwards from the global minimum.6. By taking the negative of the gradient, we now know the direction of the steepest decrease.![SegmentLocal](images/gradient-descent.gif)Figure 2: Visualization of the gradient descent. Having armed with the knowledge of which direction to go, we can optimize our loss function. After a set of periods of training usually referred as epochs, we manage to find where our weights are most optimal.<jupyter_code>##
# DEMO: FINDING THE JACOBIAN OF A VECTOR FUNCTION
#
from sympy import symbols, Matrix, Function, exp

x1, x2 = symbols('x1 x2')
f = symbols('f', cls = Function)

X = Matrix([x1,x2])
f = Matrix([-x1*x2*exp(-(x1**2 + x2**2)/2)])

f.jacobian(X)
##
# DEMO: FINDING THE HESSIAN OF A VECTOR FUNCTION
#
from sympy import hessian
hessianf = hessian(f,X)
hessianf<jupyter_output><empty_output><jupyter_text>IV. Integrals  1. What is an integral?1. An integral is an opposite of a derivate: a derivate finds a slope of a function's curve, while an integral finds an area under a function's curve.2. For example, we have a function $f(x) = x^2$, and we want to find the area between $x =0$ and $x = 2$ for this function. We can find the area by using a method called **Riemann Sum**.**** From the interval [0,2], we partitions it into a $n$ number of subintervals, each has a width of: $\Delta x = \frac{2 - 0}{n}$.* The points of in the partition will then be: $0, 0 + \Delta x, 0 + 2\Delta x, ..., 0 + (n-2)\Delta x, 0 + (n-1)\Delta x, 2$* We then compute the area under the curve:$$A = \Delta x[f(0) + f(0 + \Delta x) + f(0 + 2\Delta x) + ... f(2 - \Delta x) + f(2)]$$***        Figure 1: Visualization of the Riemann Sum for $f(x) = x^2$: to compute the area under a function's curve from point $(x_0,y_0)$ to $(x_k,y_k)$. we pack some rectangles with equal width under the curve and sum their areas. This method is extremely powerful for integrals we cannot analytically solve.3. In practice, we compute the area under a function's curve by taking a definite integral from point $a$ to point $b$:$$A = \int_a^b f(x)dx $$4. Let's move back to the Riemann Sum, there are three types of Riemann sum:    * In a **Left Riemann Sum**, we approximate the area using rectangles (usually of equal width), where the height of each rectangle is equal to the value of the function at the left endpoint of its base.    * In a **Right Riemann Sum**, we approximate the area using rectangles (usually of equal width), where the height of each rectangle is equal to the value of the function at the right endpoint of its base.    * In a **Midpoint Riemann Sum**, the height of each rectangle is equal to the value of the function at the midpoint of its base.    * In a **Trapezoidal Riemann Sum** also known as the **Trapezoidal Rule**, the area under the curve is the average of the Left Sum and the Right Sum.        Figure 2: Types of Riemann Sum: Left Riemann Sum (a); Right Riemann Sum (b); Trapezoidal Riemann Sum (c). The Midpoint Riemann Sum is not shown here.<jupyter_code>##
# DEMO: COMPUTE THE DEFINITE INTEGRAL:
#
from sympy import integrate, symbols, sin
x = symbols('x')
f = sin(x)
integrate(f,(x,0,np.pi))
##
# DEMO: COMPUTE THE RIEMANN SUM USING TRAPEZOIDAL RULE
#
def trapezoidal_riemann(a,b, n,*args):
    h = (b - a) / (n-1)
    x = np.linspace(a,b,n)
    
    f = np.sin(x)
    
    A = (h/2) * (f[0] + 2 * sum(f[1: n-1] + f[n-1]))
    
    return A

trapezoidal_riemann(a = 0, b = np.pi, n = 1000)<jupyter_output><empty_output><jupyter_text>2. Applications of Integrals1. One application of integrals is computing the finding areas under probability distributions to get probabilities. For example, we want to compute the probability that any given golden retriever’s weight is between 61 and 62 pounds, and we know that the population mean is 64.43 pounds and the population standard deviation is 2.99 pounds.        Figure 1: An example of using integrals: computing the area under the curve of a probability density function.<jupyter_code>##
# DEMO: COMPUTE THE PROBABILITY DENSITY FUNCTION
#
import math

def normal_pdf(x: float, mean: float, std_dev: float) -> float:
    return (1.0 / (2.0 * math.pi * std_dev ** 2) ** 0.5) * math.exp(-1.0 * ((x - mean) ** 2 / (2.0 * std_dev ** 2)))

def approximate_integral(a, b, n, f):
    delta_x = (b - a) / (n - 1)
    
    total_sum = 0
    for i in range(1, n + 1):
        midpoint = 0.5 * (2 * a + delta_x * (2 * i - 1))
        total_sum += f(midpoint)
        
    return total_sum * delta_x

p_between_61_and_62 = approximate_integral(a = 61, b = 62, n = 1000, f = lambda x: normal_pdf(x,64.43,2.99))
print(p_between_61_and_62)<jupyter_output>0.08263012897923018
{'fraction_non_alphanumeric': 0.1234890558640967, 'fraction_numerical': 0.022505354096337436, 'mean_word_length': 4.164979377577803, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '307987', 'license': 'Apache-2.0', 'n_tokens_mistral': 8911, 'n_tokens_neox': 8396, 'n_words': 3836, 'path': 'Multivariate Calculus/Multivariate Calculus - Lecture Note.ipynb', 'repo_name': 'lesonquan/Machine-Learning-Notes', 'size': '27686'}