<jupyter_start><jupyter_text>**Concepts Covered:**

- Matrix Operations (Linear Algebra)
- Functions and Derivatives
- Optimization: Gradient Descent
- Introduction to TensorFlow and Keras  **Matrix Operations (Linear Algebra)**<jupyter_code>import numpy as np     # import numpy library<jupyter_output><empty_output><jupyter_text>Now, let's create a matrix using functions: arange() and reshape()<jupyter_code>M = np.arange(12).reshape(4,3)
M<jupyter_output><empty_output><jupyter_text>Sometimes, we need to flip the axes/dimensions of a matrix. When we interchange a matrix’s rows and columns, the result is called the **transpose** of the matrix.<jupyter_code>M.T       # taking transpose of the matrix<jupyter_output><empty_output><jupyter_text>You can read more about the use of numpy for matrix operations and other mathematical operations in general using this link: [NumPy basics](https://numpy.org/doc/stable/user/basics.html)  Decomposition of Matrix:- Eigendecomposition: We can use a built-in function in numpy "numpy.linalg.eig" for computing the eigenvalues.<jupyter_code>import numpy as np

p, v = np.linalg.eig(np.array([[4, 2], [2, 5]]))
print(p)
print(v)<jupyter_output>[2.43844719 6.56155281]
[[-0.78820544 -0.61541221]
 [ 0.61541221 -0.78820544]]<jupyter_text>Note:- NumPy normalizes the eigenvectors to unit length, whereas we took ours to be of arbitrary length.  Norm:- In code, we can calculate the L2 norm of a vector as follows.<jupyter_code>a = np.array([-3, 4])
np.linalg.norm(a)<jupyter_output><empty_output><jupyter_text>- To calculate the L1 norm, we compose the absolute value function with a sum over the elements.<jupyter_code>np.abs(a).sum()
b = np.ones((4,4))
b
np.linalg.norm(b)<jupyter_output><empty_output><jupyter_text>Rank:<jupyter_code>mat1 = np.arange(1,7).reshape(2,3)
mat1
np.linalg.matrix_rank(mat1)
mat2 = np.ones((4,))
mat2
np.linalg.matrix_rank(mat2) # 1 dimension - rank 1 unless all 0
mat3 = np.zeros((4,))
mat3
np.linalg.matrix_rank(mat3)<jupyter_output><empty_output><jupyter_text>**Functions and Derivatives** These are the most widely used function in Deep Learning:- Linear: `y = w*x + b`- Exponential: `y = e^x` or `y = np.exp(x)`- Logarithm: `y = log(x)` or `y = np.log(x)`- Power: `y = x**4` or `y = np.power(x, 4)`- Modulus: `y = |x|` or `y = np.absolute(x)`- Trigonometric: `y = cos(x)` or `y = np.cos(x)`  Let's define a composite function and visualize that:<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

def composite_function(x):
  sns.set({'figure.figsize': (6,6)})
  plt.xlabel('x')
  plt.ylabel('z')
  y = x**x
  z = np.sin(y)
  return plt.plot(x, z);

x = np.arange(0.01, 3.01, 0.01)
composite_function(x)<jupyter_output><empty_output><jupyter_text>**Optimization: Gradient Descent** - For simplicity we choose the objective function  `y = x``2` to illustrate how to implement gradient descent.- Although we know that  `x = 0`  is the solution to minimize y, we still use this simple function to observe how  `x`  changes.<jupyter_code>y = lambda x: x**2                  # Objective function
gradient_y = lambda x: 2 * x        # Its derivative<jupyter_output><empty_output><jupyter_text>- Next, we use `x = 8` as the initial value and assume `η = 0.1`.- Using gradient descent to iterate `x` for 10 times we can see that, eventually, the value of `x` approaches the optimal solution.<jupyter_code>def gradient(epoch, learning_rate):
    x = 10
    results = [x]
    for i in range(epoch):
        x = x - learning_rate * gradient_y(x)
        results.append(float(x))
        print('Epoch %d: %f' %(epoch+1, x))
    return results<jupyter_output><empty_output><jupyter_text>**Terminology:**

- **Learning Rate:** In machine learning, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.

It metaphorically represents the speed at which a machine learning model "learns".

In this case, the learning rate determines the speed at which we reach the minima of the function.

So, here is the thing with higher values and lower values of learning rate:
- **Higher Learning Rate -** With a high learning rate we can cover more ground each step, but we risk overshooting the lowest point since the slope of the hill is constantly changing.
- **Low Learning Rate -** With a very low learning rate, we can confidently move in the direction of the negative gradient since we are recalculating it so frequently. A low learning rate is more precise, but calculating the gradient is time-consuming, so it will take us a very long time to get to the bottom.<jupyter_code>epoch1 = 5
learning_rate1 = 0.1
result1 = gradient(epoch1, learning_rate1)
# show_trajectory(result1, epoch1, learning_rate1)
epoch2 = 5
learning_rate2 = 0.2
result2 = gradient(epoch2, learning_rate2)
# show_trajectory(result2, epoch2, learning_rate2)
epoch3 = 20
learning_rate3 = 0.1
result3 = gradient(epoch3, learning_rate3)
# show_trajectory(result3, epoch3, learning_rate3)
def show_trajectory(result, epoch, learning_rate, ax=None, **kwargs):
    n = max(abs(min(result)), abs(max(result)))
    ax = ax
    domain = np.arange(-n, n, 0.01)
    sns.set({'figure.figsize': (8,8)})
    ax.plot(domain, [x**2 for x in domain], '--r', **kwargs)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_title("For Epoch = {}, learning rate = {}".format(epoch, learning_rate)) 
    ax.plot(result, [x**2 for x in result], '-o', **kwargs)
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18,6))
show_trajectory(result1, epoch1, learning_rate1, ax1)
show_trajectory(result2, epoch2, learning_rate2, ax2)
show_trajectory(result3, epoch3, learning_rate3, ax3)<jupyter_output><empty_output><jupyter_text>Insights:- Here you can see how we achieve the minimum value using different learning rates and epochs in gradient descent.- First, we initialize the minimum value randomly as 10.- As we iterate the gradient descent, we reach nearer to the minima.  **Introduction to TensorFlow and Keras**  Linear Regression using TensorFlow (To become familiar with the TensorFlow framework by implementing some trivial tasks)<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing

print(tf.__version__)
tf.random.set_seed(1)<jupyter_output><empty_output><jupyter_text>Now, let's creat a dataset for further demonstration of techniques.<jupyter_code>x = np.array([50, 80, 100 , 30, 50, 20, 90, 10, 20, 150, 180, 45, 95, 90, 70, 120],  dtype=float)
y = np.array([150, 200, 240, 100, 140, 80, 220, 60, 100, 490, 450, 180, 250, 200, 300, 400],  dtype=float)
plt.scatter(x, y)
plt.xlim(0,200)
plt.ylim(0,500)
plt.xlabel('X')
plt.ylabel('Y')
plt.show()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=42)
model = tf.keras.Sequential([
  tf.keras.layers.Dense(units=1, input_shape=[1])
])<jupyter_output><empty_output><jupyter_text>**Terminology:**

- **Sequential:** A Sequential model is appropriate for a plain stack of layers where each layer has exactly one input tensor and one output tensor.

    If you want to know when to use a Sequential model, read [here](https://www.tensorflow.org/guide/keras/sequential_modelwhen_to_use_a_sequential_model)
- **Dense:** Just your regular densely-connected NN layer.

**Note: If you don't understand much about these terms used here, don't worry about it. By the time, you'll get familiar with these terms and their use.**<jupyter_code>model.compile(loss='mean_squared_error',
              optimizer=tf.keras.optimizers.Adam(0.01))<jupyter_output><empty_output><jupyter_text>**Terminology:**

- **Adam:** Adam is an alternative optimization algorithm that provides more efficient neural network weights by running repeated cycles of "adaptive moment estimation."
    - The number inside the paranthesis of the Adam is the learning rate.<jupyter_code>history = model.fit(X_train, y_train, epochs=1000, verbose=False, validation_split=0.1)
print("Finished training the model")<jupyter_output><empty_output><jupyter_text>**Terminology:**

- **Epoch:** In terms of artificial neural networks, an epoch refers to one cycle through the full training dataset. Usually, training a neural network takes more than a few epochs. In other words, if we feed a neural network the training data for more than one epoch in different patterns, we hope for a better generalization when given a new "unseen" input (test data).<jupyter_code>sns.set({'figure.figsize': (8,8)})
plt.xlabel('Epoch Number')
plt.ylabel("Loss Magnitude")
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend(['train', 'val'], loc='upper left')<jupyter_output><empty_output><jupyter_text>Insights:

- The x-axis represents the epochs (number of iterations) and the y-axis represents the Loss magnitude.
- We see the loss is decreasing as the number of epochs increase.
- It means as we train more, the model becomes more closer to give good predictions. The error between the actual and predicted values reduces.  Key Takeaways:

- After a certain number of epochs, the loss becomes constant and there is no change. It means that we reached saturation.
- As we reached saturation around 300 epochs, there is no need of going for further iterations, as this is the waste of time/resources. [Computation time plays a vital role while you build something at large scale]
- So, TensorFlow provides some functions which take care of the above problem. It stops the training if the loss reaches a certain value and it's not changing significantly in the next iterations. It stops automatically according to the threshold you set.<jupyter_code>y_pred = model.predict(X_test)
print('Actual Values', y_test)
print('Predicted Values', y_pred.reshape(1,-1))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10758675549368599, 'fraction_numerical': 0.028537337178084914, 'mean_word_length': 5.147921760391198, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 68, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '643109', 'license': 'Apache-2.0', 'n_tokens_mistral': 3411, 'n_tokens_neox': 3191, 'n_words': 1146, 'path': 'neural_networks/prework/Hands_on_Pre_work_Deep_Learning.ipynb', 'repo_name': 'vasantbala/vb_ai_course', 'size': '10057'}