<jupyter_start><jupyter_text>Training a Regression Neural Network Model with Keras in TensorflowIn this exercise, we will show you how to train an artificial neural network using Keras in Tensorflow. We will implement the same artificial neural network you have just seen: a neural network for regression with one hidden layer, and trained using gradient descent.For more information about Keras, consult the [online documentation](https://keras.io/api/).Tensorflow is a general purpose high-performance computing language which Keras is built on top of to allow us to easily define and train neural networks.To get started, let us import Keras, along with other libraries.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import time
import tensorflow.keras as keras<jupyter_output><empty_output><jupyter_text>Create and Visualize a Data SetWe will create some data to use for training. Notice the training examples contained in `X` are two dimensional feature vectors.<jupyter_code># Generate the raw data
x = np.arange(0, 5, 0.1)
y = x ** 2 + 10*np.sin(x)

# Add a bias column vector of 1's to the vector x to get our design matrix X
X = np.concatenate([x, np.ones(x.shape)]).reshape(2, -1).T 
print(X.shape)
print(y)

# Visualize the data
plt.plot(X[:, 0], y, '*')
plt.xlabel('x')
plt.ylabel('y')
plt.show()<jupyter_output>(50, 2)
[ 0.          1.00833417  2.02669331  3.04520207  4.05418342  5.04425539
  6.00642473  6.93217687  7.81356091  8.6432691   9.41470985 10.1220736
 10.76039086 11.32558185 11.8144973  12.22494987 12.55573603 12.8066481
 12.97847631 13.07300088 13.09297427 13.04209367 12.92496404 12.74705212
 12.51463181 12.23472144 11.91501372 11.5637988  11.1898815  10.80249329
 10.41120008 10.02580662  9.65625857  9.31254306  9.00458898  8.74216772
  8.53479557  8.39163859  8.32142109  8.33233841  8.43197505  8.62722889
  8.92424228  9.32834063  9.84397926 10.47469882 11.22308996 12.09076742
 13.07835391 14.18547387]<jupyter_text>The steps to define and train a neural network using Keras are:1. Define the model structure.2. Define the optimization function, i.e., the variant of gradient descent you want to use.3. Define the loss function to minimize.4. "Fit" the model on training data for $M$ epochs.The "fitting" process propagates the data forward in the network, computes the loss value, computes the gradients for the weights, and updates the weights for $M$ times. We will walk you through those steps:  Step 1. Define Model Structure We will use the Keras [Sequential class](https://keras.io/api/models/sequential/) to group a linear stack of layers layers. We will use the [Dense class](https://keras.io/api/layers/core_layers/dense/) to create each layer. `model.summary()` is an incredibly useful tool. It shows the number of parameters per layer.<jupyter_code>model = keras.Sequential()

hidden_layer_1 = keras.layers.Dense(units=200, input_shape=(2,), activation='relu',
                                    use_bias=False, kernel_initializer='random_normal')
model.add(hidden_layer_1)

output_layer = keras.layers.Dense(units=1, activation=None, 
                                  use_bias=False, kernel_initializer='random_normal')
model.add(output_layer)

model.summary()<jupyter_output>Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_6 (Dense)              (None, 200)               400       
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 200       
=================================================================
Total params: 600
Trainable params: 600
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Let's break this down. Our network should comprise of 3 layers: an input layer, one hidden layer and an output layer.The code above creates the hidden and output layers.Hidden layerWe pass the following arguments to the Dense function:`units`: dimensionality of the output space. We can view this as the number of nodes in the layer itself.`activation`: Transition (also called activation) function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).`use_bias`: whether the layer uses a bias vector.`kernel_initializer`: Initializer for the weights matrix.`input_shape`: dimensionality of the input layer.- Using `Dense` with an input shape of 2, we are specifying the input layer's dimensions. The input layer should have 2 dimensions to match the feature dimension of each of the 50 training data points in $\mathbf{X}$.- Using `Dense` with 200 units, we specify a fully-connected layer with 200 hidden units. This is equivalent to multiplying the 2-dimensional input with $\mathbf{W}_1$ of shape $2 \times 200$. We also specify the `ReLU` transtion (also called activation) function.Output Layer:- The output layer is a single dense layer. Since our model will predict one value, we specify 1 unit. This is equivalent to multiplying the 200-dimensional hidden layer vector with $\mathbf{W}_2$ of shape $200 \times 1$. We initialize the weights of the network with random normal values. Since $\mathbf{X}$ already contains a bias value, we avoid bias terms in the network.  Step 2. Define the Optimization Function<jupyter_code>lr = 0.001 # specify learning rate

optimizer = keras.optimizers.SGD(learning_rate=lr)
training_batch_size = len(y)<jupyter_output><empty_output><jupyter_text>Keras does not provide vanilla gradient descent. Instead, the basic variant it provides is Stochastic Gradient Descent (SGD). For more information, see the [SGD online documentation](https://keras.io/api/optimizers/sgd/). We can use SGD to behave like vanilla gradient descent by setting the `training_batch_size` to the number of data points.What does this do? Although the loss is calculated using all data points, SGD computes the gradients of weights using `training_batch_size` number of data points. So, when the `training_batch_size` is smaller than the number of training data points, SGD _approximates_ the gradients of weights (this is done for various reasons, but we will not discuss it here). To compute exact gradients we just set the `training_batch_size` to number of training points `len(y)`.   Step 3. Define the loss functionSince we are constructing a neural network for regression, we will use the mean squared error as our loss function to minimize.<jupyter_code>loss = keras.losses.MeanSquaredError()<jupyter_output><empty_output><jupyter_text>Step 4. Compile the modelAttach the optimizer and loss function to the model.<jupyter_code>model.compile(optimizer=optimizer,loss=loss)<jupyter_output><empty_output><jupyter_text>Step 5. Fit the Model on Training Data<jupyter_code>t0 = time.time() # start time

M = 1000 # number of epochs to train

# Fit the model on training data
history = model.fit(
    X,
    y,
    epochs=M,
    batch_size= training_batch_size,
    verbose=0, # no progress bar. You can comment out this line to see the loss per epoch
)

t1 = time.time() # stop time

print('Elapsed time: %.2fs' % (t1-t0))<jupyter_output>Elapsed time: 1.82s<jupyter_text>It is possible that the Keras model takes much longer to train than a version built from scratch with basic Numpy. This is because of the massive overhead required to setup and train the model. However, as the number of training points increases and the network becomes more complex, Keras (and other Machine Learning libraries) scale much better than pure-Numpy implementations---even better when you use a GPU. For this reason, developers and researchers almost always use libraries like Tensorflow and Keras to train networks.  Visualize the Results<jupyter_code>outputs = [layer.output for layer in model.layers]                  # all layer outputs
get_activations = keras.backend.function(model.input, outputs)   # evaluation function
    
Z = [X] + get_activations([X]) # predicted values for raw data x are in the first column of Z

losses = history.history['loss']
average = sum(losses)/M
print('Average loss for ' + str(M) + ' epochs = ' + str(average))

fig, axarr = plt.subplots(1, 2)
fig.set_figwidth(15)
fig.set_figheight(6)

axarr[0].plot(x, y)
axarr[0].plot(x, Z[-1].flatten())
axarr[0].set_ylabel('$f(x)$',size = 20)
axarr[0].set_xlabel('$x$', size = 20)
axarr[0].legend(['Actual', 'Predicted'], fontsize = 20)

axarr[1].semilogy(losses)
axarr[1].set_title('Loss', size = 20)
axarr[1].set_xlabel('Epoch', size = 20)

plt.show()<jupyter_output>Average loss for 1000 epochs = 12.554273083686828
{'fraction_non_alphanumeric': 0.09545663148233134, 'fraction_numerical': 0.06883891693437356, 'mean_word_length': 4.594993581514762, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '103803', 'license': 'Apache-2.0', 'n_tokens_mistral': 2912, 'n_tokens_neox': 2469, 'n_words': 1004, 'path': 'ANNKeras.ipynb', 'repo_name': 'pujaroy280/AI-Foundations', 'size': '8716'}