<jupyter_start><jupyter_text>Introduction to Regression with Neural Networks in TensorFlowThere are many definitions for a regression problem but in our case, we're going to simplify it: predicting a numerical variable based on some other combination of variables, even shorter... predicting a number.<jupyter_code># Import TensorFlow
import tensorflow as tf
print(tf.__version__)<jupyter_output>2.4.1<jupyter_text>Creating data to view and fit<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

# Create features
X = np.array([-7.0,-4.0,-1.0,2.0,5.0,8.0,11.0,14.0])

# Create labels
y = np.array([3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0])

# Visualize it
plt.scatter(X,y);
y == X + 10<jupyter_output><empty_output><jupyter_text>Input and output shapes<jupyter_code># Create a demo tensor of our housing price prediction problem
house_info = tf.constant(["bedroom","bathroom","garage"])
house_price = tf.constant([939700])
house_info, house_price
X[0],y[0]
X[1],y[1]
input_shape = X[0].shape
output_shape = y[0].shape
input_shape, output_shape
X[0].ndim
X[0],y[0]
# Turn our NumPy arrays into tensors with dtype float32
X = tf.cast(tf.constant(X),dtype=tf.float32)
y = tf.cast(tf.constant(y),dtype=tf.float32)
X,y
input_shape = X[0].shape
output_shape = y[0].shape
input_shape,output_shape
plt.scatter(X,y);<jupyter_output><empty_output><jupyter_text>Steps in modeling with TensorFlow1. **Creating a model** - define the input and output layers, as well as the hidden layers of a deep learning model.2. **Compiling a model** - define the loss function (in other words, the function which tells out model how wronf it is) and the optimizer (tells our model how to improve the patterns it is learning) and evalution metrics (what we can use to interpret the performance of our model).3. **Fitting a model** - letting the model try to find patterns between X & y (features and labels).<jupyter_code># Set random seed
tf.random.set_seed(42)

# 1. Create a model using the Sequential API
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1)
])

# 2. Compile the model
model.compile(loss=tf.keras.losses.mae, # mae is short for mean absolute error
              optimizer=tf.keras.optimizers.SGD(), # SGD short for stochastic gradient descent
              metrics=["mae"])

# 3. Fit the model
model.fit(X,y,epochs=5)
# Check out X and y
X,y
# Try and make a prediction using our model
y_pred = model.predict([17.0])
y_pred
y_pred + 11<jupyter_output><empty_output><jupyter_text>Improving our modelWe can improve our model, by altering the steps we took to create a model.1. **Creating a model** - here we ,ight add more layers, increase the number of hidden units (all called neurons) within each of the hidden layers, change the activation function of each layer.2. **Compiling a model** - here we might change the optimization function or perhaps the **learning rate** of the optimization function.3. **Fitting a model** - here we might fit a model for more **epochs** (leave it training for longer) or on more data (give the model more examples to learn from).<jupyter_code># Let's rebuild our model

# 1. Create the model
model = tf.keras.Sequential([
  tf.keras.layers.Dense(100,activation="relu"),
  tf.keras.layers.Dense(100,activation="relu"),
  tf.keras.layers.Dense(100,activation="relu"),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model.compile(loss=tf.keras.losses.mae,
              optimizer=tf.keras.optimizers.SGD(),
              metrics=["mae"])

# 3. Fit the model
model.fit(X,y,epochs=100)
X,y
# Let's see if our model prediction has improved..
model.predict([17.0])
# Let's see if we can make another to improve our model

# 1. Create the model
model = tf.keras.Sequential([
  tf.keras.layers.Dense(50,activation=None),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model.compile(loss="mae",
              optimizer=tf.keras.optimizers.Adam(lr=0.01),
              metrics=["mae"])

# 3. Fit the model
model.fit(X,y,epochs=100)
X,y
model.predict([17.0])<jupyter_output><empty_output><jupyter_text>Evaluating a modelIn practice, a typical workflow you'll go through when building neural networks is:```Build a model -> fit it -> evaluate it -> tweak a model -> fit it -> evaluate it -> tweak a model -> evaluate it...``` When it comes to evaluation... there are 3 words you should memorize:> "Visualize, visualize, visualize"It's a good idea to visualize:* The data = what data are we working with? What  does it look like?* The model itself - what does our model look like?* The training of a model - how does a model perform while it learns?* The predictions of the model - how ado the predictions of a model line up against the ground truth (the original labes)?<jupyter_code># Make a bigger dataset
X = tf.range(-100,100,4)
X
# Make labels fot the dataset
y = X + 10
y
# Visualize the data 
import matplotlib.pyplot as plt

plt.scatter(X,y);<jupyter_output><empty_output><jupyter_text>The 3 sets...* **Training set** - the model learns from this data, which is typically 70-80% of the total data you have available.* **Valudation set** - the model gets tuned on this data, which is typically 10-15% of the data available.* **Test set** - the model gets evaluated on this data to test what it has learned, this set is typically 10-15% of the total data available.<jupyter_code># Check the length of how many samples we have
len(X)
# Split the data into train and test sets
X_train = X[:40] # first 40 are training samples (80% of the data)
y_train = y[:40]

X_test = X[40:] # last 10 are testing samples (20% of the data)
y_test = y[40:]

len(X_train), len(X_test)
len(y_train) , len(y_test)<jupyter_output><empty_output><jupyter_text>Visualizing the dataNow we've got data in trining and test sets... let's visualize it again!<jupyter_code>plt.figure(figsize=(10,7))

# Plot training data in blue
plt.scatter(X_train,y_train,c="b",label="Training data")

# Plot test data in green
plt.scatter(X_test,y_test,c="g",label="Testing data")

# Show legend
plt.legend();
# Let's have a look at how to build a neural network for our data

# 1. Create a model
model = tf.keras.Sequential([
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model.compile(loss="mae",
              optimizer=tf.keras.optimizers.SGD(),
              metrics=["mae"])


# 3. Fit the model
# model.fit(X_train,y_train,epochs=100)
#model.summary() #Error
"""
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-5f15418b3570> in <module>()
----> 1 model.summary()

/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py in summary(self, line_length, positions, print_fn)
   2374     """"""
   2375     if not self.built:
-> 2376       raise ValueError('This model has not yet been built. '
   2377                        'Build the model first by calling `build()` or calling '
   2378                        '`fit()` with some data, or specify '

ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
"""
X[0],y[0]
# Let's create a model which builds automatically by defining the input_shape argument
tf.random.set_seed(42)

# 1. Create a model (same as above)
model = tf.keras.Sequential([
  tf.keras.layers.Dense(10,input_shape=[1],name="input_layer"),
  tf.keras.layers.Dense(1,name="output_layer")
],name="model_1")

# 2. Compile the model
model.compile(loss="mae",
              optimizer=tf.keras.optimizers.SGD(),
              metrics=["mae"])
model.summary()<jupyter_output>Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_layer (Dense)          (None, 10)                20        
_________________________________________________________________
output_layer (Dense)         (None, 1)                 11        
=================================================================
Total params: 31
Trainable params: 31
Non-trainable params: 0
_________________________________________________________________<jupyter_text>* Total parameters - total number of parameters in the model* Trainable parameters - these are the parameters (patterns) the model can update as it trains* Non-trainable parameters - these parametees aren't updated during training (this is typical when you bring in already lean patterns or parameters from other models during **transfer learning**)<jupyter_code># Let's fit our model to the training data
model.fit(X_train,y_train,epochs=100,verbose=0)
# Get a summary of our model
model.summary()
from tensorflow.keras.utils import plot_model

plot_model(model=model,show_shapes=True)<jupyter_output><empty_output><jupyter_text>Visualizing our model's predictionsTo visualize predictions, it's a good idea to plot them against the ground truth labels.Often you'll see this is the form of `y_test` or `y_true` versus `y_pred` (ground truth versus your model)<jupyter_code># Make some predictions
y_pred = model.predict(X_test)
y_pred
y_test<jupyter_output><empty_output><jupyter_text>🔑 **Note:** If you feel like you're going to reuse some kind of functionality in the future, it's a good idea to turn it into a function.<jupyter_code># Let's create a plotting function
 def plot_predictions(train_data=X_train,
                      train_labels=y_train,
                      test_data=X_test,
                      test_labels=y_test,
                      predictions=y_pred):
   """
   Plots training data, test data and compares predictions to ground truth labels.
   """
   plt.figure(figsize=(10,7))
   # Plot training data in blue
   plt.scatter(train_data,train_labels,c="b",label="Training data")
   # Plot testing data in green
   plt.scatter(test_data,test_labels,c="g",label="Testing data")
   # Plot model's predictions in red
   plt.scatter(test_data,predictions,c="r",label="Predictions")
   # Show the legend
   plt.legend();
plot_predictions(train_data=X_train,train_labels=y_train,test_data=X_test,test_labels=y_test,predictions=y_pred)<jupyter_output><empty_output><jupyter_text>Evaluating our model's predictions with regression evaluation metricsDepending on the problem you're working on, there will be different evaluation metrics to evaluate your model's performance.Since we're working on a regression, two of the main metrics:* MAE - mean absolute error, "on average, how wrong is each of my model's predictions"* MSE - mean square error, "square the average errors"<jupyter_code># Evaluate the model on the test
model.evaluate(X_test,y_test)
# Calculate the mean absolute error
mae = tf.keras.losses.MAE(y_test,tf.constant(y_pred))
mae
tf.constant(y_pred)
y_test
tf.squeeze(y_pred)
# Calculate the mean absolute error
mae = tf.keras.losses.MAE(y_test,tf.squeeze(y_pred))
mae
# Calculate the mean square error
mse = tf.keras.losses.MSE(y_test,tf.squeeze(y_pred))
mse
# Make some functions to reuse MAE and MSE
def mae(y_true,y_pred):
  return tf.keras.losses.MAE(y_true=y_test,y_pred=tf.squeeze(y_pred))

def mse(y_true,y_pred):
  return tf.keras.losses.MSE(y_true=y_test,y_pred=tf.squeeze(y_pred))<jupyter_output><empty_output><jupyter_text>Running experiments to improve our model```Build a model -> fit it -> evaluate it -> tweak it -> fit it -> evaluate it -> tweak it -> fit it -> evaluate it..```1. Get more data - get more examples fot your model to train on (more opportunities to learn patterns or relationships between features and labels).2. Make your model larger (using a more complex model) - this might come in the form of more layers or more hidden units in each layer.3. Train for longer - give your model more of a chance to find patterns in the data.Let's do 3 modelling experiments:1. `model_1` - same as the original model, 1 layer, trained for 100 epochs.2. `model_2` - 2 layers, trained for 100 epochs3. `model_3` - 2 layers, trained for 500 epochs **Build `model_1`**<jupyter_code># Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_1 = tf.keras.Sequential([
    tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_1.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_1.fit(X_train,y_train,epochs=100)
# Make and plot predictions for model_1
y_preds_1 = model_1.predict(X_test)
plot_predictions(predictions=y_preds_1)
# Calculate model_1 evaluation metrics
mae_1 = mae(y_test,y_preds_1)
mse_1 = mse(y_test,y_preds_1)
mae_1,mse_1<jupyter_output><empty_output><jupyter_text>**Build `model_2`*** 2 dense layers, trained for 100 epochs<jupyter_code># Set random seed
tf.random.set_seed(42)

# Create the model
model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# Compile the model
model_2.compile(loss="mae",
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# Fit the model
model_2.fit(X_train,y_train,epochs=100)
# Make the plot prediction for model_2
y_preds_2 = model_2.predict(X_test)
plot_predictions(predictions=y_preds_2)
# Calculate the model_2 evaluation metrics
mae_2 = mae(y_test,y_preds_2)
mse_2 = mse(y_test,y_preds_2)
mae_2,mse_2<jupyter_output><empty_output><jupyter_text>**Build `model_3`*** 2 layers, trained for 500 epochs<jupyter_code># Set random seed
tf.random.set_seed(42)

# Create the model
model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# Compile the model
model_3.compile(loss="mae",
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# Fit the model
model_3.fit(X_train,y_train,epochs=500)
# Make and plot some predictions
y_preds_3 = model_3.predict(X_test)
plot_predictions(predictions=y_preds_3)
# Calculate model_3 evaluation metrics
mae_3 = mae(y_test,y_preds_3)
mse_3 = mse(y_test,y_preds_3)
mae_3,mse_3<jupyter_output><empty_output><jupyter_text>🔑 **Note:** You want to start with small experiments (small models) and make sure they work and then increase their scale when necessary.  Comparing the results of our experimentsWe've run a few experiments, let's compare he results.<jupyter_code># Let's compare our model's results using a pandas DataFrame
import pandas as pd

model_results = [["model_1",mae_1.numpy(),mse_1.numpy()],
                 ["model_2",mae_2.numpy(),mse_2.numpy()],
                 ["model_3",mae_3.numpy(),mse_3.numpy()]]

all_results = pd.DataFrame(model_results,columns=["model","mae","mse"])
all_results<jupyter_output><empty_output><jupyter_text>Looks like `model_2` performed the best...<jupyter_code>model_2.summary()<jupyter_output>Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_9 (Dense)              (None, 10)                20        
_________________________________________________________________
dense_10 (Dense)             (None, 1)                 11        
=================================================================
Total params: 31
Trainable params: 31
Non-trainable params: 0
_________________________________________________________________<jupyter_text>🔑 **Note:** One of your main goals should be to minimize the time between your experiments. The more experiments you do, the more thing you'll figure out which don't work and in turn, get closer to figure out what does work. Remember the machine learning practioner's motto: "experiment, experiment, experiment".  Tracking your experimentsOne really good habit in machine learning modelling is to track the result of your experiments.And when doing so, it can be tedious if you're running lots of experiments.Luckily, there are tools to help us!**Resources*** TensorBoard* Weights & Biases   Saving our modelsSaving our models allows us to use them outside of Google Colab (or whatever they were trained) such as in a web application or mobile appThere are two main formats we can save our model's too:1. The SavedModel format2. The HDF5 format<jupyter_code># Save model using SavedModel format
model_2.save("best_model_SavedModel_format")
# Save model using HDF5 format
model_2.save("best_model_HDF5_format.h5")<jupyter_output><empty_output><jupyter_text>Loading in a saved model<jupyter_code># Loading in a SavedModel format model
loaded_SavedModel_format = tf.keras.models.load_model("/content/best_model_SavedModel_format")
loaded_SavedModel_format.summary()
# Compare model_2 predictions with SavedModel format model predictions
model_2_preds = model_2.predict(X_test)
loaded_SavedModel_format_preds = loaded_SavedModel_format.predict(X_test)
model_2_preds == loaded_SavedModel_format_preds
# Load in a model using the .h5 format
loaded_h5_model = tf.keras.models.load_model("/content/best_model_HDF5_format.h5")
loaded_h5_model.summary()
# Compare model_2 predictions with .h5 model predictions
model_2_preds = model_2.predict(X_test)
loaded_h5_model_preds = loaded_h5_model.predict(X_test)
model_2_preds == loaded_h5_model_preds
# Compare mae of model_2 predictions with SavedModel format model predictions
mae(y_true=y_test,y_pred=model_2_preds) == mae(y_true=y_test,y_pred=loaded_SavedModel_format_preds)<jupyter_output><empty_output><jupyter_text>Download a model (or any other file) from Google ColabIf you want to download your files from Google Colab:1. You can go to the "files" tab and right click on the file you're after and click "download".2. Use code (see the cell below).3. Save it to Google Drive by connecting Google Drive and copying it there (see 2nd code cell below.<jupyter_code># Download a file from Google Colab
from google.colab import files
files.download("/content/best_model_HDF5_format.h5")
#from google.colab import drive
#drive.mount('/content/drive')
# Save a file from Google Colab to Google Drive (requires mounting Google Drive)
#!cp /content/best_model_HDF5_format.h5 /content/drive/MyDrive<jupyter_output><empty_output><jupyter_text>A large example<jupyter_code># Import required libraries
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
# Read in the insurance dataset
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance
# let's get one hot encode so it's all numbers
insurance_one_hot = pd.get_dummies(insurance)
insurance_one_hot.head()
# Create X & y values (features and labels)
X = insurance_one_hot.drop("charges",axis=1)
y = insurance_one_hot["charges"]
# View X
X.head()
# View y
y.head()
# Create a training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=42)
len(X), len(X_train), len(X_test)
X_train
# Build a neural network (sort of the model_2 above)
tf.random.set_seed(42)

# Create model
insurance_model = tf.keras.Sequential([
    tf.keras.layers.Dense(10),
    tf.keras.layers.Dense(1)
])

# Compile model
insurance_model.compile(loss=tf.keras.losses.mae,
                        optimizer=tf.keras.optimizers.SGD(),
                        metrics=["mae"])

# Fit model
insurance_model.fit(X_train,y_train,epochs=100)
# Check the result of the insurance model on the test data
insurance_model.evaluate(X_test,y_test)
y_train.median(), y_train.mean()<jupyter_output><empty_output><jupyter_text>Right now it looks like our model isn't performing too well... let's try and improve it!To (try) improve our model, we'll run 2 experiments:1. Add an extra layer with more hidden units and use the Adam optimizer2. Same as about but train for longer (200 epochs)3. (insert your own experiment here)<jupyter_code># Set random seed
tf.random.set_seed(42)

# Create model
insurance_model_2 = tf.keras.Sequential([
   tf.keras.layers.Dense(100),
   tf.keras.layers.Dense(10),
   tf.keras.layers.Dense(1)
])

# Compile model
insurance_model_2.compile(loss="mae",
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics = ["mae"])

# Fit model
insurance_model_2.fit(X_train,y_train,epochs=100)
# Evaluate the larger model
insurance_model_2.evaluate(X_test,y_test)
# Set random seed
tf.random.set_seed(42)

# Create a model
insurance_model_3 = tf.keras.Sequential([
   tf.keras.layers.Dense(100),
   tf.keras.layers.Dense(10),
   tf.keras.layers.Dense(1)
])

# Compile model
insurance_model_3.compile(loss="mae",
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# Fit model
history = insurance_model_3.fit(X_train,y_train,epochs=200)
# Evaluate our third model
insurance_model_3.evaluate(X_test,y_test)
insurance_model.evaluate(X_test,y_test)
# Plot history (also known as a loss curve or a training curve)
pd.DataFrame(history.history).plot()
plt.xlabel("loss")
plt.ylabel("epochs");<jupyter_output><empty_output><jupyter_text>> 🤔 **Question:** How long should you train for?It depends. Really... it depends on the problem you're woriking on. However, many people have asked this question before... so TensorFlow has a solution! It's called the EarlyStopping Callback, which is a TensorFlow component you can add to your model to stop training once it stops improving a certain metric  Preprocessing data (normalization and standardization)In terms of scaling values, neural networks tend to prefer normalization.If you're not sure which to use try both and see which performs better.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf

# Read in the insurance dataframe
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11477122995138353, 'fraction_numerical': 0.020264437275659956, 'mean_word_length': 4.576387129465417, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 111, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '526433', 'license': 'MIT', 'n_tokens_mistral': 7280, 'n_tokens_neox': 6917, 'n_words': 2147, 'path': 'TensorFlow/01_neural_network_regression_with_tensorflow.ipynb', 'repo_name': 'bhupendpatil/Practice', 'size': '22009'}