<jupyter_start><jupyter_text>Introduction to regression with tensorflow<jupyter_code>import tensorflow as tf
print(tf.__version__)<jupyter_output>2.4.1<jupyter_text>Creating some data to view and fit the model<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
# Create the features
X = np.array([-7.0, -4.0, -1.0, 2.0, 5.0, 8.0, 11.0, 14.0])

# Create the labels
y = np.array([3.0, 6.0, 9.0, 12.0 ,15.0, 18.0, 21.0, 24.0])
# Visualize the data
plt.scatter(X, y)
# The relation b/w X and y
y == X + 10<jupyter_output><empty_output><jupyter_text>Input and output features<jupyter_code>house_info = tf.constant(['bedroom', 'bathroom', 'garage'])
house_price = tf.constant([900000])
# Turn any numoy arrays in tensors
X = tf.constant(X, dtype=tf.float32)
y = tf.constant(y, dtype=tf.float32)<jupyter_output><empty_output><jupyter_text>Steps for modelling in tensorflow1. **Creating the model** - define the input and output layers, as well as the hidden layers of the deep learning model.2. **Compiling a model** - define the loss function (in other words, the function which tells our model how wrong it is) and the optimizer (tells our model on how to improve the patterns it is learning) and evaluation metrices (what we can use to interpret the performance of our model)3. **Fitting the model** - letting the model trying to find pattern b/w X and y<jupyter_code># Set the seed
tf.random.set_seed(42)

# 1. Create the model using the sequential API
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1)
])

# 2. Compile the model
model.compile(loss=tf.keras.losses.mae,
             optimizer=tf.keras.optimizers.SGD(),
             metrics=["mae"])

# 3. Fit the model
model.fit(X, y, epochs=5)
X, y
# Making a prediction on X and y
model.predict([17.0])<jupyter_output><empty_output><jupyter_text>Improving the modelWe can improve the model by tkaing the steps we took in the 3 steps1. Creating the model- We can increse the number of layers, increase the number of neurons in each of those layers, changing the activation functions of each layer.2. Compiling a model - here we might change the optimization function perhaps the learning rate of the optimization functions.3. Fitting the model - here we can fit the model for more epochs (leave it training for longer) or on more data (give the model moere examples to learn from).<jupyter_code># rebuilding the model

# 1. Creating the model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1),
    tf.keras.layers.Dense(1)
])

# 2. Complining the model
model.compile(loss=tf.keras.losses.mae,
             optimizer=tf.keras.optimizers.SGD(),
             metrics=['mae'])

# 3. Fit the model
model.fit(X, y, epochs=100)
# Predicting 
model.predict([17.0])
# Let's see if we can make another to improve our model

# 1. Create the model with 100 hidden layers
model = tf.keras.Sequential([
    tf.keras.layers.Dense(50, activation=None),
    tf.keras.layers.Dense(1)
])

# 2. Complie the model
model.compile(loss=tf.keras.losses.mae,
             optimizer=tf.keras.optimizers.Adam(lr=0.01),
             metrics=["mae"])

# 3. Fit the model
model.fit(X, y, epochs=100)
model.predict([17.0])<jupyter_output><empty_output><jupyter_text>The learning rate is the most important hyperparameter there is to improve the model  Evaluating the model```Build a model -> fit it -> evaluate it -> tweak it -> fit it -> evaluate it and so on```> When it comes to evaluation, we should visualize the dataIt's a good idea to visulaize* The data - what data are we working with? What does it look like?* The model itself - what does our model look like?* The training of a model - how does the model perform while it learns?* The prediction of the model - how does the predicitions line up against the actual truth<jupyter_code># Making a bigger data set
X = tf.range(-100, 100, 4)
X
# Maing the labels
y = X + 10
y
# Visualize the data
plt.scatter(X, y)<jupyter_output><empty_output><jupyter_text>The 3 sets...* Training set - The model learns from this data, which is usually 70 - 80% of the data* Validation set - the model get tuned on this data, whcih is 10-15% of the data available* Testing set - the model get evaluated on this data on how it has learned, this is usually 10-15% of the dataIn analogy of a real life example -* The course materail(Used to prep for the exam) is the training set* The practise exam is the validation set* The final exam is the test set<jupyter_code># Spliting the data into train and test
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X.numpy(), y.numpy(), test_size=0.2)
# Shape of the data
X_train.shape, X_test.shape, y_train.shape, y_test.shape
# Using regualar ways to get hte data
X_train = X[:40]
y_train = y[:40]

X_test = X[40:]
y_test = y[40:]

len(X_train), len(y_train), len(X_test), len(y_test)<jupyter_output><empty_output><jupyter_text>Visualizing the dataNow we've got our data, lets viz the data<jupyter_code>plt.figure(figsize=(7, 7))

plt.scatter(X_train, y_train, c="b", label="training data")

plt.scatter(X_test, y_test, c="r", label="testing data")

plt.legend();
# Let's see how this data works with the model

# 1. Create the model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1)
#     tf.keras.layers.Dense(100, activation='relu')
])

# 2. Complikte the model
model.compile(loss=tf.keras.losses.mae,
             optimizer=tf.keras.optimizers.SGD(),
             metrics=["mae"])

# 3. Fit the model
# model.fit(X_train, y_train, epochs=100)

model.summary()<jupyter_output><empty_output><jupyter_text>Let's build the model which builds automatically by defining the input_shape args<jupyter_code># 0. Setting the radome seed
tf.random.set_seed(42)

# 1. Create a model
model = tf.keras.Sequential([
  tf.keras.layers.Dense(10, input_shape=[1], name="input_layer"),
  tf.keras.layers.Dense(1, name="output_layer")
], name="another_model")

# 2. Compile the model
model.compile(loss=tf.keras.losses.mae,
             optimizer=tf.keras.optimizers.SGD(),
             metrics=['mae'])
model.summary()<jupyter_output>Model: "another_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_layer (Dense)          (None, 10)                20        
_________________________________________________________________
output_layer (Dense)         (None, 1)                 11        
=================================================================
Total params: 31
Trainable params: 31
Non-trainable params: 0
_________________________________________________________________<jupyter_text>* Total params - Total number of parameters in the model.* Trainable params - These are the params (patterns) the model can update as it training* Non trainable params - these params aren't updated during the training of the model, this is typically when you bring in the parameters from other trained models (transfer learning)[Article for Weights and Biases](https://wandb.ai/site/articles/fundamentals-of-neural-networks)[Learnable Parameters ("Trainable Params") In A Keras Convolutional Neural Network](https://deeplizard.com/learn/video/8d-9SnGt5E0)[Deep Learning Course by MIT](http://introtodeeplearning.com/)<jupyter_code># Let's fit the model to the training data
model.fit(X_train, y_train, epochs=100, verbose=1)
# Get the models summary
model.summary()
from tensorflow.keras.utils import plot_model
# from keras.utils.vis_utils import plot_model
plot_model(model=model, show_shapes=True)<jupyter_output><empty_output><jupyter_text>How to visualize the model predictionsTo viz the predictions it is a good idea to plot themOften you'll see we do `y_test` or `y_true` vs `y_pred`<jupyter_code># Predicting the output
y_pred = model.predict(X_test)
y_pred
y_test.numpy()
# Creating a plotting function

def plot_predictions(train_data=X_train,
                     train_label=y_train,
                     test_data=X_test,
                     test_label=y_test,
                     predictions=y_pred):
  """
  Plot training data, test data and compare to ground truth
  """
  plt.figure(figsize=(10, 7))
  plt.scatter(train_data, train_label, c="b", label="training data")
  plt.scatter(test_data, test_label, c="g", label="test data")
  plt.scatter(test_data, predictions, c="r", label="predictions")
  plt.legend();
plot_predictions(train_data=X_train,
                 train_label=y_train,
                 test_data=X_test,
                 test_label=y_test,
                 predictions=y_pred)<jupyter_output><empty_output><jupyter_text>Evaluation metricesThese methods tell us in a more quantitaive way of how close our `y_test` and `y_pred` areThe evaluation is based on what kind of model we are running. Since we are running the regression, the two ways of metrics are * MAE - Average of the absolute value of the error* MSE - Average of the squared difference of the error<jupyter_code>model.evaluate(X_test, y_test)
# Calcualting the Mean Absolute Error
mae = tf.metrics.mean_absolute_error(y_test, tf.squeeze(y_pred))
mae
# Calculating the mean squared error - This is to make the larger error more significant
mse = tf.metrics.mean_squared_error(y_test, tf.squeeze(y_pred))
mse
# Function for evaluations 
def mae(y_test=y_test,
        y_pred=y_pred):
  return tf.metrics.mean_absolute_error(y_test, tf.squeeze(y_pred))

  # Function to creat MSE evaluation
def mse(y_test=y_test,
        y_pred=y_pred):
  return tf.metrics.mean_squared_error(y_test, tf.squeeze(y_pred))
print(mae())
print(mse())<jupyter_output>tf.Tensor(3.1969407, shape=(), dtype=float32)
tf.Tensor(13.070143, shape=(), dtype=float32)<jupyter_text>Improving the modelThis is how we aim to achieve it1. Get more data for the model to train on2. Make your model larger3. Train for longer, give the model more epochsLet's do 3 experiments1. `model_1` - same as the original model, 1 layer, trainded on 100 epochs2. `model_2` - 2 layers, trained for 100 epochs3. `model_3` - 2 layers, trained on 500 epochs**Build `model_1`**<jupyter_code>X_train, y_train
# This is the model_1
# Set random seed
tf.random.set_seed(42)

# 1. Create the model
model_1 = tf.keras.Sequential([
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_1.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_1.fit(X_train, y_train, epochs=100)
# Make and plots predictions for model_1
y_pred_1 = model_1.predict(X_test)
y_pred_1
plot_predictions(predictions=y_pred_1)
# Calcuate model_1 evaluation metrics
mae_1 = mae(y_test, y_pred_1)
mse_1 = mse(y_test, y_pred_1)
mae_1, mse_1<jupyter_output><empty_output><jupyter_text>**Build `model_2`*** 2 dense layers trained on 100 epochs<jupyter_code># 1. Create the model
model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_2.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_2.fit(X_train, y_train, epochs=200)
# Make and plot predictions
y_pred_2 = model_2.predict(X_test)
plot_predictions(predictions=y_pred_2)

mae_2 = mae(y_test, y_pred_2)
mae_2
mse_2 = mse(y_test, y_pred_2)
mse_2<jupyter_output><empty_output><jupyter_text>**Building `model_3`**<jupyter_code>tf.random.set_seed(42)

# 1. Build the model
model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
model_3.compile(loss=tf.keras.losses.mae,
                optimizer=tf.keras.optimizers.SGD(),
                metrics=["mae"])

# 3. Fit the model
model_3.fit(X_train, y_train, epochs=500)
y_preds_3 = model_3.predict(X_test)
plot_predictions(predictions=y_preds_3)
# Calculate the evaluation metrics for model_3
mae_3 = mae(y_test, y_preds_3)
mse_3 = mse(y_test, y_preds_3)

mae_3, mse_3<jupyter_output><empty_output><jupyter_text>Compare the results of the experiments<jupyter_code># Let's create the pandas data frame
import pandas as pd
model_results = [['model_1', mae_1.numpy(), mse_1.numpy()],
                 ['model_2', mae_2.numpy(), mse_2.numpy()],
                 ['model_3', mae_3.numpy(), mse_3.numpy()]]

all_results = pd.DataFrame(model_results, columns=["model", "mae", "mse"])
all_results 
model_2.summary()<jupyter_output>Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_7 (Dense)              (None, 10)                20        
_________________________________________________________________
dense_8 (Dense)              (None, 1)                 11        
=================================================================
Total params: 31
Trainable params: 31
Non-trainable params: 0
_________________________________________________________________<jupyter_text>**Note:** The time taken between experiments must be reduced to a minimum to facilitate a maximum number of experiments in short duration of time.   Tracking the experiments* Tensorboard - This is form tensorflow's library for model tracking* Weights and Biases - A tool for trackign all of kinds of machine learngin experiments (plugs straigt into tensorflow).  Saving our modelSaving our model allows us to use out of colab, like a web app and appsThere are 2 main formats to the model1. The SaveModel format2. HDF5 Format<jupyter_code># Save the model in SavedModel format
model_2.save("best_model_SavedModel_format")
# Save the model in HDF5 format
model_2.save('best_model_HDF5.h5')
model_hdf5 = tf.keras.models.load_model("/content/best_model_HDF5.h5")
model_hdf5.summary()
# Save from the SavedModel
model_savedmodel = tf.keras.models.load_model("/content/best_model_SavedModel_format")
model_savedmodel.summary()
# Save the model from HDF5
model_hdf5 = tf.keras.models.load_model("/content/best_model_HDF5.h5")
model_hdf5.summary()
# Compare the model_2 predictions with the SavedModel and the HDF5 format
model_2_preds = model_2.predict(X_test)
savedmodel_2_preds = model_savedmodel.predict(X_test)
hdf5model_2_preds = model_hdf5.predict(X_test)

model_2_preds == savedmodel_2_preds
model_2_preds == hdf5model_2_preds<jupyter_output><empty_output><jupyter_text>Downlaod a model or any other file from Google ColabThe number of ways you can do it1. Download from the file section of the colab section2. Use the code to download from colab3. You can save it to drive, and copying it to where ever we require it to<jupyter_code># from google.colab import files
# files.download("/content/best_model_HDF5.h5")
# Save a file from Google Colab to Google Drive
!ls source dest<jupyter_output>ls: cannot access 'source': No such file or directory
ls: cannot access 'dest': No such file or directory<jupyter_text>A larger example<jupyter_code># Import the libs

import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
# Reading the dataset
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance.head()
insurance_one_hot = pd.get_dummies(insurance)
insurance_one_hot.head()
# Get all the columns of the main dataframe
insurance_one_hot.columns
# Create X and y values (features and labels)
y = insurance_one_hot[['charges']]
X = insurance_one_hot.drop(['charges'], axis=1)
# Observe all the X and y dataframe columns
X.columns, y.columns
X.head()
y.head()
# Create the training and test set
from sklearn.model_selection import train_test_split
# Splitting the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Get all the shapes
X.shape, y.shape, X_train.shape, X_test.shape, y_train.shape, y_test.shape
# Set the randome seed
tf.random.set_seed(42)

# Buildng the neural network
insurance_model = tf.keras.Sequential([
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# Compile the model
insurance_model.compile(loss=tf.keras.losses.mae,
                        optimizer=tf.keras.optimizers.SGD(),
                        metrics=["mae"])

# Fit the model
insurance_model.fit(X_train, y_train, epochs=100)
# Evaluate the model with the test_data
insurance_model.evaluate(X_test, y_test)
y_train.median(), y_train.mean()<jupyter_output><empty_output><jupyter_text>This model seems to not be working as expected, so we need to run a few experimentsWe will be doing 21. We will adding extra layers with more hidden units2. Train for longer<jupyter_code># Set radom seed
tf.random.set_seed(42)

# 1. Create the model
insurance_model_2 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
insurance_model_2.compile(
    loss=tf.keras.losses.mae,
    optimizer=tf.keras.optimizers.Adam(),
    metrics=["mae"]
)

# 3. Fit the model
insurance_model_2.fit(X_train, y_train, epochs=100)
insurance_model_2.evaluate(X_test, y_test)
# se random seed
tf.random.set_seed(42)

# 1. Create the model
insurance_model_3 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
insurance_model_3.compile(loss=tf.keras.losses.mae,
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# 3. Fit the model
history = insurance_model_3.fit(X_train, y_train, epochs=200)
# Evaluating the model
insurance_model_3.evaluate(X_test, y_test)
# Viewing the results of the previous model
insurance_model.evaluate(X_test, y_test)
# Plot history (loss curve or a training curve)
pd.DataFrame(history.history).plot()
plt.xlabel("loss")
plt.ylabel("epochs")<jupyter_output><empty_output><jupyter_text>> How long should you train for?It depends on the model and the problem to be precise. There is a solution by tensorflow called [EarlyStopping Callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping). This will stop once the metrics stops to decrease beyond a certain point.  Preprocessing of data (normalization and standardization)- Article: [Scale, Standardize, or Normalize with Scikit-Learn](https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02)<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
# Reading the data
insurance = pd.read_csv("https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/master/insurance.csv")
insurance.head()
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder 
from sklearn.model_selection import train_test_split
# Make column transformer
ct = make_column_transformer(
    (MinMaxScaler(), ["age", "bmi", "children"]),
    (OneHotEncoder(handle_unknown="ignore"), ["sex", "smoker", "region"])
)
# Creating X and Y
X = insurance.drop(["charges"], axis=1)
y = insurance[["charges"]]
# Split the data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Fit the column transformer to our training data
ct.fit(X_train)

# Transform the train data and the test data with Normalization and OneHotEncoder
X_train_normal = ct.transform(X_train)
X_test_normal = ct.transform(X_test)
X_train_normal[0]<jupyter_output><empty_output><jupyter_text>The datat is pre-processed and ready to be used in the neural network<jupyter_code># Set the random seed
tf.random.set_seed(42)

# 1. Creat the model
insurance_model_4 = tf.keras.Sequential([
  tf.keras.layers.Dense(100),
  tf.keras.layers.Dense(10),
  tf.keras.layers.Dense(1)
])

# 2. Compile the model
insurance_model_4.compile(loss=tf.keras.losses.mae,
                          optimizer=tf.keras.optimizers.Adam(),
                          metrics=["mae"])

# 3. Fit the model
insurance_model_4.fit(X_train_normal, y_train, epochs=100)
insurance_model_4.evaluate(X_test_normal, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11387953239548247, 'fraction_numerical': 0.021745591440459677, 'mean_word_length': 4.57707182320442, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 91, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '13573', 'license': 'MIT', 'n_tokens_mistral': 6998, 'n_tokens_neox': 6619, 'n_words': 1916, 'path': 'learning-notebooks/01_neural_network_regression_with_tf.ipynb', 'repo_name': 'tejeshreddy/tensorflow-deep-learning', 'size': '20188'}