<jupyter_start><jupyter_text>Validating performance of regression modelsThis notebook explains how to use CNTK metric functions to validate the performance of a regression model.We're using the [car MPG dataset](https://archive.ics.uci.edu/ml/datasets/Auto+MPG) from the UCI dataset library. This dataset is perfect for demonstrating how to build a regression model using CNTK. In the dataset, you'll find 9 columns:1. mpg: continuous 2. cylinders: multi-valued discrete 3. displacement: continuous 4. horsepower: continuous 5. weight: continuous 6. acceleration: continuous 7. model year: multi-valued discrete 8. origin: multi-valued discrete 9. car name: string (unique for each instance)All columns in the dataset contain numeric values except for the origin column which is a categorical value.We'll strip the `car name` column as it cannot be used in our model.  The modelThe model we're using features two hidden layers. Each with 64 neurons with a ReLU (Rectified Linear Unit) activation function. The output is a single neuron without an activation function. This is necessary to turn this neural network into a regression model.We're using the 8 input features and the miles per gallon as target for our neural network.<jupyter_code>from cntk import default_options, input_variable
from cntk.layers import Dense, Sequential
from cntk.ops import relu

with default_options(activation=relu):
    model = Sequential([
        Dense(64),
        Dense(64),
        Dense(1,activation=None)
    ])
    
features = input_variable(9)
target = input_variable(1)

z = model(features)<jupyter_output><empty_output><jupyter_text>PreprocessingIn this section we'll first preprocess the data so that it is compatible for use with our neural network.We need to load the data and then clean it up.<jupyter_code>import pandas as pd
import numpy as np
df_cars = pd.read_csv('auto-mpg.csv', na_values=['?'])
df_cars = df_cars.dropna()<jupyter_output><empty_output><jupyter_text>The origin column contains three possible values, as is shown in the dictionary below. To use the origin in the neural network we need to split it into three separate columns. For this we'll first replace the numeric values with a string value. After we've done that, we ask pandas to generate dummy columns. This creates three columns: usa, europa, and japan. For each sample in the dataset, one of these columns will contain a value of 1 and the rest will contain a value of 0.<jupyter_code>origin_mapping = {
    1: 'usa',
    2: 'europe',
    3: 'japan'
}

df_cars.replace({'origin': origin_mapping}, inplace=True)

categorical_origin = pd.get_dummies(df_cars['origin'], prefix='origin')

df_cars = pd.concat([df_cars, categorical_origin], axis=1)
df_cars = df_cars.drop(columns=['origin', 'car name'])<jupyter_output><empty_output><jupyter_text>The final result of this operation is the following dataset. It contains 9 columns. Of these columns the `mpg` column is used as the target output. The rest is used as a feature for the model.<jupyter_code>df_cars.head()<jupyter_output><empty_output><jupyter_text>Let's create a training and validation set for training.First we extract the car properties by dropping the mpg column. This produces a copy of the dataset without the mpg column. If we were to use the data as-is, we would quickly run into exploding gradients. To prevent this from happening we need to scale the data so that it has a mean of zero and a variance of 1. The result of this is that we get values in a range of -1 to +1.Finally we use the `train_test_split` function to produce a training and validation set.<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = df_cars.drop(columns=['mpg']).values.astype(np.float32)
y = df_cars.iloc[:,0].values.reshape(-1,1).astype(np.float32)

scaler = StandardScaler()
X = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)<jupyter_output><empty_output><jupyter_text>Training the neural networkNow that we have a neural network, let's train it using the training set.We're using a squared error loss function which is a regular loss that you will find in almost any regression model. We'll train the model using a SGD learner, which is the most basic learner around for CNTK.The criterion for training is the mean squared error function. Additionally we would like to measure the mean absolute error rate for the model. In order to do this, we need to create a CNTK function factory that produces a combination of these two. When you create a new function in python marked with `cntk.Function` annotation CNTK will automatically convert it to a function object that has a `train` method for training and a `test` method for validation. Since we don't have a mean absolute error function out-of-the-box we'll create it here as well using the standard CNTK operators.<jupyter_code>import cntk 
from cntk.losses import squared_error

def absolute_error(output, target):
    return cntk.ops.reduce_mean(cntk.ops.abs(output - target))

@cntk.Function
def criterion_factory(output, target):
    loss = squared_error(output, target)
    metric = absolute_error(output, target)
    
    return loss, metric
from cntk.logging import ProgressPrinter
from cntk.losses import squared_error
from cntk.learners import sgd

loss = criterion_factory(z, target)
learner = sgd(z.parameters, 0.001)

progress_printer = ProgressPrinter(0)

train_summary = loss.train((X_train,y_train), 
                           parameter_learners=[learner], 
                           callbacks=[progress_printer],
                           minibatch_size=16,
                           max_epochs=10)<jupyter_output>average      since    average      since      examples
    loss       last     metric       last              
 ------------------------------------------------------
Learning rate per minibatch: 0.001
      754        754       25.5       25.5            16
      697        669       24.9       24.6            48
      581        494       22.5       20.6           112
      451        337       19.5         17           240
     20.1       20.1        3.4        3.4            16
     30.7         36       4.44       4.96            48
     30.3         30       4.44       4.43           112
     25.4       21.2       3.98       3.58           240
     14.2       14.2       2.73       2.73            16
     15.7       16.5       3.16       3.37            48
     15.7       15.7       3.18        3.2           112
     13.9       12.3       2.91       2.67           240
     14.8       14.8       3.03       3.03            16
     12.7       11.6       2.87       2.79            48[...]<jupyter_text>The output of the training session is looking promising, you can see that the loss is going down quite nicely. It's not perfect, but not bad for a first attempt.  Evaluating model performanceIn order to measure the performance of our model we're going first going to use the squared error function from CNTK.This gives us an rough idea of the error rate of the model. But this is in squares so it is quite hard to read depending on your background.As an alternative we'll also use the mean absolute error metric. This gives a more understandable error rate.This metric gives us a good idea of just how much we're off predicting the miles per gallon.CNTK doesn't include a mean absolute error function, but you can easily create it yourself using the standard CNTK ops.We're using the test method on the metric to determine how well our model is doing. This is different from the classification model where we had to do quite a bit more to measure the performance of our model. The output of the `test` method tells us how many miles per gallon the model is off on average when predicting based on the test set we created earlier.<jupyter_code>loss.test((X_test, y_test))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06740442655935613, 'fraction_numerical': 0.03118712273641851, 'mean_word_length': 3.047328244274809, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '34372', 'license': 'MIT', 'n_tokens_mistral': 2321, 'n_tokens_neox': 2126, 'n_words': 987, 'path': 'ch4/Validating performance of regression models.ipynb', 'repo_name': 'AndrewChui/CNTK', 'size': '7952'}