<jupyter_start><jupyter_text>T81-558: Applications of Deep Neural Networks**Module 8: Kaggle Data Sets*** Instructor: [<NAME>](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).  Module 8 Material* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=XpGI4engRjQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=24) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)* Part 8.2: Building Ensembles with Scikit-Learn and Keras [[Video]](https://www.youtube.com/watch?v=AA3KFxjPxCo&index=25&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_keras_ensembles.ipynb)* Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters [[Video]](https://www.youtube.com/watch?v=GaKo-9c532c) [[Notebook]](t81_558_class_08_3_keras_hyperparameters.ipynb)* Part 8.4: Bayesian Hyperparameter Optimization for Keras [[Video]](https://www.youtube.com/watch?v=GaKo-9c532c) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)* **Part 8.5: Current Semester's Kaggle** [[Video]](https://www.youtube.com/watch?v=GaKo-9c532c) [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)<jupyter_code># Nicely formatted time string
def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60
    return "{}:{:>02}:{:>05.2f}".format(h, m, s)<jupyter_output><empty_output><jupyter_text>Part 8.5: Current Semester's KaggleKaggke competition site for current semester (Fall 2019):* Coming soonPrevious Kaggle competition sites for this class (NOT this semester's assignment, feel free to use code):* [Spring 2019 Kaggle Assignment](https://www.kaggle.com/c/applications-of-deep-learningwustl-spring-2019)* [Fall 2018 Kaggle Assignment](https://www.kaggle.com/c/wustl-t81-558-washu-deep-learning-fall-2018)* [Spring 2018 Kaggle Assignment](https://www.kaggle.com/c/wustl-t81-558-washu-deep-learning-spring-2018)* [Fall 2017 Kaggle Assignment](https://www.kaggle.com/c/wustl-t81-558-washu-deep-learning-fall-2017)* [Spring 2017 Kaggle Assignment](https://inclass.kaggle.com/c/applications-of-deep-learning-wustl-spring-2017)* [Fall 2016 Kaggle Assignment](https://inclass.kaggle.com/c/wustl-t81-558-washu-deep-learning-fall-2016)  Iris as a Kaggle CompetitionIf the Iris data were used as a Kaggle, you would be given the following three files:* [kaggle_iris_test.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_iris_test.csv) - The data that Kaggle will evaluate you on.  Contains only input, you must provide answers.  (contains x)* [kaggle_iris_train.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_iris_train.csv) - The data that you will use to train. (contains x and y)* [kaggle_iris_sample.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_iris_sample.csv) - A sample submission for Kaggle. (contains x and y)Important features of the Kaggle iris files (that differ from how we've previously seen files):* The iris species is already index encoded.* Your training data is in a separate file.* You will load the test data to generate a submission file.The following program generates a submission file for "Iris Kaggle".  You can use it as a starting point for assignment 3.<jupyter_code>import os
import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras.callbacks import EarlyStopping

df_train = pd.read_csv("https://data.heatonresearch.com/data/t81-558/datasets/kaggle_iris_train.csv",
                       na_values=['NA','?'])

# Encode feature vector
df_train.drop('id', axis=1, inplace=True)

num_classes = len(df_train.groupby('species').species.nunique())

print("Number of classes: {}".format(num_classes))

# Convert to numpy - Classification
x = df_train[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values
dummies = pd.get_dummies(df_train['species']) # Classification
species = dummies.columns
y = dummies.values
    
# Split into train/test
x_train, x_test, y_train, y_test = train_test_split(    
    x, y, test_size=0.25, random_state=45)

# Train, with early stopping
model = Sequential()
model.add(Dense(50, input_dim=x.shape[1], activation='relu'))
model.add(Dense(25))
model.add(Dense(y.shape[1],activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto',
                       restore_best_weights=True)

model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=0,epochs=1000)
from sklearn import metrics

# Calculate multi log loss error
pred = model.predict(x_test)
score = metrics.log_loss(y_test, pred)
print("Log loss score: {}".format(score))

# Generate Kaggle submit file

# Encode feature vector
df_test = pd.read_csv("https://data.heatonresearch.com/data/t81-558/datasets/kaggle_iris_test.csv",
                      na_values=['NA','?'])

# Convert to numpy - Classification
ids = df_test['id']
df_test.drop('id', axis=1, inplace=True)
x = df_test[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values
y = dummies.values

# Generate predictions
pred = model.predict(x)
#pred

# Create submission data set

df_submit = pd.DataFrame(pred)
df_submit.insert(0,'id',ids)
df_submit.columns = ['id','species-0','species-1','species-2']

df_submit.to_csv("iris_submit.csv", index=False) # Write submit file locally

print(df_submit)<jupyter_output>id  species-0  species-1     species-2
0   100   0.013904   0.795336  1.907604e-01
1   101   0.001614   0.381189  6.171972e-01
2   102   0.002317   0.466447  5.312361e-01
3   103   0.973577   0.026410  1.352797e-05
4   104   0.981406   0.018586  8.082319e-06
5   105   0.979300   0.020693  6.637653e-06
6   106   0.994228   0.005771  5.370086e-07
7   107   0.000977   0.256403  7.426196e-01
8   108   0.017728   0.843886  1.383867e-01
9   109   0.000364   0.235242  7.643935e-01
10  110   0.000076   0.075360  9.245642e-01
11  111   0.997202   0.002798  1.343153e-07
12  112   0.052874   0.876635  7.049178e-02
13  113   0.000059   0.068381  9.315599e-01
14  114   0.986298   0.013698  3.236879e-06
15  115   0.002562   0.562420  4.350178e-01
16  116   0.054271   0.860268  8.546133e-02
17  117   0.981537   0.018459  3.920587e-06
18  118   0.989732   0.010266  1.793094e-06
19  119   0.990169   0.009830  1.572438e-06
20  120   0.015455   0.880904  1.036411e-01
21  121   0.021652   0.883587  9[...]<jupyter_text>MPG as a Kaggle Competition (Regression)If the Auto MPG data were used as a Kaggle, you would be given the following three files:* [kaggle_mpg_test.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_auto_test.csv) - The data that Kaggle will evaluate you on.  Contains only input, you must provide answers.  (contains x)* [kaggle_mpg_train.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_auto_test.csv) - The data that you will use to train. (contains x and y)* [kaggle_mpg_sample.csv](https://data.heatonresearch.com/data/t81-558/datasets/kaggle_auto_sample.csv) - A sample submission for Kaggle. (contains x and y)Important features of the Kaggle iris files (that differ from how we've previously seen files):The following program generates a submission file for "MPG Kaggle".<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping
import pandas as pd
import io
import os
import requests
import numpy as np
from sklearn import metrics

save_path = "."

df = pd.read_csv(
    "https://data.heatonresearch.com/data/t81-558/datasets/kaggle_auto_train.csv", 
    na_values=['NA', '?'])

cars = df['name']

# Handle missing value
df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())

# Pandas to Numpy
x = df[['cylinders', 'displacement', 'horsepower', 'weight',
       'acceleration', 'year', 'origin']].values
y = df['mpg'].values # regression

# Split into train/test
x_train, x_test, y_train, y_test = train_test_split(    
    x, y, test_size=0.25, random_state=42)

# Build the neural network
model = Sequential()
model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1
model.add(Dense(10, activation='relu')) # Hidden 2
model.add(Dense(1)) # Output
model.compile(loss='mean_squared_error', optimizer='adam')
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, 
                        verbose=1, mode='auto', restore_best_weights=True)
model.fit(x_train,y_train,validation_data=(x_test,y_test),
          verbose=2,callbacks=[monitor],epochs=1000)

# Predict
pred = model.predict(x_test)
import numpy as np

# Measure RMSE error.  RMSE is common for regression.
score = np.sqrt(metrics.mean_squared_error(pred,y_test))
print("Final score (RMSE): {}".format(score))
import pandas as pd

# Generate Kaggle submit file

# Encode feature vector
df_test = pd.read_csv("https://data.heatonresearch.com/data/t81-558/datasets/kaggle_auto_test.csv",
                      na_values=['NA','?'])

# Convert to numpy - regression
ids = df_test['id']
df_test.drop('id', axis=1, inplace=True)

# Handle missing value
df_test['horsepower'] = df_test['horsepower'].fillna(df['horsepower'].median())

x = df_test[['cylinders', 'displacement', 'horsepower', 'weight',
       'acceleration', 'year', 'origin']].values




# Generate predictions
pred = model.predict(x)
#pred

# Create submission data set

df_submit = pd.DataFrame(pred)
df_submit.insert(0,'id',ids)
df_submit.columns = ['id','mpg']

df_submit.to_csv("auto_submit.csv", index=False) # Write submit file locally

print(df_submit)<jupyter_output>id        mpg
0   350  27.541994
1   351  25.862688
2   352  26.286074
3   353  28.067247
4   354  26.922823
5   355  29.997942
6   356  27.195421
7   357  27.913530
8   358  29.707598
9   359  27.877825
10  360  28.481096
11  361  27.393724
12  362  27.112352
13  363  29.838976
14  364  22.223558
15  365  11.889268
16  366  22.200274
17  367  20.895983
18  368  29.619097
19  369  29.607958
20  370  28.666338
21  371  28.795122
22  372  26.298723
23  373  26.294970
24  374  27.767382
25  375  28.209459
26  376  26.572680
27  377  28.218096
28  378  27.944262
29  379  25.759920
30  380  27.392458
31  381  27.163240
32  382  27.431566
33  383  27.360979
34  384  27.490450
35  385  27.584368
36  386  27.756899
37  387  25.689104
38  388  16.059099
39  389  24.993456
40  390  20.356920
41  391  27.254869
42  392  25.351488
43  393  26.956682
44  394  26.871660
45  395  27.191133
46  396  24.900072
47  397  28.000689
48  398  28.800188
{'fraction_non_alphanumeric': 0.123442190484854, 'fraction_numerical': 0.13736013826980806, 'mean_word_length': 4.40511307767945, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 14, 'https://': 24, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '87385', 'license': 'Apache-2.0', 'n_tokens_mistral': 5054, 'n_tokens_neox': 4121, 'n_words': 957, 'path': 't81_558_class_08_5_kaggle_project.ipynb', 'repo_name': 'sanjayssane/t81_558_deep_learning', 'size': '10998'}