<jupyter_start><jupyter_text>T81-558: Applications of Deep Neural Networks* Instructor: [<NAME>](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).**Module 4 Assignment: Classification and Regression Neural Network****Student Name: <NAME>**  Assignment InstructionsFor this assignment, you will use the **crx.csv** dataset.  This dataset is a public dataset that can you can find [here](https://archive.ics.uci.edu/ml/datasets/credit+approval). You should use the CSV file on my data site, at this location: [crx.csv](https://data.heatonresearch.com/data/t81-558/crx.csv) because it includes column headers.  The primary use for this dataset is binary classification. There are 15 attributes, plus a target column that contains only + or -.  Some of the columns have missing values.You should train a neural network and return the predictions.  You will submit these predictions to the **submit** function.  See [Assignment 1](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb) for details on how to submit an assignment or check that one was submitted.Complete the following tasks:* Your task is to replace missing values in columns *a2* and *a14* with values estimated by a neural network (one neural network for *a2* and another for *a14*).* Your submission file will contain the same headers as the source CSV: *a1*, *a2*, *s3*, *a4*, *a5*, *a6*, *a7*, *a8*, *a9*, *a10*, *a11*, *a12*, *a13*, *a14*, *a15*, and *a16*.* You should only need to modify *a2* and *a14*.* Neural networks can be much more powerful at filling missing variables than median and mean.* Train two neural networks to predict *a2* and *a14*.  * The *y* (target) for training the two nets will be *a2* and *a14*, depending on which you are trying to fill.* The x for training the two nets will be 's3','a8','a9','a10','a11','a12','a13','a15'.  These are chosen because it is important not to use any columns with missing values; also, it could cause unwanted bias if we include the ultimate target (*a16*).* ONLY predict new values for missing values in *a2* and *a14*.* You will likely get this small warning:  Warning: The mean of column a14 differs from the solution file by 0.20238937709643778. (might not matter if small)  Google CoLab InstructionsIf you are using Google CoLab, it will be necessary to mount your GDrive so that you can send your notebook during the submit process. Running the following code will map your GDrive to ```/content/drive```.<jupyter_code>try:
    from google.colab import drive
    drive.mount('/content/drive', force_remount=True)
    COLAB = True
    print("Note: using Google CoLab")
    %tensorflow_version 2.x
except:
    print("Note: not using Google CoLab")
    COLAB = False<jupyter_output><empty_output><jupyter_text>Assignment Submit FunctionYou will submit the ten programming assignments electronically.  The following **submit** function can be used to do this.  My server will perform a basic check of each assignment and let you know if it sees any underlying problems. **It is unlikely that should need to modify this function.**<jupyter_code>import base64
import os
import numpy as np
import pandas as pd
import requests
import PIL
import PIL.Image
import io

# This function submits an assignment.  You can submit an assignment as much as you like, only the final
# submission counts.  The paramaters are as follows:
# data - List of pandas dataframes or images.
# key - Your student key that was emailed to you.
# no - The assignment class number, should be 1 through 1.
# source_file - The full path to your Python or IPYNB file.  This must have "_class1" as part of its name.  
# .             The number must match your assignment number.  For example "_class2" for class assignment #2.
def submit(data,key,no,source_file=None):
    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')
    if source_file is None: source_file = __file__
    suffix = '_class{}'.format(no)
    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))
    with open(source_file, "rb") as image_file:
        encoded_python = base64.b64encode(image_file.read()).decode('ascii')
    ext = os.path.splitext(source_file)[-1].lower()
    if ext not in ['.ipynb','.py']: raise Exception("Source file is {} must be .py or .ipynb".format(ext))
    payload = []
    for item in data:
        if type(item) is PIL.Image.Image:
            buffered = BytesIO()
            item.save(buffered, format="PNG")
            payload.append({'PNG':base64.b64encode(buffered.getvalue()).decode('ascii')})
        elif type(item) is pd.core.frame.DataFrame:
            payload.append({'CSV':base64.b64encode(item.to_csv(index=False).encode('ascii')).decode("ascii")})
    r= requests.post("https://api.heatonresearch.com/assignment-submit",
        headers={'x-api-key':key}, json={ 'payload': payload,'assignment': no, 'ext':ext, 'py':encoded_python})
    if r.status_code==200:
        print("Success: {}".format(r.text))
    else: print("Failure: {}".format(r.text))<jupyter_output><empty_output><jupyter_text>Assignment 4 Sample CodeThe following code provides a starting point for this assignment.<jupyter_code>import os
import pandas as pd
from scipy.stats import zscore
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
import pandas as pd
import io
import requests
import numpy as np
from sklearn import metrics

# This is your student key that I emailed to you at the beginnning of the semester.
key = "<KEY>"  # This is an example key and will not work.

# You must also identify your source file.  (modify for your local setup)
# file='/content/drive/My Drive/Colab Notebooks/assignment_yourname_class4.ipynb'  # Google CoLab
# file='C:\\Users\\jeffh\\projects\\t81_558_deep_learning\\assignments\\assignment_yourname_class4.ipynb'  # Windows
file='/Users/jheaton/projects/t81_558_deep_learning/assignments/assignment_yourname_class4.ipynb'  # Mac/Linux

# Begin assignment
df = pd.read_csv("https://data.heatonresearch.com/data/t81-558/crx.csv",na_values=['?'])
df
from scipy.stats import zscore
from sklearn.model_selection import train_test_split
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder
from matplotlib import pyplot
from keras import metrics

def chart_regression(pred, y, sort=True):
    t = pd.DataFrame({'pred': pred, 'y': y.to_numpy().flatten()})
    if sort:
        t.sort_values(by=['y'], inplace=True)
    pyplot.plot(t['y'].tolist(), label='expected')
    pyplot.plot(t['pred'].tolist(), label='prediction')
    pyplot.ylabel('output')
    pyplot.legend()
    pyplot.show()

def onehotcoding(data):
    OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
    OH_cols = pd.DataFrame(OH_encoder.fit_transform(data.values.reshape(-1,1)))
    
    return OH_cols

def labelencoding(data):
    encoder = LabelEncoder()
    encoded = pd.DataFrame(encoder.fit_transform(data.values.reshape(-1,1)))
    
    return encoded

def preprocess_data(df, target):
    if target:
        df = labelencoding(df)
        return df
    else:
        columns = [x for x in df.columns if str(df[x].dtypes) in ('object') and x != 'a13']
        for x in columns:
            df[x] = df[x].map({'t':1, 'f':0})
            
        oh_columns = onehotcoding(df['a13'])
        oh_columns.index = df.index

        df.drop(columns='a13', axis=1, inplace=True)
        df_concat = pd.concat([df, oh_columns], axis=1, ignore_index=True)
        df_concat.index = df.index

    return df_concat


def fill_missing_numeric(df,current,target):
    X = df.drop([target], axis=1)
    y = df[target]
    
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, test_size=0.2, random_state=0)
    
    train_columns = ['s3','a8','a9','a10','a11','a12','a13','a15']
    
    X_train = preprocess_data(X_train[train_columns], False)
    X_test = preprocess_data(X_test[train_columns], False)
    y_train = preprocess_data(y_train, True)
    y_test = preprocess_data(y_test, True)

    merged_df = X_train.append(X_test).reset_index().set_index('index').sort_index()
    
    model = create_network(X_train, X_test, y_train, y_test)
    
    # Fill in as needed
    mask_nan = df.loc[pd.isna(df[current]), :].index
    predictions = model.predict(merged_df.iloc[mask_nan])
    df.loc[mask_nan, current] = predictions
    
    print(df.loc[mask_nan])


def create_network(X_train, X_test, y_train, y_test):
    model = Sequential()
    model.add(Dense(50, input_dim=X_train.shape[1], activation='relu')) # Hidden 1
    model.add(Dense(25, activation='relu')) # Hidden 2
    model.add(Dense(1)) # Output

    model.compile(loss='mean_squared_error', optimizer='adam')
    monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=10, verbose=2,
                            mode='auto', restore_best_weights=True)
    
    model.fit(X_train,y_train,validation_data=(X_test,y_test),
              callbacks=[monitor],verbose=0,epochs=100)
    
    pred = model.predict(X_test)
    print(pred.shape)
    chart_regression(pred.flatten(), y_test)
    
    return model

df_submit = fill_missing_numeric(df,'a2','a16')
df_submit<jupyter_output>/var/folders/2j/m7m8jdyd0lldlm4fstt86mrh0000gn/T/ipykernel_64347/649713166.py:41: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[x] = df[x].map({'t':1, 'f':0})
/opt/homebrew/Caskroom/miniforge/base/envs/tensorflow/lib/python3.9/site-packages/pandas/core/frame.py:4906: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  return super().drop(
/var/folders/2j/m7m8jdyd0lldlm4fstt86mrh0000gn/T/ipykernel_64347/649713166.py:41: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

[...]
{'fraction_non_alphanumeric': 0.11501865671641791, 'fraction_numerical': 0.02416044776119403, 'mean_word_length': 4.291707798617966, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '400285', 'license': 'Apache-2.0', 'n_tokens_mistral': 3568, 'n_tokens_neox': 3382, 'n_words': 1027, 'path': 'assignments/assignment_yourname_class4.ipynb', 'repo_name': 'IlkerCa/t81_558_deep_learning', 'size': '10763'}