<jupyter_start><jupyter_text>DNN AssignmentIn this assignment you are working together with your teammates from the second project. You will apply your new knowledge about dense neural networks to the data from your ML project to investigate, if you can make further improvements on prediction performance. Your data is (hopefully) already cleaned and transformed (this was part of your ML project) such that you can focus fully on feeding it to your neural network. Use TensorFlow 2.x in this assignment as it makes training with real-life data much more easier with many implemented features (e.g. early-stopping, tensorboard, regularization, etc.). In this notebook you will learn- how to apply a neural network to your own data using TensorFlow 2.x- how to tune the network and monitor learning- how to train several networks and ensemble them into a stronger model  Module loadingLoad all the necessary packages for your assignment. We give you some modules in advance, feel free to add more, if you need them.<jupyter_code>import numpy as np
import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt

from tensorflow.keras import layers
from tensorflow.keras import regularizers
    
print('Using TensorFlow version: %s' % tf.__version__)
#!pip install -q git+https://github.com/tensorflow/docs
    
import tensorflow_docs as tfdocs
import tensorflow_docs.modeling
import tensorflow_docs.plots<jupyter_output><empty_output><jupyter_text>Data loadingLoad here your data from your ML project. You can use either `pandas` or `numpy` to format your data.<jupyter_code>df = pd.read_csv("data/kickstarter_cleaned.csv")
#df.columns
features = ['country', 'created_at', 'deadline', 'goal', 'launched_at', 'state', 'cat_id',
       'location_type', 'location_state', 'location_score', 'name_length', 'name_words',
       'projects_successful', 'projects_failed'] 
df.created_at = df.created_at.astype('datetime64')
df.launched_at = df.launched_at.astype('datetime64')
df.deadline = df.deadline.astype('datetime64')
df['month_created'] = df['created_at'].dt.month
df['month_launched'] = df['launched_at'].dt.month
df['month_deadline'] = df['deadline'].dt.month

df['delta_public'] = (df['created_at'] - df['launched_at']) / pd.offsets.Day(-1)
df['delta_funding'] = (df['launched_at'] - df['deadline']) / pd.offsets.Day(-1)

df['delta_total'] = ((df['created_at'] - df['deadline']) / pd.offsets.Day(-1)).round(0)

df['delta_public'] = df.delta_public.round(0)
df['delta_funding'] = df.delta_funding.round(0)
df.columns
features = ['country', 'goal', 'cat_id',
       'location_type', 'location_state', 'location_score', 'name_length', 'name_words',
       'projects_successful', 'projects_failed', 'month_created', 
       'month_launched', 'month_deadline', 'delta_public',
       'delta_funding', 'delta_total']
X = df[features]

y = df.state.replace({"successful" : 1, "failed" : 0})
# Load the saved model
#new_large_model = tf.keras.models.load_model('saved_model/my_large_model')

# Check its architecture
#new_large_model.summary()
X = pd.get_dummies(X, columns=['country', 'cat_id', 'location_type', 'location_state'], drop_first=True)
#y = y.values
X.head()<jupyter_output><empty_output><jupyter_text>TrainingFor training you need a train/val split (hopefully you did a train/test split before (and you should use the same as in your ML project to make results comparable).<jupyter_code>#X_train.head()
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Import sklearn.preprocessing.StandardScaler
from sklearn.preprocessing import MinMaxScaler

X_train_scaled = X_train.copy()
X_test_scaled = X_test.copy()
# Initialize a scaler, then apply it to the features
scaler = MinMaxScaler()
numerical = ['goal', 'location_score', 'name_length', 'name_words',
       'projects_successful', 'projects_failed', 'month_created', 
       'month_launched', 'month_deadline', 'delta_public',
       'delta_funding', 'delta_total']
X_train_scaled[numerical] = scaler.fit_transform(X_train[numerical])
X_test_scaled[numerical] = scaler.transform(X_test[numerical])
type(X_train_scaled)
#X_train_scaled[numerical].dtype(int)
y_train.shape
N_VAL = y_train.shape[0] // 3
N_TRAIN = y_train.shape[0]
BATCH_SIZE = 128
STEPS_PER_EPOCH = N_TRAIN // BATCH_SIZE
EPOCHS = 200<jupyter_output><empty_output><jupyter_text>Build, compile and fit your modelTo become fast at retraining your (different) models it is good practice to define a function that gets fed by a model, its name, an optimizer to use and the number of epochs you want the model to be trained. If your model trains for many epochs you will receive a lot of logging from TensorFlow. To reduce the logging noise you can use a callback (provided by the `tensorflow_docs` module we installed and imported for you) named `EpochDots()` that simply prints a `.` for each epoch and a full set of metrics after a number of epochs have been trained. If you want to produce logs for using Tensorboard you also need to include the `callbacks.Tensorboard()`.<jupyter_code>import os

#def get_callbacks():
# Define path where checkpoints should be stored
checkpoint_path = "kickstarter/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=0) # Set verbose != 0 if you want output during training 
# return [list of your callbacks]
lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(
    0.01,
    decay_steps=STEPS_PER_EPOCH*1000,
    decay_rate=1,
    staircase=False)


# Define optimizer used for modelling
optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.01, name='Adam')<jupyter_output><empty_output><jupyter_text>You can implement your callbacks in the `model.fit()` method below.  Build your modelYou can build your model by using `tf.keras.Sequential()` that helps you to sequentially define your different layers from input to output.<jupyter_code># Function for medium model architecture 
def get_compiled_medium_model():
    medium_model = tf.keras.Sequential([
      tf.keras.layers.Dense(64,kernel_initializer = 'uniform', activation='relu', input_dim = X_train_scaled.shape[1]),
      tf.keras.layers.Dense(64,kernel_initializer = 'uniform', activation='relu'),
      tf.keras.layers.Dense(64,kernel_initializer = 'uniform', activation='relu'),
      tf.keras.layers.Dense(1,kernel_initializer = 'uniform', activation='sigmoid')
        
    ])

    medium_model.compile(optimizer=optimizer,
                  loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
                  metrics=['accuracy'])
    return medium_model<jupyter_output><empty_output><jupyter_text>Train your modelTrain your model by using your `model_compile_and_fit()` function you defined above.<jupyter_code>type(X_train_scaled)
#X_train_scaled = X_train_scaled.values
#X_train_scaled[0]
# Summary of medium sized model
medium_model = get_compiled_medium_model()
print(medium_model.summary())
training_history = {}
training_history['medium_model'] = medium_model.fit(X_train_scaled,
                     y_train,
                     validation_split= 0.2,
                     verbose=1, 
                     steps_per_epoch= STEPS_PER_EPOCH,
                     epochs=EPOCHS,
                     callbacks=[cp_callback])  # Pass callback to training<jupyter_output>Epoch 1/200
33/33 [==============================] - 2s 63ms/step - loss: 0.6923 - accuracy: 0.5505 - val_loss: 0.6902 - val_accuracy: 0.5650
Epoch 2/200
33/33 [==============================] - 1s 41ms/step - loss: 0.6896 - accuracy: 0.5681 - val_loss: 0.6886 - val_accuracy: 0.5650
Epoch 3/200
33/33 [==============================] - 1s 43ms/step - loss: 0.6882 - accuracy: 0.5662 - val_loss: 0.6876 - val_accuracy: 0.5650
Epoch 4/200
33/33 [==============================] - 1s 41ms/step - loss: 0.6872 - accuracy: 0.5667 - val_loss: 0.6869 - val_accuracy: 0.5650
Epoch 5/200
33/33 [==============================] - 1s 41ms/step - loss: 0.6866 - accuracy: 0.5662 - val_loss: 0.6864 - val_accuracy: 0.5650
Epoch 6/200
33/33 [==============================] - 1s 40ms/step - loss: 0.6863 - accuracy: 0.5647 - val_loss: 0.6860 - val_accuracy: 0.5650
Epoch 7/200
33/33 [==============================] - 1s 41ms/step - loss: 0.6859 - accuracy: 0.5648 - val_loss: 0.6857 - val_accuracy: 0.5650
Epoch [...]<jupyter_text>Evaluate your model trainingTensorFlow offers now (this was more cumbersome before) a simple history plotter that you can use to plot training histories and see how the model performed on training and validation data set.<jupyter_code>history_plotter = tfdocs.plots.HistoryPlotter(metric = 'your_metric', smoothing_std=10)
history_plotter.plot(your_history)<jupyter_output><empty_output><jupyter_text>Model tuningYou might have no luck with your first model (most surely you did not). In this section you will apply methods you know to tune your model's performance. An obvious way of course is to change your model's architecture (removing or adding layers or layer dimensions, changing activation functions). However, after this you might still be able to detect some overfitting and there are some more methods you can apply to improve your neural network. Some of them are regularization, learning rate decay, early stopping, or dropout. If you want to add regularization you can apply directly layer-wise L2- or L1-regularization by using a layer's `kernel_regularization` argument and an appropriate regularizer from the [`tensorflow.keras.regularizers`](https://www.tensorflow.org/api_docs/python/tf/keras/regularizers) module we imported for you.  __Optimizer schedules__Quite often your optimizer does not run efficiently through the loss function surface. Remember that theory ensures a convergence of mini-batch SGD if and only if the learing rate decreases sufficiently fast. A way to apply this to your model training is to use a learning rate scheduler (learning rate decay) that reduces the learning rate over the number of update steps. The [`tf.keras.optimizers.schedules`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules) module offers you some approaches to do that. Note that to apply this to your `model_compile_and_fit()` function you defined above you need to implement the learning rate schedule either in there or with a helper function that your function calls inside. If you want to visualize different schedulers you can define them and call them on a range of values and plot them in a line plot. __Early stopping__Earyl stopping is a procedure that enables you to stop your training earlier than defined by your `max_epochs` argument. It is used in practices to 1. determine the optimal parameter vector by monitoring the validation error closely (if it rises again too much stuck with the best parameters found until then) and2. to save expensive resources (either in terms of monetary costs or ecological costs).To implement early stopping in TensorFlow the `tf.keras` module offers you a `callback` named [`tf.keras.callback.EarlyStopping()`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) that monitors for you a certain metric (it makes sense here to use a validation metric) and to stop training after a certain number of epochs with no improvement or by defining a certain `min_delta` that defines a minimum value of improvement - if below the callback stops your training. You can add this callback simply to the callbacks defined in your `get_callbacks()` function you defined above.__Dropout__Dropout was one of the important developments in regularization for neural networks. It was developed by <NAME> and his team at Toronto University. Dropout can be applied to each layer in your network and is implemented in `tf.keras` by an own layer named `Dropout()` awaiting a dropout rate set by you. So to introduce dropout you have to rework your model design.  Make use of your knowledge and apply tuning techniques to improve your network.<jupyter_code>#===========#
# YOUR CODE #
#===========#<jupyter_output><empty_output><jupyter_text>Model ensemblingYou have learned that models can be ensembled. What is possible in `scikit-learn` is also possible in TensorFlow, just a little different as it is relying on its computation graph. However, any model is callable like a `layer` by invoking it on either an `Input` or on the output of another layer. Furthermore, you can also stack outputs together.To produce an ensemble you can define a couple of models, than use their predictions as inputs for another model and produce a final output (using `keras.Model(input, output)`). But you can also start simple and use the mean predictions over all models and then compute the `argmax()` to assign them to a class in classification (via using `layers.average([model1_preds,model2_preds,...])`). You will be surprised how well this works. Now implement your own ensemble to improve your work even a little more and to have something more to polish up your ML project on `GitHub` ;)<jupyter_code>#===========#
# YOUR CODE #
#===========#<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10412926391382406, 'fraction_numerical': 0.020870736086175942, 'mean_word_length': 4.817667536988686, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '199880', 'license': 'MIT', 'n_tokens_mistral': 4056, 'n_tokens_neox': 3786, 'n_words': 1432, 'path': '06_DNN_Assignment.ipynb', 'repo_name': 'bur-oez/Kickstarter-ML-Project', 'size': '13377'}