<jupyter_start><jupyter_text>Model Building using Deep Learning Model (Keras)<jupyter_code># Define model
import random
random.seed(42)
from tensorflow.python.keras.layers import Dense
from tensorflow.python.keras import Sequential
model = Sequential()
model.add(Dense(100, input_dim=48, activation= "relu"))
model.add(Dense(50, activation= 'relu'))
model.add(Dense(1))
model.summary() #Print model Summary
# Compile model
#model.compile(loss= "mean_squared_error" , optimizer="Adamax", metrics=["mean_squared_error"])
model.compile(loss= "mse" , optimizer="adam", metrics=["mse"])
# Fit Model
model.fit(X_train, y_train, epochs=3, batch_size=33)<jupyter_output>Epoch 1/3
1451/1451 [==============================] - 5s 3ms/step - loss: 3429727744.0000 - mse: 3430939648.0000
Epoch 2/3
1451/1451 [==============================] - 5s 3ms/step - loss: 3432653824.0000 - mse: 3433865984.0000
Epoch 3/3
1451/1451 [==============================] - 5s 3ms/step - loss: 3431179264.0000 - mse: 3432398080.0000<jupyter_text>Fit nn in-sample validation<jupyter_code>from sklearn.metrics import mean_squared_error

model.fit(X_train, y_train, epochs=3, validation_data=(X_valid, y_valid))<jupyter_output>Epoch 1/3
1496/1496 [==============================] - 6s 4ms/step - loss: 3431397376.0000 - mse: 3431825920.0000 - val_loss: 297629152.0000 - val_mse: 298047328.0000
Epoch 2/3
1496/1496 [==============================] - 6s 4ms/step - loss: 3432369920.0000 - mse: 3432799744.0000 - val_loss: 298028032.0000 - val_mse: 298443808.0000
Epoch 3/3
1496/1496 [==============================] - 6s 4ms/step - loss: 3430427648.0000 - mse: 3430852608.0000 - val_loss: 315636768.0000 - val_mse: 315985376.0000<jupyter_text>Test NN in test  put the result into one dataframe<jupyter_code>#Select the independent variables for test dataset
X_test = test[features].values

#Prediction using Neural Network
y_test_nn = model.predict(X_test)

## need to change the raw df_test dataset
ee_col_te = df_te['ee']
ncat_te = df_te['ncat']

ee_te = pd.DataFrame(ee_col_te)
ncat_te = pd.DataFrame(ncat_te)
pred_test_df = pd.DataFrame(y_test_nn)

nn_test_df = pd.concat([ncat_te, ee_te, pred_test_df], axis=1)

nn_test_df.to_csv('test31_df.csv',index = False)<jupyter_output><empty_output><jupyter_text>Save train pred to csv<jupyter_code>## predict in training set
tr = train[features].values
pred_train = model.predict(tr)

## put ee and ncat and pred to csv
ee_col_tr = df_tr['ee']
ncat_tr = df_tr['ncat']
ee_tr = pd.DataFrame(ee_col_tr)
ncat_tr = pd.DataFrame(ncat_tr)
pred_train_df = pd.DataFrame(pred_train)
nn_train_df = pd.concat([ncat_tr, ee_tr, pred_train_df], axis=1)
nn_train_df.to_csv('train31_df.csv',index = False)<jupyter_output><empty_output><jupyter_text>evaluate the result in validation dataset(Final_valid)  Load the validation daset daset<jupyter_code>df_valid = pd.read_csv('Validation_34.csv')
## cap in 300000
df_valid['ncat'] = np.where(df_valid['ncat'] > 300000, 300000, df_valid['ncat'])<jupyter_output><empty_output><jupyter_text>predit in the validation set<jupyter_code>## predict in validation set
val = df_valid[features].values
pred_val = model.predict(val)

## put ee and ncat and pred to csv
ee_col_val = df_valid['ee']
ncat_val = df_valid['ncat']
ee_val = pd.DataFrame(ee_col_val)
ncat_val = pd.DataFrame(ncat_val)
pred_val_df = pd.DataFrame(pred_val)
nn_valid_df = pd.concat([ncat_val, ee_val, pred_val_df], axis=1)
nn_valid_df.to_csv('valid7_df.csv',index = False)<jupyter_output><empty_output><jupyter_text>test in full_train and full_test<jupyter_code>## load full train data
df_full_tr = pd.read_csv('Train_full_34.csv')
## predict in full_train set
full_tr = df_full_tr[features].values
pred_full_tr = model.predict(full_tr)

## put ee and ncat and pred to csv
ee_col_full_tr = df_full_tr['ee']
ncat_full_tr = df_full_tr['ncat']
ee_full_tr = pd.DataFrame(ee_col_full_tr)
ncat_full_tr = pd.DataFrame(ncat_full_tr)
pred_full_tr_df = pd.DataFrame(pred_full_tr)
nn_full_tr_df = pd.concat([ncat_full_tr, ee_full_tr, pred_full_tr_df], axis=1)
nn_full_tr_df.to_csv('full_tr7_df.csv',index = False)<jupyter_output><empty_output><jupyter_text>Test in full_train dataset<jupyter_code>## load full test dataset 
df_full_te = pd.read_csv('Test_full_34.csv')
## predict in full_test set
full_te = df_full_te[features].values
pred_full_te = model.predict(full_te)

## put ee and ncat and pred to csv
ee_col_full_te = df_full_te['ee']
ncat_full_te = df_full_te['ncat']
ee_full_te = pd.DataFrame(ee_col_full_te)
ncat_full_te = pd.DataFrame(ncat_full_te)
pred_full_te_df = pd.DataFrame(pred_full_te)
nn_full_te_df = pd.concat([ncat_full_te, ee_full_te, pred_full_te_df], axis=1)
nn_full_te_df.to_csv('full_te7_df.csv',index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15525210084033614, 'fraction_numerical': 0.0773109243697479, 'mean_word_length': 7.22279792746114, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '432515', 'license': 'MIT', 'n_tokens_mistral': 2151, 'n_tokens_neox': 1870, 'n_words': 350, 'path': 'Scripts/Modeling/NNModel_prediction.ipynb', 'repo_name': 'ziqinm/MSBA-A7-Public', 'size': '4760'}