<jupyter_start><jupyter_text>BiDirectional LSTM classifier in keras  Load dependencies<jupyter_code>import keras
from keras.datasets import imdb
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Embedding, SpatialDropout1D, Dense, Flatten, Dropout, LSTM
from keras.layers.wrappers import Bidirectional
from keras.callbacks import ModelCheckpoint # new! 
import os # new! 
from sklearn.metrics import roc_auc_score, roc_curve # new!
import matplotlib.pyplot as plt # new!
%matplotlib inline
# output directory name:
output_dir = 'model_output/bilstm'

# training:
epochs = 6
batch_size = 128

# vector-space embedding: 
n_dim = 64
n_unique_words = 10000
max_review_length = 200
pad_type = trunc_type = 'pre'
drop_embed = 0.2

# neural network architecture: 
n_lstm = 256
droput_lstm = 0.2<jupyter_output><empty_output><jupyter_text>Load data For a given data set: * the Keras text utilities [here](https://keras.io/preprocessing/text/) quickly preprocess natural language and convert it into an index* the `keras.preprocessing.text.Tokenizer` class may do everything you need in one line:    * tokenize into words or characters    * `num_words`: maximum unique tokens    * filter out punctuation    * lower case    * convert words to an integer index<jupyter_code>(x_train, y_train), (x_valid, y_valid) = imdb.load_data(num_words=n_unique_words)<jupyter_output><empty_output><jupyter_text>Preprocess data<jupyter_code>x_train = pad_sequences(x_train, maxlen=max_review_length, padding=pad_type, truncating=trunc_type, value=0)
x_valid = pad_sequences(x_valid, maxlen=max_review_length, padding=pad_type, truncating=trunc_type, value=0)
x_train[:6]
for i in range(6):
    print len(x_train[i])<jupyter_output>200
200
200
200
200
200<jupyter_text>Design neural network architecture<jupyter_code>model = Sequential()
model.add(Embedding(n_unique_words, n_dim, input_length=max_review_length)) 
model.add(SpatialDropout1D(drop_embed))
# model.add(Conv1D(n_conv, k_conv, activation='relu'))
# model.add(Conv1D(n_conv, k_conv, activation='relu'))
# model.add(GlobalMaxPooling1D())
# model.add(Dense(n_dense, activation='relu'))
# model.add(Dropout(dropout))
model.add(Bidirectional(LSTM(n_lstm, dropout=droput_lstm)))
model.add(Dense(1, activation='sigmoid'))
model.summary()
n_dim, n_unique_words, n_dim * n_unique_words
max_review_length, n_dim, n_dim * max_review_length<jupyter_output><empty_output><jupyter_text>Configure Model<jupyter_code>model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
modelcheckpoint = ModelCheckpoint(filepath=output_dir+"/weights.{epoch:02d}.hdf5")
if not os.path.exists(output_dir):
    os.makedirs(output_dir)
model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, 
          verbose=1, validation_data=(x_valid, y_valid), callbacks=[modelcheckpoint])<jupyter_output>Train on 25000 samples, validate on 25000 samples
Epoch 1/6
25000/25000 [==============================] - 689s 28ms/step - loss: 0.5523 - acc: 0.7024 - val_loss: 0.3817 - val_acc: 0.8368
Epoch 2/6
  640/25000 [..............................] - ETA: 7:21 - loss: 0.2883 - acc: 0.8953<jupyter_text>Evaluate<jupyter_code>model.load_weights(output_dir+"/weights.01.hdf5") # zero-indexed
y_hat = model.predict_proba(x_valid)
len(y_hat)
y_hat[0]
plt.hist(y_hat)
_ = plt.axvline(x=0.5, color='orange')
pct_auc = roc_auc_score(y_valid, y_hat)*100.0
"{:0.2f}".format(pct_auc)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1292166952544311, 'fraction_numerical': 0.038593481989708404, 'mean_word_length': 6.7240618101545255, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '3531', 'license': 'Apache-2.0', 'n_tokens_mistral': 1371, 'n_tokens_neox': 1270, 'n_words': 253, 'path': 'Deep Learning/NLP Tutorials/.ipynb_checkpoints/Conv_LSTM_Stack_in_keras-checkpoint.ipynb', 'repo_name': 'surajch77/ML-Examples', 'size': '3498'}