<jupyter_start><jupyter_text>Preprocess (encode text to number)<jupyter_code># movie_reviews_train = [
#          {'review': 'this is the best movie', 'sentiment': 'positive'},
#          {'review': 'i recommend you watch this movie', 'sentiment': 'positive'},
#          {'review': 'it was waste of money and time', 'sentiment': 'negative'},
#          {'review': 'the worst movie ever', 'sentiment': 'negative'}
#     ]
# df = pd.DataFrame(movie_reviews_train)

# max_length is used for max sequence of input
max_length = get_max_length(df)

X, y= preprocess(df)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
X_train<jupyter_output><empty_output><jupyter_text>Build Model<jupyter_code># LSTM model
model = Sequential()
model.add(LSTM(32))
model.add(Dropout(0.5))
model.add(Dense(8, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(8, activation='relu'))
model.add(Dropout(0.5))
from tensorflow.keras import regularizers
model.add(Dense(3, activation='softmax'))

model.summary
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy',tf.keras.metrics.AUC(),tf.keras.metrics.Recall(),tf.keras.metrics.Precision()]
             #,metrics=[tf.keras.metrics.AUC()]
             )<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>print('Train...')
model.fit(X_train, y_train,epochs=100,
          validation_data=(X_test,y_test),
          batch_size=32)
model.summary()<jupyter_output>Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_4 (LSTM)                (None, 32)                36224     
_________________________________________________________________
dropout_13 (Dropout)         (None, 32)                0         
_________________________________________________________________
dense_13 (Dense)             (None, 8)                 264       
_________________________________________________________________
dropout_14 (Dropout)         (None, 8)                 0         
_________________________________________________________________
dense_14 (Dense)             (None, 8)                 72        
_________________________________________________________________
dropout_15 (Dropout)         (None, 8)                 0         
______________________________________________________[...]<jupyter_text>Testyour model can predict correctly even for unseen words from training.  This is the most benefit of using pretrained word embedding.  Why? pretrained embedding will encode [better], [nice] to similar vector of [best]  even if these words were not in train.  therefore, the input vector to RNN is similar, so correct answers for even these unseen words.<jupyter_code>test_df = df

test_X, test_Y = preprocess(test_df)
test_X

score, acc = model.evaluate(test_X, test_Y, verbose=2)
print('Test score:', score)
print('Test accuracy:', acc)
test_df
df.dtypes

def value_transform(x):
  if x<0.5:
    x=0
  else:
    x=1
  return x
# df['Text'] = np.asarray(df['Text']).astype(np.int64)
preds = model.predict(test_X)
preds = np.array([[value_transform(j) for j in preds[i]] for i in range(20148 - DropDataNumber)], dtype='int32')
from sklearn.metrics import accuracy_score, confusion_matrix
preds.ndim

preds.shape
test_Y.shape
accuracy_score(test_Y.argmax(axis=1), preds.argmax(axis=1))
confusion_matrix(test_Y.argmax(axis=1), preds.argmax(axis=1))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12981969486823855, 'fraction_numerical': 0.019972260748959778, 'mean_word_length': 4.122159090909091, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '235584', 'license': 'MIT', 'n_tokens_mistral': 1114, 'n_tokens_neox': 1056, 'n_words': 242, 'path': 'HateSpeechDetection.ipynb', 'repo_name': 'vikasgc09/DetectingHateSpeech', 'size': '3605'}