<jupyter_start><jupyter_text>This is the NLP modek for toxic comment challenge application. For it I use kaggle.com developing tools. Also the data is stored distantly in a storage called kaggle kernels<jupyter_code># code to see the input data

for k_directory, _, doc_names in os.walk('/kaggle/input'): # path for datasets
    for doc_names in doc_names:
        print(os.path.join(k_directory, doc_names))
        
# Python method walk() generates the file names in a directory tree by walking the tree either top-down or bottom-up.<jupyter_output><empty_output><jupyter_text>If we want to see content of any of these files we need to use read_csv() and after iloc() or head() to see the content.<jupyter_code># DataFram
import pandas as pd

df_ex = pd.read_csv("/kaggle/input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/amazon_cells_labelled.csv")

df_ex.head()
# df.iloc[103]<jupyter_output><empty_output><jupyter_text>It is necessary to write head() and tail() methods in different code blocks due to their execution sequence - if both were in the one block,then the last method result would be printed only.<jupyter_code>df_ex.tail()
print(df_ex.iloc[103])
print('\n')
print(df_ex.iloc[785])
# nrows : int, optional - change how many picese rows would be read
wdata_path = {'imdb_dataset':   '/kaggle/input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/imdb_labelled.txt',
         'yelp_dataset':   '/kaggle/input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/sentiment labelled sentences/yelp_labelled.txt',
         'amazon_dataset': '/kaggle/input/sentiment-labelled-sentences-data-set/sentiment labelled sentences/amazon_cells_labelled.txt'}

ds_arr = []
for origin, doc_name in wdata_path.items():
    wdataset = pd.read_csv(doc_name, names=['text','zscore'], sep='\t')
    wdataset['origin'] = origin
    ds_arr.append(wdataset)

#ndataset is data frame - each .txt file in our data repository, ds_arr  - list of 3 datasets<jupyter_output><empty_output><jupyter_text>As we have several different files in our dataset we need to concat them all together. It is recommended to do so as it would enable the working over all of them but not only over 1 file.<jupyter_code>wdataset = pd.concat(ds_arr)
wdataset.iloc[2015]<jupyter_output><empty_output><jupyter_text>Glob is one of Python many modules. It is often used to find a filename which suits prescribed features like format. It return filenames with their pathnames.<jupyter_code>pip install glob3
import os
import glob

path = '/kaggle/input/sentiment-labelled-sentences-data-set/sentiment labelled sentences'
extension = 'csv'
os.chdir(path)
result = glob.glob('*.{}'.format(extension))
print(result)
wdataset.info()<jupyter_output><empty_output><jupyter_text>Here we can see that sentence of the 2100-th position has Name: 100 and origin - amazon_dataset, so that illustrates how 3 datasets are places after each other. Amazon is placed last, the 3rd one.<jupyter_code>wdataset.iloc[2223]
from __future__ import print_function
#https://docs.python.org/2/library/__future__.html

wdataset = pd.concat(ds_arr)

print(wdataset.iloc[5])
print('\n') 
print(wdataset.iloc[500])
print('\n') 
print(wdataset.iloc[999])
print('\n') 
print(wdataset.iloc[1000])
print('\n') 
print(wdataset.iloc[1100])
print('\n') 
print(wdataset.iloc[1850])
print('\n') 
print(wdataset.iloc[1893])
print('\n') 
print(wdataset.iloc[2001])
print('\n') 
print(wdataset.iloc[2999])
print('\n')<jupyter_output><empty_output><jupyter_text>As we have several fileas in the dataset it is possible to proceed any of them separatly from each other. The path variable has been already defined.<jupyter_code>ndataset_imdb = wdataset[wdataset['origin'] == 'yelp_dataset']
ndataset_imdb.head()
ndataset_imdb.tail()<jupyter_output><empty_output><jupyter_text>So we can switch between different documents which are forming current dataset. MATPLOTLIB GRAPH as example:<jupyter_code># Counter 
from collections import Counter
# Matplot
import matplotlib.pyplot as plt
%matplotlib inline
decode_map = {0: "negative_sentiment", 1: "positive_sentiment"}
def decode_sentiment(sc):
    return decode_map[int(sc)]
%%time
wdataset.zscore = wdataset.zscore.apply(lambda x: decode_sentiment(x))<jupyter_output><empty_output><jupyter_text>On the Pyplot graph we can see how many sentences have either positive or negative sentiment. It comes out that positive sentences are slightely prevailing.<jupyter_code>zs_cnt = Counter(wdataset.zscore)

plt.figure(figsize=(22,12))
plt.bar(zs_cnt.keys(), zs_cnt.values())
plt.title("Distribuition of score sentiment of all Amazon, IMDB and Yelp together")<jupyter_output><empty_output><jupyter_text>On the code below we will find out the percise difference in amount of negative and positive sentences. We can navigate in pandas dataframe and sort the sentences:<jupyter_code>only_negative_zscores = ["0"]
wdataset.loc[wdataset['zscore'].isin(only_negative_zscores)]
only_positive_zscores = ["1"]
wdataset.loc[wdataset['zscore'].isin(only_positive_zscores)]<jupyter_output><empty_output><jupyter_text>> **RNN MODEL for Machine Learning ** vers. 1<jupyter_code>comments =  wdataset.text
sentpoints = wdataset.zscore

#example
comments.iloc[2000].lower()
from string import punctuation

def preprocess(wcontent):
    wcontent = "".join([ch for ch in wcontent if ch not in punctuation]) # w means whole
    wcontent = wcontent.lower() 
    tcomments = wcontent.split("\n") # t_ means total
    wcontent = " ".join(wcontent)
    twords = wcontent.split()
    
    return tcomments, twords


tcomments, twords = preprocess(comments)
print(twords)
from collections import Counter

word_counter = Counter(twords)
word_list = sorted(word_counter)
vocab_to_int = {word:idx+1for idx, word in enumerate(word_list)}
#print(vocab_to_int)
int_to_vocab = {idx:word for word, idx in vocab_to_int.items()}
# print(int_to_vocab)
#encoded_reviews = [[vocab_to_int[word] for word in comment] for comment in tcomments]
#print(len(int_to_vocab))
comments.iloc[2700]
def pad_text(comments, seq_length):
    
    comments_pad = []
    
    for comment in comments:
         comments_pad.append(comment[:seq_length])
        
    return np.array(comments_pad)


padded_comments = pad_text(comments, seq_length = 150)
import numpy as np
import torch

train_ratio = 0.8
valid_ratio = (1 - train_ratio)/2
total = padded_comments.shape[0]
train_cutoff = int(total * train_ratio)
valid_cutoff = int(total * (1 - valid_ratio))

train_x, train_y = padded_comments[:train_cutoff], sentpoints[:train_cutoff]

valid_x, valid_y = padded_comments[:train_cutoff : valid_cutoff], sentpoints[train_cutoff : valid_cutoff]
test_x, test_y = padded_comments[valid_cutoff:], sentpoints[valid_cutoff:]

from torch.utils.data import TensorDataset, DataLoader

train_data = TensorDataset(train_x, train_y)
valid_data = TensorDataset(valid_x, valid_y)
test_data = TensorDataset(test_x, test_y)<jupyter_output><empty_output><jupyter_text>**MODEL for Machine Learning ** vers. 2<jupyter_code>from keras import utils

# Word2vec
import gensim
# Utility
# import numpy as np
from collections import Counter
import logging
import time
import pickle
import itertools

# Set log
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO )#info
                                    
# DATASET_COLUMNS = ["text", "zscore"]
# DATASET_ENCODING = "ISO-8859-1"
# dataset_path ="../input/sentiment140/training.1600000.processed.noemoticon.csv"
# df = pd.read_csv(dataset_path, encoding =DATASET_ENCODING , names=DATASET_COLUMNS)
neg_values = ["0"]
wdataset.loc[wdataset['zscore'].isin(some_values)]
# nltk !!!!!!
import nltk
from nltk.corpus import stopwords
from  nltk.stem import SnowballStemmer
import re #important!

nltk.download('stopwords')
# TEXT CLENAING
TEXT_CLEANING_RE = "@\S+|https?:\S+|http?:\S|[^A-Za-z0-9]+"
stop_words = stopwords.words("english")

stemmer = SnowballStemmer("english")
def preprocess(text, stem=False):
    # Remove link,user and special characters
    text = re.sub(TEXT_CLEANING_RE, ' ', str(text).lower()).strip()
    tokens = []
    for token in text.split():
        if token not in stop_words:
            if stem:
                tokens.append(stemmer.stem(token))
            else:
                tokens.append(token)
    return " ".join(tokens)
%%time
wdataset.text = wdataset.text.apply(lambda x: preprocess(x))
from sklearn.model_selection import train_test_split

trsize = 0.8
wdataset_train, wdataset_test = train_test_split(wdataset, test_size=1-trsize, random_state=32)
print("train size:", len(wdataset_train))
print("test size:", len(wdataset_test))<jupyter_output>train size: 2198
test size: 550<jupyter_text>Add Gensim (Word2Vec) features<jupyter_code>sizewv = 150
windowwv = 5
epochwv = 24
mincontwv = 10
%%time
docs = [_text.split() for _text in wdataset_train.text] # _ before varibale means that its made for internal use
wvmodel = gensim.models.word2vec.Word2Vec(size=sizewv, 
                                            window=windowwv, 
                                            min_count=mincontwv, 
                                            workers=8)
wvmodel.build_vocab(docs)
voc = wvmodel.wv.vocab.keys()
voclen1 = len(voc) - 1 
print("Vocab length is", voclen)
%%time
wvmodel.train(docs, total_examples = len(docs), epochs = epochwv)
wvmodel.most_similar("excellent")
from keras.preprocessing.text import Tokenizer
%%time
tknzr = Tokenizer()
tknzr.fit_on_texts(wdataset_train.text)

voclen2 = len(tknzr.word_index) + 1
print("Total words", voclen2)
print("Total words", voclen2)
from keras.preprocessing.sequence import pad_sequences
seqlen = 150
%%time
x_train = pad_sequences(tknzr.texts_to_sequences(wdataset_train.text), maxlen = seqlen)
x_test = pad_sequences(tknzr.texts_to_sequences(wdataset_test.text), maxlen = seqlen)
# SENTIMENT
neg = "negative_sentiment"
pos = "positive_sentiment"
thresholds = (0, 1)
from sklearn.preprocessing import LabelEncoder
stringenc = LabelEncoder()
stringenc.fit(wdataset_train.zscore.tolist())

y_train = stringenc.transform(wdataset_train.zscore.tolist())
y_test = stringenc.transform(wdataset_test.zscore.tolist())

y_train = y_train.reshape(-1,1)
y_test = y_test.reshape(-1,1)

print("y_train",y_train.shape)
print("y_test",y_test.shape)
print("x_train", x_train.shape)
print("y_train", y_train.shape)
print()
print("x_test", x_test.shape)
print("y_test", y_test.shape)
y_train[:10]
#import numpy as np
embedding_matrix = np.zeros((voclen2, sizewv))
for word, i in tknzr.word_index.items():
  if word in wvmodel.wv:
    embedding_matrix[i] = wvmodel.wv[word]
print(embedding_matrix.shape)
from keras.layers import Embedding
embedding_layer = Embedding(voclen2, sizewv, weights=[embedding_matrix], input_length=seqlen, trainable=False)
# convernt string to binary
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout, Flatten, Conv1D, MaxPooling1D, LSTM
model = Sequential()
model.add(embedding_layer)
model.add(Dropout(0.5))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

model.summary()
model.compile(loss='binary_crossentropy',
              optimizer="Adam",
              metrics=['accuracy'])
from keras.callbacks import ReduceLROnPlateau, EarlyStopping
callbacks = [ ReduceLROnPlateau(monitor='val_loss', patience=5, cooldown=0),
              EarlyStopping(monitor='val_accuracy', min_delta=1e-4, patience=5)]
bs = 512
es = 6
%%time
history = model.fit(x_train, y_train,
                    batch_size=bs,
                    epochs=es,
                    validation_split=0.1,
                    verbose=1,
                    callbacks=callbacks)
%%time
tresults = model.evaluate(x_test, y_test, batch_size=bs)
print()
print("Train accuracy:",tresults[1])
print("Train loss:",tresults[0])
accuracy = history.history['accuracy']
validation_accuracy = history.history['val_accuracy']
loss = history.history['loss']
validation_loss = history.history['val_loss']
 
epochs = range(len(accuracy))
 
plt.plot(epochs, accuracy, 'b', label='Training acc')
plt.plot(epochs, validation_accuracy, 'r', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()
 
plt.figure()
 
plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, validation_loss, 'r', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
 
plt.show()
def decode_sentiment(sentimentsc, include_neutral=True):  
        return 'This comment has more negative sentiment' if sentimentsc < 0.5 else 'This comment has more positive sentiment'
def predict(text):
    start_at = time.time()
    # Tokenize text
    x_test = pad_sequences(tknzr.texts_to_sequences([text]), maxlen = seqlen)
    # Predict
    sentimentsc = model.predict([x_test])[0]
    # Decode sentiment
    message = decode_sentiment(sentimentsc)

    return {"Meesage": message, "Zscore": float(sentimentsc),
       "Spent time": time.time()-start_at}
predict("I love the music")
predict("But, in any case, the best part is, you can download these pictures to your laptop using IR, or even send pictures from your laptop to the phone.")
predict("My phone sounded OK ( not great - OK), but my wife's phone was almost totally unintelligible, she couldn't understand a word being said on it")
predict("I'm returning them.")
predict("The best thing is that I did this")
predict("The thing is that I did this but with no success")
predict("I an happy with new work")
predict("After this evening he was anxious")
%%time
y_pred_1d = []
y_test_1d = list(wdataset_test.zscore)
scores = model.predict(x_test, verbose=1, batch_size=8000)
y_pred_1d = [decode_sentiment(score, include_neutral=False) for score in scores]
# we will amnalyze the results with the help of sklearn
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.manifold import TSNE
from sklearn.feature_extraction.text import TfidfVectorizer
kerasm = "model_kmodel.h5"
keraswvm = "wvmodel_kmodel.w2v"
kerastknzrm = "kmodel_tokenizer.pkl"
kerasencm = "kmodel_encoder.pkl"
model.save(kerasm)
w2v_model.save(keraswvm)
pickle.dump(tokenizer, open(kerastknzrm, "wb"), protocol=0)
pickle.dump(encoder, open(kerasencm, "wb"), protocol=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11279977691020636, 'fraction_numerical': 0.01345510317902956, 'mean_word_length': 5.20995670995671, 'pattern_counts': {'":': 3, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '651617', 'license': 'MIT', 'n_tokens_mistral': 4975, 'n_tokens_neox': 4759, 'n_words': 1195, 'path': 'CURRENTGOODKagModel.ipynb', 'repo_name': 'zivlir/Model', 'size': '14344'}