<jupyter_start><jupyter_text>Introduction This problem is a old competition from Kaggle which can be found here: https://www.kaggle.com/c/movie-review-sentiment-analysis-kernels-only/overviewThe basic overview of the problem is as follows: The dataset is a set of reviews from rotten tomatoes, a website for reviewing films. We want to be able to input a review of a film and about a score of the film. In this setting the score is the label that is output from a model and can have five distinct value. The sentiment labels are:0 - negative1 - somewhat negative2 - neutral3 - somewhat positive4 - positiveThe goal is to be able to this process with as high accuracy as possible. For more information visit the kaggle competition website.<jupyter_code>import numpy as np 
import pandas as pd 
import os
print(os.listdir("../input"))
import gc

from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from keras.utils import to_categorical

from keras.models import Sequential
from keras.layers import Dense,Dropout,Embedding,LSTM,Conv1D,GlobalMaxPooling1D,Flatten,MaxPooling1D,GRU,SpatialDropout1D,Bidirectional
from keras.callbacks import EarlyStopping
from keras.utils import to_categorical
from keras.losses import categorical_crossentropy
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score,confusion_matrix,classification_report,f1_score
import matplotlib.pyplot as plt<jupyter_output>['train.tsv.zip', 'sampleSubmission.csv', 'test.tsv.zip']<jupyter_text>These are the files avalible in the Kaggle kernel.   EDA<jupyter_code>gc.collect()<jupyter_output><empty_output><jupyter_text>Loading dataset and basic visualization<jupyter_code>train=pd.read_csv('../input/train.tsv.zip',sep='\t')
print(train.shape)
train.head()
test=pd.read_csv('../input/test.tsv.zip',sep='\t')
print(test.shape)
test.head()
sub=pd.read_csv('../input/sampleSubmission.csv')
sub.head()<jupyter_output><empty_output><jupyter_text>Lets get some feeling for how a sentence is broken down into several phrases.<jupyter_code>train.loc[train.SentenceId == 17]<jupyter_output><empty_output><jupyter_text>Here we see a potential problem. Differnet phrases in the same sentence will get different sentiments. It is especially intresting that the phrase "billingual charmer" has sentiment 3, while "billingual" and "charmer" gets a "2". This means that words do not only impact seperally but can magnify each others meaning. However, in some cases that is not obious. For example "inspired it" and "inspired" have different sentiments.<jupyter_code>print('Average count of phrases per sentence in train is {0:.0f}.'.format(train.groupby('SentenceId')['Phrase'].count().mean()))
print('Average count of phrases per sentence in test is {0:.0f}.'.format(test.groupby('SentenceId')['Phrase'].count().mean()))
print('Number of phrases in train: {}. Number of sentences in train: {}.'.format(train.shape[0], len(train.SentenceId.unique())))
print('Number of phrases in test: {}. Number of sentences in test: {}.'.format(test.shape[0], len(test.SentenceId.unique())))
print('Average word length of phrases in train is {0:.0f}.'.format(np.mean(train['Phrase'].apply(lambda x: len(x.split())))))
print('Average word length of phrases in test is {0:.0f}.'.format(np.mean(test['Phrase'].apply(lambda x: len(x.split())))))<jupyter_output>Average count of phrases per sentence in train is 18.
Average count of phrases per sentence in test is 20.
Number of phrases in train: 156060. Number of sentences in train: 8529.
Number of phrases in test: 66292. Number of sentences in test: 3310.
Average word length of phrases in train is 7.
Average word length of phrases in test is 7.<jupyter_text>Wordcloud<jupyter_code>df_train_first = train.loc[train['SentenceId'].diff() != 0] #only extracts the first phrase for each sentence (full sentence)
from wordcloud import WordCloud,STOPWORDS
import matplotlib.pyplot as plt 


df_temp = df_train_first.loc[df_train_first['Sentiment']==4]

words = ' '.join(df_temp['Phrase'])
cleaned_word = " ".join([word for word in words.split()
                            if 'http' not in word
                                and not word.startswith('@')
                                and word != 'RT'
                                and word !='&amp'
                            ])

stopwords = set(STOPWORDS)
wordcloud = WordCloud(stopwords=stopwords,
                      background_color='black',
                      width=3000,
                      height=2500
                     ).generate(cleaned_word)

plt.figure(1,figsize=(12, 12))
plt.imshow(wordcloud)
plt.axis('off')
plt.show()
df_temp = df_train_first[df_train_first['Sentiment']==0]

words = ' '.join(df_temp['Phrase'])
cleaned_word = " ".join([word for word in words.split()
                            if 'http' not in word
                                and not word.startswith('@')
                                and word != 'RT'
                                and word !='&amp'
                            ])

stopwords = set(STOPWORDS)

wordcloud = WordCloud(stopwords=stopwords,
                      background_color='black',
                      width=3000,
                      height=2500
                     ).generate(cleaned_word)

plt.figure(1,figsize=(12, 12))
plt.imshow(wordcloud)
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>Pre-processing  **Adding Sentiment column to test datset and joing train and test for preprocessing**<jupyter_code>test['Sentiment']= -999
test.head()
df=pd.concat([train,test],ignore_index=True)
print(df.shape)

del train,test # free up space
gc.collect() 
from nltk.tokenize import word_tokenize
from nltk import FreqDist
from nltk.stem import SnowballStemmer,WordNetLemmatizer
stemmer=SnowballStemmer('english')
lemma=WordNetLemmatizer()
from string import punctuation
import re
def clean_review(review_col):
    review_corpus=[]
    for i in range(0,len(review_col)):
        review=str(review_col[i])
        review=re.sub('[^a-zA-Z]',' ',review)
        #review=[stemmer.stem(w) for w in word_tokenize(str(review).lower())]
        review=[lemma.lemmatize(w) for w in word_tokenize(str(review).lower())]
        review=' '.join(review)
        review_corpus.append(review)
    return review_corpus
df['clean_review']=clean_review(df.Phrase.values)
df.head()
df_train=df[df.Sentiment!= -999]
df_train.shape

df_test=df[df.Sentiment == -999]
df_test.shape

del df
gc.collect()<jupyter_output><empty_output><jupyter_text>Split the data<jupyter_code>train_text=df_train.clean_review.values
test_text=df_test.clean_review.values
target=df_train.Sentiment.values
y=to_categorical(target)
print(train_text.shape,target.shape,y.shape)

# stratify allows equal split over all classes
X_train_text,X_val_text,y_train,y_val=train_test_split(train_text,y,test_size=0.2,stratify=y,random_state=123)
print(X_train_text.shape,y_train.shape)
print(X_val_text.shape,y_val.shape)<jupyter_output>(124848,) (124848, 5)
(31212,) (31212, 5)<jupyter_text>Modeling   Tokenization<jupyter_code>#determining number of unique words
all_words=' '.join(X_train_text)
all_words=word_tokenize(all_words)
dist=FreqDist(all_words)
num_unique_word=len(dist)
num_unique_word
# Determening the maximum lenght of a review (number of words)
r_len=[]
for text in X_train_text:
    word=word_tokenize(text)
    l=len(word)
    r_len.append(l)
    
MAX_REVIEW_LEN=np.max(r_len)
MAX_REVIEW_LEN
max_features = num_unique_word
max_words = MAX_REVIEW_LEN
from keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(list(X_train_text))
X_train = tokenizer.texts_to_sequences(X_train_text)
X_val = tokenizer.texts_to_sequences(X_val_text)
X_test = tokenizer.texts_to_sequences(test_text)
from keras.preprocessing import sequence,text
''' 
This function transforms a list (of length num_samples) of sequences (lists of integers) 
into a 2D Numpy array of shape (num_samples, num_timesteps). 
num_timesteps is either the maxlen argument if provided,
or the length of the longest sequence in the list. '''

X_train = sequence.pad_sequences(X_train, maxlen=max_words)
X_val = sequence.pad_sequences(X_val, maxlen=max_words)
X_test = sequence.pad_sequences(X_test, maxlen=max_words)
print(X_train.shape,X_val.shape,X_test.shape)<jupyter_output>(124848, 48) (31212, 48) (66292, 48)<jupyter_text>logistic regressionIs used as a baseline model<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
tfidf=TfidfVectorizer(ngram_range=(1,2),max_df=0.95,min_df=10,sublinear_tf=True)
X_train_vec=tfidf.fit_transform(df_train.clean_review).toarray()
print(X_train_vec.shape)
X_test_vec =tfidf.transform(df_test.clean_review).toarray()
print(X_test_vec.shape)
from sklearn.preprocessing import LabelEncoder,OneHotEncoder
le=LabelEncoder()
y_train_enc=le.fit_transform(df_train.Sentiment.values)
y_train_enc.shape
logreg = LogisticRegression()
#ovr_logreg = OneVsRestClassifier(logreg) # this strategy consists in fitting one classifier per class
%%time
logreg.fit(X_train_vec, y_train_enc)
log_pred=logreg.predict(X_test_vec)

df_test['Logistic_Sentiment'] = log_pred
df_test.head()
del X_train_vec, X_test_vec, y_train_enc, logreg,le,tfidf
gc.collect()<jupyter_output><empty_output><jupyter_text>LSTM<jupyter_code>lstm=Sequential()
lstm.add(Embedding(max_features,100,mask_zero=True))
lstm.add(LSTM(64,dropout=0.4, recurrent_dropout=0.4,return_sequences=True))
lstm.add(LSTM(32,dropout=0.5, recurrent_dropout=0.5,return_sequences=False))
lstm.add(Dense(5,activation='softmax'))
lstm.compile(loss='categorical_crossentropy',optimizer=Adam(lr=0.001),metrics=['accuracy'])
lstm.summary()
%%time

epochs = 3
batch_size = 128

lstm_history =lstm.fit(X_train, y_train, validation_data=(X_val, y_val),epochs=epochs, batch_size=batch_size, verbose=1)
lstm_pred=lstm.predict_classes(X_test,verbose=1)

df_test['lstm_sentiment'] = lstm_pred
df_test.head()
del lstm,lstm_history,lstm_pred
gc.collect()<jupyter_output><empty_output><jupyter_text>Vader<jupyter_code>from nltk.sentiment.vader import SentimentIntensityAnalyzer

def print_sentiment_scores(sentence):
    snt = SentimentIntensityAnalyzer().polarity_scores(sentence)  #Calling the polarity analyzer
    print("{:-<40} {}".format(sentence, str(snt)))<jupyter_output><empty_output><jupyter_text>Lets try it out<jupyter_code>print_sentiment_scores('bad movie is bad')
print_sentiment_scores('bad movie is really bad')
print_sentiment_scores('ok movie is ok')
print_sentiment_scores('good movie is good')
print_sentiment_scores('good movie is really good')
print_sentiment_scores('This movie is so bad that it is good')
print_sentiment_scores('This movie is so good that it is bad')
print_sentiment_scores('This movie is the shit')
print_sentiment_scores('This movie is shit')
print_sentiment_scores('The plot was obvious')
%%time 

i=0 #counter

compval1 = [ ]  #empty list to hold our computed 'compound' VADER scores

for sentence in df_train['Phrase']:
    if i % 5000 == 0:
        print(i,'samples done')
    k = SentimentIntensityAnalyzer().polarity_scores(sentence)
    compval1.append(k['compound'])
    
    i = i+1

    
#converting sentiment values to numpy for easier usage

compval1 = np.array(compval1)

len(compval1)
df_vader = df_train.copy()
df_vader['VADER_score'] = compval1
df_vader.head()<jupyter_output><empty_output><jupyter_text>A VADER score of -1 corresponds to a negative review, a score of 1 corresponds to a positive review, and a score of 0 corresponds to a ok review. As we have five classes, the delimitations could be -0.6, -0.2, 0.2, 0.6 for a negative to positive review. However, for this data, are we sure that we should draw the boundaries there? Is a review of -0.19 really ok when it is mostly negative? What does, an ok review mean in this context and in relation to the VADER score?Lets find out...<jupyter_code>import seaborn as sns
import matplotlib as plt<jupyter_output><empty_output><jupyter_text>Many ofthe phrases are nonsensical and will give a vader score of 0. As many phrases are such in the dataset, the vader score will presumably be biased towards a neutral score.<jupyter_code>df_vader_non_zero = df_vader.loc[df_vader['VADER_score'] != 0.0000]
#sns.displot(data = df_vader, x="VADER_score", hue="species", stat="Sentiment", fill=True)
sns.distplot(df_vader_non_zero['VADER_score'])
sns.distplot(df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 0]['VADER_score'])
print('Mean value:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 0]['VADER_score'].mean())
print('Std:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 0]['VADER_score'].std())

df_temp = df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 0]
df_temp.loc[df_temp['VADER_score'] > 0.5].sample(5)
sns.distplot(df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 1]['VADER_score'])
print('Mean value:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 1]['VADER_score'].mean())
print('Std:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 1]['VADER_score'].std())
sns.distplot(df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 2]['VADER_score'])
print('Mean value:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 2]['VADER_score'].mean())
print('Std:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 2]['VADER_score'].std())
sns.distplot(df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 3]['VADER_score'])
print('Mean value:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 3]['VADER_score'].mean())
print('Std:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 3]['VADER_score'].std())
sns.distplot(df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 4]['VADER_score'])
print('Mean value:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 4]['VADER_score'].mean())
print('Std:',df_vader_non_zero.loc[df_vader_non_zero['Sentiment'] == 4]['VADER_score'].std())<jupyter_output>Mean value: 0.506208796481403
Std: 0.29923221744090833<jupyter_text>Prediction with logistic regression In this case we use the logistic regression model to map the vader score to the sentiment score.<jupyter_code>x =df_vader['VADER_score'].values.reshape(-1,1)
y = df_vader['Sentiment'].values.reshape(-1,1)
print(x.shape)
print(y.shape)
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state=0)
clf.fit(x, y)<jupyter_output><empty_output><jupyter_text>Lets get the vader score for test data<jupyter_code>%%time 


i=0 #counter

compval2 = [ ]  #empty list to hold our computed 'compound' VADER scores


for sentence in df_test['Phrase']:
    if i % 5000 == 0:
        print(i,'samples done')
    k = SentimentIntensityAnalyzer().polarity_scores(sentence)
    compval2.append(k['compound'])   
    i = i+1
    
#converting sentiment values to numpy for easier usage
print('DONE')
compval2 = np.array(compval2)

len(compval2)
df_test['VADER_score_test'] = compval2

clf_vader_pred = clf.predict(df_test['VADER_score_test'].values.reshape(-1,1))
df_test['VADER_sentiment'] = clf_vader_pred
df_test['VADER_sentiment'].value_counts()
df_test.head()
df_test.to_csv('sentiment_all_models.csv',index=False)<jupyter_output><empty_output><jupyter_text>Prediction with manual set bins<jupyter_code>%%time 


i=0 #counter

compval2 = [ ]  #empty list to hold our computed 'compound' VADER scores


for sentence in df_test['Phrase']:
    if i % 5000 == 0:
        print(i,'samples done')
    k = SentimentIntensityAnalyzer().polarity_scores(sentence)
    compval2.append(k['compound'])
    
    i = i+1
    
#converting sentiment values to numpy for easier usage
print('DONE')
compval2 = np.array(compval2)

len(compval2)
df_test['VADER_score'] = compval2

%time

#Assigning score categories and logic
i = 0

predicted_value = [ ] #empty series to hold our predicted values

while(i<len(df_test)):
    if i % 5000 == 0:
        print(i,'samples done')
    if ((df_test.iloc[i]['VADER_score'] >= 0.4)):
        predicted_value.append(4)
        i = i+1
    elif ((df_test.iloc[i]['VADER_score'] > 0.15) & (df_test.iloc[i]['VADER_score'] < 0.4)):
        predicted_value.append('3')
        i = i+1
    elif ((df_test.iloc[i]['VADER_score'] > -0.075) & (df_test.iloc[i]['VADER_score'] < 0.15)):
        predicted_value.append('2')
        i = i+1
    elif ((df_test.iloc[i]['VADER_score'] > -0.15) & (df_test.iloc[i]['VADER_score'] < -0.075)):
        predicted_value.append('1')
        i = i+1
    elif ((df_test.iloc[i]['VADER_score'] <= -0.15)):
        predicted_value.append('0')
        i = i+1
        
df_test['VADER_sentiment'] = predicted_value

del df_vader,compval1,compval2,logreg,df_temp,x,y,logreg

gc.collect()<jupyter_output><empty_output><jupyter_text>CNN<jupyter_code>cnn= Sequential()
cnn.add(Embedding(max_features,100,input_length=max_words))
cnn.add(Conv1D(64,kernel_size=3,padding='same',activation='relu',strides=1))
cnn.add(MaxPooling1D(3))
cnn.add(Conv1D(64,kernel_size=3,padding='same',activation='relu',strides=1))
cnn.add(GlobalMaxPooling1D())
cnn.add(Dense(128,activation='relu'))
cnn.add(Dropout(0.4))
cnn.add(Dense(128))
cnn.add(Dense(5,activation='softmax'))

cnn.summary()
cnn.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
%%time
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint

num_epochs = 3
batch_size = 128

cnn_history =cnn.fit(X_train, y_train, validation_data=(X_val, y_val),epochs=num_epochs, batch_size=batch_size, verbose=1)
cnn_pred =cnn.predict_classes(X_test, verbose=1)

df_test['CNN_Sentiment'] = cnn_pred
df_test.to_csv('sentiment_all_models_V2.csv',index=False)<jupyter_output><empty_output><jupyter_text>CNN-LSTM<jupyter_code>cnnlstm= Sequential()
cnnlstm.add(Embedding(max_features,100,input_length=max_words))
cnnlstm.add(Conv1D(64,kernel_size=3,padding='same',activation='relu',strides=1))
cnnlstm.add(MaxPooling1D(3))
cnnlstm.add(Conv1D(64,kernel_size=3,padding='same',activation='relu',strides=1))
cnnlstm.add(MaxPooling1D(3))
cnnlstm.add(LSTM(64,dropout=0.4, recurrent_dropout=0.4,return_sequences=False))
cnnlstm.add(Dense(128,activation='relu'))
cnnlstm.add(Dropout(0.4))
cnnlstm.add(Dense(128))
cnnlstm.add(Dense(5,activation='softmax'))

cnnlstm.summary()
cnnlstm.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
%%time
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint

num_epochs = 3
batch_size = 128

cnnlstm_history =cnnlstm.fit(X_train, y_train, validation_data=(X_val, y_val),epochs=num_epochs, batch_size=batch_size, verbose=1)
cnnlstm_pred = cnnlstm.predict_classes(X_test, verbose=1)

df_test['CNN-LSTM_Sentiment'] = cnnlstm_pred
df_test.to_csv('sentiment_all_models_V3.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12483601826100646, 'fraction_numerical': 0.021776774938342867, 'mean_word_length': 5.791874554526015, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 81, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '496642', 'license': 'MIT', 'n_tokens_mistral': 7088, 'n_tokens_neox': 6752, 'n_words': 1354, 'path': 'sentiment-analysis-rotten-tomatoes.ipynb', 'repo_name': 'LarssonHampus/Movie_Reviews_Sentiment_Analysis', 'size': '19057'}