<jupyter_start><jupyter_text>Prediction of Twitter Data using Trained Word2Seq and Word2Vec Models This notebook performs prediction on the Twitter dataset gathered by the University of Malaya Halal research group.  Therefore, the data is not publicly exposed and can be made available upon further request.  The prediction of the Twitter data uses the trained **Word2Vec** and **Word2Seq** models.  The list of models available are:* Word2Seq Convolutional Neural Network* Word2Seq Long Short Term Memory* Word2Seq Convolutional Neural Network + Long Short Term Memory* Word2Seq Convolutional Neural Nwtwork + Bi-directional Recurrent Neural Network + Bi-directional Long Short Term Memory* Word2Vec Convolutional Neural Network* Word2Vec Long Short Term Memory* Word2Vec Convolutional Neural Network + Long Short Term Memory* Word2Vec Convolutional Neural Nwtwork + Bi-directional Recurrent Neural Network + Bi-directional Long Short Term Memory  Import the required libraries<jupyter_code>import numpy as np
import pandas as pd
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
from tensorflow.python.keras.preprocessing import sequence as keras_seq
from tensorflow.python.keras.models import load_model
from tensorflow.python.keras.preprocessing import text as keras_text, sequence as keras_seq
from sklearn.utils import shuffle
from tensorflow import set_random_seed
import os
import gc

# Set seed
myrand=58584
np.random.seed(myrand)
set_random_seed(myrand)

WORDS_SIZE=8000

# To allow dynamic GPU memory allowcation for model training
config = tf.ConfigProto()
config.gpu_options.allow_growth = True  
config.log_device_placement = True

sess = tf.Session(config=config)
set_session(sess)<jupyter_output>Using TensorFlow backend.<jupyter_text>Load and prepare the collected Twitter data<jupyter_code>data = pd.read_excel('../../data.xlsx',sheet_name='Sheet1')
data.text = data.text.astype(str)
data.shape
data.head(n=10)<jupyter_output><empty_output><jupyter_text>Load and prepare the tokkenizer for Word2Seq and Word2Vec<jupyter_code>mydata = pd.read_csv('../../../../../Master (Sentiment Analysis)/Paper/Paper 3/Datasets/eRezeki/eRezeki_(text_class)_unclean.csv',header=0,encoding='utf-8')
mydata = mydata.loc[mydata['sentiment'] != "neutral"]
mydata['sentiment'] = mydata['sentiment'].map({'negative': 0, 'positive': 1})

mydata1 = pd.read_csv('../../../../../Master (Sentiment Analysis)/Paper/Paper 3/Datasets/IMDB/all_random.csv',header=0,encoding='utf-8')
mydata = mydata.append(mydata1)
mydata = shuffle(mydata)

mydata1 = pd.read_csv('../../../../../Master (Sentiment Analysis)/Paper/Paper 3/Datasets/Amazon(sports_outdoors)/Amazon_UCSD.csv',header=0,encoding='utf-8')
mydata1['feedback'] = mydata1['feedback'].astype(str)
mydata = mydata.append(mydata1)
mydata = shuffle(mydata)

mydata1 = pd.read_csv('../../../../../Master (Sentiment Analysis)/Paper/Paper 3/Datasets/Yelp(zhang_paper)/yelp_zhang.csv',header=0,encoding='utf-8')
mydata1['feedback'] = mydata1['feedback'].astype(str)
mydata = mydata.append(mydata1)

del(mydata1)
gc.collect()

mydata = shuffle(mydata)
mydata = shuffle(mydata)
mydata = shuffle(mydata)<jupyter_output><empty_output><jupyter_text>Create tokkenizer from full list of texts<jupyter_code>tokenizer = keras_text.Tokenizer(char_level=False)
tokenizer.fit_on_texts(list(mydata['feedback']))
tokenizer.num_words=WORDS_SIZE<jupyter_output><empty_output><jupyter_text>Load the trained modelsCreate dictionary for different input sizes for each model<jupyter_code>models_list = os.listdir('../Models/')
input_sizes = {'word2seq_cnn':700,
               'word2seq_cnn_birnn_bilstm':100,
               'word2seq_cnn_lstm':500,
               'word2seq_lstm':100,
               'word2vec_cnn':700,
               'word2vec_cnn_birnn_bilstm':100,
               'word2vec_cnn_lstm':500,
               'word2vec_lstm':100}<jupyter_output><empty_output><jupyter_text>Function for sequence data matrix creation from Twitter data<jupyter_code>def create_seq(input_size):
    list_tokenized = tokenizer.texts_to_sequences(list(data.text))
    x_data = keras_seq.pad_sequences(list_tokenized, 
                                     maxlen=input_size,
                                     padding='post')
    x_data = x_data.astype(np.int64)
    return(x_data)<jupyter_output><empty_output><jupyter_text>Function for predict the data<jupyter_code>def predict_data(model,x_data):
    sentiment = model.predict_classes(x_data)
    sentiment = sentiment.astype(str)
    sentiment[sentiment=='1'] = "Positive"
    sentiment[sentiment=='0'] = "Negative"
    probability = model.predict_proba(x_data)
    positive_probability = probability[:,1]
    negative_probabiltiy = probability[:,0]
    return(sentiment, positive_probability, negative_probabiltiy)<jupyter_output><empty_output><jupyter_text>Function to add new column to the excel dataframe<jupyter_code>def add_columns(data, model_name, sentiment, positive_probability, negative_probabiltiy):
    name_1 = '%s_sentiment' % (model_name)
    name_2 = '%s_posProb' % (model_name)
    name_3 = '%s_negProb' % (model_name)
    data[name_1] = sentiment
    data[name_2] = positive_probability
    data[name_3] = negative_probabiltiy
    return(data)<jupyter_output><empty_output><jupyter_text>Start looping to predict the sentiment<jupyter_code>for name in models_list:
    x_data = create_seq(input_sizes[name])
    mydir = '../Models/%s/%s.hdf5' % (name,name)
    model = load_model(mydir)
    sentiment, positive_prob, negative_prob = predict_data(model, x_data)
    data = add_columns(data, name, sentiment, positive_prob, negative_prob)

data.head(n=10)<jupyter_output>/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "<jupyter_text>Calcuate the weighted sentiments through average of probabilities* Weighted probability of positive sentiment* Weighted probability of negative sentiment* Weighted sentiment<jupyter_code>data['weighted_sentiment'] = np.str
data['weighted_posProb'] = np.zeros
data['weighted_negProb'] = np.zeros

for i in range(data.shape[0]):
    data.loc[i,'weighted_posProb'] = np.mean(data.iloc[i,[6, 9, 12, 15,18, 21, 24, 27]].values)
    data.loc[i,'weighted_negProb'] = np.mean(data.iloc[i,[7, 10, 13, 16, 19, 22, 25, 28]].values)
    
    if data.loc[i,'weighted_posProb'] > data.loc[i,'weighted_negProb']:
        data.loc[i,'weighted_sentiment'] = "Positive"
    elif data.loc[i,'weighted_posProb'] == data.loc[i,'weighted_negProb']:
        data.loc[i,'weighted_sentiment'] = "Neutral"
    else:
        data.loc[i,'weighted_sentiment'] = "Negative"
    
data.head(n=10)<jupyter_output><empty_output><jupyter_text>Save the data to a new excel file<jupyter_code>data.columns
data.to_excel('../../data_predicted.xlsx')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11518175338560228, 'fraction_numerical': 0.01924447612259444, 'mean_word_length': 5.3036837376460015, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '626730', 'license': 'MIT', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2279, 'n_words': 488, 'path': 'Scripts/.ipynb_checkpoints/Prediction_Twitter_data-checkpoint.ipynb', 'repo_name': 'hazimhanif/twitter-halal-sentiment', 'size': '7015'}