<jupyter_start><jupyter_text>Sentiment Analysis using NLP (Spacy/ntlk) Check out Spacy's API! It's very extensive.https://spacy.io/api/doc<jupyter_code># Before you begin, make sure to install spacy and download 1 of the language models that have a host of relationships already
# built in! Each model comes with prebuilt tokenization icons, Tags, dependencies, sentence segmentation, and entities!

# pip install spacy

# Download the large word vector models from spacy
# Other Language models to be downloaded here: https://spacy.io/usage/models

# python -m spacy download en_core_web_sm
# python -m spacy download en_core_web_md
# python -m spacy download en_core_web_lg<jupyter_output><empty_output><jupyter_text>If you're not familiar with Basic Machine Learning Concepts and Logistic Regression, check out these videos to get caught up! (Links are in the description)![Logistic_Regression](logisticRegression.PNG)<jupyter_code># DataFrame
import pandas as pd

# Matplot
import matplotlib.pyplot as plt
%matplotlib inline

# Scikit-learn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.manifold import TSNE
from sklearn.feature_extraction.text import TfidfVectorizer

# Keras
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout, Embedding, Flatten, Conv1D, MaxPooling1D, LSTM
from keras import utils
from keras.callbacks import ReduceLROnPlateau, EarlyStopping

import spacy

# nltk
import nltk
from nltk.corpus import stopwords
from  nltk.stem import SnowballStemmer

# Word2vec
import gensim

# Utility
import re
import numpy as np
from collections import Counter
import time<jupyter_output>c:\users\spenc\anaconda3\envs\nlp\lib\site-packages\gensim\similarities\__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package <https://pypi.org/project/python-Levenshtein/> is unavailable. Install Levenhstein (e.g. `pip install python-Levenshtein`) to suppress this warning.
  warnings.warn(msg)<jupyter_text>Preprocessing text data includes the following steps:- Tokeniziation (Break down document into words, sentences, or other categories)- Removal of unnecessary text that don't add a lot of value to our machine learning model.- Normalization (simplifying word to its root form -- you can think of this as getting the infinitive of a verb)  Let's start with Tokenization<jupyter_code>text = """
Hi! My name is Spencer and I love doing videos related to tech implementation/strategy. If you like finance, tech, or real world
Q&A videos about Data Science, Data Engineering, and Data Analyst related topics, let me know in the comments down below! *psst*
If you made it this far in the video, why don't you subscribe? I upload videos every week on a Sunday ~7:00 PM EST. If you like
this type of content make sure to hit that like button! That really helps out with the growth of this channel. :)
"""
# In the NLP pipeline world, you would typically preprocess your data before you start feeding it to your NLP model.
# For instance, let's begin by removing special characters.
# In the real world, you would have many data streams coming in in different formats. And, you would run your various regex
# functions to identify known patterns of text and store them in JSONL or other data storage files.
pattern = r'[^A-Za-z ]'
regex = re.compile(pattern)
result = regex.sub('', text)
result
# Load in the NLP model that you have chosen to downloaded; I have the large model.
nlp = spacy.load("en_core_web_lg")
doc = nlp(result)
# Let's get each individual word as an element.
tokens = [token for token in doc]
tokens
'''
    The idea behind normalizing words (lemmatization) seeks to convert your text to the 'base' format.
    Print out the following: Token, Check if stop word, lemma version.
'''
for t in tokens:
    print('Token is : ', t,'--- Is this a stop word? ', t.is_stop, '--- Lemmatized token is: ', t.lemma_)

# Store the lemmas without the words.
lemmas = [t.lemma_ for t in tokens if not t.is_stop]
tokens[1]<jupyter_output><empty_output><jupyter_text>From this point on, I used a lot of the code from this kaggle notebook (And gained much of my inspiration to look into NLP). SO DO CHECK IT OUT! Link is in the description.  https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis/output<jupyter_code># Reading in twitter data on sentiment. (NEGATIVE, POSITIVE for target)
# Already cleaned and preprocessed...
df = pd.read_csv('twitter_data.csv')
df = df.sample(frac=1).reset_index()
df = df.drop(['index'], axis = 1)
df
nltk.download('stopwords')
stop_words = stopwords.words("english")
stemmer = SnowballStemmer("english")

def preprocess(text, stem=False):
    # Remove link,user and special characters
    text = re.sub("@\S+|https?:\S+|http?:\S|[^A-Za-z0-9]+", ' ', str(text).lower()).strip()
    tokens = []
    for token in text.split():
        if token not in stop_words:
            if stem:
                tokens.append(stemmer.stem(token))
            else:
                tokens.append(token)
    return " ".join(tokens)
%%time
df.text = df.text.apply(lambda x: preprocess(x)) # preprocessing the text data.
df.text[0]
# Split into train and test dataset.
df_train, df_test = train_test_split(df, test_size=0.2, random_state=42)
print("TRAIN size:", len(df_train))
print("TEST size:", len(df_test))
df.text[0:10]<jupyter_output><empty_output><jupyter_text>Word2vecThis neural network vectorizes the words so that each 'text' can be understood in a neural network. But, the main purpose of the 2 layer neural network is to convert text to a vector, where the vector can be read in to a future neural network AND has a relationship embeddings behind the vectorized values based on cosine similarity.<jupyter_code>%%time
documents = [_text.split() for _text in df_train.text] 
w2v_model = gensim.models.word2vec.Word2Vec(vector_size =300, # vector size 
                                            window=7, # distance between current and predicted word within a sentence
                                            min_count=10, # ignores words with total frequency less than the parameter
                                            workers=8) # threads
w2v_model.build_vocab(documents)
words = w2v_model.wv.index_to_key
vocab_size = len(words)
print("Vocab size", vocab_size)
%%time
w2v_model.train(documents, total_examples=len(documents), epochs=32)
w2v_model.wv.most_similar("like")
w2v_model.wv.most_similar("comment")
# w2v_model.wv.most_similar("subscribe")<jupyter_output><empty_output><jupyter_text>Tokenize Text & Create Embedding Layer.Once you have created a word2vec model, go back and your observations to a token.<jupyter_code>%%time
tokenizer = Tokenizer()
tokenizer.fit_on_texts(df_train.text)

vocab_size = len(tokenizer.word_index) + 1
print("Total words", vocab_size)
%%time
x_train = pad_sequences(tokenizer.texts_to_sequences(df_train.text), maxlen=300)
x_test = pad_sequences(tokenizer.texts_to_sequences(df_test.text), maxlen=300)
print(len(x_train), len(x_test))
# Creating an embedding layer that will act as an input layer for the neural network.
embedding_matrix = np.zeros((vocab_size, 300))
for word, i in tokenizer.word_index.items():
    if word in w2v_model.wv:
        embedding_matrix[i] = w2v_model.wv[word]
print(embedding_matrix.shape)
# used in the future.
embedding_layer = Embedding(vocab_size, 300, weights=[embedding_matrix], input_length=300, trainable=False)<jupyter_output>(48018, 300)<jupyter_text>Label Encoder (Encoding the Dependent variables to 0 or 1)<jupyter_code>labels = df_train.target.unique().tolist()
labels
encoder = LabelEncoder()
encoder.fit(df_train.target.tolist())

y_train = encoder.transform(df_train.target.tolist())
y_test = encoder.transform(df_test.target.tolist())

y_train = y_train.reshape(-1,1)
y_test = y_test.reshape(-1,1)

print("y_train",y_train.shape)
print("y_test",y_test.shape) 
df_train
y_train<jupyter_output><empty_output><jupyter_text>Creating the Model.<jupyter_code>model = Sequential()
model.add(embedding_layer)
model.add(Dropout(0.5))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2)) # This can be BERT which may have better performance. 
model.add(Dense(1, activation='sigmoid'))
model.summary()
model.compile(loss='binary_crossentropy',
              optimizer="adam",
              metrics=['accuracy'])
callbacks = [ ReduceLROnPlateau(monitor='val_loss', patience=5, cooldown=0),
              EarlyStopping(monitor='val_accuracy', min_delta=1e-4, patience=5)]<jupyter_output><empty_output><jupyter_text>Train Note to self... I already ran the model and it took 2 hours to run...<jupyter_code>%%time
history = model.fit(x_train,
                    y_train,
                    batch_size=1024,
                    epochs=8,
                    validation_split=0.1,
                    verbose=1,
                    callbacks = callbacks)<jupyter_output>Epoch 1/8
71/71 [==============================] - 564s 8s/step - loss: 0.6145 - accuracy: 0.6527 - val_loss: 0.5263 - val_accuracy: 0.7469
Epoch 2/8
71/71 [==============================] - 722s 10s/step - loss: 0.5496 - accuracy: 0.7168 - val_loss: 0.5160 - val_accuracy: 0.7541
Epoch 3/8
71/71 [==============================] - 847s 12s/step - loss: 0.5356 - accuracy: 0.7303 - val_loss: 0.5173 - val_accuracy: 0.7499
Epoch 4/8
71/71 [==============================] - 909s 13s/step - loss: 0.5266 - accuracy: 0.7357 - val_loss: 0.5033 - val_accuracy: 0.7588
Epoch 5/8
71/71 [==============================] - 930s 13s/step - loss: 0.5194 - accuracy: 0.7400 - val_loss: 0.5028 - val_accuracy: 0.7619
Epoch 6/8
71/71 [==============================] - 989s 14s/step - loss: 0.5152 - accuracy: 0.7444 - val_loss: 0.4983 - val_accuracy: 0.7610
Epoch 7/8
71/71 [==============================] - 1033s 15s/step - loss: 0.5055 - accuracy: 0.7487 - val_loss: 0.4979 - val_accuracy: 0.7634
Epoch 8/8
71/[...]<jupyter_text>Evaluate<jupyter_code>%%time
score = model.evaluate(x_test, y_test, batch_size=32)
print()
print("ACCURACY:",score[1])
print("LOSS:",score[0])
history.history
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
 
epochs = range(len(acc))
 
plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'r', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()
 
plt.figure()
 
plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'r', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
 
plt.show() # Seems like the model may be overfitting. training loss << validation loss
def decode_sentiment(score):
    return 'NEGATIVE' if score < 0.5 else 'POSITIVE'
    
def predict(text, include_neutral=True):
    start_at = time.time()
    # Tokenize text
    x_test = pad_sequences(tokenizer.texts_to_sequences([text]), maxlen=300)
    # Predict
    score = model.predict([x_test])[0]
#     print(score)
    # Decode sentiment
    label = decode_sentiment(score[0])

    return {"label": label, "score": float(score),
       "elapsed_time": time.time()-start_at}  
predict("Leave a like on this video, comment, and subscribe for more!")
predict("I hope you like it!")
predict("I love you")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1215050051777701, 'fraction_numerical': 0.02865032792544011, 'mean_word_length': 4.529103053435114, 'pattern_counts': {'":': 3, '<': 42, '<?xml version=': 0, '>': 39, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '148324', 'license': 'MIT', 'n_tokens_mistral': 3967, 'n_tokens_neox': 3650, 'n_words': 1133, 'path': 'Sentiment_Analysis/Recommendation.ipynb', 'repo_name': 'SpencerPao/Natural-Language-Processing', 'size': '11588'}