<jupyter_start><jupyter_text>Sentiment analysis with Logistic Regression  Logistic Regression> Some of the explanations in this tutoarial are inspired by chapters 3 and 8 in the book _"Python Machine Learning"_ by <NAME>First, let's introduce what Logistic Regression is. [Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression) is a classification model that is very easy to implement and performs very well on linearly separable classes. It is one of the most widely usedalgorithms for classification in industry too, which makes it attractive to play with._Very_ simplistically explained, Logistic Regression works as follows:![Logistic Regression](./images/logistic_regression.png)First we will define the input for our algorithm. The imput will be each sample in whatever dataset we are working with. Each sample will consist of several features. For example, if we're working with housing price prediction, the features for each sample could be the size of the house, number of rooms, etc. We'll call the input vector **X**.For the algorythm to learn, we need to define variables that we can adjust accordingly to what we want to predict. We will create a vector of _weights_ (**W**) that the model will adjust in order to predict more accurately. The process of adjusting those weights is what we call **learning**.For every input sample, we will perform a dot product of the features by the weights **XW**. This product is sometimes referred as _net input_. This will give us a real number. Since in this particular problem we want to _classify_ (positive/negative), we need squash this number in the range [0, 1]. This will give us the _probability_ of a positive event. A function that does precisely that is called **sigmoid**. The sigmoid function looks like this:![sigmoid](./images/sigmoid.svg)What sigmoid is doing is basically transforming big inputs into a value close to 1, and small inputs into a value close to 0. This is exactly what we want. We will do this for every sample in our training set and compute the errors. To calculate the error we only need to compare our prediction with the true label for each sample. We will sum the square errors of all the samples to get a global prediction error. This will be our **cost function**.A cost function is then something we want to minimize. **Gradient descent** is a method for finding the minimum of a function of multiple variables, such like the one we're dealing with here.  Gradient descent> Watch [this](https://www.youtube.com/watch?v=IHZwWFHWa-w) video for a visual introduction to Gradient DescentOnce all our training samples have been computed and the error calculated with our cost function, we need to _minimize_ that cost function. A method for doing that is gradient descent. There are many [articles](https://en.wikipedia.org/wiki/Gradient_descent) that contain detailed explanations _and_ implementations of GD, so let's not do this here. However is good to have an intuition.For illustration purposes, let's think about a function with two parameters. Something like this one:![Gradient Descent](./images/Gradient_descent.png)Gradient descent will try to find the minimum of the function. To do so, we calculate the slope of the function at a certain point, and move towards the direction that makes the function decrease. There are some things to have in mind though.As you can see a function can have one or several _local minimum_. In a local minimum, the slope will be zero and GD will "think" it's found the global minimum. To avoid this, we can choose a bigger "step" when we move towards the minimum. The "size" of the step towards the minimun is what we call the **learning rate**, and it's another adjustable parameter.We need to be careful here: If we choose a too small learning rate, we can get stuck in a local minimum. If we instead choose a too big learning rate, we risk overshooting the global minimum. We need to experiment, and the adecuate learning rate depends on the particular problem and the data.  The training processIn order for logistic regression to learn, we need to repeat the process descrived before several times. Each one of these times is called an **epoch**. The number of epochs to run depends on the problem and the training data. It is... yes, another tunnable parameter of the algorithm.The set of all tunnable parameters is called **hyperparameters** of the model.Like with the leatning rate, we need to be careful when choosing the number of epochs: If we train too many epochs, we risk **overfitting**. This means that our model will "memorize" the training data and will generalize badly when presented new data. If we train too little, it will fail to find any pattern and the prediction accuracy will be very low. This is known as **underfitting**.There are techniques that help prevent overfitting. These **regularization** techniques are out of the scope of this tutorial, but... guess! It's also something to tune and experiment with :)This is why when training a model you need to set aside a _test dataset_ in order to know the accuracy of your algorithm in unknown data. The test dataset will **never** be used during training  Sentiment analysisLet's first of all have a look at the data<jupyter_code>import pandas as pd

#Load the data into a DataFrame
train = pd.read_csv('data/train.csv', encoding='latin-1')
test = pd.read_csv('data/test.csv', encoding='latin-1')

train.head(10)<jupyter_output><empty_output><jupyter_text>As we can see, the structure of a twit varies a lot between twit and twit. They have different lengths, letters, numbers, extrange characters, etc. It is also important to note that **a lot** of words are not correctly spelled, for example the word _"Juuuuuuuuuuuuuuuuussssst"_ or the word _"frie"_ instear of _"friend"_This makes it hard to mesure how positive or negative are the words withing the corpus of twits. If they were all correct dictionary words, we could use a **lexicon** to punctuate words. However because of the nature of social media language, we cannot do that. So we need a way of scoring the words such that words that appear in positive twits have greater score that those that appear in negative twits.But first... how do we represent the twits as vectors we can input to our algorithm? Bag of wordsOne thing we could do to represent the twits as equal-sized vectors of numbers is the following:* Create a list (vocabulary) with all the unique words in the whole corpus of twits. * We construct a feature vector from each twit that contains the counts of how often each word occurs in the particular twit_Note that since the unique words in each twit represent only a small subset of all the words in the bag-of-words vocabulary, the feature vectors will mostly consist of zeros_Let's construct the bag of words. We will work with a smaller example for illustrative purposes, and at the end we will work with our real data.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

twits = [
    'This is amazing!',
    'ML is the best, yes it is',
    'I am not sure about how this is going to end...'
]

count = CountVectorizer()
bag = count.fit_transform(twits)

count.vocabulary_<jupyter_output><empty_output><jupyter_text>As we can see from executing the preceding command, the vocabulary is stored in a Python dictionary that maps the unique words to integer indices. Next, let's print the feature vectors that we just created:<jupyter_code>bag.toarray()<jupyter_output><empty_output><jupyter_text>Each index position in the feature vectors corresponds to the integer values that are stored as dictionary items in the CountVectorizer vocabulary. For example, the first feature at index position 0 resembles the count of the word 'about' , which only occurs in the last document, and the word 'is' , at index position 7, occurs in all three twits (two times in the second twit). These values in the feature vectors are also called the **raw term frequencies**: `tf(t,d )` —the number of times a term `t` occurs in a document `d`.  How relevant are words? Term frequency-inverse document frequencyWe could use these raw term frequencies to score the words in our algorithm. There is a problem though: If a word is very frequent in _all_ documents, then it probably doesn't carry a lot of information. In order to tacke this problem we can use **term frequency-inverse document frequency**, which will reduce the score the more frequent the word is accross all twits. It is calculated like this:\begin{equation*}tf-idf(t,d) = tf(t,d) ~ idf(t,d)\end{equation*}_tf(t,d)_ is the raw term frequency descrived above. _idf(t,d)_ is the inverse document frequency, than can be calculated as follows:\begin{equation*}\log \frac{n_d}{1+df\left(d,t\right)}\end{equation*}where `n` is the total number of documents and _df(t,d)_ is the number of documents where the term `t` appears. The `1` addition in the denominator is just to avoid zero term for terms that appear in all documents. Ans the `log` ensures that low frequency term don't get too much weight.Fortunately for us `scikit-learn` does all those calculations for us:<jupyter_code>import numpy as np

from sklearn.feature_extraction.text import TfidfTransformer

tfidf = TfidfTransformer(use_idf=True,
                         norm='l2',
                         smooth_idf=True)

np.set_printoptions(precision=2)

# Feed the tf-idf transformer with our previously created Bag of Words
tfidf.fit_transform(bag).toarray()<jupyter_output><empty_output><jupyter_text>As you can see, words that appear in all documents like _is_ (with 0.47 ), get a lower score than others that don't appear in all documents, like _amazing_ (with 0.72).Note also that `norm='l2'` parameter: This is an important one, and what is doing is normalize the tf-idfs so that they're all in the same scale and thus work better with Logistic Regression.  Data clean up (yay...) Removing stop wordsNow that we know how to format and score our input, we can start doing the analysis! Can we?... Well, we _can_, but let's look at our **real** vocabulary. Specifically, the most common words:<jupyter_code>from collections import Counter

vocab = Counter()
for twit in train.SentimentText:
    for word in twit.split(' '):
        vocab[word] += 1

vocab.most_common(20)<jupyter_output><empty_output><jupyter_text>As you can see, the most common words are meaningless in terms of sentiment: _I, to, the, and_... they don't give any information on positiveness or negativeness. They're basically **noise** that can most probably be eliminated. Let's see the whole distribution to convince ourselves of this:<jupyter_code>from bokeh.models import ColumnDataSource, LabelSet
from bokeh.plotting import figure, show, output_file
from bokeh.io import output_notebook
output_notebook()
import math

def plot_distribution(vocabulary):

    hist, edges = np.histogram(list(map(lambda x:math.log(x[1]),vocabulary.most_common())), density=True, bins=500)

    p = figure(tools="pan,wheel_zoom,reset,save",
               toolbar_location="above",
               title="Word distribution accross all twits")
    p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color="#555555", )
    show(p)

plot_distribution(vocab)<jupyter_output><empty_output><jupyter_text>It's clear now that a porcion of the words are overly represented. These kind of words are called _stop words_, and it is a common practice to remove them when doing text analysis. Let's do it and see the distribution again:<jupyter_code>import nltk

nltk.download('stopwords')
from nltk.corpus import stopwords
stop = stopwords.words('english')

vocab_reduced = Counter()
for w, c in vocab.items():
    if not w in stop:
        vocab_reduced[w]=c

vocab_reduced.most_common(20)<jupyter_output><empty_output><jupyter_text>This looks better, only in the 20 most common words we already see words that make sense: _good, love, really_... Let's see the distribution now<jupyter_code>plot_distribution(vocab_reduced)<jupyter_output><empty_output><jupyter_text>Removing special characters and "trash"We still se a very uneaven distribution. If you look closer, you'll see that we're also taking into consideration punctuation signs ('-', ',', etc) and other html tags like `&amp`. We can definitely remove them for the sentiment analysis, but we will try to keep the emoticons, since those _do_ have a sentiment load:<jupyter_code>import re

def preprocessor(text):
    """ Return a cleaned version of text
    """
    # Remove HTML markup
    text = re.sub('<[^>]*>', '', text)
    # Save emoticons for later appending
    emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text)
    # Remove any non-word character and append the emoticons,
    # removing the nose character for standarization. Convert to lower case
    text = (re.sub('[\W]+', ' ', text.lower()) + ' ' + ' '.join(emoticons).replace('-', ''))
    
    return text

print(preprocessor('This!! twit man :) is <b>nice</b>'))<jupyter_output>this twit man is nice :)<jupyter_text>We are almost ready! There is another trick we can use to reduce our vocabulary and consolidate words. If you think about it, words like: love, loving, etc. _Could_ express the same positivity. If that was the case, we would be  having two words in our vocabulary when we could have only one: lov. This process of reducing a word to its root is called **steaming**.We also need a _tokenizer_ to break down our twits in individual words. We will implement two tokenizers, a regular one and one that does steaming:<jupyter_code>from nltk.stem import PorterStemmer

porter = PorterStemmer()

def tokenizer(text):
    return text.split()

def tokenizer_porter(text):
    return [porter.stem(word) for word in text.split()]

print(tokenizer('Hi there, I am loving this, like with a lot of love'))
print(tokenizer_porter('Hi there, I am loving this, like with a lot of love'))<jupyter_output>['Hi', 'there,', 'I', 'am', 'loving', 'this,', 'like', 'with', 'a', 'lot', 'of', 'love']
['Hi', 'there,', 'I', 'am', 'love', 'this,', 'like', 'with', 'a', 'lot', 'of', 'love']<jupyter_text>Training Logistic RegressionWe are finally ready to train our algorythm. We need to choose the best hyperparameters like the _learning rate_ or _regularization strength_. We also would like to know if our algorithm performs better steaming words or not, or removing html or not, etc...To take these decisions methodically, we can use a Grid Search. Grid search is a method of training an algorythm with different variations of parameters to latter select the best combination<jupyter_code>from sklearn.model_selection import train_test_split

# split the dataset in train and test
X = train['SentimentText']
y = train['Sentiment']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)<jupyter_output><empty_output><jupyter_text>In the code line above, `stratify` will create a train set with the same class balance than the original set<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(strip_accents=None,
                        lowercase=False,
                        preprocessor=None)

param_grid = [{'vect__ngram_range': [(1, 1)],
               'vect__stop_words': [stop, None],
               'vect__tokenizer': [tokenizer, tokenizer_porter],
               'vect__preprocessor': [None, preprocessor],
               'clf__penalty': ['l1', 'l2'],
               'clf__C': [1.0, 10.0, 100.0]},
              {'vect__ngram_range': [(1, 1)],
               'vect__stop_words': [stop, None],
               'vect__tokenizer': [tokenizer, tokenizer_porter],
               'vect__preprocessor': [None, preprocessor],
               'vect__use_idf':[False],
               'vect__norm':[None],
               'clf__penalty': ['l1', 'l2'],
               'clf__C': [1.0, 10.0, 100.0]},
              ]

lr_tfidf = Pipeline([('vect', tfidf),
                     ('clf', LogisticRegression(random_state=0))])

gs_lr_tfidf = GridSearchCV(lr_tfidf, param_grid,
                           scoring='accuracy',
                           cv=5,
                           verbose=1,
                           n_jobs=-1)
# Note: This may take a long while to execute, like... 1 or 2 hours
gs_lr_tfidf.fit(X_train, y_train)
print('Best parameter set: ' + str(gs_lr_tfidf.best_params_))
print('Best accuracy: %.3f' % gs_lr_tfidf.best_score_)<jupyter_output>Best parameter set: {'clf__C': 1.0, 'clf__penalty': 'l2', 'vect__ngram_range': (1, 1), 'vect__preprocessor': <function preprocessor at 0x7f8daa8109d8>, 'vect__stop_words': None, 'vect__tokenizer': <function tokenizer at 0x7f8da0c3d378>}
Best accuracy: 0.772<jupyter_text>Interestingly, the set of parameters that best results give us are:* A regularization strength of `1.0` using `l2` regularization* Using our `preprocessor` (removing html, keeping emoticons, etc) _does_ improve the performance* Surprisingly, removing stop words does not improve accuracy* word steming doesn't seem to help eitherAs youcan see, sometimes intuition may lead to wrong decisions, and it's important to _test_ all our assumptions. Let's see what's our best accuracy then:<jupyter_code>clf = gs_lr_tfidf.best_estimator_
print('Accuracy in test: %.3f' % clf.score(X_test, y_test))<jupyter_output>Accuracy in test: 0.770<jupyter_text>If we would like to use the classifier in another place, or just not train it again and again everytime, we can save the model in a pickle file:<jupyter_code>import pickle
import os

pickle.dump(clf, open(os.path.join('data', 'logisticRegression.pkl'), 'wb'), protocol=4)<jupyter_output><empty_output><jupyter_text>Finally, let's run some tests :-)<jupyter_code>twits = [
    "This is really bad, I don't like it at all",
    "I love this!",
    ":)",
    "I'm sad... :("
]

preds = clf.predict(twits)

for i in range(len(twits)):
    print(f'{twits[i]} --> {preds[i]}')<jupyter_output>This is really bad, I don't like it at all --> 0
I love this! --> 1
:) --> 1
I'm sad... :( --> 0
{'fraction_non_alphanumeric': 0.08208873270371184, 'fraction_numerical': 0.006424335602899187, 'mean_word_length': 4.428614008941878, 'pattern_counts': {'":': 1, '<': 62, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '625240', 'license': 'MIT', 'n_tokens_mistral': 5198, 'n_tokens_neox': 4979, 'n_words': 2382, 'path': 'Prediction/Logistic regression/Sentiment analysis with Logistic Regression.ipynb', 'repo_name': 'INSEAIT/Data-Science-Introduction', 'size': '18223'}