<jupyter_start><jupyter_text>Sentiment Analysis of Tweets in Russian using Multinomial Naive BayesMultinomial Naive Bayes classification algorithm tends to be a baseline solution for sentiment analysis task. The basic idea of Naïve Bayes technique is to find the probabilities of classes assigned to texts by using the joint probabilities of words and classes. Given the dependent feature vector $(x_1, \dots, x_n)$ and the class $C_k$. Bayes' theorem is stated mathematically as the following relationship:$$\begin{align}P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)P(x_1,\dots,x_n \mid C_k)}{P(x_1,\dots,x_n)}\end{align}$$According to the "naive" conditional independence assumptions, for the given class $C_k$ each feature $x_{i}$ is conditionally independent of every other feature $x_{j}$ for$j\neq i$.$$\begin{align}P(x_i \mid C_k,x_1,\dots,x_n) = P(x_i \mid C_k)\end{align}$$Thus, the relation can be simplified to$$\begin{align}P(C_k \mid x_1,\dots,x_n) = \frac{P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)}{P(x_1,\dots,x_n)}\end{align}$$Since $P(x_1,...x_n)$ is constant, if the values of the feature variables are known, the following classification rule can be used:$$\begin{align}P(C_k \mid x_1,\dots,x_n) \propto  P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k) \\ \Downarrow \\ \hat{y} = \underset{k}{\arg\max} P(C_k)\prod_{i=1}^{n}P(x_i \mid C_k)\end{align}$$To avoid underflow, log probabilities can be used.$$\begin{align}\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{P(x_i \mid C_k)})\end{align}$$The variaty of naive Bayes classifiers primarly differs between each other by the assumptions they make regarding the distribution of $P(x_i \mid C_k)$, while $P(C_k)$ is usually defined as the relative frequency of class $C_k$ in the training dataset.The multinomial distribution is parametrized by vectors $\theta_k = (\theta_{k1},\ldots,\theta_{kn})$ for each class $C_k$, where $n$ is the number of features (i.e. the size of the vocabulary) and $\theta_{ki}$ is the probability $P(x_i \mid C_k)$ of feature $i$ appearing in a sample that belongs to the class $C_k$.The parameters $\theta_y$ is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:$$\begin{align}\hat{\theta}_{ki} = \frac{ N_{ki} + \alpha}{N_k + \alpha n}\end{align}$$where $N_{ki} = \sum_{x \in T} x_i$ is the number of times feature $i$ appears in a sample of class y in the training set $T$, and $N_{y} = \sum_{i=1}^{|T|} N_{ki}$ is the total count of all features for class $C_k$. The smoothing priors $\alpha \ge 0$ accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting $\alpha = 1$ is called Laplace smoothing, while $\alpha < 1$ is called Lidstone smoothing.Thus, the final decision rule is defined as follows:$$\begin{align}\hat{y} = \underset{k}{\arg\max}(\ln{P(C_k)}+\sum_{i=1}^{n}\ln{\frac{ N_{ki} + \alpha}{N_k + \alpha n}})\end{align}$$  1. Loading dataSentiment dataset of Tweets in Russian [1] is avaialble at http://study.mokoron.com/. The files positive.csv and negative.csv contain positively labelled and negatively labelled tweets, respectively.<jupyter_code>import pandas as pd
import numpy as np
# Read the data from CSV files
n = ['id', 'date','name','text','typr','rep','rtw','faw','stcount','foll','frien','listcount']
data_positive = pd.read_csv('positive.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
data_negative = pd.read_csv('negative.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])
# Create balanced dataset
sample_size = min(data_positive.shape[0], data_negative.shape[0])
raw_data = np.concatenate((data_positive['text'].values[:sample_size], 
                           data_negative['text'].values[:sample_size]), axis=0) 
labels = [1]*sample_size + [0]*sample_size<jupyter_output><empty_output><jupyter_text>2. Preprocessing dataTexts generated by humans in social media sites contain lots of noise that can significantly affect the results of the sentiment classification process. Moreover, depending on the features generation approach, every new term seems to add at least one new dimension to the feature space. That makes the feature space more sparse and high-dimensional. Consequently, the task of the classifier has become more complex.To prepare messages, such text preprocessing techniques as replacing URLs and usernames with keywords, removing punctuation marks and converting to lowercase were used in this program.<jupyter_code>import re

def preprocess_text(text):
    text = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','URL', text)
    text = re.sub('@[^\s]+','USER', text)
    text = text.lower().replace("ё", "е")
    text = re.sub('[^a-zA-Zа-яА-Я1-9]+', ' ', text)
    text = re.sub(' +',' ', text)
    return text.strip()

data = [preprocess_text(t) for t in raw_data]<jupyter_output><empty_output><jupyter_text>3. Training MultinomialNBA Pipeline class was used to make the vectorizer => transformer => classifier easier to work with. Such hyper-parameters as n-grams range, IDF usage, TF-IDF normalization type and Naive Bayes alpha were tunned using grid search. The performance of the selected hyper-parameters was measured on a test set that was not used during the model training step.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split, GridSearchCV

text_clf = Pipeline([('vect', CountVectorizer()),
                     ('tfidf', TfidfTransformer()),
                     ('clf', MultinomialNB())])

tuned_parameters = {
    'vect__ngram_range': [(1, 1), (1, 2), (2, 2)],
    'tfidf__use_idf': (True, False),
    'tfidf__norm': ('l1', 'l2'),
    'clf__alpha': [1, 1e-1, 1e-2]
}<jupyter_output><empty_output><jupyter_text>The dataset was plited into train and test subsets.<jupyter_code>x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, random_state=42)
from sklearn.metrics import classification_report

score = 'f1_macro'
print("# Tuning hyper-parameters for %s" % score)
print()
np.errstate(divide='ignore')
clf = GridSearchCV(text_clf, tuned_parameters, cv=10, scoring=score)
clf.fit(x_train, y_train)

print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()
print("Grid scores on development set:")
print()
for mean, std, params in zip(clf.cv_results_['mean_test_score'], 
                             clf.cv_results_['std_test_score'], 
                             clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))
print()

print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
print(classification_report(y_test, clf.predict(x_test), digits=4))
print()<jupyter_output># Tuning hyper-parameters for f1_macro

Best parameters set found on development set:

{'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}

Grid scores on development set:

0.732 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 1)}
0.741 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (1, 2)}
0.705 (+/-0.008) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': True, 'vect__ngram_range': (2, 2)}
0.728 (+/-0.004) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}
0.730 (+/-0.005) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (1, 2)}
0.702 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l1', 'tfidf__use_idf': False, 'vect__ngram_range': (2, 2)}
0.734 (+/-0.006) for {'clf__alpha': 1, 'tfidf__norm': 'l2', 'tfidf__use_idf': True, 'vect__ngram_range': ([...]
{'fraction_non_alphanumeric': 0.14783699059561128, 'fraction_numerical': 0.01818181818181818, 'mean_word_length': 5.236121970289289, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '284593', 'license': 'MIT', 'n_tokens_mistral': 2821, 'n_tokens_neox': 2658, 'n_words': 831, 'path': 'Sentiment Analysis of Tweets in Russian using Multinomial Naive Bayes.ipynb', 'repo_name': 'sismetanin/sentiment-analysis-of-tweets-in-russian', 'size': '7975'}