<jupyter_start><jupyter_text>PreprocessingIn this lab, we will be exploring how to preprocess tweets for sentiment analysis. We will provide a function for preprocessing tweets during this week's assignment, but it is still good to know what is going on under the hood. By the end of this lecture, you will see how to use the [NLTK](http://www.nltk.org) package to perform a preprocessing pipeline for Twitter datasets.  SetupYou will be doing sentiment analysis on tweets in the first two weeks of this course. To help with that, we will be using the [Natural Language Toolkit (NLTK)](http://www.nltk.org/howto/twitter.html) package, an open-source Python library for natural language processing. It has modules for collecting, handling, and processing Twitter data, and you will be acquainted with them as we move along the course.For this exercise, we will use a Twitter dataset that comes with NLTK. This dataset has been manually annotated and serves to establish baselines for models quickly. Let us import them now as well as a few other libraries we will be using.<jupyter_code>import nltk
nltk.download('stopwords')
import nltk                                # Python library for NLP
from nltk.corpus import twitter_samples    # sample Twitter dataset from NLTK
import matplotlib.pyplot as plt            # library for visualization
import random                              # pseudo-random number generator
import pandas as pd
import numpy as np
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>About the Twitter datasetThe sample dataset from NLTK is separated into positive and negative tweets. It contains 5000 positive tweets and 5000 negative tweets exactly. The exact match between these classes is not a coincidence. The intention is to have a balanced dataset. That does not reflect the real distributions of positive and negative classes in live Twitter streams. It is just because balanced datasets simplify the design of most computational methods that are required for sentiment analysis. However, it is better to be aware that this balance of classes is artificial. You can download the dataset in your workspace (or in your local computer) by doing:<jupyter_code>pip install pyarrow
# downloads sample twitter dataset.
from pathlib import Path

import pyarrow.parquet as pq

data = pq.read_table('psoriasis_all_posts.parquet').to_pandas()
data = data[['text', 'forum_id']]
data.reset_index(drop=True, inplace=True)
data
data['forum_id'].unique(), data['forum_id'].nunique()<jupyter_output><empty_output><jupyter_text>We can load the text fields of the positive and negative tweets by using the module's `strings()` method like this:<jupyter_code>pd.isnull(data['forum_id']).sum()<jupyter_output><empty_output><jupyter_text>Next, we'll print a report with the number of positive and negative tweets. It is also essential to know the data structure of the datasets<jupyter_code>data['text'].apply(lambda x: len(x.split(' '))).sum()<jupyter_output><empty_output><jupyter_text>We have more than 7 million words in the data.Distribution of classes.<jupyter_code>plt.figure(figsize=(17,6))
sns.countplot(data['forum_id'])<jupyter_output>C:\python\python39\lib\site-packages\seaborn\_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>From the above visualisation we can see classes are not very well balanced.<jupyter_code>def post_f_id(index):
    post = data[data.index==index].values[0]
    print(post[0])
    print('forum_id:', post[1])<jupyter_output><empty_output><jupyter_text>Looking few posts and tags.<jupyter_code>post_f_id(5)
post_f_id(11)<jupyter_output>Hal<NAME>,  klar mag ich Märchen und deshalb hab ich mir die Verfilmungen auch angesehen. Ich fand sie wirklich gut gemacht. Natürlich kenn ich auch die alten Verfilmungen, sowohl die Deutschen als auch die Tschechischen.  Liebe Grüße und ein frohes neues Jahr  Bella                      Quote      
forum_id: 4-community<jupyter_text>post_f_id(5) The texts needs to be cleaned up.<jupyter_code>pip install bs4
import re
import nltk
from nltk.corpus import stopwords
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Let's import a few more libraries for this purpose.<jupyter_code>def clean_text(text):
    text = BeautifulSoup(text,'lxml').text
    text = text.lower()
    text = re.sub('[/(){}\[\]\|@,;]', " ",text)
    text = re.sub('[^0-9a-z #+_]'," ",text)
    text = ' '.join(word for word in text.split() if word not in set(stopwords.words('english')))
    return text
data['text'] = data['text'].apply(clean_text)
post_f_id(5)

post_f_id(11)
data['text'].apply(lambda x: len(x.split(" "))).sum()<jupyter_output><empty_output><jupyter_text>Now, we have over 5 million words to work with.Splitting the dataset.<jupyter_code>from sklearn.model_selection import train_test_split
X = data['text']
y = data['forum_id']
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2, random_state=0)
X_train.shape
X_test.shape
X_test.head()<jupyter_output><empty_output><jupyter_text>The next steps includes feature engineering and model building using pipeline. Naive Bayes classifier for multinomial models<jupyter_code>from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
nb = Pipeline([('vect', CountVectorizer()),
               ('tfidf', TfidfTransformer()),
               ('clf', MultinomialNB()),
              ])
nb.fit(X_train, y_train)
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

y_pred = nb.predict(X_test)
print(confusion_matrix(y_pred,y_test))
print(classification_report(y_pred, y_test,target_names=data['forum_id'].unique()))
print(accuracy_score(y_pred,y_test))<jupyter_output>precision    recall  f1-score   support

             4-community       0.95      0.46      0.62      9144
           1-allgemeines       0.03      0.89      0.07        90
            2-behandlung       0.00      0.00      0.00         0
3-klinik-reha-kur-urlaub       0.40      0.75      0.52      1944

                accuracy                           0.51     11178
               macro avg       0.34      0.52      0.30     11178
            weighted avg       0.85      0.51      0.60     11178

0.5137770620862409<jupyter_text>Linear support vector machine.<jupyter_code>from sklearn.linear_model import SGDClassifier
svm = Pipeline([('vect',CountVectorizer()),
               ('tfidf',TfidfTransformer()),
               ('svm_clf', SGDClassifier(loss='hinge', penalty='l2',alpha=0.001,n_jobs=-1,random_state=0,max_iter=5))])
svm.fit(X_train,y_train)   
%%time
y_pred_svm = svm.predict(X_test)
print(accuracy_score(y_test,y_pred_svm))
print(classification_report(y_test,y_pred_svm, target_names=data['forum_id'].unique()))<jupyter_output>0.5807836822329576
                          precision    recall  f1-score   support

             4-community       0.55      0.76      0.64      4443
           1-allgemeines       0.70      0.26      0.38      2308
            2-behandlung       0.75      0.41      0.53       775
3-klinik-reha-kur-urlaub       0.59      0.60      0.59      3652

                accuracy                           0.58     11178
               macro avg       0.65      0.51      0.53     11178
            weighted avg       0.61      0.58      0.56     11178

Wall time: 1.61 s<jupyter_text>Creating bag of words with Keras.<jupyter_code>from tensorflow import keras
from keras import utils
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.preprocessing import text, sequence<jupyter_output><empty_output><jupyter_text>Using TensorFlow backend.<jupyter_code>train_size = int(len(data) * 0.7)
print('Train size: %d' %train_size)
print('Test_size: %d' %(len(data)-train_size))
train_post = data['text'][:train_size]
train_tag = data['forum_id'][:train_size]
test_post = data['text'][train_size:]
test_tag = data['forum_id'][train_size:]
train_post.head()
train_tag.head()
test_tag.head()
token = text.Tokenizer(num_words=1000,char_level=False)
token.fit_on_texts(train_post)
x_train = token.texts_to_matrix(train_post)
x_test = token.texts_to_matrix(test_post)
print(x_train)
print(x_train.shape)
print(x_test)
print(x_test.shape)
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
Y_train = encoder.fit_transform(train_tag)
Y_test = encoder.transform(test_tag)
print(Y_train)
print(Y_train.shape)
print(Y_test)
print(Y_test.shape)
num_classes = np.max(Y_train) + 1
from tensorflow.keras import utils as np_utils


Y_train = np_utils.to_categorical(Y_train, num_classes)
Y_test = np_utils.to_categorical(Y_test, num_classes)
print(Y_test)
print(Y_test.shape)
print(Y_train)
print(Y_train.shape)
model = Sequential()
model.add(Dense(512, input_shape=(1000,), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
%%time
model.fit(x_train,Y_train, batch_size=32, epochs=2)
score = model.evaluate(x_test, Y_test,
                       batch_size=32, verbose=1)
print('Test accuracy:', score[1])<jupyter_output>524/524 [==============================] - 3s 5ms/step - loss: 1.0514 - accuracy: 0.5336
Test accuracy: 0.5336076617240906<jupyter_text>Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier

rfc = Pipeline([('vect', CountVectorizer()),
                ('tfidf', TfidfTransformer()),
                ('rfc', RandomForestClassifier(n_estimators=200, criterion='entropy'))])
rfc.fit(X_train,y_train)
y_pred_rfc = rfc.predict(X_test)
print(accuracy_score(y_pred_rfc,y_test))
print(classification_report(y_pred_rfc,y_test))<jupyter_output>0.5468777956700662
                          precision    recall  f1-score   support

           1-allgemeines       0.85      0.49      0.62      7617
            2-behandlung       0.11      0.80      0.20       333
3-klinik-reha-kur-urlaub       0.19      0.89      0.32       168
             4-community       0.53      0.63      0.58      3060

                accuracy                           0.55     11178
               macro avg       0.42      0.70      0.43     11178
            weighted avg       0.73      0.55      0.59     11178<jupyter_text>Logistic regression.<jupyter_code>from sklearn.linear_model import LogisticRegression
lr = Pipeline([('vect',CountVectorizer()),
               ('tfidf', TfidfTransformer()),
               ('lr_clf', LogisticRegression(n_jobs=-1,C=0.01))])
lr.fit(X_train,y_train)
y_pred_lr = lr.predict(X_test)

print(accuracy_score(y_pred_lr,y_test))
print(classification_report(y_pred_lr,y_test,target_names= data['forum_id'].unique()))<jupyter_output>0.4825550187869028
{'fraction_non_alphanumeric': 0.09996429846483398, 'fraction_numerical': 0.0408782577650839, 'mean_word_length': 3.711942809083263, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '329416', 'license': 'MIT', 'n_tokens_mistral': 3890, 'n_tokens_neox': 3591, 'n_words': 943, 'path': 'case_study.ipynb', 'repo_name': 'msalehsaudi/NLP', 'size': '11211'}