<jupyter_start><jupyter_text>Text classification ![](./img/categorization.png) How can a computer learn to classify things? ![](./img/classification.png) But how documents are represented as vectors? ![Document term matrix](./img/docterm.png) ![Tfidf](./img/tfidf.png)  Building a classification pipeline  ...with `scikit-learn` Notations:* `X` is a 2D matrix, rows represents data points, columns contains feature values* `y` is a 1D array containing the labels<jupyter_code>class SklearnPredictor:
    def fit(self, X, y):
        """
        Learning to classify from the data
        """
        return self
    
    def predict(X):
        """
        Predict a label for all examples in X
        """
        return predictions
        
class SklearnTransformer:
    def fit(self, X, y=None):
        """
        Learning to transforming the data
        """
        return self
    
    def transform(X):
        """
        Transform all the data in X
        """
        return transformed_data
from sklearn.pipeline import Pipeline

Pipeline([
    ("transformer1", SklearnTransformer()),
    ("transformer2", SklearnTransformer()), # ...
    ("mypredictor", SklearnPredictor()),
])<jupyter_output><empty_output><jupyter_text>Categorizing Hungarian news articles Have a look at the data<jupyter_code>import pandas as pd

df = pd.read_csv("./data/index_articles.csv")
df.head()
len(df)
df.category.value_counts()<jupyter_output><empty_output><jupyter_text>We usually split the data into training and test sets<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df.text, df.category, test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Now we can build the preprocessing pipeline<jupyter_code>from spacy.hu import Hungarian

nlp = Hungarian()

def tokenize(text):
    return [tok.text for tok in nlp(text) if tok.is_alpha and not tok.is_stop]

tokenize("Hello világ. Itt vagyok.")<jupyter_output><empty_output><jupyter_text>As `spaCy` does not support lemmatization currently, we rely on simple stemming<jupyter_code>import snowballstemmer

stemmer = snowballstemmer.stemmer('hungarian')
stemmer.stemWord("baglyom")
def tokenize_stem(text):
    return [stemmer.stemWord(tok.text) for tok in nlp(text) if tok.is_alpha and not tok.is_stop]

tokenize_stem("Hello világ. Itt vagyok.")
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(tokenizer=tokenize_stem, ngram_range=(1,2), lowercase=True)
vectorizer.fit(X_train)<jupyter_output><empty_output><jupyter_text>Note that we used `ngram_range=(1,2)` that means we rely on words **and neighbouring word pairs** as features.<jupyter_code>example_vector = vectorizer.transform(["Az oktatási miniszter is levélben szögezte le: covfafa"])
example_vector.nonzero()
doc_vector = example_vector[example_vector.nonzero()].tolist()[0]
doc_features = vectorizer.inverse_transform(example_vector)[0]
for feature, score in zip(doc_features, doc_vector):
    print(feature, score)<jupyter_output>szögezt 0.45421955882361387
oktatás miniszter 0.4848001115368416
oktatás 0.2783541455813643
miniszter levél 0.5695599453959549
miniszter 0.23767731020511645
levél 0.3166755100853776<jupyter_text>Build a predictor<jupyter_code>X_transformed = vectorizer.transform(X_train)
from sklearn.svm import LinearSVC

classifier = LinearSVC()
classifier.fit(X_transformed, y_train)
classifier.predict(vectorizer.transform(["Az oktatási minisztere is levélben szögezte le: covfafa"]))<jupyter_output><empty_output><jupyter_text>By building a pipeline we simplify the whole process<jupyter_code>from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer


pipeline = make_pipeline(
    TfidfVectorizer(tokenizer=tokenize_stem, ngram_range=(1,2)),
    LinearSVC()
)

pipeline.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>We used a simple linear SVM model, but good candidates could be:* Multinomial Naive Bayes* Logistic regression* Decision trees  Evaluate the model **Accuracy** is a common metric used in classification evaluation.$\mathrm{Accuracy} = \frac{\mathrm{\ correct\ labels}}{N}$<jupyter_code>from sklearn.metrics import accuracy_score

y_tpred = pipeline.predict(X_train)
y_pred = pipeline.predict(X_test)

print("Train accuracy: {}".format(accuracy_score(y_train, y_tpred)))
print("Test accuracy : {}".format(accuracy_score(y_test, y_pred)))<jupyter_output>Train accuracy: 1.0
Test accuracy : 0.890920170627666<jupyter_text>Other metrics ![Precision, Recall](https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/440px-Precisionrecall.svg.png) $\mathrm{F\ score} = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{ \mathrm{Precision} + \mathrm{Recall}}$<jupyter_code>from sklearn.metrics import classification_report

y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

    belfold       0.88      0.94      0.91       778
   gazdasag       0.89      0.88      0.88       523
     kultur       0.93      0.55      0.69       118
       tech       0.94      0.91      0.93       222

avg / total       0.89      0.89      0.89      1641<jupyter_text>So far, so good, but what are the common errors of the classifier?<jupyter_code>from sklearn.metrics import confusion_matrix

confusion_matrix(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Improve your classifier* Collect more data.* Collect even more data.* [Diagnose whether your classifier suffers from  bias or variance](https://www.coursera.org/learn/machine-learning/lecture/yCAup/diagnosing-bias-vs-variance)* Analyze the confusion matrix, and add new features if necessary.* Fine tune your model through optimizing your hyperparameters (`GridSearch`)* Try alternative methods such as [FastText](https://github.com/facebookresearch/fastText) (a neural network based text classification library from Facebook)  Sentiment analysis ![Sentiment scale](./img/sentiment.png) Classification!  Data<jupyter_code>import pandas as pd

pd.set_option("display.max_colwidth", 500)

sent_df = pd.read_csv("./data/twitter_emotion.csv", index_col=None)
sent_df.head(5)
sent_df.sentiment = sent_df.sentiment.apply(lambda x: 1 if x >3 else -1 if x <3 else 0)
sent_df.head(3)<jupyter_output><empty_output><jupyter_text>Tfidf pipeline<jupyter_code>from spacy.hu import Hungarian
nlp = Hungarian()
def is_useful_token(tok):
    return not (tok.like_url or tok.like_num) and "@" not in tok.text and "rt" != tok.text.lower()
    
def sent_tokenize(text):
    return [tok.text for tok in nlp(text) if is_useful_token(tok)]

def sent_tokenize_stem(text):
    return [stemmer.stemWord(tok) for tok in sent_tokenize(text)]


sent_tokenize_stem("rt @kadarmatyas Én élőben tettem http://index.hu 89.2 21,3 :) !")
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC

pipeline = make_pipeline(
    TfidfVectorizer(
        tokenizer=sent_tokenize_stem,
        ngram_range=(1, 2),
        lowercase=True,
        token_pattern=".*"),
    LinearSVC())
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix
import numpy as np
import random

X, y = sent_df.tweet, sent_df.sentiment

random.seed(42)
np.random.seed(42)

y_pred = cross_val_predict(pipeline, X, y, cv=5)
print(classification_report(y, y_pred))

cm = confusion_matrix(y, y_pred)
print(cm)
accuracy_score(y, y_pred)<jupyter_output>precision    recall  f1-score   support

         -1       0.49      0.62      0.55      1129
          0       0.53      0.36      0.43      1261
          1       0.63      0.66      0.65      1610

avg / total       0.56      0.56      0.55      4000

[[ 704  199  226]
 [ 408  455  398]
 [ 335  205 1070]]<jupyter_text>Evaluation in context* A random classifier would result in 0.33 accuracy* Agreement rate between humans is below 0.8  Sentiment lexicons to the rescue?<jupyter_code>pos_words = set(open("./data/PrecoSenti/PrecoPos.txt").read().strip().split())
neg_words = set(open("./data/PrecoSenti/PrecoNeg.txt").read().strip().split())
print(list(pos_words)[:5])
print(list(neg_words)[:5])
# Taken from the Pattern library
pos_emoticons = {
    "<3", u"♥", u"❤", ">:D", ":-D", ":D", "=-D", "=D", "X-D", "x-D", "XD",
    "xD", "8-D", ">:P", ":-P", ":P", ":-p", ":p", ":-b", ":b", ":c)", ":o)",
    ":^)", ">:)", ":-)", ":)", "=)", "=]", ":]", ":}", ":>", ":3", "8)", "8-)",
    ">;]", ";-)", ";)", ";-]", ";]", ";D", ";^)", "*-)", "*)"
}
neg_emoticons = {
    ">:/", ":-/", ":/", ":\\", ">:\\", ":-.", ":-s", ":s", ":S", ":-S", ">.>",
    ">:[", ":-(", ":(", "=(", ":-[", ":[", ":{", ":-<", ":c", ":-c", "=/",
    ":'(", ":'''(", ";'("
}
pos_emojis = {
    u"❤️", u"💜", u"💚", u"💙", u"💛", u"💕", u"😀", u"😄", u"😃", u"😆", u"😅", u"😂",
    u"😁", u"😻", u"😍", u"😈", u"👌", u"😛", u"😝", u"😜", u"😋", u"😇", u"😊", u"😌",
    u"😏", u"😎", u"☺", u"👍", u"😉"
}
neg_emojis = {
    u"😕", u"😬", u"😟", u"😒", u"😔", u"😞", u"😠", u"😩", u"😫", u"😡", u"👿", u"😢",
    u"😥", u"😓", u"😪", u"😭", u"😿"
}
positives = pos_words | pos_emoticons | pos_emojis
negatives = neg_words | neg_emoticons | neg_emojis<jupyter_output><empty_output><jupyter_text>Can't we just count the ratio of sentiment tokens?<jupyter_code>def sentiment_polarizer(text):
    words = sent_tokenize(text)
    return [(1 if tok.lower() in positives else 
             (-1 if tok.lower()in negatives else 0)) 
            for tok in words]
            
sentiment_polarizer("Gagyi, :) nagyon jó!")
from sklearn.base import ClassifierMixin, BaseEstimator
from collections import Counter

X, y = sent_df.tweet, sent_df.sentiment

class SimpleSentiment(ClassifierMixin, BaseEstimator):
    def __init__(self, ratio_threshold=1.5):
        self.threshold = ratio_threshold
        
    def fit(self, X, y):
        return self
    
    def _decide(self, x):
        values = sentiment_polarizer(x) + [1, -1]
        val_counts = Counter(values)
        val_ratio = val_counts[1]/val_counts[-1]
        if val_ratio > self.threshold:
            return 1
        elif val_ratio < 1 / (self.threshold):
            return -1
        else:
            return 0
                                   
    def predict(self, X, *args):
        return [self._decide(x) for x in X]
y_pred = SimpleSentiment(1.5).predict(X)
print(classification_report(y, y_pred))

cm = confusion_matrix(y, y_pred)
print(cm)
accuracy_score(y, y_pred)<jupyter_output>precision    recall  f1-score   support

         -1       0.64      0.26      0.37      1129
          0       0.38      0.76      0.51      1261
          1       0.64      0.40      0.50      1610

avg / total       0.56      0.48      0.46      4000

[[295 682 152]
 [ 89 956 216]
 [ 79 880 651]]<jupyter_text>Combining multiple information sources<jupyter_code>from sklearn.pipeline import make_union
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import MaxAbsScaler

random.seed(42)
np.random.seed(42)

pipeline = make_pipeline(
    make_union(
        TfidfVectorizer(
            tokenizer=sent_tokenize,
            ngram_range=(1, 2),
            lowercase=True,
            token_pattern=".*"),
        make_pipeline(
            CountVectorizer(tokenizer=sentiment_polarizer),
            MaxAbsScaler()
        )
    ),
    LinearSVC()

)
y_pred = cross_val_predict(pipeline, X, y, cv=10)
print(classification_report(y, y_pred))

cm = confusion_matrix(y, y_pred)
print(cm)
accuracy_score(y, y_pred)<jupyter_output>precision    recall  f1-score   support

         -1       0.55      0.57      0.56      1129
          0       0.51      0.48      0.50      1261
          1       0.65      0.65      0.65      1610

avg / total       0.58      0.58      0.58      4000

[[ 648  265  216]
 [ 292  610  359]
 [ 242  318 1050]]
{'fraction_non_alphanumeric': 0.13849745684982906, 'fraction_numerical': 0.042941716001000586, 'mean_word_length': 3.6006904487917146, 'pattern_counts': {'":': 35, '<': 63, '<?xml version=': 0, '>': 71, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '208331', 'license': 'MIT', 'n_tokens_mistral': 4584, 'n_tokens_neox': 4259, 'n_words': 976, 'path': '2_TextCategorization.ipynb', 'repo_name': 'oroszgy/hunlp-text-mining-workshop', 'size': '11993'}