<jupyter_start><jupyter_text>Solving the Definition Extraction Problem Approach 1: Using Classical Embedding Techniques and Classifiers.In this notebook, we demonstrate our first approach for solving the problem in hand; Definition Extraction or what generally can be thought as the Text Classification problem.   Embedding Techniques Used: BOW and TF-IDFWe can’t simply use text strings in our machine learning model; we need a way to convert our text into something that can be represented numerically.- One tool we can use is **Bag of Words.** BoW converts text into the matrix of occurrence of words within a given document. It focuses on whether given words occurred or not in the document, and it generates a matrix that we might see referred to as a **BoW matrix or a document term matrix.** - Another tool is the **TF-IDF (Term Frequency-Inverse Document Frequency)**. It’s simply a way of **normalizing our Bag of Words(BoW)** by looking at each word’s frequency in comparison to the document frequency. In other words, it’s a way of representing how important a particular term is in the context of a given document, based on how many times the term appears and how many other documents that same term appears in. **The higher the TF-IDF, the more important that term is to that document.**- Luckily for us, we only need to understand how this works not work it by hand ourselves and that thanks to the beautiful library **sklearn**. We can generate a BoW matrix for our text data by using scikit-learn‘s **CountVectorizer**. Also, we can generate TF-IDF automatically using scikit-learn‘s **TfidfVectorizer.**- **N-grams** are combinations of adjacent words in a given text, where n is the number of words that incuded in the tokens. In all the embedding techniques used here we have **N-grams range** set with the lower and upper bounds suitable for every expriement.  Classifiers Used: LR, NB and DTsWe already assume the reader have good understanding of the classifiers we have in hand but for the sake of reporting everything, we list all those used with a brief description. If you don't feel familar with them, we recommend you read about them first then come back.- **LR (Logistic Regression):** is a statistical model that in its basic form uses a logistic function to model a binary dependent variable. - **NB (Naive Bayes):** are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable. We use the Multinomial variation here.- **DTs (Decsion Trees):** are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.<jupyter_code># imports cell
import pandas as pd
import os
import sys 
sys.path.append("../")
from source.data_loader import DeftCorpusLoader
# classification imports 
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import MultinomialNB
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline
from sklearn import metrics<jupyter_output>Using TensorFlow backend.<jupyter_text>Apply Loading, Preprocessing and Cleaning the dataset.Refer to "Deft Corpus Loading and Preparation" notebook for a detailed explaination on this.<jupyter_code>deft_loader = DeftCorpusLoader("../deft_corpus/data")
trainframe, devframe = deft_loader.load_classification_data(preprocess=True, clean=True)<jupyter_output><empty_output><jupyter_text>Expriement 1: Logistic Regression with CountVectorizer- Note 1: We use here **class weights** implemented inside sklearn's LR to handle class imbalance in data, that is done by passing `class_weight='balanced'` to the classifier constructor.- Note 2: Had to increase number of **maximum iterations** for classifier to converge on data (i.e. data-classifer specific not a general case), that is done by passing `max_iter = 1000`, default is 100 iterations only.- Note 3: We tuned `ngram_range` upper bound with some trials to find out that a value of 2 maximizes results in this experiement.<jupyter_code>classifier = LogisticRegression(class_weight='balanced', max_iter =1000)
pipe = Pipeline([('vectorizer', CountVectorizer(tokenizer= deft_loader._spacy_preprocessor, ngram_range=(1,2))),
                 ('classifier', classifier)]);

pipe.fit(trainframe["Sentence"], trainframe["HasDef"]);
predicted = pipe.predict(devframe["Sentence"])
print(metrics.classification_report(list(devframe["HasDef"]), predicted))<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.85      0.83       512
           1       0.69      0.63      0.66       267

    accuracy                           0.78       779
   macro avg       0.75      0.74      0.74       779
weighted avg       0.77      0.78      0.77       779<jupyter_text>Expriement 2: Naive Bayes with TF-IDF Vectorizer- Note 1: We use here **SMOTE** (Synthetic Minority Over-sampling) technique to handle class imbalance in data, that is done by passing to adding the SMOTE layer to the pipeline. We use the `imblearn.over_sampling.SMOTE` class implementation of SMOTE.- Note 2: We had to use `Pipeline class from imblearn`, because sklearn's Pipeline won't work with SMOTE Layer.- Note 3: Like above, we tuned `ngram_range` upper bound with some trials to find out that a value of 2 maximizes results in this experiement.<jupyter_code>classifier = MultinomialNB()
pipe = Pipeline([('vectorizer', TfidfVectorizer(tokenizer= deft_loader._spacy_preprocessor, ngram_range=(1,2))),
                 ('imbalance', SMOTE(random_state=88)),
                 ('classifier', classifier)]);

pipe.fit(trainframe["Sentence"], trainframe["HasDef"]);
predicted = pipe.predict(devframe["Sentence"])
print(metrics.classification_report(list(devframe["HasDef"]), predicted))<jupyter_output>precision    recall  f1-score   support

           0       0.82      0.70      0.76       510
           1       0.57      0.72      0.63       275

    accuracy                           0.71       785
   macro avg       0.70      0.71      0.70       785
weighted avg       0.73      0.71      0.72       785<jupyter_text>Expriement 3: Decision Trees with TF-IDF Vectorizer- Note 1: We here used a higher upper bound for `ngram_range`(i.e. = 3) rather than 2 used in the above two experiments after trying and finding out that maximizies result scores for this experiment.<jupyter_code>classifier = DecisionTreeClassifier()
pipe = Pipeline([('vectorizer', TfidfVectorizer(tokenizer= deft_loader._spacy_preprocessor, ngram_range=(1,3))),
                 ('imbalance', SMOTE(random_state=88)),
                 ('classifier', classifier)]);

pipe.fit(trainframe["Sentence"], trainframe["HasDef"]);
predicted = pipe.predict(devframe["Sentence"])
print(metrics.classification_report(list(devframe["HasDef"]), predicted))<jupyter_output>precision    recall  f1-score   support

           0       0.76      0.83      0.79       510
           1       0.62      0.52      0.57       275

    accuracy                           0.72       785
   macro avg       0.69      0.67      0.68       785
weighted avg       0.71      0.72      0.71       785
{'fraction_non_alphanumeric': 0.07234899328859061, 'fraction_numerical': 0.02778523489932886, 'mean_word_length': 3.683218101822753, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '444281', 'license': 'MIT', 'n_tokens_mistral': 2136, 'n_tokens_neox': 1957, 'n_words': 872, 'path': 'notebooks/Classical Baseline Approach.ipynb', 'repo_name': 'Elzawawy/DeftEval', 'size': '7450'}