<jupyter_start><jupyter_text>[Detecting the difficulty level of French texts](https://www.kaggle.com/c/detecting-the-difficulty-level-of-french-texts/overview/evaluation) First approach---In this notebook, we will try differents classification models with very basic text cleaning and no hyper-parameters tuning. This will allow us to have a baseline and to select the most promising models for further study.  Imports<jupyter_code># Download the french spacy model
!python -m spacy download fr_core_news_md
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import spacy
from spacy import displacy
import string
import numpy as np
import string
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.base import TransformerMixin
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, LabelEncoder
from spacy.lang.en.stop_words import STOP_WORDS
from spacy.lang.en import English
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression, LogisticRegressionCV
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.utils.multiclass import unique_labels
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import recall_score, precision_score, f1_score, accuracy_score, confusion_matrix, ConfusionMatrixDisplay
import re
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer, SnowballStemmer, WordNetLemmatizer
nltk.download('punkt')
np.random.seed = 0

def evaluate(y_true, pred):
  """
  Calculate the models performance metrics. 
  Since it is a multi-class classification, we take the weighted average 
  for the metrics that are calculated for each class.

  """

  report = {
      'accuracy':accuracy_score(y_true, pred),
      'recall':recall_score(y_true, pred, average='macro'),
      'precision':precision_score(y_true, pred, average='macro'),
      'f1_score':f1_score(y_true, pred, average='macro')
  }

  return report

def plot_confusion_matrix(y_true, pred, model):
  
  """
  A function to plot the models confusion matrix.
  """

  cf_matrix = confusion_matrix(y_test, pred)

  disp = ConfusionMatrixDisplay(confusion_matrix=cf_matrix,
                              display_labels=model.classes_)
    
  disp.plot()


reports = {}

df = pd.read_csv("https://raw.githubusercontent.com/LaCrazyTomato/Group-Project-DM-ML-2021/main/data/training_data.csv")
unlabeled_df = pd.read_csv("https://raw.githubusercontent.com/LaCrazyTomato/Group-Project-DM-ML-2021/main/data/unlabelled_test_data.csv")

display(df.head())
display(unlabeled_df.head())<jupyter_output><empty_output><jupyter_text>2. Tokenizer & vectorizer<jupyter_code># Define cleaning function
def nltk_tokenizer(doc):

    # Tokenize
    doc = word_tokenize(doc)

    # Remove uppercase and white spaces
    doc = [word.lower().strip() for word in doc]
    
    return doc


print(nltk_tokenizer(df.loc[2, 'sentence']))<jupyter_output>['le', 'test', 'de', 'niveau', 'en', 'français', 'est', 'sur', 'le', 'site', 'internet', 'de', "l'école", '.']<jupyter_text>We selected a TFIDF vectorizer because it gives more importance to rare words and we believe that a rare word can be a sign of complexity.<jupyter_code>tfidf_vectorizer = TfidfVectorizer(tokenizer=nltk_tokenizer)<jupyter_output><empty_output><jupyter_text>3. ClassificationWe are now ready to test the different classification models. We will start with the ones we have seen in class and then we will try others provided by Sklearn.<jupyter_code># Select features
X = df['sentence']
y = df['difficulty']

# Train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)

X_train<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code># Define classifier
LR_model = LogisticRegression()

LR_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', LR_model)])

LR_pipe.fit(X_train, y_train)

pred = LR_pipe.predict(X_test)

LR_report = evaluate(y_test, pred)

# Store model performance results
reports['Logistic Regression'] = LR_report

LR_report
plot_confusion_matrix(y_test, pred, LR_pipe)<jupyter_output><empty_output><jupyter_text>Let's look at some errors to maybe understand better why the model gets it wrong.<jupyter_code>badly_predicted = pd.DataFrame({'sentence':X_test[pred != y_test],
              'predicted':pred[pred != y_test],
              'true':y_test[pred != y_test]})


for i, row in badly_predicted.sample(3).iterrows():
    print(row.sentence)
    print(f"Predicted: {row.predicted}")
    print(f"Actual: {row.true}")<jupyter_output>Je pense que fréquenter les galeries est la meilleure manière d'éduquer son oeil à l'art contemporain.
Predicted: A2
Actual: B2
Si tu pars en vacances avec nous, je serais ravie.
Predicted: A1
Actual: B1
Je vais étudier l'anglais
Predicted: A2
Actual: A1<jupyter_text>Hard to tell where the errors comes from.. Probably the vectorizer has to be tuned.  kNN<jupyter_code>knn_model = KNeighborsClassifier()

knn_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', knn_model)])


knn_pipe.fit(X_train, y_train)

pred = knn_pipe.predict(X_test)

knn_report = evaluate(y_test, pred)

# Store model performance results
reports['kNN'] = knn_report


knn_report
plot_confusion_matrix(y_test, pred, knn_pipe)<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>tree_model = DecisionTreeClassifier()

tree_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', tree_model)])


tree_pipe.fit(X_train, y_train)

pred = tree_pipe.predict(X_test)

tree_report = evaluate(y_test, pred)

# Store model performance results
reports['Decision Tree'] = tree_report

tree_report
plot_confusion_matrix(y_test, pred, tree_pipe)<jupyter_output><empty_output><jupyter_text>Random Forest<jupyter_code>forest_model = RandomForestClassifier()

forest_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', forest_model)])

forest_pipe.fit(X_train, y_train)

pred = forest_pipe.predict(X_test)

forest_report = evaluate(y_test, pred)

# Store model performance results
reports['Random Forest'] = forest_report

forest_report
plot_confusion_matrix(y_test, pred, forest_pipe)<jupyter_output><empty_output><jupyter_text>Ridge classifier<jupyter_code>from sklearn.linear_model import RidgeClassifier

ridge_model = RidgeClassifier()

ridge_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', ridge_model)])


ridge_pipe.fit(X_train, y_train)

pred = ridge_pipe.predict(X_test)

ridge_report = evaluate(y_test, pred)

# Store model performance results
reports['Ridge Classifier'] = ridge_report

ridge_report
plot_confusion_matrix(y_test, pred, ridge_pipe)<jupyter_output><empty_output><jupyter_text>Perceptron classifier<jupyter_code>from sklearn.linear_model import Perceptron

perceptron_model = Perceptron()

perceptron_pipe = Pipeline([('vectorizer', tfidf_vectorizer),
                 ('classifier', perceptron_model)])


perceptron_pipe.fit(X_train, y_train)

pred = perceptron_pipe.predict(X_test)

perceptron_report = evaluate(y_test, pred)

# Store model performance results
reports['Perceptron Classifier'] = perceptron_report

perceptron_report

plot_confusion_matrix(y_test, pred, perceptron_pipe)<jupyter_output><empty_output><jupyter_text>Select best models<jupyter_code>fig, ax = plt.subplots(1, 1, figsize=(9,6))

width = 0.35
labels = reports.keys()

ax.bar(labels, [x['accuracy'] for x in reports.values()], 
        label='Accuracy', width=width)

plt.xticks(rotation=65)
ax.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09900990099009901, 'fraction_numerical': 0.004315816197004316, 'mean_word_length': 5.89326334208224, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '260757', 'license': 'MIT', 'n_tokens_mistral': 2700, 'n_tokens_neox': 2592, 'n_words': 612, 'path': 'code/first_approach.ipynb', 'repo_name': 'MartinsAlex/Group-Project-DM-ML-2021', 'size': '7878'}