<jupyter_start><jupyter_text>Initial Modelling notebook<jupyter_code>import os
import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
import bay12_solution_eposts as solution<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>post, thread = solution.prepare.load_dfs('train')
post.head(2)
thread.head(2)<jupyter_output><empty_output><jupyter_text>I will set the thread number to be the index, to simplify matching in the future:<jupyter_code>thread = thread.set_index('thread_num')
thread.head(2)<jupyter_output><empty_output><jupyter_text>We'll load the label map as well, which tells us which index goes to which label<jupyter_code>label_map = solution.prepare.load_label_map()
label_map<jupyter_output><empty_output><jupyter_text>Create features from thread dataframe We will fit a CountVectorizer, which is a simple transformation that counts the number of times the word was found.The parameter `min_df` sets the minimum number of occurances in our set that will allow a word to join our vocabulary.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer(ngram_range=(1, 1), min_df=3)
word_vectors_raw = cv.fit_transform(thread['thread_name'])<jupyter_output><empty_output><jupyter_text>To save space, this outputs a sparse matrix:<jupyter_code>word_vectors_raw<jupyter_output><empty_output><jupyter_text>However, since we'll be using it with a DataFrame, we need to convert it into a Pandas DataFrame:<jupyter_code>word_df = pd.DataFrame(word_vectors_raw.toarray(), columns=cv.get_feature_names(), index=thread.index)
word_df.head()<jupyter_output><empty_output><jupyter_text>The only other feature we have from our thread data is the number of replies. Let's add one to get the number of replies. Also, let's use the logarithm of post count as well, just for fun.We'll concatenate those into our X dataframe (Note that I'm renaming the columns, to keep track more easily):<jupyter_code>X = pd.concat([
        (thread['thread_replies'] + 1).rename('posts'), 
        np.log(thread['thread_replies'] + 1).rename('log_posts'), 
        word_df,
    ], axis='columns')
X.head()<jupyter_output><empty_output><jupyter_text>Our target is the category number. Remember that this isn't a regression task - there is no actual order between these categories! Also, our Y is one-dimensional, so we'll keep it as a Series (even though it prints less prettily).<jupyter_code>y = thread['thread_label_id']
y.head()<jupyter_output><empty_output><jupyter_text>Split dataset into "training" and "validation" In order to check the quality of our model in a more realistic setting, we will split all our input (training) data into a "training set" (which our model will see and learn from) and a "validation set" (where we see how well our model generalized). [Relevant link](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html).<jupyter_code>from sklearn.model_selection import train_test_split
# NOTE: setting the `random_state` lets you get the same results with the pseudo-random generator
validation_pct = 0.25
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=validation_pct, random_state=99)
X_train.shape, y_train.shape
X_val.shape, y_val.shape<jupyter_output><empty_output><jupyter_text>Fit a model Since we are fitting a multiclass model, [this scikit-learn link](https://scikit-learn.org/stable/modules/multiclass.html) is very relevant. To simplify things, we will be using an algorithm that is inherently multi-class.<jupyter_code>from sklearn.tree import DecisionTreeClassifier

# Just using default parameters... what can do wrong?
cls = DecisionTreeClassifier(random_state=1337)
# Fit
cls.fit(X_train, y_train)
# In-sample and out-of-sample predictions
# NOTE: we 
y_train_pred = pd.Series(
    cls.predict(X_train), 
    index=X_train.index, 
)
y_val_pred = pd.Series(
    cls.predict(X_val), 
    index=X_val.index, 
)
y_val_pred.head()<jupyter_output><empty_output><jupyter_text>Score the model To find out how well the model did, we'll use the [model evaluation functionality of sklearn](https://scikit-learn.org/stable/modules/model_evaluation.html); specifically, the [multiclass classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-metrics).<jupyter_code>from sklearn.metrics import confusion_matrix, accuracy_score, classification_report<jupyter_output><empty_output><jupyter_text>The [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) shows how our predictions differ from the actual values.It's important to note how strongly our in-sample (training) and out-of-sample (validation/test) metrics differ.<jupyter_code>def confusion_df(y_actual, y_pred):
    res = pd.DataFrame(
        confusion_matrix(y_actual, y_pred, labels=label_map.values),
        index=label_map.index.rename('predicted'),
        columns=label_map.index.rename('actual'),
    )
    return res
confusion_df(y_train, y_train_pred).style.highlight_max()
confusion_df(y_val, y_val_pred).style.highlight_max()<jupyter_output><empty_output><jupyter_text>Oh boy. That's pretty bad - we didn't predict anything for several columns! Let's look at the metrics to confirm that it is indeed bad.<jupyter_code>print("Test accuracy:", accuracy_score(y_train, y_train_pred))
print("Validation accuracy:", accuracy_score(y_val, y_val_pred))
report = classification_report(y_val, y_val_pred, labels=label_map.values, target_names=label_map.index)
print(report)<jupyter_output>precision    recall  f1-score   support

        bastard       1.00      0.40      0.57         5
beginners-mafia       0.67      0.50      0.57         4
           byor       0.00      0.00      0.00         2
        classic       0.40      0.25      0.31         8
   closed-setup       0.33      0.71      0.45         7
         cybrid       0.00      0.00      0.00         1
           kotm       0.00      0.00      0.00         1
 non-mafia-game       0.00      0.00      0.00         0
          other       0.88      0.84      0.86        55
     paranormal       0.60      1.00      0.75         3
   supernatural       0.00      0.00      0.00         0
        vanilla       0.00      0.00      0.00         1
       vengeful       0.67      0.67      0.67         3

      micro avg       0.69      0.69      0.69        90
      macro avg       0.35      0.34      0.32        90
   weighted avg       0.73      0.69      0.69        90<jupyter_text>Well, that's pretty bad. We seriously overfit our training set... which is sort-of what I expected. Oh well.By the way, the warnings at the bottom say that we have no real Precision or F-score to use, with no predictions for some classes.   Predict with the modelHere, we will predict on the test set (predicitions to send in), then save the results and the model.**IMPORTANT NOTE**: In reality, you need to re-train your same model on the entire set to predict! However, I'm just using the same model as before, as it will bad anyways. ;)<jupyter_code>post_test, thread_test = solution.prepare.load_dfs('test')
thread_test = thread_test.set_index('thread_num')
thread_test.head(2)<jupyter_output><empty_output><jupyter_text>We need to attach a `thread_label_id` column, as given in the training set:<jupyter_code>thread.head(2)<jupyter_output><empty_output><jupyter_text>Use the fitted CountVectorizer and other features to make our X dataframe:<jupyter_code>word_vectors_raw_test = cv.transform(thread_test['thread_name'])
word_df_test = pd.DataFrame(word_vectors_raw_test.toarray(), columns=cv.get_feature_names(), index=thread_test.index)
word_df_test.head()
X_test = pd.concat([
        (thread_test['thread_replies'] + 1).rename('posts'), 
        np.log(thread_test['thread_replies'] + 1).rename('log_posts'), 
        word_df_test,
    ], axis='columns')
X_test.head()<jupyter_output><empty_output><jupyter_text>Now we predict with our model, then paste it to a copy of `thread_test` as column `thread_label_id`.<jupyter_code>y_test_pred = pd.Series(
    cls.predict(X_test), 
    index=X_test.index, 
)
y_test_pred.head()
result = thread_test.copy()
result['thread_label_id'] = y_test_pred
result.head()<jupyter_output><empty_output><jupyter_text>We need to reshape to conform to the submission format specified [here](https://www.kaggle.com/c/ni-mafia-gametypeevaluation).<jupyter_code>result = result.reset_index()[['thread_num', 'thread_label_id']]
result.head()<jupyter_output><empty_output><jupyter_text>Export predictions, modelOur model consists of the text vectorizer `cv` and classifier `cls`. We already formatted our results, we just need to make sure not to write an extra index column.<jupyter_code># NOTE: Exporting next to the notebooks - the files are small, but usually you don't want to do this.
out_dir = os.path.abspath('1_output')
os.makedirs(out_dir, exist_ok=True)
result.to_csv(
    os.path.join(out_dir, 'baseline_predict.csv'),
    index=False, header=True, encoding='utf-8', 
)
import joblib

joblib.dump(cv, os.path.join(out_dir, 'cv.joblib'))
joblib.dump(cls, os.path.join(out_dir, 'cls.joblib'))
print("Done. :)")<jupyter_output>Done. :)
{'fraction_non_alphanumeric': 0.09707158351409978, 'fraction_numerical': 0.020607375271149676, 'mean_word_length': 4.2963813900057435, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '1644', 'license': 'Apache-2.0', 'n_tokens_mistral': 3068, 'n_tokens_neox': 2951, 'n_words': 821, 'path': 'notebooks/1_initial_model.ipynb', 'repo_name': 'NowanIlfideme/kaggle_ni_mafia_gametype', 'size': '9220'}