<jupyter_start><jupyter_text>Exploration of key words occurances in thread names notebook<jupyter_code>import os
import sys
from src import data_prepare

import pandas as pd
import numpy as np

import warnings
warnings.filterwarnings('ignore')
post, thread=data_prepare.load_train_data()
label_map=data_prepare.load_label_map()
thread.head()
key_words=pd.Series(label_map.index)
key_words
key_words[1]="beginner"
key_words[6]="king"
thread["name"]=thread["thread_name"].apply(lambda x: data_prepare.clean(x)).as_matrix()<jupyter_output><empty_output><jupyter_text>Let's just try to calculate how efficient it is to look for names themselves in thread names for different classes<jupyter_code>num_classes=len(key_words)
correct=pd.Series(np.zeros(num_classes),index=label_map.index)
over=pd.Series(np.zeros(num_classes),index=label_map.index)
under=pd.Series(np.zeros(num_classes),index=label_map.index)
al=pd.Series(np.zeros(num_classes),index=label_map.index)
for item in thread.itertuples():
    label=item.thread_label_id
    if key_words[label] in item.name:
        correct[label]+=1
    else:
        under[label]+=1
    for index,s in enumerate(key_words):
        if index==label:
            continue
        if s in item.name:
            over[index]+=1
            break
    al[label]+=1
x = pd.DataFrame({'all':al,'correct':correct,'under': under,'over':over,
                  'per cent correct':correct/al,'per cent under':under/al,
                 'per cent over':over/al},
                 index=correct.index)
x
x["efficiency"]=(x["correct"]-x["over"])/x['all']
x<jupyter_output><empty_output><jupyter_text>Not the best metric for efficiency, but it seems like it makes sense to use this kind of raw prediction for classes that score more than 40-50%, especially if they don't have a lot of representatives in the dataset, like cybrid or supernatural<jupyter_code>post_test, thread_test=data_prepare.load_test_data()
test_stat=pd.Series(np.zeros(13),index=label_map.index)
thread_test["name"]=thread_test["thread_name"].apply(lambda x: data_prepare.clean(x)).as_matrix()
for item in thread_test.itertuples():
    for index,s in enumerate(key_words):
        if s in item.name:
            test_stat[index]+=1
test_stat<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11959399823477493, 'fraction_numerical': 0.005736981465136805, 'mean_word_length': 5.094086021505376, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '353246', 'license': 'MIT', 'n_tokens_mistral': 750, 'n_tokens_neox': 736, 'n_words': 163, 'path': 'notebooks/Key words in thread_name exploration.ipynb', 'repo_name': 'Nikishul/Kaggle-NMA-Competition', 'size': '2266'}