<jupyter_start><jupyter_text>Balancing Process<jupyter_code>data_y.value_counts()
non_irr_df = df[df.topic!='IRRELEVANT']
irr_df = df[df.topic=='IRRELEVANT']
irr_df = irr_df.iloc[:data_y.value_counts()[1]]
new_df = pd.concat([non_irr_df,irr_df])
data_x = new_df.article_words
data_y = new_df.topic
train_x,dev_x,train_y,dev_y = train_test_split(data_x,data_y,test_size = 0.1,shuffle=False)
le = preprocessing.LabelEncoder()
le.fit(train_y)
en_train_y = le.transform(train_y)
en_dev_y = le.transform(dev_y)
topics = list(set(df['topic']))
print(le.inverse_transform(en_train_y))
print(train_y)<jupyter_output>0        FOREX MARKETS
1        MONEY MARKETS
2               SPORTS
3        FOREX MARKETS
5       SHARE LISTINGS
             ...      
2087        IRRELEVANT
2088        IRRELEVANT
2090        IRRELEVANT
2091        IRRELEVANT
2092        IRRELEVANT
Name: topic, Length: 5795, dtype: object<jupyter_text>Transform Training Text to TF-IDF<jupyter_code>vectorizer = TfidfVectorizer()
train_vectors = vectorizer.fit_transform(train_x)
dev_vectors = vectorizer.transform(dev_x)
print(train_vectors.shape,dev_vectors.shape)
print(train_vectors.nnz/float(train_vectors.shape[0]))<jupyter_output>(5795, 27230) (644, 27230)
81.13960310612597<jupyter_text>From the output above, it could be seen that the extracted TF-IDF vectors from training set are sparse, as there are only 81 non-zero features over 27230 features.  Using SMOTE to Solve Data Imbalance  First, by plotting each topic's frequency, it could be seen that the distribution of topics is imbalanced.<jupyter_code>train_y.to_frame()<jupyter_output><empty_output><jupyter_text>Considering in the data set, the topic 'MONEY MARKET' occupies the majority, which might, indirectly, lead to biased prediction to itself, we need to adjust each topic's proportion in the training data set. In this project, SMOTE (Synthetic Minority Over-sampling TEchnique) is used to solve the imbalance problem.<jupyter_code>smote = SMOTE(random_state=1)
smo_x,smo_y = smote.fit_sample(train_vectors,en_train_y)<jupyter_output><empty_output><jupyter_text>Optimal Hyper-parameter Warning: The following block runs for a long time, don't run it if not necessary<jupyter_code># # alpha = [0.001*n for n in range(1,1001)]
# alpha = [0.01*n for n in range(1,101)]

# fit_prior = [True,False]
# # After using SMOTE to improve the imbalance, we can assume the distribution is uniform distribution
# prior = [1/len(topics) for i in range(len(topics))]
# class_prior = [None,prior]
# kfold = StratifiedKFold(n_splits=10,shuffle=True,random_state=1)
# param_grid = dict(alpha=alpha,fit_prior=fit_prior,class_prior=class_prior)
# grid_search = GridSearchCV(MultinomialNB(),param_grid,scoring='neg_log_loss',n_jobs=-1,cv=kfold)
# grid_result = grid_search.fit(smo_x,smo_y)
# print("The best parameters are {0}, with score {1:4}".format(grid_result.best_params_,grid_result.best_score_))
def get_scores(en_dev_y,pred_y,model_name,topic=None):
    if(topic==None):
        print("F1 score for ",model_name," model is ",f1_score(en_dev_y,pred_y,average='macro'))
        print("Accuracy score for ",model_name," model is ",accuracy_score(en_dev_y,pred_y))
        print("Recall score for ",model_name," model is ",recall_score(en_dev_y,pred_y,average='macro'),"\n")
    else:
        print("F1 score of ",topic," for ",model_name," model is ",f1_score(en_dev_y,pred_y,average='macro'))
        print("Accuracy score of ",topic," for ",model_name," model is ",accuracy_score(en_dev_y,pred_y))
        print("Recall score of ",topic," for ",model_name," model is ",recall_score(en_dev_y,pred_y,average='macro'),"\n")<jupyter_output><empty_output><jupyter_text>MultinomialNB Method<jupyter_code># Fit data
clf = MultinomialNB(alpha=0.003)
clf.fit(smo_x,smo_y)
pred_y = clf.predict(dev_vectors)
get_scores(en_dev_y,pred_y,'MultinomialNB')
proba_y = clf.predict_proba(dev_vectors)
print(min([max(lst) for lst in proba_y]))
print(roc_auc_score(en_dev_y,proba_y,multi_class='ovr'))
irr_data = df[df.topic=='IRRELEVANT']
irr_x = irr_data.article_words
irr_y = irr_data.topic
en_irr_y = le.fit_transform(irr_y)
irr_vectors = vectorizer.transform(irr_x)
irr_proba_y = clf.predict_proba(irr_vectors)
print(max([max(lst) for lst in irr_proba_y]))<jupyter_output><empty_output><jupyter_text>Result for Test Set<jupyter_code>df_test = pd.read_csv('test.csv')
print(df_test.isnull().sum())
df_test.head(5)
test_y= df_test.topic
le.fit(train_y)
en_test_y = le.transform(test_y)<jupyter_output><empty_output><jupyter_text>Scores for Topics<jupyter_code>def scores_for_topics(df,topics,model,le):
    for topic in topics:
        topic_scores(df,topic,model,le)

def topic_scores(df,topic,model,le):
    new_df = df[df.topic==topic]
    test_x = new_df.article_words
    test_y = new_df.topic
    test_vectors = vectorizer.transform(test_x)
    le.fit(train_y)
    en_test_y = le.transform(test_y)
    prediction = model.predict(test_vectors)
    print(en_test_y)
    print(prediction)
    get_scores(en_test_y,prediction,type(model).__name__,topic)
    
    
    
le.inverse_transform([8,0,6,5])
scores_for_topics(df_test,topics,clf,le)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1187487932033211, 'fraction_numerical': 0.021432709017184786, 'mean_word_length': 6.224546722454672, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '444961', 'license': 'Apache-2.0', 'n_tokens_mistral': 2085, 'n_tokens_neox': 1935, 'n_words': 366, 'path': "Raymond_work/Raymond's_Work_temp.ipynb", 'repo_name': 'HenryLiangzy/COMP9417_Project', 'size': '5179'}