<jupyter_start><jupyter_text>Mass Evaluation Result analysis Macro F1 3-Way<jupyter_code>f1_results = evaluation.evaluate_models(f1_score, dataset_test, all_results, 
                                        dataframe=True, average='macro')
(f1_results * 100).round(2)
vis_f1 = copy.deepcopy(f1_results)
vis_f1.columns = [col.replace(' 3 Class Macro F1', '') for col in vis_f1.columns]
vis_f1.index = ['\n'.join(wrap(index, 10)) for index in vis_f1.index]
fig, ax = plt.subplots(figsize=(9,7))
sns.heatmap(vis_f1, annot=True, annot_kws={'size':15},
            cmap=sns.color_palette("Blues"), ax=ax)
ax.tick_params(axis='both', which='major', labelsize=15)
ax.set_title('Macro F1', fontdict={'fontsize':20})<jupyter_output><empty_output><jupyter_text>Accuracy<jupyter_code>acc_results = evaluation.evaluate_models(accuracy_score, dataset_test, 
                                         all_results, dataframe=True)
(acc_results * 100).round(2)<jupyter_output><empty_output><jupyter_text>Macro F1 2-wayWe can then look at the 2 way Macro F1 score which evaluates how well the models can predict Positive or Negative sentiment while still taking into account prediciting the neutral class.<jupyter_code>(evaluation.evaluate_models(f1_score, dataset_test, all_results, 
                           dataframe=True, average='macro', labels=[-1, 1]) * 100).round(2)<jupyter_output><empty_output><jupyter_text>Macro F1, 1 Distinct Sentiment subsetNow we look at the results of trying to predict the sentiment of all targets that are only in sentence that contain one distinct sentiment. This should be a result that is good for the LSTM model as it does not take into account any target information. However as we can see below the LSTM is one of the worse models.<jupyter_code>(evaluation.evaluate_models(distinct_sentiment_metrics, dataset_test, all_results, custom_metric=True,
                            dataframe=True, distinct_sentiments=1, 
                            metric_func=f1_score, average='macro') * 100).round(2)<jupyter_output><empty_output><jupyter_text>Macro F1, 2 Distinct Sentiment subsetWe do the same as above but for 2 distinct sentiments per sentence. We remove the Dong twitter dataset from this experiment due to it only containing Tweets with one distinct sentiment.<jupyter_code>two_dist_sent = evaluation.evaluate_models(distinct_sentiment_metrics, dataset_test, 
                                           all_results, custom_metric=True,
                                           dataframe=True, distinct_sentiments=2, 
                                           metric_func=f1_score, average='macro')
two_dist_sent = two_dist_sent.drop('Dong Twitter')
two_dist_sent = two_dist_sent.drop('Mean')
two_dist_mean = pd.Series(two_dist_sent.mean(), name='Mean')
(two_dist_sent.append(two_dist_mean) * 100).round(2)<jupyter_output>/home/andrew/Envs/Bella/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)<jupyter_text>Macro F1, 3 Distinct Sentiment subsetLastly the same as above but for 3 distinct sentiments. As the majority of the datasets have very few 3 distinct sentiments per sentence we only look at the Twitter Elections datases (8.78% of the dataset contains 3 distinct sentiments).<jupyter_code>three_dist_sent = evaluation.evaluate_models(distinct_sentiment_metrics, dataset_test, 
                                             all_results, custom_metric=True,
                                             dataframe=True, distinct_sentiments=3, 
                                             metric_func=f1_score, average='macro')
(three_dist_sent.loc['Election Twitter'].sort_values(ascending=False) * 100).round(2)<jupyter_output>/home/andrew/Envs/Bella/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
{'fraction_non_alphanumeric': 0.08957117482872368, 'fraction_numerical': 0.02106064450647044, 'mean_word_length': 3.676156583629893, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '297583', 'license': 'MIT', 'n_tokens_mistral': 1214, 'n_tokens_neox': 1133, 'n_words': 293, 'path': 'notebooks/Mass Evaluation Result Analysis.ipynb', 'repo_name': 'LaudateCorpus1/Bella-5', 'size': '3941'}