<jupyter_start><jupyter_text>Metrics vs PredictionsIn `PhiSpy` there are lots of metrics that you can adapt to suit your needs, but how important is each metric in each prediction?We have a [snakefile](https://github.com/linsalrob/PhiSpy/blob/master/scripts/phispy_metrics.snakefile) that will make all possible combinations of metrics and run them for you. The [parsing code](https://github.com/linsalrob/PhiSpy/blob/master/scripts/summarize_metrics_output.py) generates two files: the first is a list of a code (just an `int`) and the combination of metrics. This is just so the other table is not so big. The second output is the table of Accuracy, Precision, Recall, Specificity, and F1 score for each metric.<jupyter_code>import os, sys
import itertools
import re
import json
%matplotlib inline
from random import randint
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import gzip
from math import log, e
from scipy import stats
from math import sqrt<jupyter_output><empty_output><jupyter_text>Read the metrics<jupyter_code>mdf = pd.read_csv('metric_codes.tsv', delimiter="\t", index_col=0, names=['Code', 'Metric'], header=None)
mdf.head(30)
mdf.tail(30)<jupyter_output><empty_output><jupyter_text>Read the results file.Note that we deliberately set the data types here, because pandas was not guessing them correctly.<jupyter_code>dtypes={'Genome': object, 'Metric':np.int64, 'Accuracy':np.float64, 'Precision':np.float64, 'Recall':np.float64, 'Specificity':np.float64, 'F1 score':np.float64}
measures = pd.read_csv("phispy_metrics_tptn.tsv", delimiter="\t", dtype=dtypes, header=None, names=["Genome", "Metric", "Accuracy", "Precision", "Recall", "Specificity", 'F1 score'])
measures.head()<jupyter_output><empty_output><jupyter_text>Find the option with _no_ metricsWe need to find the option with `--metrics none`, `--phage_genes 0`, and `--ignore_annotations`. These are coded as `none`, `pg0`, and `noannotation` respectively.<jupyter_code>mdf[mdf['Metric'].str.contains('none') & mdf['Metric'].str.contains('noannotation') & mdf['Metric'].str.contains('pg0')]<jupyter_output><empty_output><jupyter_text>But no metrics == no phages!If we turn everything off, (code 45: no metrics, phage genes, or hmms) we do not find any phages! (remove the `.head()` to see more)<jupyter_code>measures[(measures['Metric'] == 45)].head()<jupyter_output><empty_output><jupyter_text>Find the option with _all_ metricsSimilarly, we need to find the option that includes `orf_length_med`, `shannon_slope`, `at_skew`, `gc_skew`, and `max_direction phmms`, but includes neither `noannotation` nor `pg0`<jupyter_code>mdf[~mdf['Metric'].str.contains('none') & ~mdf['Metric'].str.contains('noannotation') & ~mdf['Metric'].str.contains('pg0') & mdf['Metric'].str.contains('orf_length_med') & mdf['Metric'].str.contains('shannon_slope') & mdf['Metric'].str.contains('phmms')]<jupyter_output><empty_output><jupyter_text>Plot _none_ and _all_ metricsThese sort of represent "none" and "all". These are the two options from above<jupyter_code>fig, ax = plt.subplots(figsize=(22,16))

m_an = measures[(measures['Metric'] == 45) | (measures['Metric'] == 464)]
sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_an, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_an, jitter=True, color="Black")
ax.set_xticklabels(['None', 'All'])<jupyter_output><empty_output><jupyter_text>Plot some selected measuresWe have too much data to plot them all, so lets just plot a few selected measures. Chosen almost at random, but not really<jupyter_code>m_some = measures[(measures['Metric'] == 254) | (measures['Metric'] == 0) | (measures['Metric'] == 223)
                  | (measures['Metric'] == 57)
                  | (measures['Metric'] == 464)
                 ]
fig, ax = plt.subplots(figsize=(22,16))


sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_some, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_some, jitter=True, color="Black")
ax.set_xticklabels(list(mdf.iloc[m_some['Metric'].unique(),0]), rotation=45)<jupyter_output><empty_output><jupyter_text>Plot each metric aloneNote that for each metric, we need _none_ plus the metric, otherwise the results are misleading. Especially for `--phmms` and `--phagegenes` where the other metrics are set to the default.So here we plot each metric alone, together with a version of _all_ metrics.<jupyter_code>m_alone = measures[((measures['Metric'] >8) & (measures['Metric'] <17))
                   | (measures['Metric'] == 464)
                 ]
fig, ax = plt.subplots(figsize=(22,16))

labels = list(mdf.iloc[m_alone['Metric'].unique(),0])
labels[-1] = "All"
labels = [i.replace('none ', '') for i in labels]

sns.violinplot(ax = ax, x="Metric", y="F1 score", data=m_alone, scale="count" )
sns.stripplot(ax = ax,  x="Metric", y="F1 score", data=m_alone, jitter=True, color="Black")
ax.set_xticklabels(labels)
fig.savefig('metrics_alone.png')<jupyter_output><empty_output><jupyter_text>Summary of the metricsHow do the F1 scores vary across the metrics?<jupyter_code>st = measures[['Metric', 'F1 score']].groupby('Metric').describe()
st.head()<jupyter_output><empty_output><jupyter_text>Plot the different metricsWell there are a lot of colours. The more metrics we add, the better we get!<jupyter_code>fig, ax = plt.subplots(figsize=(22,16))


sns.barplot(ax = ax, x=st.index.values, y=('F1 score',  'mean'), data=st)<jupyter_output><empty_output><jupyter_text>Find some good, and bad, metricsLets see which combinations of metrics are good and bad.Note that we join the tables here, so we get the definitions of the metrics, not just the numbers.<jupyter_code>f_good = st[st[('F1 score',  'mean')] > 0.84]
f_good
pd.merge(f_good, mdf, left_index=True, right_index=True)
f_bad = st[st[('F1 score',  'mean')] < 0.5]
f_bad_code = pd.merge(f_bad, mdf, left_index=True, right_index=True)
f_bad_code
max(st[('F1 score',  'mean')])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14714714714714713, 'fraction_numerical': 0.015015015015015015, 'mean_word_length': 6.293187347931873, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '154535', 'license': 'MIT', 'n_tokens_mistral': 2110, 'n_tokens_neox': 2015, 'n_words': 550, 'path': 'jupyter_notebooks/metrics_vs_genomes.ipynb', 'repo_name': 'pdec/PhiSpy_deprekate', 'size': '5994'}