<jupyter_start><jupyter_text>Eexamples of calculating interpretation quality metrics.  Typically,  you might need to obtain whole dataset metrics, then please use ready script *metrics.py* (which will do all calculations under the hood and return dataset-aggregated metrics and per-molecule metrics too, if  needed.This notebook does the same job. However, there is a possibility to use functions from the script individually. The notebook shows how to use them.  If you are interested in the details about metrics calculations,  the notebook explains them step-by-step. Notebook may be convinient if you are interested in analysis of individual molecules.<jupyter_code>import metrics as mt
from rdkit import Chem
from rdkit.Chem.Draw import SimilarityMaps
import numpy as np
import pandas as pd<jupyter_output>RDKit WARNING: [12:19:39] Enabling RDKit 2019.09.3 jupyter extensions<jupyter_text>We will use as an example N dataset, where activity is defined as the count of Nitrogen.   Load file with contributions<jupyter_code>contribs = mt.read_contrib("example_notebook_data/N_contrib_per_atom_dc.txt")
contribs<jupyter_output><empty_output><jupyter_text>Load ground thruth labels<jupyter_code>lbls = mt.read_lbls_from_sdf(
        "example_notebook_data/N_train_lbl_small.sdf", lbls_field_name="lbls", sep=",")
lbls<jupyter_output><empty_output><jupyter_text>Merge contributions with ground truth. Note: the merge is *inner*, only molecules present in both will remain<jupyter_code>merged = mt.merge_lbls_contribs(contribs, lbls)
merged<jupyter_output><empty_output><jupyter_text>Now we are ready to calculate metrics<jupyter_code>## Let's calculate AUC. which_lbls="positive" indicates to calculate AUC for positively contributing atoms. Specifying "negative"
## you switch to negatively contributing atoms, which in this dataset are absent.
auc = mt.calc_auc(merged, which_lbls="positive", contrib_col_name="contribution")
auc
## Lets compute another metric - top-n score, with variable n (indicated by n_list parameter="infinity"), and also with fixed n=3,5
top_n = mt.calc_top_n(merged, n_list=[np.inf,3,5], contrib_col_name="contribution") 
top_n.keys() # top-n is a dictionary with bunch of dataframes<jupyter_output><empty_output><jupyter_text>We will need only  'variable_top_n', 'top3' and 'top5'  dataframes,  In each dataframe "top_score" column contains scores (per molecule).<jupyter_code>top_n["variable_top_n"]
top_n["top3"].top_score
top_n["top5"].top_score<jupyter_output><empty_output><jupyter_text>Now we can aggregate metrics per dataset<jupyter_code>summary = mt.summarize([auc, top_n])
summary<jupyter_output><empty_output><jupyter_text>We can look at individual molecules, if we are interested  Let's have a look at first molecule:<jupyter_code>mols = Chem.SDMolSupplier("example_notebook_data/N_train_lbl_small.sdf")
m = mols[0]
nm = m.GetProp("_Name")
nm
m
auc["auc_pos"][nm] # look at AUC for this molecule, it's not perfect (<1)
top_n['top3']['top_score'][nm] # look at top-n score for this molecule, it's only 0.5<jupyter_output><empty_output><jupyter_text>Let's visualize contributions of atoms. Nitrogen has ground truth label "1". So contributions for nitrogens should be the highest  We will use RDKit's Similarity Maps  for highlighting atoms:<jupyter_code>wt = {}
for n,atom in enumerate(Chem.rdmolfiles.CanonicalRankAtoms(m)):
                wt[atom] = contribs.loc[contribs.molecule==nm,"contribution"].iloc[n]
sim_map = SimilarityMaps.GetSimilarityMapFromWeights(m,wt)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08929076010172365, 'fraction_numerical': 0.007629273806159932, 'mean_word_length': 6.831858407079646, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '505969', 'license': 'MIT', 'n_tokens_mistral': 1168, 'n_tokens_neox': 1112, 'n_words': 342, 'path': 'example_notebook.ipynb', 'repo_name': 'ci-lab-cz/ibenchmark', 'size': '3539'}