<jupyter_start><jupyter_text>maxsmi Analysis of resultsThis notebook serves to analyse the results of the simulations ran on the Curta cluster from the Freie Universität Berlin._Note_:The notebook will run using the results stored in the `output_` folder. They were generated using the two following commands:```(maxsmi) $ python maxsmi/full_workflow.py --task ESOL --string-encoding smiles --aug-strategy-train no_augmentation --aug-strategy-test no_augmentation --aug-nb-train 0 --aug-nb-test 0 --ml-model CONV1D --eval-strategy True```and```(maxsmi) $ python maxsmi/full_workflow.py --task ESOL --string-encoding smiles --aug-strategy-train augmentation_with_duplication --aug-strategy-test augmentation_with_duplication --aug-nb-train 1 --aug-nb-test 1 --ml-model CONV1D --eval-strategy True```📝 Have a look at the [README](https://github.com/volkamerlab/maxsmi/blob/main/README.md) page for more details.  Canonical VS Random SMILES: Performance Comparison  GoalThe aim of this notebook is to compare the results on the test set for the models that use the canonical or one random SMILES, and try to explain why the performance is systemically better for canonical SMILES.<jupyter_code>import os
from pathlib import Path

from rdkit import Chem
from rdkit.Chem import AllChem, Draw, PandasTools

from maxsmi.utils_analysis import load_results
from maxsmi.utils_smiles import get_num_heavy_atoms
# Path to this notebook
HERE = Path(_dh[-1])
path_to_output = HERE.parents[0]<jupyter_output><empty_output><jupyter_text>DatasetWe consider the following datasets:- ESOL- FreeSolv- lipophilicityComment/uncomment the dataset of choice in the cell below.<jupyter_code># TASK = "lipophilicity"
# TASK = "FreeSolv"
TASK = "ESOL"<jupyter_output><empty_output><jupyter_text>Load results Load results from random SMILES.<jupyter_code>test_data = load_results(path_to_output,
                         TASK,
                         "augmentation_with_duplication",
                         1,
                         "augmentation_with_duplication",
                         1,
                         "CONV1D",
                         "smiles",
                         True)
test_data = test_data.rename(columns={"average_\
prediction": "random_prediction"})
test_data = test_data.drop(columns=["std_prediction", "new_smiles"])
test_data.head()<jupyter_output><empty_output><jupyter_text>Load results from canonical SMILES.<jupyter_code>test_data_canonical = load_results(path_to_output,
                                   TASK,
                                   "no_augmentation",
                                   0,
                                   "no_augmentation",
                                   0,
                                   "CONV1D",
                                   "smiles",
                                   True)
test_data_canonical = test_data_canonical.rename(columns={"average_\
prediction": "canonical_prediction"})
test_data_canonical = test_data_canonical.drop(columns=["std_prediction",
                                                        "new_smiles"])
test_data_canonical.head()<jupyter_output><empty_output><jupyter_text>Merge results into one dataframe Add canonical result to random result dataframe, (`canonical_prediction` and `random_prediction`, respectively).<jupyter_code>test_data["canonical_prediction"] = test_data_canonical["canonical_prediction"]
test_data.head()<jupyter_output><empty_output><jupyter_text>Prediction errorLook at the errors between the target, or true value, and the prediction for the canonical vs random SMILES (`canonical_error` and `random_error`, respectively).<jupyter_code>test_data["random_error"] = (test_data["target"] -
                             test_data["random_prediction"]).abs()
test_data["canonical_error"] = (test_data["target"] -
                                test_data["canonical_prediction"]).abs()
test_data.head()<jupyter_output><empty_output><jupyter_text>Size of moleculesCompute the number of heavy atoms in each molecule.<jupyter_code>test_data["num_heavy_atoms"] = test_data["canonical_\
smiles"].apply(get_num_heavy_atoms)<jupyter_output><empty_output><jupyter_text>Canonical VS random SMILESConsider molecules that are better predicted if canonical SMILES are used.<jupyter_code>better_canonical = test_data[(test_data['random_error'] >
                              test_data['canonical_error'])]
better_canonical.head()
for ind, (index, row) in enumerate(better_canonical.iterrows()):
    if ind < 3:
        if row["num_heavy_atoms"] >= 5:
            print(f"Index: {index}\n"
                  f"Target: {row['target']:.2f}\n"
                  f"Canonical SMILES: {row['canonical_smiles']}\n"
                  f"Canonical prediction (& error): "
                  f"{row['canonical_prediction']:.2f}\t"
                  f"{row['canonical_error']:.2f}\n"
                  f"Random SMILES: {row['augmented_smiles'][0]}\n"
                  f"Random prediction (& error): "
                  f"{row['random_prediction']:.2f}\t"
                  f"{row['random_error']:.2f}"
                  f"\n\n")<jupyter_output>Index: 405
Target: -4.63
Canonical SMILES: CC(=O)OCC(=O)C1CCC2C3CCC4=CC(=O)CCC4(C)C3CCC12C
Canonical prediction (& error): -4.66	0.03
Random SMILES: C(OCC(=O)C1CCC2C3CCC4C(C)(C3CCC12C)CCC(=O)C=4)(C)=O
Random prediction (& error): -3.41	1.22


Index: 1039
Target: -4.17
Canonical SMILES: CC(=O)SC1CC2=CC(=O)CCC2(C)C2CCC3(C)C(CCC34CCC(=O)O4)C12
Canonical prediction (& error): -5.22	1.05
Random SMILES: C12C(CCC3(C)C4(OC(CC4)=O)CCC31)C1(C(CC2SC(=O)C)=CC(CC1)=O)C
Random prediction (& error): -5.92	1.74


Index: 485
Target: -0.22
Canonical SMILES: COP(=O)(OC)C(O)C(Cl)(Cl)Cl
Canonical prediction (& error): -1.18	0.96
Random SMILES: O(P(=O)(C(C(Cl)(Cl)Cl)O)OC)C
Random prediction (& error): -1.35	1.13<jupyter_text>ESOL selection for paper tableFor ESOL data set:<jupyter_code>if TASK == "ESOL":
    ESOL_data = better_canonical.loc[[859, 395, 609]][["canonical_smiles",
                                                       "augmented_smiles",
                                                       "target",
                                                       "canonical_prediction",
                                                       "canonical_error",
                                                       "random_prediction",
                                                       "random_error"]]
    # Add molecular graph to dataframe
    PandasTools.AddMoleculeColumnToFrame(ESOL_data, smilesCol="canonical_smiles")
    ESOL_data = ESOL_data.style.format("{:.2f}",
                                       subset=["target",
                                               "canonical_prediction",
                                               "canonical_error",
                                               "random_prediction",
                                               "random_error"])
    display(ESOL_data)
else:
    pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10736721873215306, 'fraction_numerical': 0.01756139348943461, 'mean_word_length': 2.4388807069219443, 'pattern_counts': {'":': 3, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '370370', 'license': 'MIT', 'n_tokens_mistral': 2194, 'n_tokens_neox': 2034, 'n_words': 430, 'path': 'maxsmi/results_analysis/results_canonical_vs_random.ipynb', 'repo_name': 't-kimber/maxsmi', 'size': '7004'}