<jupyter_start><jupyter_text>Implementation and evaluation of a computational standardization pipeline for chemical compounds--------------------------------------------------------------> Based on ["Trust, But Verify: On the Importance of Chemical Structure Curation in Cheminformatics and QSAR Modeling Research" from 2010 (D. Fourches, ...)"](https://pubmed.ncbi.nlm.nih.gov/20572635/)By <NAME>; reviewed by <NAME>, PhD.  Introduction This notebook serves to display the functionality of the `opencadd.compounds.standardization` subpackage. We are following the recommended standardization steps of ["Trust, But Verify" (Fourches et al., 2010)](https://pubmed.ncbi.nlm.nih.gov/20572635/), and using a modified¹ version of the dataset from the following paper: [Cheminformatics Analysis of Assertions Mined from Literature That Describe Drug-Induced Liver Injury in Different Species](https://pubs.acs.org/doi/10.1021/tx900326k).¹ We added some entries to trigger curation steps not covered by the original data.  Overview over the pipeline------------------------------------------This pipeline has **five** main steps:1. Structural Conversion2. Filtering of Inorganics and Mixtures3. Structural Cleaning 4. Normalization of Specific Chemotypes5. Removal of DuplicatesEach step consists of action performing tasks on the dataset. Actions are:- filtering- cleaning- normalizing**Filtering** actions will result in a score applied to the entries. The score is the number of the filtering task. You can use it to select subsets of the dataset sorting by the column `filtered_at`.**Cleaning** actions will result in a modification of the mol-representation of the entry, overwriting with the recent version calculated in the task. You can use it to select subsets of the dataset sorting by the column `cleaned_at`.**Normalizing** actions also will result in a modification of the mol-representation of the entry.You can use it to select subsets of the dataset sorting by the column `normalized_at`.At the end of the script, there is the possibility to export subsets of the dataset as a CSV.<jupyter_code>from pathlib import Path

HERE = Path(_dh[-1])
REPO = HERE.parents[1]

print("Tutorial location:", HERE)
print("Repo location:    ", REPO)
# Import pandas and numpy
import pandas as pd
import numpy as np

# Importing functions from the standardization API
from opencadd.compounds.standardization import (
    convert_format,
    detect_mixtures,
    detect_metals,
    detect_salts,
    detect_inorganics,
    handle_fragments,
    handle_tautomers,
    handle_charges,
    handle_tautomers,
    disconnect_metals,
    remove_salts,
    normalize_molecules,
    validate_molecules,
)
# Utility function to compare SMILES
# JRG: Why is this function needed? You can use
#     `operator.eq` builtin!
def smiles_string_changed(smiles_old, smiles_new):
    """
    Compares SMILES strings. If they are identical, the value returned is False,
    if they differ the value returned is True.

    Parameters
    ----------
    smiles_old: str
        SMILES string
    smiles_new: str
        SMILES string

    Returns
    -------
    bool
        True if changes were detectrd, false otherwise
    """
    return smiles_old != smiles_new<jupyter_output><empty_output><jupyter_text>Initial dataset import and cleaning of empty entries------------------------------------------------Before any curation steps are can be applied, we need to import the dataset as a Pandas Dataframe. At this point you have the possibility to select the columns you need for the curation process. For our example dataset we will use columns IDs, Names and SMILEs.After that, we search for all entries which have empty strings saved under SMILES and remove them from the dataset.After the import, we add a Filtered_at column to track which standardization step filtered the entry. The initial `task_number` will be 0, which leads to a default Filtered_at-value of 0 for all entries, where null stands for all the entries that passed without any filtering.<jupyter_code>task_number = 0

# Import test-dataset
dataset = pd.read_csv(HERE / "data" / "standardization_test_data.csv", delimiter=";")

# Filter columns
dataset = dataset[["IDs", "Names", "SMILEs"]]
# Rename a column, due to an typo in the original dataset
dataset = dataset.rename(columns={"SMILEs": "SMILES"})

# Delete empty entries from the main set.
# JRG: The usual thing here is to use df.dropna() function, possibly with a subset=XXX option
dataset = dataset[(dataset["SMILES"].notna())]

# Initializing the score to null at the 'Filtered_at'-column
# JRG: You can use a constant here, I believe: dataset[X] = task_number
dataset["Filtered_at"] = dataset["SMILES"].apply(
    lambda x, task_number=task_number: task_number
)

# Initializing the score to null at the 'Cleaned_at'-column
# JRG: Same as above
dataset["Cleaned_at"] = dataset["SMILES"].apply(
    lambda x, task_number=task_number: task_number
)

# Initializing the score to null at the 'Normalized_at'-column
# JRG: Same as above
dataset["Normalized_at"] = dataset["SMILES"].apply(
    lambda x, task_number=task_number: task_number
)


# Reset the index to correct the deletion of the empty entries
dataset = dataset.reset_index(drop=True)

# [Optional] Display empty entries for manual inspection.
# dataset[(dataset["SMILES"].isnull())]
dataset.head()<jupyter_output><empty_output><jupyter_text>Step 1: Encoding Converison------------------------------------------__Convert the SMILES representation format of the compounds into Mol-files__RDKit performs a sanitization of molecules converted to mol by default. In addition to some Nitro and Perchlorate transformations the following steps are taken²:- Calculate explicit and implicit valence of all atoms. Fails when atoms have illegal valence.- Calculate symmetrized SSSR. The slowest step fails in rare cases.- Kekulize. Fails if a Kekule form cannot be found or non-ring bonds are marked as aromatic.- Assign radicals if hydrogens set and bonds+hydrogens+charge < valence.- Set aromaticity, if none set in input. Go round rings, Huckel rule to set atoms+bonds as aromatic.- Set a conjugated property on bonds where applicable.- Set hybridization property on atoms.- Remove chirality markers from sp and sp2 hybridized centers.If the conversion from SMILES to mol fails, then those SMILES will get a **Filtered_at** marker added. > JRG: SMILES contains the end S already. It is not a plural form!To avoid molecule sanitization `convert_smiles_to_mol` can be called with the argument `sanitize=False`. Keep in mind that the generation of different Lewis structures serves to find alternative representation formats of the same molecule. __Overwrite the SMILES representation with ones compiled from our generated Mol-files__ In order to register the changes we make to the entries, we have to calculate canonical SMILES with our function `convert_format`, which by default returns a canonical representation. The conversion back to SMILES has to happen since SMILES encodings vary depending on the algorithm used to calculate them. The newly calculated SMILES will be used as a validation parameter to determine any changes made to our entries further down the curation pipeline. References:² https://molvs.readthedocs.io/en/latest/guide/standardize.html?highlight=sanitizerdkit-sanitize* https://chemistry.stackexchange.com/questions/116498/what-is-kekulization-in-rdkit* https://rdkit-discuss.narkive.com/QwnqcKcM/another-can-t-kekulize-mol-observation* https://www.rdkit.org/docs/Cookbook.html* https://www.rdkit.org/docs/source/rdkit.Chem.rdmolfiles.html  Task 1: Convert to RDKit Molecule Objects<jupyter_code># Setting up the task_number
task_number = 1

# A column called mol is being added to the dataframe to store the mol-files
# JRG: I think you can use dataset["SMILES"].apply here directly, without loc (huge overhead)
dataset["mol"] = dataset.loc[:, ("SMILES")].apply(convert_format.convert_smiles_to_mol)

# Add task_number to failed entries
dataset.loc[dataset["mol"].isnull(), ["Filtered_at"]] = task_number

# Overwrite SMILES with canonical one which are created by default by the conversion algorithm used in convert_format

dataset["SMILES"] = dataset.apply(
    lambda row: convert_format.convert_mol_to_smiles(row.mol)
    if row.Filtered_at == 0
    else row.SMILES,
    axis=1,
)

dataset.tail(16)<jupyter_output><empty_output><jupyter_text>Step 2: Filtering of Inorganics and Mixtures--------------------------------------------------Since most cheminformatics applications are not capable of processing inorganic structures,  there is a need to remove those entries before any processing.Detecting inorganic structures is divided into two steps:First removing all entries not containing any Carbon at all, which are therefore not organic.Secondly, filtering out all compounds with inorganic substructures. Similar problems occur for mixtures. Since most applications can not calculate descriptors for mixtures, filtering has to happen before processing. Additionally, since "*inorganic compounds are known to have biological effects, like toxic effects*" (Fourches 2010), we can often not distinguish if its organic or inorganic part causes the recorded activity of a mixed compound. Therefore the entry is useless and can be discarded. Since the treatment is not as simple as it appears, the paper recommends deleting records containing mixtures.Common and widely used practice is to retain molecules with the highest molecular weight or the largest number of atoms. Still, the paper states this might not be the best solution, and investigation in mixtures should only happen if there is a reason to believe the largest molecule and not the mixture itself is causing the biological activity.  Task 2: Filter entries without CarbonThe first task to determine if an entry is an organic molecule is to check for the presence of Carbon. `detect_carbon` is a function able to do this. It searches for the existence of carbon atoms. If the function finds at least one Carbon atom, it returns a boolean value of **TRUE**, if not **FALSE**. All entries that return **FALSE** will get the current task number (2) assigned into the *Filtered_at* column.<jupyter_code># Setting up the task_number
task_number = 2

# Check for Carbon
dataset["Carbon_present"] = dataset.apply(
    lambda row: detect_inorganics.detect_carbon(row.mol)
    if row.Filtered_at == 0
    else None,
    axis=1,
)

# Add task_number to failed entries
dataset.loc[dataset["Carbon_present"] == False, ["Filtered_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries that do not contain any Carbon and thereby are inorganic molecules.<jupyter_code>dataset[dataset["Filtered_at"] == 2]<jupyter_output><empty_output><jupyter_text>Task 3: Filter entries with inorganic componentsWhile we filtered out all molecules not containing any Carbon, now we further inspect the entries for elements that do not occur in organic molecules. The elements that might fall into here might vary slightly depending on the definition and scope. `detect_inorganic` is a suitable function for this task.A recommendation is to check which elements can be managed by the software later used. Customization of the allowed elements in `detect_inorganic` can easily be provided by a set of SMARTS, as described shortly below. The default set of accepted elements in an organic molecule are Hydrogen, Carbon, Nitrogen, Oxygen, Fluorine, Phosphorus, Sulfur, Chlorine, Selenium, Bromine, Iodine (nonmetals and halogens). *While Astatine and Tennessine are also considered halogens, they are not included due to their radioactivity and rarity.* An example of how to set up a custom set of elements and implement them in `detect_inorganic` -----------------------------------------------------------------------------------------------------------Defining a set:`elements = Chem.MolFromSmarts("[!1&!6&!7&!8&!9&!15&!16&!17&!35&!53]")` (If you want to run this, import the following before: from rdkit import Chem)Pass the set as a parameter, where the `detect_inorganic` function is getting called:`lambda row: detect_inorganics.detect_inorganic(row.mol, elements)`<jupyter_code># Setting up the task_number
task_number = 3


# Check for inorganic structures
dataset["Inorganics"] = dataset.apply(
    lambda row: detect_inorganics.detect_inorganic(row.mol)
    if row.Filtered_at == 0
    else None,
    axis=1,
)

# Add task_number to failed entries
dataset.loc[dataset["Inorganics"] == True, ["Filtered_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries that contain other than our allowed elements. (Hydrogen, Carbon, Nitrogen, Oxygen, Fluorine, Phosphorus, Sulfur, Chlorine, Selenium, Bromine, Iodine)<jupyter_code>dataset[dataset["Filtered_at"] == 3]<jupyter_output><empty_output><jupyter_text>Task 4: Filter entries containing mixturesWe will use the function `detect_mixtures` as shown below for the filtering of mixtures.<jupyter_code># Setting up the task_number
task_number = 4

# Check for inorganic structures
dataset["mixture"] = dataset.apply(
    lambda row: detect_mixtures(row.mol) if row.Filtered_at == 0 else None,
    axis=1,
)

# Add task_number to failed entries
dataset.loc[dataset["mixture"] == True, ["Filtered_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries that are mixtures.<jupyter_code>dataset[dataset["Filtered_at"] == 4]<jupyter_output><empty_output><jupyter_text>Step 3: Structural Cleaning --------------------------------------------------"Some drugs need to be transformed "into their salt form to enhance how the drug dissolves (...) and (to) increase its effectiveness."³ Therefore it is common for chemical compound databases to contain records of salts. If possible, a suggestion is to delete the records containing salts completely, since, similar to in-organic compounds, "most descriptor-generating software (can not process salts)" (Fourches 2010). While not desirable, it is still an acceptable procedure to convert compounds into their neutral forms. But cases like this should be tagged, filtered, and afterward manually curated or compared to the concrete neutral form of that compound. In case that we want to continue working on the converted records, we should perform the following steps:- check if records contain compounds with the presence of metals- removing the salts from the record- neutralize the record (normalization or essential standardization)- neutralize the charges__Note:__While it is possible to clean and reuse entries containing metals or salts, those entries will not be curated here since it does not fit the scope of this tutorial. Nevertheless, we hint at the steps to do and which functions of the standardization API to use. ³ (https://www.drugs.com/article/pharmaceutical-salts.html (03/12/21))  Task 5: Filter entries containing metalsEntries can contain metals in different forms. Either as a regular compound in a mixture or as a counterion. In the following steps, we search for those metals. When they are a counterion, we disconnect them from the non-metals they are bonding. We might not find any metals due to previous filtering steps detecting mixtures and inorganics. Therefore we could search in the flagged entries and clean those entries later on.<jupyter_code># Setting up the task_number
task_number = 5

# Check for metals
dataset["metals"] = dataset.apply(
    lambda row: detect_metals(row.mol) if row.Filtered_at == 0 else None,
    axis=1,
)

# Add task_number to failed entries
dataset.loc[dataset["metals"] == True, ["Filtered_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries containing metals. We didn't find any entries in our filtered set, as already assumed.<jupyter_code>dataset[dataset["Filtered_at"] == 5]<jupyter_output><empty_output><jupyter_text>So the next step would be to examine our filtered entries.For that, we make a copy of our current status of the dataset.<jupyter_code>score = [3, 4]
failed_entries_copy = dataset[dataset["Filtered_at"].isin(score)].copy()<jupyter_output><empty_output><jupyter_text>And we check for the presence of metals here<jupyter_code># Check for metals
failed_entries_copy["metals"] = failed_entries_copy.apply(
    lambda row: detect_metals(row.mol) if row.Filtered_at != 0 else None,
    axis=1,
)
failed_entries_copy[failed_entries_copy["metals"] == True]<jupyter_output><empty_output><jupyter_text>One entry has a counterion that can be disconnected. We might consider removing the metals in those mixtures and re-run this standardization script with the cleaned entry.  But for this case, this does not make much sense since the resulting molecules after removing Zirconium would be not functional, and the covalent metal would be deleted entirely since all of its substructures are salts.However, what we could have done if it made sense:1. disconnect_metals2. handle_charges.uncharge3. normalize_molecule.normalize4. remove_salts5. handle_charges.uncharge6. normalize_molecule.normalize7. handle_fragments.choose_largest_fragment8. Apply a `cleaned_at` marker  Task 6: Removing salts This curation step can be applied to different subsets of the dataset.First, we will apply this to our entries that passed all steps before. Since we filtered all mixtures out in previous steps, all salts found in this step are the only compound in the entry. Therefore they need to be deleted (filtered).More interesting might be the inspection of the *inorganics* **(Task 3)** or *mixtures* **(Task 4)**. We could check if any of those mixtures contain salts known in our dictionary. If so, we can delete those salts and reuse the entries if they are free of mixtures. First, we will search for salts in our dataset.<jupyter_code># Setting up the task_number
task_number = 6

# Check for salts
dataset["salts"] = dataset.apply(
    lambda row: detect_salts(row.mol) if row.Filtered_at == 0 else None,
    axis=1,
)

# Add task_number to failed entries
dataset.loc[dataset["salts"] == True, ["Filtered_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries containing salts.<jupyter_code>dataset[dataset["Filtered_at"] == 6]<jupyter_output><empty_output><jupyter_text>To demonstrate the removal of salts, we can generate SMILES out of the mol after the deletion of the salts.We will observe the generation of an empty SMILES string.<jupyter_code># First we make a deep copy of the original dataframe
demo_df = dataset.copy()
demo_df.head()<jupyter_output><empty_output><jupyter_text>We then apply our changes to the copy to look at the SMILES generated after removal.<jupyter_code># Applying the remove_salts function on the molecules detected as salts in Tasks 6.
demo_df["mol"] = demo_df.apply(
    lambda row: remove_salts(row.mol) if row.Filtered_at == 6 else row.mol,
    axis=1,
)

# Generate a SMILES of the entries. (Only for demonstration purposes)
demo_df["SMILES"] = demo_df.apply(
    lambda row: convert_format.convert_mol_to_smiles(row.mol)
    if row.Filtered_at == 6
    else None,
    axis=1,
)<jupyter_output><empty_output><jupyter_text>Then we can look at our entries containing salts<jupyter_code>demo_df[demo_df["Filtered_at"] == 6]<jupyter_output><empty_output><jupyter_text>Next, we can filter for salts in the entries screened for inorganics and mixtures (as we did for the metals) to see if any inorganics or mixtures might have been salts.<jupyter_code>score = [3, 4]
failed_entries_copy = dataset[dataset["Filtered_at"].isin(score)].copy()
# Check for salts
failed_entries_copy["salts"] = failed_entries_copy.apply(
    lambda row: detect_salts(row.mol) if row.Filtered_at != 0 else None,
    axis=1,
)
failed_entries_copy<jupyter_output><empty_output><jupyter_text>One of the filtered entries in this example contained a salt. We might consider removing the salt in this mixture and do some normalization actions on it, similar to those in the previous step covering the disconnection of metals. The steps would be:1. remove_salts2. handle_charges.uncharge3. normalize_molecule.normalize4. handle_fragments.choose_largest_fragment  Step 4: Normalization of Specific ChemotypesAfter we filtered all problematic entries in the previous steps and created subsets to curate entries containing metals and salts, the next task is to apply normalization transformations to the remaining entries to correct functional groups and recombine charges. The standardization API utilizes the Normalization transformations embedded in the rdMolStandardize-Package, which derives the rules described in the InChI technical manual. *If available, custom conversions rules can be used and implemented but require modifying the `normalize_molecules.normalize` function to use them. (This might be covered in further development of this API.*   Task 7: Normalization<jupyter_code># Setting up the task_number
task_number = 7

# Normalize the entries, overwrite the previous mol
dataset["mol"] = dataset.apply(
    lambda row: normalize_molecules.normalize(row.mol)
    if row.Filtered_at == 0
    else row.mol,
    axis=1,
)

# Calculate new SMILES for the entries to determine which entries needed to be normalized
dataset["SMILES_after_normalization"] = dataset.apply(
    lambda row: convert_format.convert_mol_to_smiles(row.mol)
    if row.Filtered_at == 0
    else row.SMILES,
    axis=1,
)


# Compare the SMILES for changes after the normalization --> save as Boolean Value
dataset["normalized"] = dataset.apply(
    lambda row: smiles_string_changed(row.SMILES, row.SMILES_after_normalization)
    if row.Filtered_at == 0
    else None,
    axis=1,
)


# Add task_number to normalized entries
dataset.loc[dataset["normalized"] == True, ["Normalized_at"]] = task_number<jupyter_output><empty_output><jupyter_text>Below you can see all entries where normalization steps took place.<jupyter_code>dataset[dataset["Normalized_at"] == 7]<jupyter_output><empty_output><jupyter_text>FInal Conversion back to SMILESSince we have finished all our actions taken on the mol-files, we can render new SMILES strings generated by our final mol-files<jupyter_code>dataset["SMILES"] = dataset.apply(
    lambda row: convert_format.convert_mol_to_smiles(row.mol)
    if row.Filtered_at == 0
    else row.SMILES,
    axis=1,
)<jupyter_output><empty_output><jupyter_text>(Task 8): TautomersConsidering compounds can exist in various tautomeric forms, it can be advantageous to calculate those forms. Due to tautomerism being a broad and specific field, this notebook will not allow a deeper focus on interpreting the problems related to tautomers. We will only provide the tools to generate a canonicalized tautomer and enumerate all possible tautomers of the final SMILES.<jupyter_code># Generate a canonicalized tautomer
dataset["canonicalized_tautomer_smiles"] = dataset.apply(
    lambda row: handle_tautomers.canonicalize_tautomer(row.SMILES)
    if row.Filtered_at != 1
    else None,
    axis=1,
)

# Compare the SMILES for changes after the generation of a canonicalized SMILES --> save as Boolean Value
dataset["new_canonical_tautomer"] = dataset.apply(
    lambda row: smiles_string_changed(row.SMILES, row.canonicalized_tautomer_smiles)
    if row.Filtered_at == 0
    else None,
    axis=1,
)<jupyter_output>WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardize.tautomer:Tautomer enumeration stopped at maximum 1000
WARNING:rdkit.Chem.MolStandardiz[...]<jupyter_text>Below you can see all entries where the canonicalized tautomer differs to the SMILES, that resulted from the curation process.<jupyter_code>dataset[dataset["new_canonical_tautomer"] == True]<jupyter_output><empty_output><jupyter_text>Additionally to the calculation of a canonicalized tautomer, we can also use the function `enumerate_tautomer`, which returns a list of all possible tautomers. While this might be practical for a detailed check of the tautomers of an entry, storing the results in a data frame like here would be impractical. Therefore it is recommended to generate those lists only for specifically selected entries.An Example of how this can be done for the entry with the ID 12 is shown below:<jupyter_code># Extract the SMILES for the entry with the ID you want to enumerate the tautomers for
smiles_to_enumerate_tautomer = "".join(
    dataset.loc[dataset["IDs"] == 12, ["SMILES"]].values[0]
)

# Apply the enumerate_tautomer function on that SMILES
handle_tautomers.enumerate_tautomer(smiles_to_enumerate_tautomer)<jupyter_output><empty_output><jupyter_text>Step 5: Removal of duplicatesSince RDKit can calculate the canonical version of SMILES, we can try to find all duplicate entries in our data frame through a SMILES string comparison.> JRG: Are you actually comparing canonical smiles here? I think they are just the raw values present in the dataset, aren't they? You might need to do a round trip MolFromSmiles->MolToSmiles to get the canonical version. Molecule comparison is tricky! Best way is to resort to graph homology, but we are not doing that now. Just ensure you are indeed using canonical smiles.<jupyter_code>i = 0

all_SMILES = dataset.loc[:, "SMILES"]
IDs = dataset.loc[:, "IDs"]
list_len = len(all_SMILES)
list_len

while i < list_len:
    SMILES = all_SMILES[i]
    ID = IDs[i]
    j = i + 1
    while j < list_len:
        compared_SMILES = all_SMILES[j]
        compared_ID = IDs[j]
        j += 1
        if SMILES == compared_SMILES and ID != compared_ID:
            print("Following IDs have identical SMILES", ID, "and", compared_ID)
            continue  # JRG: What's this continue for? It's the last statement... do you mean `break`?
    i += 1<jupyter_output>Following IDs have identical SMILES 2 and 552
Following IDs have identical SMILES 23 and 799
Following IDs have identical SMILES 23 and 950
Following IDs have identical SMILES 30 and 507
Following IDs have identical SMILES 43 and 764
Following IDs have identical SMILES 47 and 883
Following IDs have identical SMILES 55 and 407
Following IDs have identical SMILES 58 and 864
Following IDs have identical SMILES 65 and 218
Following IDs have identical SMILES 168 and 169
Following IDs have identical SMILES 200 and 357
Following IDs have identical SMILES 211 and 212
Following IDs have identical SMILES 253 and 513
Following IDs have identical SMILES 322 and 326
Following IDs have identical SMILES 327 and 646
Following IDs have identical SMILES 606 and 614
Following IDs have identical SMILES 736 and 737
Following IDs have identical SMILES 751 and 935
Following IDs have identical SMILES 795 and 796
Following IDs have identical SMILES 799 and 950<jupyter_text>Export of the DatasetVarious exporting possibilities are now open.You can: - export the whole dataset, with its scores in an dedicated row- filter for only the entries that passed all filtering step (ergo have a Score of null in the *Filtered_at*-column)- filter for the entries that where filtered at a step to perform some transformations on them (e.g. remove salts)- export all the mol files you need into a SDF with the function `convert_mol_to_sdf` by firstly generating an array of all mol-entries (`mol_array`) and pass them to the function as a parameter, togheter with a filename `fn` (e.g. `convert_mol_to_sdf(mol_array, fn="my_sdf_export")`)<jupyter_code>dataset.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08044961295111519, 'fraction_numerical': 0.011523099218832844, 'mean_word_length': 4.852709971038477, 'pattern_counts': {'":': 1, '<': 112, '<?xml version=': 0, '>': 115, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '82968', 'license': 'MIT', 'n_tokens_mistral': 8221, 'n_tokens_neox': 7538, 'n_words': 3317, 'path': 'docs/tutorials/standardization.ipynb', 'repo_name': 'Allend95/opencadd', 'size': '28313'}