<jupyter_start><jupyter_text>Creating and adding to the multitask databaseIndividual datasets and targets are cleaned and connected to a compounds codex containing unique molecules and already computed features. This minimizes the required time featurizing such that data in the database can be quickly called upon or even to speed up the use of data not in the dataset.<jupyter_code>import cytoxnet.dataprep.io as io
import pandas as pd<jupyter_output><empty_output><jupyter_text>Initialize the compounds database<jupyter_code>help(io.create_compound_codex)<jupyter_output>Help on function create_compound_codex in module cytoxnet.dataprep.io:

create_compound_codex(db_path='./database', id_col='smiles', featurizers=None, **kwargs)
    Create a compound codex for a combined database.
    
    Creates a master csv file that tracks the unique canonicalized smiles of
    all data in the database, and stores deatures for those data.
    
    Parameters
    ----------
    db_path : str
        The path to the folder to contain database files. Will create direcory
        if it does not exist.
    id_col : str
        The column in all dataframes representing the compound id.
    featurizers : str or list of str
        The featurizer/s to initialize the compounds codex with.<jupyter_text>- A folder to store the cleaned data files and the master compounds codex must be specified, default './database'- The column name to contain unique identifiers must be specified, default 'smiles'- Features initially specified will be computed for future data.<jupyter_code>io.create_compound_codex(featurizers=['CircularFingerprint'])
pd.read_csv('./database/compounds.csv', index_col=0)<jupyter_output><empty_output><jupyter_text>It is an empty file ready to have datasets added  Adding data to the database<jupyter_code>help(io.add_datasets)<jupyter_output>Help on function add_datasets in module cytoxnet.dataprep.io:

add_datasets(dataframes, names, id_col='smiles', db_path='./database', new_featurizers=None, **kwargs)
    Add a new set of data to the tracked database.
    
    Update the compounds csv with new dataset/s canonicalized, and saves
    csvs to the database folder with foreign keys tracked.
    
    Parameters
    ----------
    dataframes : dataframe or string or list of them
        The datasets to add. If it is a string object, we will attempt to load
        the file at the string path or a file in the package data.
    id_col : str
        The column in all dataframes representing the compound id
    db_path : str
        The path to the folder containing database files.
    new_featurizers : str or list of str, default None
        Featurizer names to apply to the new data as well as all current data.<jupyter_text>- Datasets can be added from dataframes in memory, csv files, or datasets already in the package- New features can be asked for- datasets added must be names<jupyter_code>## adding datasets from the package
io.add_datasets(['zhu_rat_LD50', 'lunghini_algea_EC50'], names = ['rat', 'algea'])<jupyter_output><empty_output><jupyter_text>We can see that the compounds we added to the codex and featurized<jupyter_code>codex = pd.read_csv('./database/compounds.csv', index_col=0)
codex<jupyter_output><empty_output><jupyter_text>Additionally, the cleaned datasets were added to the database under the specified names, and contain the foreign key that matches them to the compounds codex.<jupyter_code>algea = pd.read_csv('./database/algea.csv', index_col=0)
algea
## a specific example in the dataset
print('Example in dataset: ')
print('=====================')
print(algea.iloc[1200])
print('Foreign key: ')
print('=====================')
print(algea.iloc[1200]['foreign_key'])
print('That compound with features in the codex: ')
print('=====================')
print(codex.loc[algea.iloc[1200]['foreign_key']])<jupyter_output>Example in dataset: 
=====================
chemical_formula                        C9H9N5
smiles                 Nc1nc(N)nc(-c2ccccc2)n1
casnum                                 91-76-9
molecular_weight                       187.201
species                                  algea
algea_EC50                             4.23048
units                                log(mg/L)
source              ECHA, NITE, Literature set
foreign_key                               8085
Name: 1200, dtype: object
Foreign key: 
=====================
8085
That compound with features in the codex: 
=====================
smiles                 Nc1nc(N)nc(-c2ccccc2)n1
CircularFingerprint    [0. 0. 0. ... 0. 0. 0.]
Name: 8085, dtype: object<jupyter_text>We can also add datasets in memory an add new features<jupyter_code>my_data = pd.DataFrame({'smiles': ['O', 'C'], 'target': [1, 2]})
io.add_datasets([my_data], names = ['my_data'], new_featurizers=['RDKitDescriptors'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1028653885796743, 'fraction_numerical': 0.015048443619872192, 'mean_word_length': 3.701550387596899, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '521774', 'license': 'MIT', 'n_tokens_mistral': 1466, 'n_tokens_neox': 1370, 'n_words': 500, 'path': 'examples/initialize_database.ipynb', 'repo_name': 'Zymergen-DIRECT-2021/cytoxnet', 'size': '4851'}