<jupyter_start><jupyter_text>Multitask learning capability evaluationRandom forests performed well both as classifiers and as regressors, with the descripor based features (Mordred and RDKit) performing the best. Can a multitask model with imputation or a graph neural net do better?<jupyter_code>import deepchem as dc
import numpy as np
import pandas as pd
import optuna
from functools import reduce

import cytoxnet.dataprep.io as io
import cytoxnet.dataprep.dataprep as dataprep
import cytoxnet.dataprep.featurize as feat
from cytoxnet.models.models import ToxModel
import cytoxnet.models.opt as opt<jupyter_output><empty_output><jupyter_text>ContentsThis notebook contains en evaluation of the optimized models for classification tasks. The overall goal was to leverage all of our datasets (see data report section) in order to produce the best possible classifier of toxicity in a single test microbe, in this case algea. Continuous toxicity metrics are binirized by labeling the most toxic 90% of each dataset as toxic. For this reason precision score is a more valuable metric than recal score.Models tested:- RFC w/ RDKitDescriptors baseline: train on algea data alone- RFC w/ RDKitDescriptor imputed with mean: train on all species (originally sparse)- RFC w/ RDKitDescriptor imputed with iterative imputer: train on all species (originally sparse)- GCNN w/ 0.0 wighted sparse data: train on all species (sparse)  Create the datasets to useMultitask learning would have the most benefit for small target datasets, so we will use the smallest in the package (Lunghini algea data) as the ultimate goal<jupyter_code>## !!!!!!temporary until database query works
fish = io.load_data('../database/fish.csv', cols=['smiles', 'fish_LC50'])
daphnia = io.load_data('../database/daphnia.csv', cols=['smiles', 'daphnia_EC50'])
algea = io.load_data('../database/algea.csv', cols=['smiles', 'algea_EC50'])
rat  = io.load_data('../database/rat.csv', cols=['smiles', 'rat_LD50'])
ecoli  = io.load_data('../database/ecoli.csv', cols=['smiles', 'ecoli_MIC'])

raw = reduce(
    lambda x, y: pd.merge(x, y, how='outer', on = 'smiles'),
    [fish, daphnia, algea, rat, ecoli]
)
multitask_names = [
    'fish_LC50',
    'daphnia_EC50',
    'algea_EC50',
    'rat_LD50',
    'ecoli_MIC'
]
raw.describe()<jupyter_output><empty_output><jupyter_text>add features - RDKit for RFCs, graphs for graph cnn<jupyter_code>data_f = feat.add_features(raw, method='RDKitDescriptors', codex='../database/compounds.csv')
data_f = feat.add_features(data_f, method='ConvMolFeaturizer')<jupyter_output><empty_output><jupyter_text>identify an independant algea test set by index<jupyter_code>algea_only = data_f[~data_f.isna()['algea_EC50']]
algea_index = algea_only.index
test_index = algea_only.sample(frac=.2, random_state=0).index
baseline_index = algea_only.drop(index=test_index).index<jupyter_output><empty_output><jupyter_text>Baseline modelAs a baseline we are using random forest classifier, which we know is capable for single tasks.<jupyter_code>baseline_frame = data_f.loc[algea_index]
baseline_frame = dataprep.binarize_targets(baseline_frame, target_cols=['algea_EC50'], percentile=.9)
# create the dataset
baseline = dataprep.convert_to_dataset(
    baseline_frame,
    X_col='RDKitDescriptors',
    y_col=[
        'algea_EC50'
    ]
)
# split out dev and test
baseline_test = baseline.select(np.isin(baseline.ids, test_index))
baseline_dev = baseline.select(np.isin(baseline.ids, baseline_index))<jupyter_output><empty_output><jupyter_text>retrieve hyperparmeter optimization<jupyter_code>baseline_study = optuna.load_study(
    study_name='opt',
    storage="sqlite:///classification/baseline_c.db"
)
baseline_results = baseline_study.trials_dataframe()
baseline_results
baseline_params = baseline_study.best_params
baseline_params
baseline_study.best_value<jupyter_output><empty_output><jupyter_text>train the baseline model with best identified hyperparameters<jupyter_code>baseline_model = ToxModel('RFC', **baseline_params)
baseline_model.fit(baseline_dev)
baseline_model.evaluate(baseline_test, ['precision_score', 'recall_score'], untransform=False, n_classes=2)<jupyter_output><empty_output><jupyter_text>> The final precision score is 0.906<jupyter_code>baseline_model.visualize('roc', baseline_test)<jupyter_output><empty_output><jupyter_text>Evaluate the multitask modelRandom forests require imputation for sparse datasets, try a few methods: mean imputation, interpolation, and RFR interpolation. Additionally attempt graphs with weights.<jupyter_code>import sklearn.impute<jupyter_output><empty_output><jupyter_text>Impute by mean<jupyter_code>mean = data_f.copy()<jupyter_output><empty_output><jupyter_text>First do the imputation and prepare the dataset<jupyter_code>mean[multitask_names] = sklearn.impute.SimpleImputer().fit_transform(
    mean[multitask_names].values
)
mean = dataprep.binarize_targets(mean, target_cols=multitask_names, percentile=.9)
mean_set = dataprep.convert_to_dataset(
    mean,
    X_col='RDKitDescriptors',
    y_col=multitask_names
)
mean_test = mean_set.select(np.isin(mean_set.ids, test_index))
mean_dev = mean_set.select(~np.isin(mean_set.ids, test_index))<jupyter_output><empty_output><jupyter_text>retrieve the best paramaters for this model found via hyperparameter optimization<jupyter_code>mean_study = optuna.load_study(
    study_name='opt',
    storage="sqlite:///classification/mean_c.db"
)
mean_results = mean_study.trials_dataframe()
mean_results
mean_params = mean_study.best_params
mean_params
mean_study.best_value<jupyter_output><empty_output><jupyter_text>train the multitask model using the best found parameters<jupyter_code>mean_model = ToxModel('RFC', transformers=[], tasks=multitask_names, **mean_params)
mean_model.fit(mean_dev)
mean_model.evaluate(
    mean_test, ['precision_score', 'recall_score'],
    untransform=False,
    per_task_metrics=True,
    n_classes=2)<jupyter_output><empty_output><jupyter_text>> The precision score for the independant algea test set is. 0.4545<jupyter_code>mean_model.visualize('roc', mean_test, task='algea_EC50')<jupyter_output><empty_output><jupyter_text>Impute by interpolation<jupyter_code>from sklearn.experimental import enable_iterative_imputer<jupyter_output><empty_output><jupyter_text>First impute and prepare the data. This requires sklearns  experimental imputer.<jupyter_code>iterpute = data_f.copy()
iterpute[multitask_names] = sklearn.impute.IterativeImputer(random_state=0).fit_transform(
    iterpute[multitask_names].values
)
iterpute = dataprep.binarize_targets(iterpute, target_cols=multitask_names, percentile=.9)
iterpute_set = dataprep.convert_to_dataset(
    iterpute,
    X_col='RDKitDescriptors',
    y_col=multitask_names
)
iterpute_test = iterpute_set.select(np.isin(iterpute_set.ids, test_index))
iterpute_dev = iterpute_set.select(~np.isin(iterpute_set.ids, test_index))<jupyter_output><empty_output><jupyter_text>Retrieve hyperparameter optimization for this model<jupyter_code>iterpute_study = optuna.load_study(
    study_name='opt',
    storage="sqlite:///classification/inter_c.db"
)
iterpute_results = iterpute_study.trials_dataframe()
iterpute_results
iterpute_params = iterpute_study.best_params
iterpute_params
iterpute_study.best_value<jupyter_output><empty_output><jupyter_text>train the model using the hyperparameters found best for this model on the development set.<jupyter_code>iterpute_model = ToxModel('RFC', transformers=[], tasks=multitask_names, **iterpute_params)
iterpute_model.fit(iterpute_dev)
iterpute_model.evaluate(iterpute_test, ['precision_score', 'recall_score'], untransform=False, per_task_metrics=True, n_classes=2)<jupyter_output><empty_output><jupyter_text>> The final precision score for multitask RFC with interpolated imputation is 0.4771<jupyter_code>iterpute_model.visualize('roc', iterpute_test, task='algea_EC50')<jupyter_output><empty_output><jupyter_text>Try a Graph multitask In this case, instead of imputation, the neural network architectures can accept a weight matrix in the same shape as the targets, thus sparse data can be masked out.<jupyter_code>graph = data_f.copy()<jupyter_output><empty_output><jupyter_text>Prepare the data. During binarizing, we weight all targets originally without data as 0.0.<jupyter_code>graph = dataprep.binarize_targets(graph, target_cols=multitask_names, percentile = .9)
graph_set = dataprep.convert_to_dataset(
    graph,
    X_col='ConvMolFeaturizer',
    y_col=multitask_names,
    w_label='w'
)
graph_test = graph_set.select(np.isin(graph_set.ids, test_index))
graph_dev = graph_set.select(~np.isin(graph_set.ids, test_index))<jupyter_output><empty_output><jupyter_text>Retrieve best hyperparameters for the graph multitask model.<jupyter_code>graph_study = optuna.load_study(
    study_name='opt',
    storage="sqlite:///classification/graph_c.db"
)
graph_results = graph_study.trials_dataframe()
graph_results
graph_params = graph_study.best_params
graph_params
graph_study.best_value<jupyter_output>/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/optuna/distributions.py:430: UserWarning: Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains [32] which is of type list.
  warnings.warn(message)
/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/optuna/distributions.py:430: UserWarning: Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains [64] which is of type list.
  warnings.warn(message)
/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/optuna/distributions.py:430: UserWarning: Choices for a categorical distribution should be a tuple of None, bool, int, float and str for persistent storage but contains [128] which is of type list.
  warnings.warn(message)
/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/optuna/distributions.py:430: UserWarning: Choices for a categorical distribution [...]<jupyter_text>Train the graph model using the best parameters on the entire dev set.<jupyter_code>graph_model = ToxModel(
    'GraphCNN',
    tasks=multitask_names,
    transformers=[],
    mode='classification',
    **graph_params
)
graph_model.fit(graph_dev, nb_epoch=50)
graph_model.evaluate(
    graph_test,
    ['precision_score', 'recall_score'],
    untransform=False,
    use_sample_weights=True,
    per_task_metrics=True,
    n_classes=2
)<jupyter_output>/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/Users/ek/miniconda3/envs/cytoxnet/lib/python3.6/site-packages/sklearn/metrics/_classification.py:1245: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))<jupyter_text>> The precision score for the test set is 0.919. This is approximately 1% better than the baseline single task model.<jupyter_code>graph_model.visualize('roc', graph_test, task='algea_EC50')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09787496693413279, 'fraction_numerical': 0.010140199276959704, 'mean_word_length': 7.038270729978739, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 99, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '575391', 'license': 'MIT', 'n_tokens_mistral': 3908, 'n_tokens_neox': 3707, 'n_words': 847, 'path': 'docs/multitask_learning/multitask_evalutation_classification.ipynb', 'repo_name': 'Zymergen-DIRECT-2021/cytoxnet', 'size': '11341'}