<jupyter_start><jupyter_text>Loading packages<jupyter_code>!pip install fastai --upgrade
!pip install dtreeviz
!pip install fastbook
import fastbook
fastbook.setup_book()
from fastbook import *

from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from dtreeviz.trees import *
from IPython.display import Image, display_svg, SVG

pd.options.display.max_rows = 20
pd.options.display.max_columns = 8<jupyter_output><empty_output><jupyter_text>Loading Data<jupyter_code>path = "/content/gdrive/MyDrive/archivos_tfm/ch1_train_combination_and_monoTherapy.csv"
df = pd.read_csv(path, low_memory=False)
df.head(5)
df.columns

# A little bit of data analysis
df.describe()
dep_var = 'SYNERGY_SCORE'

procs = [Categorify, FillMissing]

# We shuffle the data
df = df.sample(frac=1).reset_index(drop=True)
df[1:5]
# We will erase Combination ID as it offers no additional information
# We only want perfect samples, so only QA = 1

df_nocomb = df.drop(["COMBINATION_ID"], 1)
df_def = df_nocomb[df_nocomb['QA'] == 1]
df_def.describe()
# We create the train/validation splits
dataset_size = df_def.shape[0]
cutoff = int(dataset_size * 0.7)
train_idx = df_def.index[:cutoff] 
valid_idx = df_def.index[cutoff:] 

splits = (list(train_idx),list(valid_idx))
cont,cat = cont_cat_split(df_def, 1, dep_var=dep_var)
to = TabularPandas(df_nocomb, procs, cat, cont, y_names=dep_var, splits=splits)
len(to.train),len(to.valid)
xs,y = to.train.xs,to.train.y
valid_xs,valid_y = to.valid.xs,to.valid.y<jupyter_output><empty_output><jupyter_text>Baseline model: mean and median<jupyter_code>train_df = df_def[:cutoff]
train_df.describe()<jupyter_output><empty_output><jupyter_text>Let's see how good it performs a model whose only information is the median<jupyter_code>mean = np.mean(train_df["SYNERGY_SCORE"])
median = np.median(train_df["SYNERGY_SCORE"])
print(f" Median = {median} \n Mean = {mean}")<jupyter_output>Median = 10.002977 
 Mean = 12.892230668515133<jupyter_text>We create our metrics<jupyter_code>def r_mse(pred,y): return round(math.sqrt(((pred-y)**2).mean()), 6)
def m_rmse(m, xs, y): return r_mse(m.predict(xs), y)
error_mean = r_mse(mean, valid_y)
error_median = r_mse(median, valid_y)
print(f" Error Median = {error_median} \n Error Mean = {error_mean}")<jupyter_output>Error Median = 27.760743 
 Error Mean = 27.660492<jupyter_text>Decision Trees<jupyter_code># Now that we have preprocessed our dataset, we build the tree



Tree = DecisionTreeRegressor(max_leaf_nodes=4)
Tree.fit(xs, y);
draw_tree(Tree, xs, size=10, leaves_parallel=True, precision=2)


samp_idx = np.random.permutation(len(y))[:500]
dtreeviz(Tree, xs.iloc[samp_idx], y.iloc[samp_idx], xs.columns, dep_var,
        fontname='DejaVu Sans', scale=1.6, label_fontsize=10,
        orientation='LR')<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but DecisionTreeRegressor was fitted with feature names
  "X does not have valid feature names, but"<jupyter_text>Let's now have the decision tree algorithm build a bigger tree. Here, we are not passing in any stopping criteria such as max_leaf_nodes:<jupyter_code>m = DecisionTreeRegressor()
m.fit(xs, y);

# In the training set
m_rmse(m, xs, y)<jupyter_output><empty_output><jupyter_text>This just means that the model fits well in the training dataset, but we have to check how well it generalizes over unseen data:<jupyter_code>m_rmse(m, valid_xs, valid_y)<jupyter_output><empty_output><jupyter_text>Now we will check for overfitting:<jupyter_code>m.get_n_leaves(), len(xs)<jupyter_output><empty_output><jupyter_text>We see that it has as many leafs as datapoints, let's see what happens if we restrict the model.<jupyter_code>m = DecisionTreeRegressor(min_samples_leaf=25)
m.fit(to.train.xs, to.train.y)
m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)
m.get_n_leaves()<jupyter_output><empty_output><jupyter_text>**The RMSE is almost the same as the baseline model. That's not good, let's try some hyperparameter tuning.**<jupyter_code>leafs = np.arange(500)+1
error_list = list()
for n_leafs in leafs:
  m = DecisionTreeRegressor(min_samples_leaf=n_leafs)
  m.fit(to.train.xs, to.train.y)
  error_list.append( m_rmse(m, valid_xs, valid_y) )
error_list = np.asarray(error_list)
best_error = min(error_list)
best_leaf = leafs[error_list== min(error_list)][0]
print(f"Best number of leafs = {best_leaf} \n Error = {best_error}")<jupyter_output>Best number of leafs = 179 
 Error = 26.609312<jupyter_text>Not outstanding, barely better. We should try another algorithm  Random Forest<jupyter_code>def rf(xs, y, n_estimators=100, max_samples=300,
       max_features=0.5, min_samples_leaf=5, **kwargs):
    return RandomForestRegressor(n_jobs=-1, n_estimators=n_estimators,
        max_samples=max_samples, max_features=max_features,
        min_samples_leaf=min_samples_leaf, oob_score=True).fit(xs, y)
m = rf(xs, y);
m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)<jupyter_output><empty_output><jupyter_text>A little better than the Tree regressor, but not that great.  Feature importance<jupyter_code>def rf_feat_importance(m, df):
    return pd.DataFrame({'cols':df.columns, 'imp':m.feature_importances_}
                       ).sort_values('imp', ascending=False)

fi = rf_feat_importance(m, xs)
fi[:10]
def plot_fi(fi):
    return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False)

plot_fi(fi[:30]);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11909871244635194, 'fraction_numerical': 0.019492131616595135, 'mean_word_length': 6.58887381275441, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '357770', 'license': 'Apache-2.0', 'n_tokens_mistral': 2220, 'n_tokens_neox': 2091, 'n_words': 462, 'path': 'Word_embeddings_colaboratory_8_2_2021.ipynb', 'repo_name': 'edmanft/Drug_Synergy', 'size': '5592'}