<jupyter_start><jupyter_text>Train a baseline model for Subchallenge 2We train a COX model with LASSO regularization on all inputs:- RNA seq    * log(cpm), normalized per-specimen    * most variable 1000 genes- drug response    * AUC for ex-vivo drug sensitivity    * nullable- clinical categorical data    * one-hot encoding for each category- clinical numerical data    * different units for each column    * nullableWe take the z-score of every column, then fill nulls with 0.  First, setup environment and download training data.**You may have to restart the kernel after running this cell.** This only needs to be run once, to populate the `training/` directory.<jupyter_code>%pip install scikit-survival synapseclient
import getpass
import synapseclient
import synapseutils

syn = synapseclient.Synapse()
syn.login(input(prompt="Enter Synapse Username"), getpass.getpass("Enter Synapse Password"))
downloaded_files = synapseutils.syncFromSynapse(syn, 'syn21212904', path='training')<jupyter_output><empty_output><jupyter_text>Now, load the data, and train a model!<jupyter_code># Auto-reload the custom python modules, for easy development.

%load_ext autoreload
%aimport input_manager
%aimport model
%autoreload 1

from input_manager import RawInputs

raw_inputs = RawInputs('training')
raw_inputs.load()
from input_manager import InputManager

im = InputManager(raw_inputs)
im.prepInputs()
im.printStats()
most_variant_genes = im.rnaseq_by_spec.var().nlargest(1000).index
inhibitors = im.aucs.columns
import numpy
import pandas

from model import makeFullFeatureVector

lab_ids = im.getAllSpecimens()
feature_matrix = pandas.DataFrame()
for lab_id in lab_ids:
    feature_vector = makeFullFeatureVector(im, most_variant_genes, inhibitors, lab_id)
    feature_series = pandas.Series(data=feature_vector, name=lab_id)
    feature_matrix = feature_matrix.append(feature_series)
feature_means = feature_matrix.mean()
feature_stds = feature_matrix.std()
normed_features = (feature_matrix - feature_means) / feature_stds
normed_features = normed_features.fillna(0.0)
from sksurv.datasets import get_x_y
full_dataset = pandas.read_csv('training/response.csv').set_index('lab_id').join(normed_features)
X, Y = get_x_y(full_dataset, ['vitalStatus', 'overallSurvival'], pos_label='Dead')
import numpy
from sksurv.linear_model import CoxnetSurvivalAnalysis
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold

# This package allows general elastic net tuning, but by setting
# l1_ratio=1, we restrict to LASSO.
regr = CoxnetSurvivalAnalysis(l1_ratio=1, alpha_min_ratio=0.1, max_iter=3e5)

n_folds = 10

alphas = numpy.logspace(-1.3, 0, num=50)
cv = KFold(n_splits=5, shuffle=True, random_state=328)
gcv = GridSearchCV(
    regr,
    {"alphas": [[v] for v in alphas]},
    cv=cv).fit(X, Y)
import numpy
from matplotlib import pyplot

scores = gcv.cv_results_['mean_test_score']
scores_std = gcv.cv_results_['std_test_score']
std_error = scores_std / numpy.sqrt(n_folds)

pyplot.figure().set_size_inches(8, 6)
pyplot.semilogx(alphas, scores)
pyplot.fill_between(alphas, scores + std_error, scores - std_error, alpha=0.2)
pyplot.xlabel('alpha')
pyplot.ylabel('Concordance Index')
pyplot.axvline(gcv.best_params_['alphas'][0], color='r', ls='--', label=('Best alpha, CI=%0.3f' % gcv.best_score_))
pyplot.legend()
pyplot.title('Cross Validation Concordance Index, LASSO')
# TODO: Investigate which features were chosen...
pandas.Series(gcv.best_estimator_.coef_[:,0]).to_numpy().nonzero()<jupyter_output><empty_output><jupyter_text>Pickle the data for evaluation.We have:- feature mean and variance (for computing z-score)- feature weights- most variable genes<jupyter_code>import numpy
numpy.save('model/feature_means.npy', feature_means.to_numpy())
numpy.save('model/feature_stds.npy', feature_stds.to_numpy())
numpy.save('model/estimator_coef.npy', gcv.best_estimator_.coef_[:,0])
numpy.save('model/most_variant_genes.npy', most_variant_genes.to_numpy())
numpy.save('model/inhibitors.npy', numpy.array(inhibitors))<jupyter_output><empty_output><jupyter_text>Run the modelWe run the model with```bashSYNAPSE_PROJECT_ID=docker build -t docker.synapse.org/$SYNAPSE_PROJECT_ID/sc2_model .docker run \    -v "$PWD/training/:/input/" \    -v "$PWD/output:/output/" \    docker.synapse.org/$SYNAPSE_PROJECT_ID/sc2_model Maybe push to Synapse.docker login docker.synapse.orgdocker push docker.synapse.org/$SYNAPSE_PROJECT_ID/sc2_model```**Note: you must pass the Synapse [certified user test](https://docs.synapse.org/articles/accounts_certified_users_and_profile_validation.htmlcertified-users) before you can push to the Synapse docker hub.** Look at predictions vs goldstandard for training dataAssumes predictions are in `output/predictions.csv`. Note that the performance on training dataset is going to be better than on the leaderboard data. Therefore, this is a good test of formatting / sanity check, but not of predictive performance.<jupyter_code>from sksurv.metrics import concordance_index_censored

groundtruth = pandas.read_csv('training/response.csv').set_index('lab_id')
predictions = pandas.read_csv('output/predictions.csv').set_index('lab_id')
data = groundtruth.join(predictions)
# data = data[data.vitalStatus == 'Dead']
cindex = concordance_index_censored(
    data.vitalStatus == 'Dead', data.overallSurvival, -data.survival)[0]
print(cindex)
import seaborn
seaborn.scatterplot(
    x='overallSurvival',
    y='survival',
    data=data,
    hue='vitalStatus',
    alpha=1)
pyplot.title('SC2 baseline predictor')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10749418708638885, 'fraction_numerical': 0.009837238418887497, 'mean_word_length': 6.367588932806324, 'pattern_counts': {'":': 1, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '480893', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1913, 'n_tokens_neox': 1818, 'n_words': 445, 'path': 'index.ipynb', 'repo_name': 'sererenaa/conda', 'size': '5591'}