<jupyter_start><jupyter_text>Looks like both linear models are more or less comparable in quality.Let's visualize that:<jupyter_code>rcv_corrs = [pearsonr(l, r) for l, r in zip(yv, rcv.predict(Xv))]
lcv_corrs = [pearsonr(l, r) for l, r in zip(yv, lcv.predict(Xv))]

data = pd.DataFrame(
    {
        'ridge':[elem[0] for elem in rcv_corrs],
        'lasso':[elem[0] for elem in lcv_corrs],
    }
)

fig, ax = plt.subplots(2, sharex=True)

ax[0].hist(data['ridge'].values, bins=100, label='ridge')
ax[1].hist(data['lasso'].values, bins=100, label='lasso');
data.describe()<jupyter_output><empty_output><jupyter_text>Looks like that Ridge regression outperforms Lasso:    - mean & median Pearson correlation with target cell component types are larger    - less outliers (min is larger, 25th percentile as well) Let's try to join both methods: select feature by Lasso and use only them in Ridge<jupyter_code>(lcv.coef_ == 0).sum(), (lcv.coef_.shape[0] * lcv.coef_.shape[1])<jupyter_output><empty_output><jupyter_text>Roughly quarter of all coefficients are 0. What if we ignore them in Ridge?<jupyter_code>(lcv.coef_ != 0).sum(axis=0)
genes_upd = [gene for gene, val in zip(s.index, (lcv.coef_ != 0).sum(axis=0)) if val > 0]
len(genes_upd)<jupyter_output><empty_output><jupyter_text>Let's use them to construct X array again, and train RidgeCV only<jupyter_code>s_upd = s.loc[genes_upd]

# construct X and y arrays
s_upd = s.loc[genes_upd]

X = s_upd.T.values
y = np.array([pair[1] for pair in gt])
X.shape, y.shape
Xt, Xv, yt, yv = train_test_split(X, y, random_state=0)
rcv = RidgeCV(alphas=np.linspace(-5,5,25), cv=11)
rcv = RidgeCV(alphas=np.linspace(-5,5,25), cv=11)
rcv.fit(Xt, yt)

rcv.best_score_
rcv.score(Xv, yv)<jupyter_output><empty_output><jupyter_text>Doesn't help, meaning that all features were valuable. Well, at least we tried :)<jupyter_code>!tail -2 README.md<jupyter_output>Второй дополнительный пункт задания: обучить нелинейные модели для этого (например, Random Forest). Для моделей выбрать оптимальный набор генов из представленных и выбрать оптимальные гиперпараметры обучения. Обосновать набор параметров и генов.<jupyter_text>Lets compare our fine-tuned (a little bit) Ridge with simple non-linear model, RandomForest.For simplicity, we'll use max_depth=2, n_estimators=100.<jupyter_code>from sklearn.multioutput import MultiOutputRegressor
from sklearn.ensemble import RandomForestRegressor

X = s.T.values
y = np.array([pair[1] for pair in gt])
X.shape, y.shape
Xt, Xv, yt, yv = train_test_split(X, y, random_state=0)

max_depth=5
n_estimators=150
regr_multirf = MultiOutputRegressor(RandomForestRegressor(n_estimators=n_estimators,
                                                          max_depth=max_depth,
                                                          random_state=0),
                                   n_jobs=8)

regr_multirf.fit(Xt, yt)
regr_multirf.score(Xv, yv)
regr_multirf.predict(Xv[0].reshape(1,-1))
yv[0]
rcv = RidgeCV(alphas=np.linspace(-5,5,25), cv=11)
rcv.fit(Xt, yt)

rcv_corrs = [pearsonr(l, r) for l, r in zip(yv, rcv.predict(Xv))]
lcv_corrs = [pearsonr(l, r) for l, r in zip(yv, lcv.predict(Xv))]
rf_corrs = [pearsonr(l, r) for l, r in zip(yv, regr_multirf.predict(Xv))]

data = pd.DataFrame(
    {
        'ridge':[elem[0] for elem in rcv_corrs],
        'lasso':[elem[0] for elem in lcv_corrs],
        'RF_multi':[elem[0] for elem in rf_corrs],
    }
)

fig, ax = plt.subplots(3, sharex=True)

ax[0].hist(data['ridge'].values, bins=100, label='ridge')
ax[1].hist(data['lasso'].values, bins=100, label='lasso')
ax[2].hist(data['RF_multi'].values, bins=100, label='RF_multi');
data.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13561160835367508, 'fraction_numerical': 0.02034174125305126, 'mean_word_length': 4.488095238095238, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '278131', 'license': 'MIT', 'n_tokens_mistral': 1515, 'n_tokens_neox': 1470, 'n_words': 334, 'path': 'Task_2/notebook.ipynb', 'repo_name': 'marinegor/bostongene_test', 'size': '3687'}