<jupyter_start><jupyter_text>Variance Threshold<jupyter_code>thr = VarianceThreshold()
thr.fit(df_arn_ad)<jupyter_output><empty_output><jupyter_text>On représente le nombre de gènes en fonction d'un seuil de variance. (le nombre de gènes ayant une variance inférieure à ce seuil)<jupyter_code>def threshold(alpha):
    thr = VarianceThreshold(alpha)
    x = thr.fit_transform(df_arn_ad)
    return x.shape[1]

x_thr = np.linspace(0,2,501)
y_thr = [df_arn_ad.shape[1]-threshold(x) for x in x_thr]
fig, ax = plt.subplots(figsize=(10,8))

sns.lineplot(x_thr, y_thr)
ax.set_xlabel('Threshold', size = 15)
ax.set_ylabel('Removed variables', size = 15)

ax.axhline(21500,ls='--', color='red')
plt.text(0.09, 22000, '(0.069388, 21500)')
df_arn_ad.shape[1] - threshold(0.069388)<jupyter_output><empty_output><jupyter_text>On choisit une threshold of variance de sorte à éliminer 21500 gènes<jupyter_code>chosen_thr = 0.069388
thr = VarianceThreshold(chosen_thr)

df_arn_ad_filtered = thr.fit_transform(df_arn_ad)
useful_var_genes = [g for m, g in zip(thr.get_support(), df_arn_ad.columns.tolist()) if m]
useless_var_genes = [g for m, g in zip(thr.get_support(), df_arn_ad.columns.tolist()) if not m]

df_arn_ad_filtered = pd.DataFrame(df_arn_ad_filtered)
df_arn_ad_filtered.columns = useful_var_genes
df_arn_ad_filtered.index = df_arn_ad.index
df_arn_ad_filtered.shape
df_arn_ad_filtered.head()<jupyter_output><empty_output><jupyter_text>Test Stat : f_regression On teste l'influence des gènes sur le scorad par un test statistique et on représente la distribution du score résultant de ce test<jupyter_code>k_best = SelectKBest(f_regression, k = 200)
df_filtered_two = k_best.fit_transform(df_arn_ad_filtered.iloc[:,1:], df_arn_ad_filtered.iloc[:,0])
fig, ax = plt.subplots(figsize=(10,8))
sns.kdeplot(k_best.scores_)
ax.set_xlabel('Feature Score', size = 15)
ax.set_ylabel('Density of Features')
ax.axvline(1.78, ls='--', color='red')

plt.text(2.4 ,0.12, '(2.24, 0.112)')<jupyter_output><empty_output><jupyter_text>On sélectionne les gènes ayant un score supérieur à 1.78%<jupyter_code>x, y  = sns.kdeplot(k_best.scores_).get_lines()[0].get_data()
thr_score = x[8]
thr_score
y[8]
df_arn_ad_filtered.shape
mask_k = [(score > thr_score) for score in k_best.scores_]
useful_filtered_genes = [g for m, g in zip(mask_k, df_arn_ad_filtered.iloc[:,1:].columns.tolist()) if m]

df_filtered_best = df_arn_ad_filtered.loc[:,['scorad'] + useful_filtered_genes]
len(useful_filtered_genes)
df_filtered_best.shape<jupyter_output><empty_output><jupyter_text>ElasticNet<jupyter_code>df_filtered_best.head()
from sklearn.linear_model import ElasticNet

elastic = ElasticNet(l1_ratio = .001)
elastic.fit(df_filtered_best.iloc[:,1:], df_filtered_best.iloc[:,0])
len(elastic.coef_[elastic.coef_!=0])
elastic_genes = df_filtered_best.iloc[:,1:].columns[elastic.coef_!=0]
result = pd.DataFrame( {'Genes': elastic_genes, 'ElasticNet coef (abs)' : abs(elastic.coef_[elastic.coef_!=0]),
                       'ElasticNet coef' : elastic.coef_[elastic.coef_!=0]}).sort_values(by='ElasticNet coef (abs)', ascending=False)
result.head()
result.to_csv('genes_elasticnet.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13556116015132408, 'fraction_numerical': 0.03215636822194199, 'mean_word_length': 7.263020833333333, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '522259', 'license': 'MIT', 'n_tokens_mistral': 1380, 'n_tokens_neox': 1279, 'n_words': 221, 'path': 'Modeling/gene-selection.ipynb', 'repo_name': 'dataforgoodfr/batch6_eczema', 'size': '3172'}