<jupyter_start><jupyter_text>MELANOMA Creating the merge dataset from query and reference without Neg.cell samples. This created dataset will use in cross-validation benchmark analysis.<jupyter_code># experiment source
experiment = 'exper_melanoma'
loc = 'external'
# the output location
loc_output = os.path.join(src.DIR_DATA, loc, experiment)
src.define_folder(loc_=loc_output)
df_query = pd.read_pickle(os.path.join(src.DIR_DATA, loc, experiment, 'query.pck'))
print(df_query.shape)
print(df_query.info())
df_query.head()
df_reference = pd.read_pickle(os.path.join(src.DIR_DATA, loc, experiment, 'reference.pck'))
print(df_reference.shape)
print(df_reference.info())
df_reference.head()
df_merged = pd.concat([df_query, df_reference], axis=0)
print(df_merged.shape)
print(df_merged.info())
df_merged.head()
## For cross-validation benchmark analysis, the merged dataset without Neg.cell samples will use.
df_merged = df_merged[df_merged['label']!='Neg.cell'].reset_index(drop=True)

df_merged['label'].value_counts()
df_merged.to_pickle(os.path.join(loc_output, 'query_reference_wo_negcell.pck') )
print('Experiment datasets are exported into ', os.path.join(loc_output, 'query_reference_wo_negcell.pck' ))<jupyter_output>Experiment datasets are exported into  /home/pgundogdu/projects/signalization_prior_knowledge_based_nn/data/external/exper_melanoma/query_reference_wo_negcell.pck<jupyter_text>MOUSE Selecting specific cell types in retrieval dataset. This created dataset will use in cross-validation benchmark analysis.<jupyter_code># experiment source
experiment = 'exper_mouse'
loc = 'external'
# the output location
loc_output = os.path.join(src.DIR_DATA, loc, experiment)
src.define_folder(loc_=loc_output)
df_retrieval = pd.read_pickle(os.path.join(src.DIR_DATA, loc, experiment, '3-33_integrated_retrieval_set.pck'))
print(df_retrieval.shape)
print(df_retrieval.info())
df_retrieval.head()
df_retrieval['cell_type'] = df_retrieval['Label'].str.split('_', expand=True)[0]

df_retrieval.loc[df_retrieval['cell_type'].isin(['E12', 'E14', 'adult']), 'cell_type'] = 'HSC'
df_retrieval.loc[df_retrieval['cell_type'].isin(['celebral-cortex', 'CNS', 'brain']), 'cell_type'] = 'neuron'
df_retrieval = df_retrieval.loc[df_retrieval['cell_type'].isin(['HSC','4cell','ICM','spleen','8cell','neuron','zygote','2cell','ESC'])]#.reset_index(drop=True)
df_retrieval.drop(columns=['Label'], inplace=True)
df_retrieval
df_retrieval.to_pickle(os.path.join(loc_output, '3-33_integrated_retrieval_set_cv.pck') )
print('Experiment datasets are exported into ', os.path.join(loc_output, '3-33_integrated_retrieval_set_cv.pck' ))<jupyter_output>Experiment datasets are exported into  /home/pgundogdu/projects/signalization_prior_knowledge_based_nn/data/external/exper_mouse/3-33_integrated_retrieval_set_cv.pck
{'fraction_non_alphanumeric': 0.13709389503748662, 'fraction_numerical': 0.007497322384862549, 'mean_word_length': 8.97153024911032, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '331619', 'license': 'RSA-MD', 'n_tokens_mistral': 1029, 'n_tokens_neox': 988, 'n_words': 159, 'path': 'notebooks/1.1-pg-exporting-additional-datasets.ipynb', 'repo_name': 'pelingundogdu/signalization_prior_knowledge_based_nn', 'size': '2801'}