<jupyter_start><jupyter_text>Aim: Integrate gene expression data from three online data portals: Stemformatics, ENCODE and Haemosphere. In previous notebooks, we have combined the gene expression data of selected samples in each individual data portals. Here, we will integrate the three pre-combined expression datasets together into a larger dataset of mouse blood.<jupyter_code>import pandas as pd
import atlas
import handler
# Load data 
expression_s4m    = pd.read_csv('../data/interim/mouse_integrate/expression_s4m.tsv', sep='\t', index_col=0)
expression_encode = pd.read_csv('../data/interim/mouse_integrate/expression_encode.tsv', sep='\t', index_col=0)
expression_haemosphere = pd.read_csv('../data/interim/mouse_integrate/expression_Haemosphere.tsv', sep='\t', index_col=0)

samples_s4m = pd.read_csv('../data/interim/mouse_integrate/samples_s4m.tsv', sep='\t', index_col=0)
samples_encode = pd.read_csv('../data/interim/mouse_integrate/samples_encode.tsv', sep='\t', index_col=0)
samples_haemosphere = pd.read_csv('../data/interim/mouse_integrate/samples_Haemosphere.tsv', sep='\t', index_col=0)
print(expression_s4m.shape, expression_encode.shape, expression_haemosphere.shape)<jupyter_output>(17947, 240) (22457, 90) (14462, 794)<jupyter_text>Merge expression tables<jupyter_code>dfs = [expression_encode, expression_haemosphere, expression_s4m]
common_genes = handler.find_common_genes(dfs)
mouse_atlas = atlas.rankTransform(handler.merge_columns(dfs, common_genes))

print(mouse_atlas.shape)
mouse_atlas.head()<jupyter_output>(14030, 1124)<jupyter_text>Standardise sample metadata Since the sample metadata of different datasets may have recorded differnet variables, or using different names for similar attributes, to merge them, we will need to identify a minimal list of necessary attributes to describe samples, and unify the naming of these attributes. The essential list of attributes includes:- celltype- cell_lineage- description- markers- dataset_name- platformAlso, for each data attribute, manual reformatting is necessary to make sure the data is recorded in an uniformed format. e.g. 'NK' and 'natrual killer' should not be existed in the same table.<jupyter_code>samples_s4m.head()
set(samples_s4m.final_cell)
set(samples_s4m.sample_type)
samples_s4m_std = samples_s4m[['']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0993576017130621, 'fraction_numerical': 0.020985010706638114, 'mean_word_length': 6.972696245733788, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '408109', 'license': 'MIT', 'n_tokens_mistral': 766, 'n_tokens_neox': 732, 'n_words': 196, 'path': 'notebooks/.ipynb_checkpoints/2_data_integration-checkpoint.ipynb', 'repo_name': 'xinyip1/mouse_atlas', 'size': '2335'}