<jupyter_start><jupyter_text>So now we have the unzipped version of the file, "`GSE41265_allGenesTPM.txt`". I wonder how much space they saved by zipping it?Let's use the flags "`-l`" for "long listing" which will show us the sizes<jupyter_code>ls -1<jupyter_output>1.0_Introduction_to_bioinformatics.ipynb
1.1_Overview_of_analysis_steps.ipynb
1.2_Downloading_public_data_Shalek2013.ipynb
1.3_Single-cell_overview_additional_reading.ipynb
1.X_unused_code.ipynb
2.2_Introduction_to_dimensionality_reduction.ipynb
2.4_ICA.ipynb
2.5_Manifold_learning.ipynb
2.6_Compare_dimensionality_reduction.ipynb
2.7_Apply_dimensionality_reduction_on_Shalek2013_Macaulay2016.ipynb
2.8_Additional_reading.ipynb
2016-06-09_darmanis2015_concatenate_data.ipynb
3.0_Introduction_to_clustering.ipynb
3.1_Classifiers.ipynb
3.2_Hierarchical_clustering.ipynb
3.3_Apply_clustering_to_Shalek2013_Macaulay2016.ipynb
3.X_Unused.ipynb
4.0_Introduction_to_classifiers.ipynb
4.3_Apply_classifiers_to_Shalek2013_Macaulay2016_with_gene_ontology.ipynb
4.4_Additional_reading.ipynb
5.0_Pseudotime_introduction.ipynb
5.1_Pseudotime_ordering_algorithms_overiew.ipynb
5.3_Pseudotime_additional_reading.ipynb
6.1_Dealing_with_technical_noise.ipynb
6.2_Batch_Correction.ipynb
6.3_Te[...]<jupyter_text>oof, this is in pure bytes and I can't convert to multiples of 1024 easily in my head (1024 bytes = 1 kilobyte, 1024 kilobytes = 1 megabtye, etc -  the 1000/byte is a lie that the hard drive companies use!). So let's use the `-h` flag, which tells the computer to do th conversion for us. We can combine multiple flags with the same dash, so    ls -l -hCan be shortened to:    ls -lh<jupyter_code>! ls -1 | wc -l
! gunzip --help<jupyter_output>Apple gzip 251
usage: gunzip [-123456789acdfhklLNnqrtVv] [-S .suffix] [<file> [<file> ...]]
 -1 --fast            fastest (worst) compression
 -2 .. -8             set compression level
 -9 --best            best (slowest) compression
 -c --stdout          write to stdout, keep original files
    --to-stdout
 -d --decompress      uncompress files
    --uncompress
 -f --force           force overwriting & compress links
 -h --help            display this help
 -k --keep            don't delete input files during operation
 -l --list            list compressed file contents
 -N --name            save or restore original file name and time stamp
 -n --no-name         don't save original file name or time stamp
 -q --quiet           output no warnings
 -r --recursive       recursively compress files in directories
 -S .suf              use suffix .suf instead of .gz
    --suffix .suf
 -t --test            test compressed file
 -V --version         display program versio[...]<jupyter_text>See, "`GSE41265_allGenesTPM.txt.gz`" is there!Since the file ends in "`.gz`", this tells us its a "gnu-zipped" or "gzipped" ("gee-zipped") file, which is a specific flavor of "zipping" or compressing a file. We need to use a gnu-zipping-aware program to decompress the file, which is "`gunzip`" ("gnu-unzip").Run the next cell to unzip the file<jupyter_code>! gunzip -f *gz
3+3
asdf = 'beyonce'
asdf
asdf + ' runs the world'<jupyter_output><empty_output><jupyter_text>Let's "`ls`" again to see what files have changed<jupyter_code>ls
! head GSM1657872_1772078217.C04.csv 
import glob

import pandas as pd
pd.read_table('GSM1657872_1772078217.C04.csv')
pd.read_table('GSM1657872_1772078217.C04.csv', index_col=0)
dataframe = pd.read_table('GSM1657872_1772078217.C04.csv', index_col=0, header=None)
dataframe
series = pd.read_table('GSM1657872_1772078217.C04.csv', index_col=0, header=None, squeeze=True)
series
dataframe.shape
series.shape
series.name
filename = 'GSM1657872_1772078217.C04.csv'
filename
filename.split('.')
filename.split('.csv')
filename.split('.csv')[0]
cells = []

for filename in glob.iglob('*.csv'):
    cell = pd.read_table(filename, index_col=0, squeeze=True, header=None)
    name = filename.split('_')[0]
    cell.name = name
    cells.append(cell)
expression = pd.concat(cells, axis=1)
expression.index = expression.index.map(lambda x: x.strip(' '))
print(expression.shape)
expression.head()<jupyter_output>(22088, 466)<jupyter_text>Read metadata<jupyter_code>! gunzip /Users/kirkreardon/Downloads/*_series_matrix.txt.gz
! head /Users/kirkreardon/Downloads/*_series_matrix.txt
! head -n 20 /Users/kirkreardon/Downloads/*_series_matrix.txt
"Whooo!!!!!!!!!".strip("!")
"Whooo!!!!!!!!!".strip("o")
metadata1 = pd.read_table('/Users/kirkreardon/Downloads/GSE67835-GPL15520_series_matrix.txt', 
                          skiprows=37, header=None, index_col=0)
metadata1.index = metadata1.index.map(lambda x: x.strip('!'))
# Transpose so each row is a cell
metadata1 = metadata1.T
metadata1.head()
metadata2 = pd.read_table('/Users/kirkreardon/Downloads/GSE67835-GPL18573_series_matrix.txt', 
                          skiprows=37, header=None, index_col=0)
metadata2.index = metadata2.index.map(lambda x: x.strip('!'))
# transpose
metadata2 = metadata2.T
metadata2.head()
dataframes = [metadata1, metadata2]
metadata = pd.concat(dataframes)
print(metadata.shape)
metadata.head()
metadata = metadata.set_index('Sample_geo_accession')
metadata.head()
mkdir -p ~/projects/darmanis2015/processed_data
expression.to_csv('~/projects/darmanis2015/processed_data/expression.csv')
metadata.to_csv('~/projects/darmanis2015/processed_data/metadata.csv')
expression.GSM1657884
bad_rows = ['no_feature', 'ambiguous', 'alignment_not_unique']
good_genes = expression.index[~expression.index.isin(bad_rows)]
good_genes
expression.shape
expression_actually_genes = expression.loc[good_genes]
expression_actually_genes.shape
expression_actually_genes.tail()
expression_actually_genes.to_csv("/Users/kirkreardon/projects/darmanis2015/processed_data/expression_actually_genes.csv")
expression_actually_genes.dtypes
expression_actually_genes.tail().index<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10940660822656777, 'fraction_numerical': 0.05563047875927175, 'mean_word_length': 4.5088207985143915, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '628444', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2334, 'n_tokens_neox': 2054, 'n_words': 453, 'path': 'notebooks/2016-06-09_darmanis2015_concatenate_data.ipynb', 'repo_name': 'YeoLab/single-cell-bioinformatics', 'size': '5932'}