<commit_before>Title: From the Shell and the Cloud - Common Bioinformatics on the Example of Gene Expression Analysis using Unix and freely available Open Source Tools</br>
Author: <NAME></br>
EMail: ak at silico-sciences.com</br>
Seminar Ruprecht-Karls-Universität Heidelberg 2016-01-20 - 2016-01-22

# Bioinformatics Data in the Shell and -Cloud

### Common Bioinformatics Using Unix and Freely Available Open Source Tools on the Example of Gene Expression Analysis

# Working with Bioinformatics Data

## Retrieving Data

Find data at

0. UCSC
`http://hgdownload.cse.ucsc.edu/goldenPath/`

0. Ensembl
`http://www.ensembl.org/info/data/ftp/index.html`

0. NCBI

---

`wget`: the non-interactive network downloader

    $ wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/chr22.fa.gz http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/md5sum.txt

### Options to remember:

0. `--accept`
0. `--no-directories`
0. `--recursive`
0. `-c`
0. `-q`
0. `-b`
0. `-p`


---

`rsync`:  a fast, versatile, remote (and local) file-copying tool

---

## Filesizes

    $ ls -lh
    total 11M
    -rw-r--r-- 1 alex alex  11M Mar 20  2009 chr22.fa.gz
    -rw-r--r-- 1 alex alex 4,9K Mar 20  2009 md5sum.txt
    
---

### Verifying Data

`md5sum`: compute and check MD5 message digest

`sha1sum`: compute and check SHA1 message digest

    $ echo "bioinfo is fun" | sha1sum 
    3df832201564894a11734910cdb6b53783c7dd24  -
    
    $ echo "bioinfo is fun " | sha1sum 
    0db0ec42650cab76bd2e2cfec2ca1990f6f5a290  -

    $ grep "chr22" md5sum.txt 
    2baab477ebcc5a0f4163608ec93e4744  chr22.fa.gz

    $ md5sum chr22.fa.gz 
    2baab477ebcc5a0f4163608ec93e4744  chr22.fa.gz

    $ sha1sum chr22.fa.gz 
    d012edd46f50d674380460d3b4e91f450688e756  chr22.fa.gz




### Extracting Data

    $ for i in *.tar.gz; do tar -xvzf ${i}; done
    
    ls -lh
    total 625M
    drwxrwxr-x 3 bi bi 4,0K Dec 12 00:10 Caenorhabditis_elegans
    -rw-rw-r-- 1 bi bi 557M Dec 11 23:43 Caenorhabditis_elegans_UCSC_ce10.tar.gz
    -rw-rw-r-- 1 bi bi 8,7K Dec 12 00:09 CHECKSUMS.txt
    -rwxrwxr-x 1 bi bi 5,3K Jun 18  2014 README.txt
    drwxrwxr-x 3 bi bi 4,0K Dec 12 00:11 Saccharomyces_cerevisiae
    -rw-rw-r-- 1 bi bi  68M Oct  2 08:32 Saccharomyces_cerevisiae_UCSC_sacCer3.tar.gz
    
---

### Inspecting Data

    $ pwd
    /home/bi/Caenorhabditis_elegans/UCSC/ce10/Sequence/WholeGenomeFasta/
    
---

    $ ls -lh
    total 98M
    -rwxrwxr-x 1 bi bi 1,2K Jun 10  2014 genome.dict
    -rwxrwxr-x 1 bi bi  98M Jun 16 18:24 genome.fa
    -rwxrwxr-x 1 bi bi  197 Jun 10  2014 genome.fa.fai
    -rwxrwxr-x 1 bi bi 1,3K Jun 10  2014 GenomeSize.xml
    
#### `head` and `tail`<commit_msg>Update files<commit_after>Title: From the Shell and the Cloud - Common Bioinformatics on the Example of Gene Expression Analysis using Unix and freely available Open Source Tools</br>
Author: <NAME></br>
EMail: ak at silico-sciences.com</br>
Seminar Ruprecht-Karls-Universität Heidelberg 2016-01-20 - 2016-01-22

# Bioinformatics Data in the Shell and -Cloud

### Common Bioinformatics Using Unix and Freely Available Open Source Tools on the Example of Gene Expression Analysis

# Working with Bioinformatics Data

### Retrieving Data

Find data at

0. UCSC
`http://hgdownload.cse.ucsc.edu/goldenPath/`

0. Ensembl
`http://www.ensembl.org/info/data/ftp/index.html`

0. NCBI

---

`wget`: the non-interactive network downloader

    $ wget http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/chr22.fa.gz http://hgdownload.cse.ucsc.edu/goldenPath/hg19/chromosomes/md5sum.txt

#### Options to remember:

0. `--accept`
0. `--no-directories`
0. `--recursive`
0. `-c`
0. `-q`
0. `-b`
0. `-p`


---

`rsync`:  a fast, versatile, remote (and local) file-copying tool

*Trailing slashes (data vs data/) matter! No slash meains rsync the folder, slash means rsync content. Forgetting the slash can e.g. result in data/data/... instead of data/....*

---

### Filesizes

    $ ls -lh
    total 11M
    -rw-r--r-- 1 alex alex  11M Mar 20  2009 chr22.fa.gz
    -rw-r--r-- 1 alex alex 4,9K Mar 20  2009 md5sum.txt
    
---

### Verifying Data

`md5sum`: compute and check MD5 message digest

`sha1sum`: compute and check SHA1 message digest

    $ echo "bioinfo is fun" | sha1sum 
    3df832201564894a11734910cdb6b53783c7dd24  -
    
    $ echo "bioinfo is fun " | sha1sum 
    0db0ec42650cab76bd2e2cfec2ca1990f6f5a290  -

    $ grep "chr22" md5sum.txt 
    2baab477ebcc5a0f4163608ec93e4744  chr22.fa.gz

    $ md5sum chr22.fa.gz 
    2baab477ebcc5a0f4163608ec93e4744  chr22.fa.gz

    $ sha1sum chr22.fa.gz 
    d012edd46f50d674380460d3b4e91f450688e756  chr22.fa.gz
    
    $ md5sum -c md5sum.txt 
    [..]
    chr21_gl000210_random.fa.gz: FAILED open or read
    chr22.fa.gz: OK
    md5sum: chr3.fa.gz: No such file or directory
    [..]

### Compressed Data

`gzip`: gzip, gunzip, zcat - compress or expand files

`tar`: The GNU version of the tar archiving utility

    $ ls -lh
    total 11M
    -rw-r--r-- 1 alex alex  11M Mar 20  2009 chr22.fa.gz
    -rw-r--r-- 1 alex alex 4,9K Mar 20  2009 md5sum.txt
  
    $ gunzip chr22.fa.gz 
    
    $ ls -lh
    total 50M
    -rw-r--r-- 1 alex alex  50M Mar 20  2009 chr22.fa
    -rw-r--r-- 1 alex alex 4,9K Mar 20  2009 md5sum.txt
    
    $ gzip chr22.fa
    
    $ ls -lh
    total 11M
    -rw-r--r-- 1 alex alex  11M Mar 20  2009 chr22.fa.gz
    -rw-r--r-- 1 alex alex 4,9K Mar 20  2009 md5sum.txt
    
standard    |  compressed
--------|---------
`grep`	| `zgrep`
`less`	| `zless`
`cat`   | `zcat`
`diff`	|`zdiff`

### Inspecting Data

    $ pwd
    /home/bi/Caenorhabditis_elegans/UCSC/ce10/Sequence/WholeGenomeFasta/
    
---

    $ ls -lh
    total 98M
    -rwxrwxr-x 1 bi bi 1,2K Jun 10  2014 genome.dict
    -rwxrwxr-x 1 bi bi  98M Jun 16 18:24 genome.fa
    -rwxrwxr-x 1 bi bi  197 Jun 10  2014 genome.fa.fai
    -rwxrwxr-x 1 bi bi 1,3K Jun 10  2014 GenomeSize.xml
    
#### `head` and `tail`
{'fraction_non_alphanumeric': 0.11538461538461539, 'fraction_numerical': 0.10441970310391363, 'mean_word_length': 2.610840438489647, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1108653', 'n_tokens_mistral': 2877, 'n_tokens_neox': 2380, 'n_words': 654}