<jupyter_start><jupyter_text>Reading high-throughput sequencing files ```@SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACCAAGTTACCCTTAACAACTTAAGGG+SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9ICIIIIIIIIIIIIIIIIIIIIDIII``` + `@` line: basic read information+ sequence line+ `+` metadata line+ quality score line  Big files<jupyter_code>f = open("files/simple-file.txt")
for l in f.readlines():
    print(l,end="")
f.close()<jupyter_output><empty_output><jupyter_text>Problem`f.readlines` is powerful **but** you just loaded the whole contents of the file into memory.If the file is 10GB, you may have just crashed your computer!  Solution: turn the reader into an `iterator`<jupyter_code>with open("files/simple-file.txt") as f:
    for l in f:
        print(l.strip())<jupyter_output><empty_output><jupyter_text>```pythonwith open("files/simple-file.txt") as f:    for l in f:        do_something(l)```This bizarro syntax goes through the file line-by-line, wiping out each old line with the new one.  What if you have a *compressed* input file?<jupyter_code>with open("files/simple-file.txt.gz") as f:
    for l in f:
        print(l.strip())<jupyter_output><empty_output><jupyter_text>Use the `gzip` module<jupyter_code>import gzip

with gzip.open("files/simple-file.txt.gz") as f:
    for l in f:
        l_ascii = l.decode("ascii")
        print(l_ascii.strip())<jupyter_output><empty_output><jupyter_text>```pythonimport gzipwith gzip.open("files/simple-file.txt.gz") as f:    for l in f:        l_ascii = l.decode("ascii")        print(l_ascii.strip())```+ `gzip.open` is just like `open` except it takes a .gz file+ `l.decode("ascii")` species that the binary blob that just came out of the file is normal text  + What sequence occurs most often the in the file `files/example.fastq.gz`?  + **Bonus**: create a histogram of read quality. The last line gives the PHRED score--[Probability of error](https://en.wikipedia.org/wiki/FASTQ_formatQuality)--for each nucleotide. It ranges from `1.00000` (encoded as the letter `0`) to `0.00006` (encoded by the letter `K`).  Sum up the score for all bases in the alignment. Table is here: (http://www.drive5.com/usearch/manual/qscores.gif) ```@SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACCAAGTTACCCTTAACAACTTAAGGG+SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9ICIIIIIIIIIIIIIIIIIIIIDIII```<jupyter_code>import gzip

get_line = False
seqs = {}
with gzip.open("files/example.fastq.gz") as f:
    for l in f:
        l_ascii = l.decode("ascii")
        if l_ascii[0] == "@":
            get_line = True
            continue
        if get_line:
            try:
                seqs[l_ascii.strip()] += 1
            except KeyError:
                seqs[l_ascii.strip()] = 1
            get_line = False


seq_as_list = [(seqs[s],s) for s in seqs.keys()]
seq_as_list.sort()
print(seq_as_list[-1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12315270935960591, 'fraction_numerical': 0.040722495894909685, 'mean_word_length': 4.325174825174825, 'pattern_counts': {'":': 1, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '404737', 'license': 'Unlicense', 'n_tokens_mistral': 1210, 'n_tokens_neox': 1098, 'n_words': 263, 'path': 'chapters/05_big-files/00_fastq-files_key.ipynb', 'repo_name': 'lperezmo/pythonic-science', 'size': '3045'}