<jupyter_start><jupyter_text>Welcome to the BI Python Sequence Tutorial! In this tutorial, we are going to go over some command line basics and use google colaboratory to run python code in the cloud. We will also go over downloading files from Github, processing sequencing samples and installing python packages. Before we dive right in, I wanted to provide you with a little background information about google colab and describe why we want to use some type of jupyter service to run our code. Then there is a short description on the sequence file formats, followed by a list of several unix commands that will come in handy during the tutorial.  What is Google Colaboratory?Colaboratory, or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education. More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs. The most important feature that distinguishes Colab from other free cloud services is; Colab provides access to free Tesla K80 GPUs and comes with several python packages pre-installed!Now you can develop deep learning applications at your fingertips with no cost to you or in our case, learn how to effectively use python to process sequencing samples! If you would like to use Jupyter notebooks for your own projects, some advice on best practices can be found here: https://towardsdatascience.com/jupyter-notebook-best-practices-f430a6ba8c69  Sequencing File FormatsJust make sure everyone is on the same page about the format of the files we will be working with, lets talk about sequencing file formats briefly.* The FASTQ format is used with sequencing files to include a numeric quality score to each base in the sequence. The FASTQ format is widely used to store high-throughput sequencing data, which is reported with a per-base quality score indicating the confidence of each base call. Unfortunately, FASTQ has variants and pitfalls that can make the seemingly simple format frustrating to work with. An Example FASTQ format looks like: @DJB775P1:248:D0MDGACXX:7:1202:12782:49716 CTCTGCGTTGATACCACTGCTTACTCTGCGTTGATACCACTGCTTAGATCGG \+ IIIIIIIIIIIIIIIHHHHHHFFFFFFEECCCCBCECCCCCCCCCCCCCCCC * The description line, beginning with @. This contains the record identifier and other information. Sequence data, which can be on one or many lines. The line beginning with +, following the sequence line(s) indicates the end of the sequence. In older FASTQ files, it was common to repeat the description line here, but this is redundant and leads to unnecessarily large FASTQ files. The quality data, which can also be on one or many lines, but must be the same length as the sequence. Each numeric base quality is encoded with ASCII characters. This will be important later if you are not familiar with these types of files.   Working with the command promptNormally on your local computer, you would need to open the terminal to begin.While in the cloud using colab, we are actually running a linux computer so it will be useful to learn a few simple unix commands to help you navigate the environment.* **pwd** --- pwd is the command to tell you the present working directory. Running this command should show you the current folder that you are working in.*	**ls** --- the ls command stands for list, and will list any files and folders in your current directory.* **cd** --- cd stands for change directory. You can use this command to move into folders. A special version of this command using periods like this: cd .. will move you up one folder. In programming, two periods in a row means “up one folder”*	**head** --- this command will show you the “head” of a file. By default, it will print out the first 10 lines of your file. * **tail** --- this command will show you the “tail” of a file. By default, it will print out the last 10 lines of your file. * **curl** --- the curl command will let you download a file using a url or website address. * **tar** --- The tar command is used to package and unpackage tarball files. Think of this like creating a zipped folder. Tar works the same way on linux systems for packaging files/folders together into a single file, which then must be unpacked.* **gzip** --- the gzip command works to zip a folder or file, compressing it into a smaller folder/file. The resulting file will have the file extension of “.gz”* **gunzip** --- gunzip performs the opposite action of gzip to unzip a gz file back to a regular uncompressed file type.* **unzip** --- similar to gunzip, this unzips a file that was compressed with the traditional zipping algorithm common on windows. * **rm** --- this command stands for remove and allows you to delete a file or folder. There is no way to retrieve a file once it is deleted using the command line! * **nano** --- this is a simple text editor that will allow you to alter files from the command line. For each of these commands, you can find more information on google by searching “[name of command] bash examples”. Now lets get started!   How to run code on colabIn this tutorial, code will be provided in the text boxes here and can be typed into the code block below. To run the python code, press enter and you should see output below the cell, like in this example!    this is example code    Use this as a reference for what you run in the box below<jupyter_code>i_am_a_variable = 5

second_variable = 10

print("This is the output. The answer is: ", i_am_a_variable + second_variable)<jupyter_output><empty_output><jupyter_text>Using linux commandsIf we want to use linux commands in the notebook, we need to first type an exclamation point before the command to tell the colab notebook that we are running a command from the command line, not from inside of the jupyter notebook. Try a running the linux command list like this in the box below:    !ls<jupyter_code><jupyter_output><empty_output><jupyter_text>Making Files & Intro to PythonTo start with, lets download the project folder from github to our colab notebook using the linux git clone command. Throughout this tutorial, there will be code blocks that you can use to type the commands. We can begin by using the git pull as follows:     !git clone https://github.com/joelwebb/Source-Code-Tutorial<jupyter_code><jupyter_output><empty_output><jupyter_text>To check where we are at, lets run the pwd command below:    !pwd<jupyter_code><jupyter_output><empty_output><jupyter_text>Here we can see we are in the content folder on our cloud computer. Lets print all the files and folders in our current director using the ls command.    !ls<jupyter_code><jupyter_output><empty_output><jupyter_text>If you don't have the git package installed, you can also use the linux curl command to download a zipped folder from Github using the following example command.    !curl -LkSs https://github.com/joelwebb/Source-Code-Tutorial/archive/master.zip -o master.tar.gzThis command will download a ziped tarball file that contains everything you need to do the tutorial! go ahead and type the command below followed by the ls command to make sure you downloaded the file.<jupyter_code><jupyter_output><empty_output><jupyter_text>Next, let us use the unzipping utility from the example commands above to unzip the folder as follows:    !unzip master.tar.gz<jupyter_code><jupyter_output><empty_output><jupyter_text>While both of these methods work to download files, it will be useful for the future to know how to download projects directly from github and also from any website online using the curl command. Since we don't need two copies of these folders, lets use the linux rm command to remove the folder that we don't need.    !rm -r Source-Code-Tutorial-master master.tar.gzThe rm command will work with a file, or if we use the -r option, it works recursively. Meaning, it will delete all the files and folders inside of the folder. Be careful deleting on the command line because once a file is lost, it is gone forever!<jupyter_code><jupyter_output><empty_output><jupyter_text>To move into our folder we downloaded, we could use the linux command cd command to change directories... But hey, this is a python course, right! Lets change folders using python!First we need to import the os package (importing packages only need to be done once), and then use the function os.chdir("name of the directory") --- try it with this command below:    import os    os.chdir("Source-Code-Tutorial")<jupyter_code><jupyter_output><empty_output><jupyter_text>To check that we moved into the folder, we could run the linux pwd command Or using python we can print the current working directory using this command:    print(os.getcwd())<jupyter_code><jupyter_output><empty_output><jupyter_text>You should now see that we are located in"/content/Source-Code-Tutorial" and see a path printed to your screen. If you look on the left hand size of your screen, there is a folder icon and you can click it to visually view the files and folders as you go along.Lets use the os.chdir command again to move down two folders at once,into the Source-Code-Tutorial-master folder and then into the Tutorial-1 folder.os.chdir("Source-Code-Tutorial-master/Tutorial-1")print(os.getcwd()) prints our current directory<jupyter_code><jupyter_output><empty_output><jupyter_text>In this first tutorial folder, we are going to learn how to create a python script and use python to examine a fastq file that has ambiguous degenerate DNA sequences. We will also learn how to call programs in other languages from the command line using python. First, we will use the nano command to edit a file using the code below. You can either copy & paste the code, or type it out yourself.  Lines that start with a  are comments, where lines that do not start with a  are actual code. To start, lets create a python file that will print the contents of the fastq file to the command prompt. Normally in the terminal we can open a text editor like nano to create a file. In jupyter, we can just use a little "jupyter magic" to write a file for us!In this file, anything following a hastag () will represent a comment and the other information is actual code. Type the following command to magically write our code to a file and create the python script:     %%writefile python_script_1.py    with open("example_bad_file.fastq", "r") as file: this line opens an example fastq file        data = file.read() this line reads the file and saves it to a variable called data        print(data)this line prints the data<jupyter_code><jupyter_output><empty_output><jupyter_text>To run a python file from the command line, we would use the command:    !python python_script_1.pyThis will open the fastq file, read the file and save it to a variable. Then it will print the contents to your screen.<jupyter_code><jupyter_output><empty_output><jupyter_text>One of the best parts about jupyter notebooks is that you can even run python code straight from the notebook too! Try just copy and pasting this code into the code block below and you should see the same result!    with open("example_bad_file.fastq", "r") as file: this line opens an example fastq file        data = file.read() this line reads the file and saves it to a variable called data        print(data)this line prints the data<jupyter_code><jupyter_output><empty_output><jupyter_text>Next, we are going to write a second python script that will help us run some code in another programming language ( Perl ) to filter reads that have different lengths of quality scores and sequences.    %%writefile python_script_2.py    import os load the operating system package, os    os.system("perl filter.pl example_bad_file.fastq  > Filtered_example_bad_file.fastq ") use the os.system function to call a perl script from the command line<jupyter_code><jupyter_output><empty_output><jupyter_text>To run this script, type the following into the command line:         !python python_script_2.pyWe should see that the perl filtering works, and we removed a read where the sequence score and quality score  were different lengths in our file.<jupyter_code><jupyter_output><empty_output><jupyter_text>Next, we will make a third python script to run a sample bash script to examine if our newly created filtered file matches the content of the original or it will tell us if they are different.Type the following command:    %%writefile python_script_3.py    load the operating system package, os    import os    use the os.system function to call a perl script from the command line    os.system("./compare_files.sh example_bad_file.fastq Filtered_example_bad_file.fastq")<jupyter_code><jupyter_output><empty_output><jupyter_text>Now lets run the python script using the following command:    !python python_script_3.pyThis will use this other bash file to tell us if the two files are the same, or different - indicating whether or not our trimming command from earlier actually removed any files.<jupyter_code><jupyter_output><empty_output><jupyter_text>Oh No! We don't have the correct permissions to run the bash file. Specific files have permissions (whether you can only read the file, or you can run it, or edit it) - and to change permissions on a file we can use the linux chmod command:    !chmod 777 compare_files.shThis will allow us to have the correct permissions. Run the command below.<jupyter_code><jupyter_output><empty_output><jupyter_text>Now we should be able to run our python script we created and see if you have corrected the permissions errors.    !python python_script_3.py<jupyter_code><jupyter_output><empty_output><jupyter_text>Here you should see some output similar to the output below highlighting that these two files are different from one another:    The file "example_bad_file.fastq" is different from "Filtered_example_bad_file.fastq" Congratulations! The first half of the tutorial is done and you have learned: 1) how to use a jupyter notebook2) how to write files using jupyter3) a few basic basic linux commands4) how to run python files from the command line5) how to use python to run other programs or scripts6) how to use python to read data from files7) how to download files from the internet or Github8) change file permissions &9) navigate the command prompt (moving up or down into folders) using python!In the next section of the tutorial, we will focus more on using python specific packages to process fastq files.  Tutorial 2 - installing python packages and working with sequences To start this second half of the tutorial, we need to move up one directory from the Tutorial-1 folder, and then down into the Tutorial-2 folder. Lets use the os.chdir() function we learned earlier to move up one folder.    os.chdir("..")In programming, two periods means "up one directory".<jupyter_code><jupyter_output><empty_output><jupyter_text>Now that you are in the main directory, use the command again to move into the Tutorial-2 folder. Without me giving you the command, see if you can figure out how to move into the right folder. If you get stuck while programming, googling to try to find the answer is always acceptable!<jupyter_code><jupyter_output><empty_output><jupyter_text>To make sure you are in the correct folder, lets run the following command:    os.getcwd()<jupyter_code>os.getcwd()<jupyter_output><empty_output><jupyter_text>You should see that you are in the following folder:     '/content/Source-Code-Tutorial/Source-Code-Tutorial-master/Tutorial-2'--- Now that we are in the Tutorial-2 folder, we’re going to learn how to install python packages in order to process fastq files.  First we’re going to learn how to trim reads on fastq files and then use a  package called pysam to extract all of the individual sequences out of a fastq file. pysam was developed for working with BAM,CRAM, or SAM formatted genetics files but will work great for our purposes of processing fastq/fasta files too. More details can be found in their documentation here: https://pysam.readthedocs.io/en/latest/api.htmlRun the following code from the command line to install the python package pysam:     pip install pysam<jupyter_code><jupyter_output><empty_output><jupyter_text>You have installed your first package with Pip, the python package manager! You can learn more about pip here: - Next, we need to download a big fasta file that is too big to store on github. We need to make a another bash script to help us download the big file to our google colab notebook:    %%writefile downloader.sh    !/bin/bash    fileid="1A4f8nFITTMGpRIqOdG77j9q4dowiWbZ4"    filename="contaminated_1.fastq.gz"    curl -c ./cookie -s -L "https://drive.google.com/uc?export=download&id=${fileid}" > /dev/null    curl -Lb ./cookie "https://drive.google.com/uc?export=download&confirm=`awk '/download/ {print $NF}' ./cookie`&id=${fileid}" -o ${filename}<jupyter_code><jupyter_output><empty_output><jupyter_text>To run our downloader script, we need to give it executable permissions using chmod again. Do you remember the command we used last time?See if you can figure out how to add the correct permissions to this downloader file on your own. If you can't figure it out, feel free to email Joe at <EMAIL> and he will get back to you =)Dont forget that since this will be a unix command, we need to start with an ! to run unix commands.<jupyter_code><jupyter_output><empty_output><jupyter_text>Now that you have figured out how to add the correct permissions to the downloader file, lets run it using bash.    !bash downloader.sh<jupyter_code><jupyter_output><empty_output><jupyter_text>Now we have our large gzipped file. We need to unzip it using the following command:    !gunzip contaminated_1.fastq.gz<jupyter_code><jupyter_output><empty_output><jupyter_text>Now that our file is unzipped, we can use python to list all of the files in our current directory by combining two commands:    os.listdir(os.getcwd())<jupyter_code><jupyter_output><empty_output><jupyter_text>Now that you can see you have the regular fastq file (contaminated_1.fast)and not the gzipped version, lets copy our filter script into this directory after loading the shutil package using the following commands:    import shutil loads shutil    shutil.copyfile("../Tutorial-1/filter.pl", "filter.pl")     copies the file from its source location, to our current folder<jupyter_code><jupyter_output><empty_output><jupyter_text>Now that we have our filter script, lets use python to run it and save the output to a new filtered file using OS.system    os.system("perl filter.pl contaminated_1.fastq > Filtered_contaminated_1.fastq")    print("Finished processing!")<jupyter_code><jupyter_output><empty_output><jupyter_text>Now we are going to write a large python script using pysam to filter out any reads in our fastq file that have ambigious DNA code, such as K, N, or other degenerate letters using this code that I pasted below. Just remember that each line starting with a hash tag () is a comment that I made, helping to indicate what the lines of code are doing. This code will take a little bit of time to run because this is a big 1.3gb fastq fileWe can also include some jupyter magic with the %time function that allows us to track time metrics of the code:    %time    import pysam    import time    use the first item in the command line following the code as the file to process    filename = "Filtered_contaminated_1.fastq"    specify the output name    out_filename = "clean_Filtered_contaminated_1.fastq"     ambigious = set(['M', 'D', 'R', 'N', 'K', 'Y', 'S', 'B', 'H', '-', 'V', 'W'])    specify a set of good nucleotides    good = set(["A", "C", "G", "T"])    use pysam to open the input file and output file at the same time    with pysam.FastxFile("Filtered_contaminated_1.fastq") as fh, open(out_filename, mode='w') as fout:        loop over each sequence entry in the fastq file         for entry in fh:            set a variable for a bad nucleotide equal to false            found_a_bad_nuc = False             print(entry.name)             print(entry.sequence)             print(entry.comment)             print(entry.quality)            check for ambigous bases that are not A, T, C or G            by looping over every nucleotide within the sequence of the fastq file            for nuc in entry.sequence:                check if the nucleotide in uppercase matches the good set listed above                if nuc.upper() not in good:                    if there is a nucleotide that isn't in the good list, set our variable to true as a flag                    found_a_bad_nuc = True                    stop going over this sequence and move to the next sequence                    break                if we don't find a bad nucleotide                else:                    go to the next step                    pass            after going through a sequence, check if we found a bad nucleotide or not            if found_a_bad_nuc == False:                if we didn't find any bad nucleotides, write the entire4 line of fastq data to the output file                fout.write(str(entry))                add a new line at the end of the entry so the following fastq file starts on its own line                fout.write("\n")            if we did find a bad nucleotide, pass and go to the next sequence            else:                pass<jupyter_code><jupyter_output><empty_output><jupyter_text>Now we have a new file, clean_Filtered_contaminated_1.fastq that has significantly fewer reads.To see the new file this has created, lets use the ls command to print the files in our current directory. Here, we will use some special commands to also look at the sizes of the files in our current directory with the ls command. Since so many reads were removed from the file, it actually changed the size of the file too. Use the following command:    !ls -lah<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05274844790864186, 'fraction_numerical': 0.003942538632346944, 'mean_word_length': 4.288281811646297, 'pattern_counts': {'":': 0, '<': 134, '<?xml version=': 0, '>': 137, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '336090', 'license': 'FSFAP', 'n_tokens_mistral': 5852, 'n_tokens_neox': 5637, 'n_words': 3141, 'path': 'Ipython_Tutorial (1).ipynb', 'repo_name': 'joelwebb/Source-Code-Tutorial', 'size': '22092'}