<jupyter_start><jupyter_text>About the dataset: The Human microbiome project is a large-scale metagenomic project aiming to describe the microbial diversity associated to the human body. This project collected almost 5,000 samples from 300 healthy individuals, allowing an extensive description of the intra and inter-individual variability of the human microbiome.For this project, the flora from several body sites was investigated: * the oral cavity and oropharynx (saliva; buccal mucosa (cheek), keratinized gingiva (gums), palate, tonsils, throat and tongue soft tissues, and supra- and subgingival dental plaque (tooth biofilm above and below the gum))* the skin (retroauricular creases (behind each ear) and the two antecubital fossae (inner elbows), and one specimen for the anterior nares (nostrils))* the stool* the vagina (vaginal introitus, midpoint and posterior fornix)**For this assignment, we will use 3 stool samples from 1 healthy female individual sampled at 3 different visits.**  Exercise 1: Assembly, mapping and assembly metrics For this exercise, we’ll use the files **ex1_pair1.fastq** and **ex2_pair2.fastq** in the ex1_data folder. As you know, assembly is a memory-intensive computational task. To make sure that the exercise can be performed on your machine, the provided example is a lightweight artificial metagenome composed of 100,000 illumina reads from 5 organisms.These illumina fastq files (**ex1_pair1.fastq and ex1_pair2.fastq**) have been QCed and trimmed, and singletons (reads without a paired counterpart) have been filtered.  Q1: Assemble your small metagenome using Megahit. The usage for this tool is the following:***megahit -1 [pair1.fastq] -2 [pair2.fastq] -o [output_directory]***for macOS users: megahit may give you back an error saying that it cannot estimate the memory. Try fix the issue by adding the parameter -m 9e9 to the command. The tool will create a folder using the name you provided, inside, the final output will be called final.contigs.fa<jupyter_code>#here run your code !<jupyter_output><empty_output><jupyter_text>Q2: Let’s look at your assembly. In a new cell, print the number of contigs that you have created. You can use the method that you like the most. (hint : you can also use IPython grep command to calculate the number of “>” in your file).<jupyter_code># here your code !<jupyter_output><empty_output><jupyter_text>Q3: Now, let’s calculate contig coverage and extract unassembled reads.To do so, we’ll use bowtie2 to map the reads back to the contigs. The usage of the tool takes two steps.  You first need to create a database of your contigs :***bowtie2-build [contigs.fa] [database_name]***then run the mapping command:***bowtie2 -x [database_name] -1 [pair1.fastq] -2 [pair2.fastq] -S bowtie.sam***<jupyter_code># here your code to run bowtie2 database creation!
# here your code to run bowtie2 mapping !<jupyter_output><empty_output><jupyter_text>This will produce a **bowtie.sam** file. A **.sam** file is a standard alignment map file, composed of a header section and an alignement section. More information on .sam file can be found on our favorite web-ressource : [Wikipedia](https://en.wikipedia.org/wiki/SAM_(file_format)).Additionally, the tool will print out a summary of the alignment, where information about the number of read mapped to the contigs can be found. Look carefully at this output and find the % of aligned reads.<jupyter_code># here report the % of aligned reads.
# here add a comment. Is it a good alignment? What could be improved?<jupyter_output><empty_output><jupyter_text>Most downstream tools may not use a **.sam** file but rather a **.bam**. The BAM file format is the binary representation of the Sequence Alignment/Map (SAM). It will store exactly the same data but in a compressed format.To obtain a **.bam** from a **.sam** file, we can use samtools view command: ***samtools view -F 4 -bS [your sam file] > aln.bam***<jupyter_code># here run the command to convert the sam file into bam.<jupyter_output><empty_output><jupyter_text>The next step is to use the sam file to produce a coverage file (**cov.txt**). To do so, you’ll need to use the command:***pileup.sh in=[your_sam_file] out=cov.txt***<jupyter_code># here run pileup to get the cov.txt file<jupyter_output><empty_output><jupyter_text>This cov.txt file is a tab-delimited format file that you can open using excel or a notepad.<jupyter_code># here briefly describe (1-2 sentences) what information did you find in the cov.txt file?<jupyter_output><empty_output><jupyter_text>Once you evaluated the quality of your assembly, you may want to extract any reads that were not assembled for further investigation. To do so, we can use samtools view command.***samtools view -u -f4 [your sam file] | samtools bam2fq -s unmapped.se.fq - > unmapped.pe.fq***This command will extract unmapped pairs of reads into the **unmapped.pe.fq** file and unmapped singleton in the **unmapped.se.fq**.<jupyter_code># here run samtools view to extract unmaped reads<jupyter_output><empty_output><jupyter_text>Exercise 2: Binning contigs using metabat2 and checkMRaw reads from the 3 stool samples from 1 healthy female described above were QC’d and trimmed before being co-assembled using megahit. The output of this co-assembly is available to you as **contigs.fa** in the **ex2_data** folder.<jupyter_code># here, briefly (1-2 sentences)describe the difference between a co-assembly and a regular assembly. 
# Why do you think a co-assembly is valuable in our example?<jupyter_output><empty_output><jupyter_text>After assembly, the reads from the three samples where mapped and bam files generated. These bam files were used to bin the contigs using a tool called Metabat2. The goal here is to generate bins corresponding to the different organisms. You can see the bins in the **bins** folder in the **ex2_data** folder.<jupyter_code># Using bash commands only, print the number of bins generated by metabat2 (in the folder ex2_bins)<jupyter_output><empty_output><jupyter_text>In order to explore the bins created by Metabat2, we can use checkM, a tool aiming to help to assess the quality of bins and genomes recovered from metagenomes. Because checkM doesn’t run on Python 3, but only on python 2 (and therefore would not run on your current environment), we’ll provide you the tool’s output when necessary.First, we run the “checkm unique” command. This command checks each bin and ensures no sequences has been assigned to bins. For most automated binning methods, the assignment of a sequence to multiple putative genomes would indicate a serious binning error.  ![alt text](data/img/checkM.png "Title")<jupyter_code># Look at the github repo for checkM:  https://github.com/Ecogenomics/CheckM.
# Here describe briefly (1-2 sentence) what workflow you would use to assess the quality of the bins as genomes.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0719247467438495, 'fraction_numerical': 0.009551374819102749, 'mean_word_length': 5.248643761301989, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 1, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '637770', 'license': 'MIT', 'n_tokens_mistral': 2006, 'n_tokens_neox': 1886, 'n_words': 962, 'path': 'Assembly.ipynb', 'repo_name': 'bhurwitz33/metag-BE587', 'size': '6910'}