<jupyter_start><jupyter_text>PIPITS Fungal ITS-dedicated Pipeline* The default pair merge algorithm in vsearch discards 90% of the data. This was observed in other datasets and is believe to be overly conservative. PIPITs offers support for using Pear  is a dedicated alternative Dependencies  || PIPITS ||* Follow instructions provided at: * https://github.com/hsgweon/pipits* Note: all dependencies which require 'sudo' will already be met (i.e. don't bother running those commands... they won't work anyways) || deML ||* Follow instructions provided at: * https://github.com/grenaud/deML || phyloseq ||* conda install -c r-igraph * Rscript -e "source('http://bioconductor.org/biocLite.R');biocLite('phyloseq')"  || FUNGuild ||* download FUNGUild script:* https://raw.githubusercontent.com/UMNFuN/FUNGuild/master/Guilds_v1.1.py || PEAR ||* download at: https://sco.h-its.org/exelixis/web/software/pear/ Citations * <NAME>., <NAME>., <NAME>., <NAME>., <NAME>., <NAME>., et al. (2015). PIPITS: an automated pipeline for analyses of fungal internal transcribed spacer sequences from the Illumina sequencing platform. Methods in ecology and evolution, 6(8), 973-980.* <NAME>., <NAME>., <NAME>., <NAME>., & <NAME>. (2014). deML: robust demultiplexing of Illumina sequences using a likelihood-based approach. Bioinformatics, 31(5), 770-772.* McMurdie and Holmes (2013) phyloseq: An R Package for Reproducible Interactive Analysis and Graphics of Microbiome Census Data. PLoS ONE. 8(4):e61217* <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, Schilling JS, <NAME>. 2016. FUNGuild: An open annotation tool for parsing fungal community datasets by ecological guild. Fungal Ecology 20:241–248.* <NAME>, <NAME>, <NAME>, <NAME>. 2013. PEAR: a fast and accurate Illumina Paired-End reAd mergeR. Bioinformatics, 30(5): 614-620. Last Modified by <NAME> on January 2nd, 2018   Step 1: User Input<jupyter_code>import os

# Provide the directory for your index and read files
ITS = '/home/roli/FORESTs_BHAVYA/WoodsLake/raw_seq/ITS/'

# Provide 
datasets = [['ITS',ITS,'ITS.metadata.pipits.Woods.tsv']]

# Ensure your reads files are named accordingly (or modify to suit your needs)
readFile1 = 'read1.fq.gz'
readFile2 = 'read2.fq.gz'
indexFile1 = 'index_read1.fq.gz'
indexFile2 = 'index_read2.fq.gz'

# Example of metadata file
#Index1	Index2	Name
#AATTCAA	CATCCGG	RG1
#CGCGCAG	TCATGGT	RG2
#AAGGTCT	AGAACCG	RG3
#ACTGGAC	TGGAATA	RG4

## Again, for our pipeline Index1 typically is the reverse complement of the reverse barcode, while Index2 is the forward barcode.<jupyter_output><empty_output><jupyter_text>Step 2: Demultiplex Raw Reads<jupyter_code># Ignore all the 'conflict' errors. The reads are paired so the conflicts are bogus (i.e. it gives a warning everytime an barcode appears in multiple samples, but no pairs are duplicated)

for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    metadata = directory+dataset[2]
    index1 = directory+indexFile1
    index2 = directory+indexFile2
    read1 = directory+readFile1
    read2 = directory+readFile2
    
    # Make output directory
    %mkdir $directory/pipits_input/
    
    # Run deML   ## Note: you may get error involving 'ulimit'. If so, exit your notebook. Enter 'ulimit -n 9999' at the command line, then restart a new notebook.
    !deML -i $metadata -f $read1 -r $read2 -if1 $index1 -if2 $index2 -o $directory/pipits_input/$name

    # Remove unnecessary 'failed' reads and index files
    %rm $directory/pipits_input/*.fail.* $directory/pipits_input/unknown*<jupyter_output><empty_output><jupyter_text>Step 3: Make Sample Mapping File (aka. 'readpairlist')<jupyter_code>import glob, re
for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    
    # Remove Previously Prepended Name (PIPITS wanted something)
    for file in glob.glob(directory+"pipits_input/"+name+"_*"):
        new_name = re.sub(name+"_","",file)
        os.rename(file, new_name)
    
    # Rename files with with extension .fq (PIPITS is PICKY)
    for file in glob.glob(directory+"pipits_input/*.fq.gz"):
        new_name = re.sub(".fq.gz",".fastq.gz",file)
        os.rename(file, new_name)
    
    # Remove Unbinned Reads
    %rm $directory/pipits_input/unknown*        
    
    # Run PIPITS List Prep
    input_dir = directory+"pipits_input/"
    output_dir = directory+name+".readpairslist.txt"
    
    !pipits_getreadpairslist -i $input_dir -o $output_dir -f<jupyter_output><empty_output><jupyter_text>Step 4: Pre-process Data with PIPITS (merge and QC)<jupyter_code>for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    
    input_dir = directory+"pipits_input/"
    output_dir = directory+"pipits_prep/"
    readpairfile = directory+name+".readpairslist.txt"
    
    !pipits_prep -i $input_dir -o $output_dir -l $readpairfile<jupyter_output><empty_output><jupyter_text>Step 4: Extract Variable Region (**User Input Required**)<jupyter_code>ITS_Region = "ITS1"

for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    
    input_file = directory+"pipits_prep/prepped.fasta"
    output_dir = directory+"pipits_funits/"
    
    !pipits_funits -i $input_file -o $output_dir -x $ITS_Region<jupyter_output><empty_output><jupyter_text>Step 5: Cluster and Assign Taxonomy<jupyter_code>for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    
    input_file = directory+"pipits_funits/ITS.fasta"
    output_dir = directory+"PIPITS_final/"

    !pipits_process -i $input_file -o $output_dir --Xmx 20G<jupyter_output><empty_output><jupyter_text>Step 6: Push OTU Table through FUNGuild<jupyter_code>for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    
    # Prepare PIPITS output for FUNGuild
    !pipits_funguild.py -i $directory/PIPITS_final/otu_table.txt -o $directory/PIPITS_final/otu_table_funguild.txt
   
    # Run FUNGuild
    !python /home/db/FUNGuild/Guilds_v1.1.py -otu $directory/PIPITS_final/otu_table_funguild.txt -db fungi -m -u<jupyter_output><empty_output><jupyter_text>Step 7: Import into R<jupyter_code>## Setup R-Magic for Jupyter Notebooks
import rpy2
import pandas as pd
%load_ext rpy2.ipython
%R library(phyloseq)

for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
    metadata = dataset[2]
    
    # Input Biom
    biom = directory+"/PIPITS_final/otu_table.biom" 
    %R -i biom
    %R x <- import_biom(biom)

    # Fix taxonomy table
    %R colnames(tax_table(x)) <- c("Domain","Phylum","Class","Order","Family","Genus","Species")
    %R tax_table(x) = gsub("k__| p__| c__| o__| f__| g__| s__","",tax_table(x)) 

    # Merge Mapping into Phyloseq  
    sample_file = pd.read_table(directory+metadata, keep_default_na=False)
    %R -i sample_file
    %R rownames(sample_file) <- sample_file$X.SampleID
    %R sample_file$X.SampleID <- NULL
    %R sample_file <- sample_data(sample_file)
       
    %R p <- merge_phyloseq(x, sample_file)
                        
    # Save Phyloseq Object as '.rds'
    output = directory+"/PIPITS_final/p_"+name+".pipits.final.rds"
    %%R -i output
    %%R saveRDS(p, file = output)
    
    # Confirm Output
    %R print(p)<jupyter_output><empty_output><jupyter_text>Step 7: Clean-up Intermediate Files and Final Outputs<jupyter_code>for dataset in datasets:
    name = dataset[0]
    directory = dataset[1]
   
    %rm -r $directory/pipits_prep/
    %rm -r $directory/pipits_funits/
    %rm -r $directory/pipits_input/
    
    del_me = directory+name+".readpairslist.txt"
    %rm $del_me<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11494252873563218, 'fraction_numerical': 0.01823226317875545, 'mean_word_length': 4.0265604249667994, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 60, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '182621', 'license': 'MIT', 'n_tokens_mistral': 2754, 'n_tokens_neox': 2569, 'n_words': 702, 'path': 'sequence_analysis_walkthrough/PIPITS_Fungal_ITS_Pipeline.ipynb', 'repo_name': 'buckleylab/Buckley_Lab_SIP_project_protocols', 'size': '7634'}