<jupyter_start><jupyter_text>miRNA-Seq Pipeline  Be sure to install paramiko and scp with pip before using this notebook  1. Configure AWS key pair, data location on S3 and the project information This cell only contains information that you, the user, should input. String Fields**s3_input_files_address**: This is an s3 path to where your input fastq files are found. This shouldn't be the path to the actual fastq files, just to the directory containing all of them. All fastq files must be in the same s3 bucket.**s3_output_files_address**: This is an s3 path to where you would like the outputs from your project to be uploaded. This will only be the root directory, please see the README for information about exactly how outputs are structured**design_file**: This is a path to your design file for this project. Please see the README for the format specification for the design files. **your_cluster_name**: This is the name given to your cluster when it was created using ParallelCluster. **private_key**: The path to your private key needed to access your cluster.**project_name**: Name of your project. There should be no whitespace.**workflow**: The workflow you want to run for this project. For the miRNASeq pipeline the possible workflow is "bowtie2". **genome**: The name of the reference you want to use for your project. Currently only "human" is supported here. analysis_stepsThis is a set of strings that contains the steps you would like to run. The order of the steps does not matter.posible bowtie2 steps: "fastqc", "trim", "cut_adapt", "align_and_count", "multiqc"<jupyter_code>import os
import sys
sys.path.append("../../src/cirrus_ngs")
from util import PipelineManager
from util import DesignFileLoader
from util import ConfigParser

## S3 input and output addresses.
# Notice: DO NOT put a forward slash at the end of your addresses.
s3_input_files_address = "s3://path/to/fastq"
s3_output_files_address = "s3://path/to/output"
    
## Path to the design file
design_file = "../../data/cirrus-ngs/smallrnaseq_design_example.txt"

## ParallelCluster name
your_cluster_name = "clustername"

## The private key pair for accessing cluster.
private_key = "/path/to/your_aws_key.pem"

## Project information
# Recommended: Specify year, month, date, user name and pipeline name (no empty spaces)
project_name = "test_project"

## Workflow information: only "bowtie2" now
workflow = "bowtie2"

## Genome information: currently available genomes: human, mouse
genome = "mouse"

## "fastqc", "trim", "cut_adapt", "align_and_count", "merge_counts", "multiqc"
analysis_steps = {"fastqc", "trim", "cut_adapt", "cut_adapt_fastqc", "align_and_count","multiqc"}


print("Variables set.")<jupyter_output>Variables set.<jupyter_text>2. Create ParallelCluster Following cell connects to your cluster. Run before step 3.<jupyter_code>sys.path.append("../../src/cirrus_ngs")
from awsCluster import ClusterManager, ConnectionManager
## Create a new cluster
master_ip_address = ClusterManager.create_aws_cluster(cluster_name=your_cluster_name)
ssh_client = ConnectionManager.connect_master(hostname=master_ip_address,
                                              username="ec2-user",
                                              private_key_file=private_key)<jupyter_output><empty_output><jupyter_text>3. Run the pipeline This cell actually executes your pipeline. Make sure that steps 1 and 2 have been completed before running.<jupyter_code>## DO NOT EDIT BELOW
## print the analysis information
reference_list, tool_list = ConfigParser.parse(os.getcwd())
ConfigParser.print_software_info("SmallRNASeq", workflow, genome, reference_list, tool_list)

print (analysis_steps)

sample_list, group_list, pair_list = DesignFileLoader.load_design_file(design_file)

PipelineManager.execute("SmallRNASeq", ssh_client, project_name, workflow, analysis_steps, s3_input_files_address,
                       sample_list, group_list, s3_output_files_address, reference, "NA", pair_list)<jupyter_output>#Primary analysis details
#Author: <NAME>
#Date: 2019-07-25 14:05:29

#Reference used:
Reference genome: hairpin_mouse
#Tools used:
FASTQC: 0.11.3
Trimmomatic: 0.36
samtools: 1.9
MultiQC: v1.3
bowtie2: 2.3.3-linux
cutadapt: v1.14



{'cut_adapt', 'multiqc', 'trim', 'fastqc', 'align_and_count'}<jupyter_text>4. Check status of pipeline This allows you to check the status of your pipeline. You can specify a step or set the step variable to "all". If you specify a step it should be one that is in your analysis_steps set. You can toggle how verbose the status checking is by setting the verbose flag (at the end of the second line) to False.<jupyter_code>step="all"
PipelineManager.check_status(ssh_client, step, "SmallRNASeq", workflow, project_name, analysis_steps,verbose=True)<jupyter_output>checking status of jobs...

Your project will go through the following steps:
	fastqc, trim, cut_adapt, align_and_count, multiqc

The fastqc step calls the fastqc.sh script on the cluster
The fastqc step has finished running without failure

The trim step calls the trim.sh script on the cluster
The trim step has finished running without failure

The cut_adapt step calls the cutadapt.sh script on the cluster
The cut_adapt step has finished running without failure

The align_and_count step calls the bowtie2_and_count.sh script on the cluster
The align_and_count step has finished running without failure

The multiqc step calls the multiqc.sh script on the cluster
The multiqc step has finished running without failure


Your pipeline has finished<jupyter_text>5. Display MultiQC report  Note: Run the cells below after the multiqc step is done<jupyter_code># Download the multiqc html file to local
notebook_dir = os.getcwd().split("notebooks")[0] + "data/"
!aws s3 cp $s3_output_files_address/$project_name/$workflow/multiqc_report.html $notebook_dir
from IPython.display import IFrame

IFrame(os.path.relpath("{}multiqc_report.html".format(notebook_dir)), width="100%", height=1000)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07928856382978723, 'fraction_numerical': 0.011469414893617021, 'mean_word_length': 5.190329218106996, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '55619', 'license': 'MIT', 'n_tokens_mistral': 1798, 'n_tokens_neox': 1686, 'n_words': 680, 'path': 'notebooks/cirrus-ngs/miRNA-Seq Pipeline.ipynb', 'repo_name': 'ucsd-ccbb/cirrus-ngs', 'size': '6020'}