Motivation: The Oxford Nanopore MinION sequencer, currently in pre-release testing through the MinION Access Programme (MAP), promises long reads in real-time from an inexpensive, compact, USB device. Tools have been released to extract FASTA/Q from the MinION base calling output and to provide basic yield statistics. However, no single tool yet exists to provide comprehensive alignment-based quality control and error profile analysis—something that is extremely important given the speed with which the platform is evolving.Results: NanoOK generates detailed tabular and graphical output plus an in-depth multi-page PDF report including error profile, quality and yield data. NanoOK is multi-reference, enabling detailed analysis of metagenomic or multiplexed samples. Four popular Nanopore aligners are supported and it is easily extensible to include others.Availability and implementation: NanoOK is an open-source software, implemented in Java with supporting R scripts. It has been tested on Linux and Mac OS X and can be downloaded from https://github.com/TGAC/NanoOK. A VirtualBox VM containing all dependencies and the DH10B read set used in this article is available from http://opendata.tgac.ac.uk/nanook/. A Docker image is also available from Docker Hub—see program documentation https://documentation.tgac.ac.uk/display/NANOOK.Contact:richard.leggett@tgac.ac.ukSupplementary information:Supplementary data are available at Bioinformatics online. ONT’s technology involves the detection of current changes across biological nanopores through which DNA molecules move. The degree of current change depends on the bases (5-mer) present in the pore at any given time and multiple measurements are made as the molecule advances, resulting in a characteristic ‘squiggle’ plot which ONT’s basecalling software (Metrichor) processes to a nucleotide sequence. The MinION will read both strands of DNA (Template and Complement) and the software will attempt to call a consensus 2D  read. Individual reads (one per file) are output in FAST5 format, an implementation of the HDF5 standard. Two tools—poretools , has been developed which provides a wide range of very useful QC metrics in real-time. As reads emerge, minoTour will align them against a reference but this analysis is currently limited to coverage assessment and variant calling. Without a comprehensive post-alignment tool, analysis can be unnecessarily time consuming and can require specialized programming skills. NanoOK is designed to address this need, providing quick and intuitive alignment-based analysis and quality control of Nanopore runs, facilitating comparison across chemistry, flow cell, base calling changes and alignment tools. NanoOK itself extracts FASTA files, but users can also use third party tools. NanoOK carries out alignments via a number of supported alignment tools and will produce tabular output files from which it also generates graphs and a multi-page PDF report.The Oxford Nanopore Technologies (ONT) MinION is one of the latest of a new generation of single molecule sequencing technologies. Originally revealed at the 2012 Advances in Genome Biology and Technology conference, it arrived in the labs of MinION Access Programme (MAP) members in May 2014. Offering multi-kilobase reads, the MinION attracted interest due to its compact size, USB connection, relatively inexpensive expected purchase price and a streamed mode of operation that enables analysis of data as it generated. Though not out of pre-release testing, the device is in the hands of research groups around the world who are evaluating the performance and suitability of the platform for a wide range of applications including medical diagnostics, environmental surveillance and oretools  and poReoretools  and BWA-oretools , which inanook extract -s MyNanoporeRun -aAlignments are initiated with:nanook align -s MyNanoporeRun -r refname.faFinally, alignments are analysed and plain text machine readable analysis files, graphs and PDF report are generated with:nanook analyse -s MyNanoporeRun -r refname.faNanoOK will create further subdirectories for FASTA or FASTQ files, alignments, analysis files, logs and LaTeX files as it runs.NanoOK expects runs to be organized in sample directories—initially these contain the reads output by the Metrichor software, but NanoOK will add further subdirectories containing analysis and output. There are three simple steps involved in running NanoOK. Firstly, FASTA or FASTQ reads are extracted:Read reference FASTA file and store IDs and lengths.For each set of reads  within pass and fail directories, store length of each read.For each set of reads  within the pass and fail directories, parse the alignment and store details of errors, quality and accuracy.Write a set of analysis files—tabbed plain text files suitable for machine reading or graph plotting.Initiate graph plotting through the R environment.Build a LaTeX report file and then PDF with pdflatex.NanoOK currently supports four alignment tools—LAST (the default), BWA-MEM, BLASR and marginAlign. However, NanoOK facilitates the addition of new parsers by implementing Java classes supporting an AlignmentFileParser interface. The basis for NanoOK’s analysis of sequencing errors  is base-by-base parsing of the alignment strings found in the MAF format files used by LAST. BWA, BLASR and marginAlign output SAM format files, so NanoOK uses a CIGARString class to convert the CIGAR-format alignment strings in SAM files into MAF style strings.When executing the analysis phase, NanoOK performs the following sequence of actions:An individual read may produce multiple alignments to one or more references. In this instance, NanoOK takes the highest scoring alignment and merges this with any alignments for the same reference that map close to the initial alignment. Merging is performed by expanding the co-ordinates of the original alignment to include the nearby alignment; analysis of matches, mismatches and indels is then performed on the combined alignment. In the interests of execution speed, NanoOK supports multithreading at all stages. Users can specify threads using a –t parameter.Supplementary material contains an example NanoOK report for a run of Escherichia coli K12 substr. DH10B produced using an R7.3 flow cell. The report contains data on the whole run, as well as separate sections for each reference . At the run level, data include:Pre-alignment summary—including pass/fail counts and read length distributions for Template, Complement and 2D reads.Alignment summary—including counts of reads aligning to each reference, mean length, aligned bases, coverage.Substitution analysis—like Error kmers—3/4/5-mers occuring before substitutions, insertions and deletions, along with error motif images. Here, we find a high abundance of low complexity kmers, indicative of homopolymer problems, as observed in For each reference:Error analysis—identity, insertions, deletions, substitutions.Identity—read identity histograms, scatter plots of read identity versus length, alignment identity versus percent of read aligned, percentage of read aligned versus length.Coverage—coverage of reference in Template, Complement and 2D reads and GC content of reference.Perfect kmers—analysis of longest perfect sequence without any errors compared with reference.Over- and under-represented kmers—tables and scatter plots of 5-mers in reference versus 5-mers in reads.The NanoOK provides comprehensive alignment-based analysis of Nanopore reads through a simple, easy to use interface. During our progress through the MAP, we have found it to be an invaluable tool for understanding the data emerging from the sequencer as the platform evolves and we believe it will have wide applicability to other groups working with ONT’s technology. The speed of change within the MAP is rapid and requires a tool such as NanoOK to evaluate changes: at the time of writing there have been four flowcell versions, five genomic sequencing kits, multiple versions of sequencer operating software and base caller, and at the recent London Calling meeting ONT announced forthcoming new MinIONs (MK1), flowcell chemistries (R8.0) and a new fast run mode.


K-mer-based methods of genome analysis have attracted great interest because they do not require genome assembly and can be performed directly on sequencing reads. Many analysis tasks require one to compare k-mer lists from different sequences to find words that are either unique to a specific sequence or common to many sequences. However, no stand-alone k-mer analysis tool currently allows one to perform these algebraic set operations.We have developed the GenomeTester4 toolkit, which contains a novel tool GListCompare for performing union, intersection and complement (difference) set operations on k-mer lists. We provide examples of how these general operations can be combined to solve a variety of biological analysis tasks.GenomeTester4 can be used to simplify k-mer list manipulation for many biological analysis tasks.The online version of this article (doi:10.1186/s13742-015-0097-y) contains supplementary material, which is available to authorized users. On the other hand, these new technologies have introduced new bioinformatic problems resulting from short read lengths, a large number of sequencing errors, and a huge amount of data that must be processed. Analysis of genomic data often requires either In recent years, oligomer-frequency-based methods of genome analysis have attracted great interest because they do not require genome assembly and can be performed directly on sequencing reads , 2. ThesOligomer frequency analysis is typically conducted by k-mers (oligomers of length k). The first step involves counting k-mers from raw sequencing reads or assembled sequences and is performed in an analogous manner for all subsequent k-mer analysis methods. Several k-mer counting programs have been developed in recent years, both as part of assembly tools or as separate programs. One of the fastest and most widely used k-mer counting tools is Jellyfish , which rThe second step - k-mer analysis - is task-specific. Analysis of k-mer frequencies has been used to estimate the size of the genome , to deteMany k-mer analysis tasks would benefit from the use of mathematical set operations to find sub- and supersets of k-mers from different nucleotide sequences. Here we present the GenomeTester4 toolkit that aids in both the creation and the modification of k-mer lists. GenomeTester4 can generate lists of k-mer counts from nucleotide sequences and perform basic algebraic set operations - union, intersection and difference (complement) - on these lists. Currently there is no other public toolkit that provides this functionality. We describe the toolkit and provide examples of how to apply this functionality to perform specific biological analyses.The GenomeTester4 software package consists of three programs - GListMaker, GListCompare and GListQuery Fig. . The GLi32 - 1. The full specification of the k-mer list format is given in Additional file The central data structure of all three programs is a binary k-mer list that consists of a small header and an array of k-mers together with their counts. K-mers are encoded as 64-bit unsigned integers with two bits representing each nucleotide. Counts are 32-bit unsigned integers. To reduce memory usage, the list contains only the canonical form of each k-mer. This means that one data entry represents both a k-mer and its reverse complement and is stored using the smaller of the two possible integer encodings. The maximum length of k-mers is thus 32 nucleotides and the maximum possible total count of any given k-mer and its reverse complement is 2GListMaker computes k-mer counts from nucleotide sequences stored in FASTA or FASTQ format. It uses temporary arrays to collect all k-mers from the input file during the reading phase. The arrays are then sorted and the adjacent instances of the same k-mer are counted during the collation phase. Multiple CPU threads can be used if there is more than one file of input sequences.GListCompare performs the basic set-algebraic operations - union, intersection and difference - with two user-specified k-mer lists Fig. . All opeThe union of two lists contains all entries that were present in either of the original lists, intersection contains only the entries that were present in both lists, and complement provides the entries that were present in the first but not in the second list. The precise definition of what counts as the presence of k-mer and what will be the final count in the output list can be specified by rules. By default, applying a union operation results in a list in which the final k-mer count is the sum of the counts from both initial lists, whereas calculating the intersection results in a default list where the smaller of the two counts is reported. Calculating the complement provides the count from the first list by default Fig. .To speed up the generation of lists of sample-specific k-mers from a collection of many genomes we have also implemented a function to find what we term the complement from union Fig. . In thisGListQuery is used to query the counts of k-mers from k-mer lists. The input can be single k-mer sequences from the command line, lists of k-mers supplied in a text file, or other k-mer lists generated with GListMaker or GListCompare. Input k-mers can also be read from FASTA/FASTQ files with a sliding window. K-mers are looked up by binary search, resulting in O(logN) complexity .GListQuery can also search for k-mers that differ from the query k-mer by up to a user-supplied number of mismatches. To accomplish this, it generates all possible mismatched k-mer versions of each query sequence and searches for each of them from the input list.GListCompare and GListQuery do not read lists directly into RAM but instead memory-map these using the Linux mmap system function. This guarantees that only those parts of each file that are being used will be read from disk to RAM. Parts that are never used will not be read and those that were not used recently will be silently cleaned from memory without virtual memory swapping. This allows the system to use lists larger than the available amount of physical memory, albeit with a performance penalty if many look-ups are performed sequentially.GListMaker uses a pool of CPU worker threads to speed up list generation. Each input file is processed in parallel if enough worker threads are available. The sorting and merging of temporary tables into the final k-mer list is also performed using separate worker threads.To perform the basic set-algebraic operations of union, intersection and difference on k-mer lists, we made additional design decisions. The semantics of standard mathematical set operations are defined only by the presence or absence of specific elements in the set. In case of k-mer lists we have richer structure, as for each k-mer there is also the count of its occurrences.To accommodate the occurrence count within k-mer lists, we extended these operations by defining the semantics of presence or absence together with a method to calculate the final count. For all operations, the k-mer is counted as being present in one of the input lists only if its count is equal to or above a minimum cutoff value. After determining the presence in the two individual lists, the standard set operation is then applied. If by the result of the operation the k-mer should be included in the output set, its final count is calculated by predetermined rule and the k-mer is written to the output list.1 - set count value to 12 - set count value to 2add - add up both countssubtract - subtract the second count from the firstmin - smaller of the countsmax - larger of the countsfirst - count in the first listsecond - count in the second listThe default rule for intersection is min, for union add and for difference first.The possible rules are:To avoid the possibility of zero valued counts in the output list, union and difference operations implement a subset of these rules.All bacterial genomes used were downloaded from the RefSeq microbial genomes ftp site  and wereHuman genome build 37.1 was downloaded from the RefSeq genomes ftp site .S. aureus and human chromosome 14 sequencing read datasets were downloaded from GAGE project [ project .Bos taurus genome build 6.1 were downloaded from RefSeq genomes ftp site [ftp site .30 bytes) RAM.All software performance tests were conducted on a CentOS 5.10 Linux server with 32 cores (2.27 GHz) and 512 GiB  in the initial reading phase and each unique k-mer requires 12 bytes (one 64-bit and one 32-bit integer) during the collation phase. As GListMaker does not implement Bloom filter and keeps all k-mers are in memory, singleton sequencing errors consume both memory and disk space. For a more detailed analysis of memory usage, please refer to Additional file The running speed and memory requirements of GListMaker depend on both the total number of k-mers in the input files and the number of unique k-mers. We found that the performance of GListMaker is comparable to that of other k-mer counting tools. Tables The speed of GListQuery depends on the size of the list file being searched. Using a list of 32-mers from a single bacterial genome we obtained a speed of up to 35 million look-ups per minute. Using a 64 GiB union of all NCBI bacterial genomes, the look-up speed was roughly 2.7 million 32-mers per minute.Because the list files are memory-mapped, more parts of the file will be read into RAM as the number of query sequences increases, which also speeds up subsequent look-ups. This guarantees a near-immediate start-up for the case where only a few look-ups are required because the entire file does not need to be read from disk, while also providing near-optimal speeds for a large number of look-ups.We have found that many k-mer analysis tasks require one to combine or partition k-mer lists into sub- and supersets of k-mers. By implementing optimized set-algebraic operations within a single tool, GenomeTester4 allows researchers to significantly simplify k-mer analysis pipelines. Still, because most research questions cannot be implemented using only simple set operations on k-mer lists, we expect that both the initial and final steps of analysis will be performed using custom task-specific tools.Separate k-mer lists were created for each bacterial genome with GListMaker.These lists were recursively combined pairwise with a script MakeUnion.pl (included in GenomeTester4 package) that uses the GListCompare union function.If the input sequence is composed of many smaller files, one can combine GListMaker and GListCompare to generate the final list by trading less RAM usage for longer list generation time. For example, we used the following approach to generate the union of all k-mers in NCBI bacterial genome database (2776 whole genomes):The running time of this operation is given in Table de novo repeats; however, none of these allows the user to use control sequences to find repeats that are specific to certain sequences. In general, we expect that repeats in regions with significant homology share a set of common k-mers. We further expect that most of these k-mers are not present elsewhere in significant numbers. To demonstrate how the set operations implemented in GenomeTester4 can be used to accomplish this task, we present an example of how to find sex-chromosome-specific repeats in a cow genome for fluorescence in situ hybridization. The following procedure was used to restrict the search space by finding a specific set of repeated k-mers:Created separate 16-mer lists from X and Y chromosomes with GListMakerCreated a single 16-mer list from all autosomesCreated subsets of all k-mers whose count was at least 10 in the sex-chromosome lists using the GListCompare cutoff optionSubtracted the autosome list from repeated k-mer lists with GListCompare using the difference function with a cutoff value of 2Subtracted the other sex-chromosome list from the repeated and unique k-mer lists using GListCompare with a cutoff value of 1There are many specific tools that have been developed to find 6.Creating a subset of all k-mers that occur at least 50 times in the chromosome Y list7.Subtracting the list of k-mers of over 50 copies from the list of Y-specific k-mers using GListCompare difference functionThe resulting lists contained 112,387 Y-specific k-mers. We removed those that had more than 50 copies so as to ignore well-known repeats by:In total, we identified 4878 X-specific and 137,868 Y-specific 16-mers.The final steps in our analysis were performed using custom Perl scripts. First, we located the regions in chromosome that had significant over-representation of unique repeated k-mers. Then we grouped these regions by similarity using BLAST alignment. Finally, these regions were aligned against the full genome with BLAST. The more complete protocol with command used is outlined in Additional file Streptococcus, of S. pneumoniae species and of S. pneumoniae strain G54.The detection of bacteria at various phylogenetic levels is often required during medical diagnosis and in both epidemiological and ecological studies. To identify whether a certain bacterium belongs or is closely related to a predefined group of strains, one can find the k-mers that are unique to that group of strains and search for those from sequencing reads of the bacterium of interest. We used GenomeTester4 to generate lists of specific 32-mers from the genus First, we created a union of all k-mers contained in all bacteria in the NCBI database, as described above. During this step we also obtained k-mer lists from all strains of interest.Next, we found the intersection of all lists of the strains of interest. This list contains all k-mers that are present in all bacteria from the set. It is reasonable to expect that many of these k-mers will be present in any new strain as long as it is closely related to any known strain from this set. Because we want to use the difference from union operation for finding unique k-mers, the intersection operation must use the -sum rule (i.e. the counts in the intersection list are sum of the counts of source lists). Finally, we found the difference between the target list and the list of all bacterial genomes using the difference from union option of GListCompare.The running times of GListMaker or GListCompare and the sizes of all lists used in this example are provided in Table GenomeTester4 is a universal toolbox for creating and using k-mer lists from genomic sequences and its computational speed is competitive with other k-mer counting programs. This package is unique in its ability to perform fast set operations and list queries with a user-specified number of mismatches. These routines have proven to be useful in our research and, because of their universal nature, we expect that others may find them to be useful for many possible tasks that require k-mer counting and/or operations with k-mer lists. This makes GenomeTester4 a potentially valuable addition to many k-mer and sequence analysis toolkits.Project name: GenomeTester4https://github.com/bioinfo-ut/GenomeTester4Project (source code) home page: Operating systems: Linux (64-bit)Programming language: COther requirements (when recompiling): GCC version 4License: GNU General Public License version 3.0 (GPLv3)Any restrictions to use by non-academics: noneThe data sets supporting the results of this article are available in the GigaDB repository .


Current volumes of sequence data can lead to large numbers of hits identified on a search, typically in the range of 10s to 100s of thousands. It is often quite difficult to tell from these raw results whether the search has been a success or has picked-up sequences with little or no relationship to the query. The best approach to this problem is to cluster and align the resulting families, however, existing methods concentrate on fast clustering and either do not align the sequences or only perform a limited alignment.A method (MULSEL) is presented that combines fast peptide-based pre-sorting with a following cascade of mini-alignments, each of which are generated with a robust profile/profile method. From these mini-alignments, a representative sequence is selected, based on a variety of intrinsic and user-specified criteria that are combined to produce the sequence collection for the next cycle of alignment. For moderate sized sequence collections (10s of thousands) the method executes on a laptop computer within seconds or minutes.MULSEL bridges a gap between fast clustering methods and slower multiple sequence alignment methods and provides a seamless transition from one to the other. Furthermore, it presents the resulting reduced family in a graphical manner that makes it clear if family members have been misaligned or if there are sequences present that appear inconsistent.The online version of this article (doi:10.1186/s12859-016-1059-9) contains supplementary material, which is available to authorized users. Many sequence analysis methods, such as correlated mutation analysis  rely on The selection of a representative subset of sequences is most simply achieved by calculating the pairwise similarity of the sequences and setting a cutoff, say, 80 % identity, and selecting one member of the pair as a representative. A more complex algorithm has been proposed for protein structure based on finding representatives that are equally spaced, given some distance metric. This worked by applying cycles of culling to remove the proteins with most neighbours (within a given cutoff) until only isolated representatives remain .With large numbers of sequences, two problems arise. The first is purely practical: if the sequences are not aligned the pairwise calculation of similarity can be computationally very expensive and even if the sequences are aligned, with very large families  just the evaluation of the similarity can be a burden. The second difficulty arises less from computational limitations and more from the problem of which criteria should be used to select a representative sequence from a pair or subfamily.The practical step of comparing many sequences has been approached through the use of peptide (or oligonucleotide) counting statistics which have been used as a heuristic for sequence similarity for many years , 4 and hThere are many criteria that can be considered in selecting a representative sequence. Perhaps most important is to avoid errors such as premature truncation or missing segments. Although not strictly an error, many databank search programs are prone to lose positions at the termini of sequences where the degree of similarity tends to be weak and this problem of "terminus shrinkage" can be undesirable especially where it can lead to loss of a long-range (sequence-wise) interactions that can be important in structure prediction .Equally important, but not so easily quantified, is the benefit of selecting sequences that have the best annotation and most importantly, for homology modelling, have a known structure. The existence of a known structure, depending on the sequence database, can be trivially identified as having an identifier associated the the Protein Structure Database (PDB). Without taking the time consuming step of making links to external databases such as the Gene Ontology database, the information included in the sequence title is generally all that is available. From this, however, useful key-words can be recognised such as: "hypothetical" or "mutant" that can be scored and used to down-weight the selection probability for the sequence.In this work, I present a method, based on an existing multiple sequence alignment program , that inThe aim of MULSEL is to produce a small alignment of representative sequences that can be annotated with derived secondary information (such as secondary structure prediction) to inform on the consistency and validity of the alignment. It does this through a series of hierarchical reductions (or condensations) into mini-alignments and at each stage a representative sequence is selected from each mini-alignment to be passed to the next stage of the condensation. This selection process is based on some generic properties of the sequence  but is augmented by user specified criteria, such as quality of annotation  or species or any quality the user thinks important to their study, including the ability to add a bias to include sequences that have been marked by the user, irrespective of any annotation, as being of special interest.In the following sections, the robustness of this approach will be tested and compared to pairwise comparison based methods (BlastClust and CD-hit). To do this requires a set of sequences that contain sub-families that include members reaching to the limits at which sequences similarity can be detected but with the additional constraint that there should be no false connections made between members of different families. Test data of this nature could be constructed by combining sequences from two biologically distinct families. However, it is difficult to find challenging examples with pairs of families that lie at the limits of separation as it is then often unclear whether the two families are distinct .This difficulty was overcome in two ways by generating artificial sequence families and pseudo-artificial families. The first approach was to construct sequences that have an over-representation of one particular amino acid type and the second is to use real biological sequence families but combine each with a family of "anti-sequences" which are the same sequences but in reversed order. . In the first approach, clearly, 20 distinct sequences will remain in the reduced selection whereas in the second, just a sequence/anti-sequence pair will remain.In principle, the hierarchic condensation through mini-alignments can begin with sequences in any order but to reduce the amount of computation required using a full pairwise dynamic programming alignment algorithm , it is m.The operation of the method was tested using the artificial families generated by random mutation of 20 seed sequences, with each starting as a homopolymer of all twenty amino acid types. The data was designed so that all sequences that are enriched in the same amino acid can be aligned with each other but not with a sequence that has been enriched with a different amino acid type.  sequences combined with their reversed “anti”-sequence dopplegangers. If successful, this test should result in just two distinct sequences remaining.span parameter setting of 50, meaning that sequences separated by more than 50 in the input order are not compared.  followed by increasing numbers of mini-alignment/selection cycles, each to a limit of 50 % residue identity. . Each alignment cycle used a a).On the 10 K sequence test data set see , the res1span) be reduced, as any failure to find a match at the early stages will be corrected at a later stage. The times taken for each stage are now greatly reduced, with the peptide presort now becoming the limiting factor in the early stages. Importantly, the number of final sequences has now almost reached the expected 20, and after a final multiple sequence alignment, only 20 family representatives remain (Table 1b).Additional fast sorting stages were introduced by alternating these with sequence alignment stages and with each alignment stage, the threshold above which pairs were clustered was gradually reduced from 90 to 50 % in stages of 10 %. As the initial alignment stages are only considering closely related sequences, their alignment parameters can be constrained to allow inserts of only several positions and the range of adjacent sequences (c).To redistribute the initial load carried by the first peptide presort, this was partitioned using the blocked-diagonal methods described in the “1a and b) in which a similar rate of condensation was observed with both runs reducing the number of subfamilies to 22. The reason why the target number of 20 families failed to be reached was investigated and found, in both cases to be due to two isolated outliers in sequence composition, one of poly-A and the other poly-E, that failed to align with their expected A-rich and E-rich families. This was not due to a failure to be considered but because the larger profiles for each family were now so diverse that the two homopolymers failed to score more than the cutoff.This protocol was tested on the larger test sequence sets of 30 and 90K sequences Table , a and bN2/4, seconds for N thousand sequences) so sequence volumes over 90K would be best calculated initially in parts as suggested above or by using a fast program like cd-hit as a pre-filter. This method follows a similar short peptide based sorting strategy but invests less time in the alignment stage  and as such does not find such remote sequence similarities, reducing the 30K and 90K collections to 173 and 525 clusters, respectively, rather than the target 20  to a manageable size, of 0.3, 2.9 and 35.2 minutes (elapsed time on a laptop) are well within the range that is acceptable for routine sequence analysis. Most of this time is taken by the sequence alignment stages following the peptide pre-sort and, despite the efficiency of the peptide sorting stage, the times still have a quadratic dependency on the number of sequences (ftp://ftp.ncbi.nlm.nih.gov/blast/executables/LATEST). However, blastclust is not intended to reduce the sequence selection so direct comparison is not totally valid. Nevertheless on the same test data, the times required by blastclust  were: 11.9, 110.3 and 876.2 minutes. The latter time (over 14 hours) would not be ideal if repeated runs were required. The number of clusters returned by blastclust were in excess of 1000, somewhat above the expected 20. Although the parameters had not been adjusted for the degree of sequence divergence in the test data, even if this were improved, the heavy computation time makes the method less attractive.An alternative approach to clustering large numbers of sequences takes advantage of the fast search and alignment algorithm in the blast program combined with single-linkage clustering  test data set, the effects of changing the peptide length used in sorting and the reduction in the number of distinguished amino acids (referred to as ‘softening’ above) were investigated. The results in Table Two parameters remain to be tested which are: the cutoff on the minimum score below which a pair are not stored and the number of top matches retained per sequence. The nature of the test data is not suited to testing the first as homopolymers can only ever score 1 and tests on the second indicated that saving just the top hit may be sufficient but that the time saved is not significant. These parameters were also tested on real sequence data (below) but had little effect on the quality or execution time of the algorithm (data not shown). As such, they were kept at their ‘default’ values of 3 top hits held per protein with any score under 10 discarded.Tests were carried out using the data generated from sets of native sequences combined with their corresponding reversed “anti-sequence” partner, or doppleganger . AlUsing the PFAM family PF00072 , 24, whiThe massive reduction of sequence numbers by MULSEL to a few representatives, starting from several thousand sequences, is an encouraging test but does not represent how MULSEL would normally be used, which is to halt at around 100 sequences and then perform a full multiple sequence alignment. The previous tests were therefore rerun with a condition to stop when the reduced number of sequences fell below 100 and the sequences aligned. The resulting sequence order was then examined to see how cleanly the true sequences had partitioned from their reversed dopplegangers. In all-bar-one of the five runs, a clean division between true and anti-sequences was obtained.  which has often been adopted for test purposes , 26. A sThe selected sequences were combined, forming a collection of just over 70 K sequences which was reduced again down to a collection of 1665 sequences after a first pass at the 40 % level and cycles at this level were repeated until the number of sequences was further reduced. Normally, a family of up to 100 sequences would be retained so as to preserve a good spread of variation but to reduce the size of the printed alignment (while keeping a font-size that remains legible), the family was reduced until it reached 14 at which point they were aligned  . With juFor a given peptide length , a binary tree is created in which each overlapping peptide in a sequence (encoded by a unique numeric identifier) is allocated to a leaf-node in a binary tree. Unlike algorithms that allocate space for every possible peptide, a binary tree only allocates space for observed peptides and is therefore not limited by the length of the peptide.For simplicity, a fragment of sequence will be referred to as a “peptide”. However, the algorithms described below treat all characters equally using the alphabet of single-letter amino acid codes. This means that the RNA base, uracil (U) should be encoded as ‘T’. The smaller nucleic acid alphabet will generally require a longer “peptide” length to behave in a similar way to proteins.L log(L) time in the length, L, of the sequence . Ideally, this value should be stored for all pairs to be used later in clustering but for a large number of sequences, N2−N)/2 storage requirement. Firstly, the simple step of applying a cutoff to discard low-scoring pairs was used, and secondly, only the top M similarities were held for each protein, giving an N×M requirement, with M typically just 3. The additional computational complexity of resorting each list of hits every time a new pair was evaluated was treated in a similar way to the peptide sorting with the use of a binary tree construct. . These ordered pairs, beginning with the highest, were then used to grow clusters by a single-linkage cluster algorithm  which rLists of sequences extracted from a databank search are seldom in random order, with adjacent sequences commonly having some similarity. To capture this, adjacent sequences in the input list were given a small bonus to their pairwise score that helps to preserve the original sequence order in the absence of any clear indication to re-sort it.N) as it is possible for a pair of sequences/profiles to be aligned at each stage, giving a maximum of 2N).Given a list of sequences roughly sorted by peptide composition, the elimination of redundant sequences followed the strategy outlined in the original MULTAL multiple sequence alignment program. As with the peptide pre-sort described above, this calculates a pairwise sequence similarity and aligns those joined by single-linkage clustering. However, as the sequence order is now sorted, only relatively adjacent sequences need to be compared and aligned only if they have a similarity above a predefined threshold. This process generates a series of small local multiple sequence alignments of, typically, between 2 and 10 sequences from which a representative sequence can be selected to generate a reduced list of sequences. As outlined in the Introduction, a number of criteria were employed in this selection. , where d is the deviation from the target length. The penalty has the value 5 at a deviation of 12 residues rising to 9 by 100 residues. When there is a specific seed sequence, this score was modified to give half the penalty value to sequences that are longer than the seed. This bias directs the sequence selection away from fragmentary sequences and helps reduce the “terminal shrinkage” problem mentioned in the introduction.If the input includes a sequence of special interest, such as a sequence for which a model or prediction is to be made, this can be identified by including the string “SEED” in its code. (That is this is the string following the “>” character at the start of a sequence entry). The sequence selected from the mini-alignment is then biased to keep a close match to the seed-sequence length. Otherwise, the selection is biased towards a sequence close to the average length of those in the mini-alignment. This was implemented as penalty score, Additional biases were extracted from both the sequence identification code and other information contained in the annotation. A strong bias (-60) was added for selection of sequences with a PDB code and a lesser bias (-20) for sequences from UNIPROT. A strong bias was added against sequences that had the words: “mutant” (+40) or “fragment” (+50) in their titles and smaller penalties for the words “probable” (+1), “precursor” (+2), “uncharacterised” (+5) and “hypothetical” (+5). If there is an identified seed sequence it is, of course, highly desirable to retain it in the selection and a large penalty (-100) was assigned. Other sequences that may have constituted a seed alignment can be identified by the inclusion of the (lower-case) “seed” key-word in their code and these were also biased for retention (-50).The units of these penalties are the number of common peptides (unique N-tuples) counted between two sequences. This will be slightly length dependent, for example two sequences of length 3 have a score of 1 and as the sequences become longer, the number of common peptides encountered by chance will increase but because of recurring peptides, not as fast as say, a full alignment score. However, the biases are all imposed within the context of a mini-alignment in which the sequences are all close in length.After the summation of all the penalties, the sequence with the lowest (best) score is written to the new sequence collection. The whole process is then repeated on this reduced collection, either with or without a reduction in the cutoff threshold that determines the size of the mini-alignments.The starting points for the generation of test data were twenty sequences consisting of just a single amino acid type. Each sequence was then successively substituted with all 26 letters and any non amino acid 1-letter codes  were replaced by the gap character (‘-’). When the mutated sequence had less than 1/3 of the starting amino acid (33 % is close to the limit that can be detected by sequence similarity), then substitutions were made with the original type until the composition was restored to over 1/3. An example of the sequence test data and a family alignment is shown in the Additional file Each sequence was assigned a random number which was used to shuffle their order. Collections of 10, 30 and 90 thousand test sequences were generated and their gap-characters removed. Each sequence collection was then passed to MULSEL and a variety of parameter options tested with both the quality of the clustering and the computation time being monitored. .To assess the speed and quality of the method, the test data described above has the advantage that it contains twenty separate families each enriched with a character in the single-letter amino acid alphabet. As these were limited to no less than 1/3 of their starting identity, they should all align within a family but not between families. If MULSEL has been successful in uniting all members of a family, then at the end, a count of sub-families will equal twenty. The rate of clustering can also be followed by performing this count at intermediate stages.Having reduced a large sequence collection to a small multiple sequence alignment, it is important to gain a clear overview of the quality of the alignment and including derived features can help in this assessment. Derived features for different types of sequences are described below while the overall quality of the alignment is aided by colouring amino acids as specified by a physico-chemical based colouring scheme  and for A good indication of a protein’s structure and a guide to the quality of the alignment can be gained from their predicted secondary structure. For this the psipred program was used  but rathFor transmembrane (TM) proteins, psipred predictions were made as described above, but in addition, TM segments were predicted using the memsat program . AlthougRNA secondary structures were predicted using the RNAfold method from the Vienna Package . AlthougMany bioinformatics based research projects either begin or include a search of the sequence databanks with one or more query sequences of interest. With the huge size of the current sequence collections, the number of sequences of potential similarity identified on such searches can be enormous, typically in the range of 10s to 100s of thousands of sequences, depending on the query and the search parameters.Faced with such a volume of data it is often quite difficult to tell from the raw data whether the results make sense or whether, through profile drift on iterated searches, they may include sequences with little or no relationship to the query. The best approach to this problem is to cluster and align the resulting families and some methods that perform this task have been discussed above. However, existing methods concentrate on fast clustering and either do not align the sequences or only perform a limited alignment, leaving a still considerable task to align and combine the clusters to represent the full range of family diversity.In this work a method was presented that combines fast peptide-based pre-sorting with a following cascade of mini-alignments, each of which are generated with a robust profile/profile method. From these mini-alignments, a representative sequence is selected, based on a variety of intrinsic and user-specified criteria that are combined to produce the sequence collection for the next cycle of alignment. For moderate sized sequence collections (10s of thousands) the method executes on a laptop computer within seconds or minutes, so allowing various parameters and cutoffs to be tested in repeated runs.An artificial sequence collection was generated under a random residue substitution model in which 20 distinct equal sized families were evolved, each of which had a bias towards one amino acid type. The evolution of these sequences was constrained to maintain a minimum 33 % bias towards their starting homopolymer which is sufficient to allow alignment within a family but prevent alignment between families. Therefore, irrespective of the number of starting sequences, the final result should be just 20 sequences: one from each family. This metric was used to evaluate both various parameter choices for the current method and other methods.On real sequences, the resulting sequence collection can be reduced to any extent but one would typically aim for between 10 and 100 sequences that can then be aligned by almost any multiple sequence alignment program. Having a small number of sequences also means that secondary analysis programs can then be applied easily, even when they are computationally demanding. Examples were provided for three families of differing type and size, from a very large globular protein family of over 350,000 members to a small RNA sequence family of just 400 sequences.α and β secondary structure, transmembrane segments and (for RNA) the base-paired stem-loops. For the TM-protein example, the TM-segments were used to improve the final alignment of the reduced family and for the RNA family, the alignment produced by the search program was retained through the reduction process.For each of these applications, visualisation of the alignment was produced using derived structure predictions, including The MULSEL program described here bridges a gap between fast clustering methods with little or no capacity to align diverse sequences (being limited by the pairwise comparison of single sequences) and any number of slower multiple sequence alignment methods that use profile comparison but make no reduction in the number of sequences.It would be possible to combine existing methods, such as CD-HIT  or MAFFTBy contrast, the current method provides a seamless transition from one to the other and furthermore, presents the resulting reduced sequences in a graphical manner that makes it clear if members have been misaligned or if there are sequences present that appear inconsistent. In this situation, the method is fast enough that it can be easily re-run after manual pruning or with a more restrictive set of parameters.All data and program source codes can be found in the Additional file


Genomic sequence alignment is an important technique to decode genome sequences in bioinformatics. Next-Generation Sequencing technologies produce genomic data of longer reads. Cloud platforms are adopted to address the problems arising from storage and analysis of large genomic data. Existing genes sequencing tools for cloud platforms predominantly consider short read gene sequences and adopt the Hadoop MapReduce framework for computation. However, serial execution of map and reduce phases is a problem in such systems. Therefore, in this paper, we introduce Burrows-Wheeler Aligner's Smith-Waterman Alignment on Parallel MapReduce (BWASW-PMR) cloud platform for long sequence alignment. The proposed cloud platform adopts a widely accepted and accurate BWA-SW algorithm for long sequence alignment. A custom MapReduce platform is developed to overcome the drawbacks of the Hadoop framework. A parallel execution strategy of the MapReduce phases and optimization of Smith-Waterman algorithm are considered. Performance evaluation results exhibit an average speed-up of 6.7 considering BWASW-PMR compared with the state-of-the-art Bwasw-Cloud. An average reduction of 30% in the map phase makespan is reported across all experiments comparing BWASW-PMR with Bwasw-Cloud. Optimization of Smith-Waterman results in reducing the execution time by 91.8%. The experimental study proves the efficiency of BWASW-PMR for aligning long genomic sequences on cloud platforms. Bioinformatics involves the biological, genomic, statistics, mathematics, and computer science disciplines of study. Analysis of genomic and biological sequence data is one of the important parts of bioinformatics . The ana(1)Short read aligners: used to align genomic sequences whose read length is between 32 bp and 200 bp, that is, BWA  and Bowt(2)Long read aligners: used to align genomic sequences whose length is greater than 1000 bp, that is, Illumina.Genomic sequencing data is obtained from the developed Next-Generation Sequencing (NGS) technologies . Most of the genomic data available consists of millions of genomic sequences of short reads . With thExamples of short read aligners include FASTA , BLAST , BLAT 88, SOAP , and BurExisting bioinformatics applications require both management of huge amounts of data and heavy computation for analysis. Nearly 3 billion US dollars and 10 years were required to produce the initial human reference genome containing about 3.5 billion base pairs. The latest developments in sequencing technologies available produce enormous amount of data in terms of gigabytes per run . In NGS Using cloud platforms, one can solve the storage and computing problems that exist in gene sequencing. Cloud computing platforms provide flexibility, scalability, and on-demand access to resources. Moreover, adoption of cloud computing technologies eliminates the costs incurred in establishing, maintaining large physical storage systems, and computing clusters. Cloud users pay for the utilized storage and computing resources without worrying about maintenance, availability, and reliability related issues. Although cloud computing provides scalable and flexible infrastructure, parallel computing models on cloud platforms/infrastructure are required to achieve the desired goal of analyzing gene sequences. One such framework for the cloud, called MapReduce model , was intThe Hadoop framework  adopts aNext-Generation Sequencing (NGS) tools like CloudAligner , Cloud BAll the existing long read aligners for cloud environments consider the Hadoop framework. Genome alignment is an iterative process and Hadoop incurs considerable performance overheads when iterative applications are hosted on the framework , 35. TheOptimization of Smith-Waterman in the BWA-SW long read sequence alignment.A custom MapReduce framework to support the required computations for long sequence alignment.Parallel map and reduce workers execution strategy.Parallel execution of the map and reduce functions at worker nodes.Therefore, in this paper, we present Burrows-Wheeler Aligner's Smith-Waterman Alignment on Parallel MapReduce (BWASW-PMR) to perform long read alignments on a cloud platform. BWASW-PMR adopts the BWA-SW presented in  to perfoThis paper is organized as follows. q0 and q1 represent two genomic sequences obtained from the seed matches. The Smith-Waterman algorithm computes the similarity matrix score initially. Using the similarity matrix and backtracking technique, optimal alignment is obtained. Let 𝒳 represent the size of q0 and 𝒴 represents the size of q1. For sequence q0, there are 𝒳 + 1 possible prefixes and, for q1, there are 𝒴 + 1 possible prefixes including the empty sequence. Let qseq denote a prefix of 𝒴 characters and qseq[𝒳] represents the 𝒳th character of a sequence qseq. The similarity score between prefixes q0 and q1 is represented as qa,b. The similarity matrix is denoted by Z and its size is . The first row and column of Z are initialized to 0. The remaining elements of Z ) are computed usingR is a function providing values of exact match or mismatch; that is, if q0[a] = q1[b] then R = Xi ; otherwise, R = Mi  (if characters among sequences q0 and q1 are unique). I and J represent the matrices in accordance with the affine gap model. The matrices I and  J are used to determine the gaps and are defined asGE and GF represent the first and successive gap penalty.Burrows-Wheeler Aligner's Smith-Waterman (BWA-SW) Alignment relies on SW algorithm to align the seed matches of sequences. Let q0 and q1. The backtracking begins with a cell in Z that holds the highest score and proceeds till a zero valued cell is reached. Smith-Waterman algorithm provides optimal alignments and similarity scores. It must be noted that computation of matrix Z is bounded by the running time of the slowest Z task due to the dependencies it exhibits.Backtracking algorithm is used to find the optimal alignment between Q and the reference sequence R. A prefix directed acyclic word graph (Prefix  DAWG) is built using sequence Q. prefix  trie is built using the sequence R. The prefix trie is a tree representation of sequence R. The tree is constructed by concatenating all edge symbols from any node in the graph as a route to the root node providing a unique string. The unique string obtained is a substring of the sequence R. Each node of the tree is represented using suffix  array  interval. Traversing from nodes generates strings that are lexicographically sorted. A node in prefix  trie represents a string. The suffix  array  interval  of the node and the string it represents are considered to be equivalent to each other. From prefix  trie, nodes exhibiting identical or similar suffix  array  interval are collapsed to form Prefix  DAWG. Each node of Prefix  DAWG established represents one or more substrings of R. prefix  trie, Prefix  DAWG, and suffix  array of the input string “BANANA$”.BWA-SW algorithm constructs a full-text index in minute space (FM-index)  of the qX) and DG(X) represent two functions that are used to form prefix  trie and Prefix  DAWG graph. In the BWA-SA algorithm, PT(R) and DG(Q) are initially computed. Let a be the root node of PT(R), and b represents the root node of DG(Q). The best score between the sequences Q and R is computed using a dynamic programming mechanism. Initialize Eab = Fab = Kab = null considering the root nodes of PT(R) and DG(Q). For each of the parents, that is, ap, of the ath node in DG(Q), computation of Fab∣ap, Kab∣ap, and Eab∣ap is considered. The computation of Fab∣ap isgo is the gap open penalty and ge is the gap extension penalty. Kab∣ap is computed usingbp is the parent of the bth node in PT(R). The computation of Eab∣ap is achieved using O represents the score between the symbol on edge  and . The computation of Eab, Fab, and Kab is defined as a′ = argmaxap∈ar(a)Eab∣ap and ar(a) is a set containing parent nodes of a. Variable Eab represents the best matching score between the substrings, that is, substring a and substring b. Consider Eab > 0 when the substrings b and a match. It is known that the Smith-Waterman algorithm provides accurate alignment results but requires large computation time. To optimize the computations, the reverse postorder traversal scheme on DG(X) and PT(X) is adopted. The dynamic programming mechanism in the BWA-SW algorithm enables identifying the seed matches of the genomic sequences. Based on the partial matches, the Smith-Waterman Alignment is considered to the extended matches. seed  interval  pair, that is, , is formed when the best score, that is, Eab, is high and suffix  array  interval size of b is within the threshold set. The seed matches are derived from seed  interval  pairs by analyzing the suffix array of sequences Q and R. Matching of the extended seed matches using Smith-Waterman algorithm is considered only when high matching scores are observed.Let PT, where kr is a key and vr contains the overlapping offset data. The key value pair of Q is represented as , where kq is a key and vq is the query sequence. In each of the w map workers, sequence Q is further split into Q′ chunks and stored in the local memory available. Alignment is performed using the BWA-SW algorithm considering R′ and each Q′ in a parallel fashion utilizing the p cores. The Smith-Waterman algorithm in BWA-SW is optimized by a parallelization technique to reduce execution time. Let 𝒯MAP represent the average execution time of the w map worker nodes. The map workers after computation provide alignment locations between Q and chunk R′ along with the score. Multiple alignment locations and scores  are stored in the temporary cloud memory as list. The map function of BWASW-PMR can be defined as map, ) → list. Reduce worker nodes w obtain intermediate data, that is, list, to perform the shuffle and sort function. In the reduce phase, aggregation of all alignment locations, that is, list(vd), that are nonredundant and nonoverlapping is considered. The reduce operation can be defined as reduce) → list(vd). Let 𝒯REDUCE represent the average time required by w reduce worker nodes to perform the shuffle, sort, and reduce computation. The total makespan of the BWASW-PMR cloud platform to align the sequence Q against R is computed as Let Q and chunk R′ are computed. The required data, that is, Q and chunk R′, is obtained from the cloud memory. Let 𝓉Get_Data_M represent the time taken to obtain the data. To reduce computation time using parallel computing techniques, the genomic sequence Q is split into Q′ chunks. Let 𝓉QI represent the time taken to split sequence Q into Q′ chunks. Sequence alignment considering one chunk of Q′ and R′ is performed using the BWA-SW algorithm. DG(Q′) and PT(R′) are initially constructed. The seed matches of sequences Q′ and R′ are computed using a dynamic programming mechanism. The seeds computed are extended to ensure rule alignment of the genomic sequences using the SW algorithm. To reduce computation time, parallelization of the SW algorithm is considered in BWASW-PMR. The parallelization technique to optimize computation is discussed in the latter subsection.In the map phase of the BWASW-PMR cloud platform, alignment locations and corresponding scores between 𝓉BWASW represent the time taken to align Q′th chunk and R′ using the BWA-SW algorithm. The total time taken to align the total Q′ chunks is (Q′ × 𝓉BWASW). As p computing cores are available with each worker node, the parallel computation of p number of chunks of Q is possible. The computation time considering all Q′ chunks and utilizing p cores is defined as ((Q′ × 𝓉BWASW)/p). The alignment locations and scores obtained are stored in the cloud for reduce workers. The time taken to store this data per map worker is represented as 𝓉Store_Data_M. The makespan of the wth map worker node can be defined asLet w map worker nodes is defined asThe average execution time of all the w reduce worker nodes and w map worker nodes simultaneously. This parallel initialization mechanism enables reducing the total makespan 𝒯. In the reduce phase, the shuffle function obtains the intermediate data (produced by the worker nodes) from the cloud storage. The alignment locations obtained from the intermediate data are sorted based on the offset data. The time taken by the wth reduce worker node to obtain the intermediate data and perform shuffle and sort operations is represented as 𝓉Get_Data_R. The reduce function in BWASW-PMR is used to aggregate the alignment locations. The overlapping and redundant alignments are neglected. Parallelization of the reduce function is achieved by utilizing all p computing cores available with worker nodes. Let 𝓉Fn_R/p represent the time taken to compute the reduce function utilizing the p computing cores. The reduce function provides the alignment results between sequences R and Q that are written to cloud storage for further analysis. Let 𝓉Store_Data_R represent the time taken by the wth reduce worker node to write alignment results into the cloud storage. The makespan of the wth reduce worker node can be defined asw reduce worker nodes, that is, 𝒯REDUCE, is defined as The master node in BWASW-PMR initializes fined as TREDUCE=∑Ew.Using , the tot𝒯 (the computing time or makespan of BWASW-PMR) depends on the computation time of map and reduce worker nodes. The core alignment steps  are performed by the map worker nodes, that is, 𝒯REDUCE ≪ 𝒯MAP. The time taken to initialize the map and reduce computing clusters based on VMs, that is, 𝒯VM_Config, is dependent on the cloud platform considered for deployment and can be neglected. Therefore, it can be stated by the total makespan: From , it can 𝒯.Optimizing the SW algorithm in BWA-SW is a possible solution to reduce the total makespan Z, requires the maximum time. Adopting a parallelization technique for computation of Z is considered in BWASW-PMR. Let us consider a query sequence Q and reference sequence R, whose sizes are 4 and 8, respectively. Therefore, the matrix Z to be computed is of size  and is shown in Z make the parallelization technique difficult to implement. For computation of Z3,4 , is considered as the reference R. Query sequences considered are obtained from the influenza virus database [Optimization of the SW algorithm in BWA-SW aligner for BWASW-PMR is a novel approach considering cloud deployment. To analyze performance of the optimized SW algorithm, comparison with the standard SW algorithm  available within BWA-SW is considered. The optimization is achieved using a wavefront parallelization technique. The optimized version of SW is hereafter referred to as “SW-Optimized” in this section. Uniform SW parameter settings  are considered for analysis. For analysis, the sequences database . The queExecution time of SW-Optimized and SW algorithm is noted for the four alignment pairs described. The results obtained are graphically shown in  Saccharomyces cerevisiae S288c) is considered for evaluation [To evaluate the performance of BWASW-PMR, comparison with Bwasw-Cloud is considered. Deployments of Bwasw-Cloud and BWASW-PMR are considered with one computing node . BWASW-PMR is deployed on the Azure cloud platform. Bwasw-Cloud is designed using the Hadoop framework. Apache Hadoop & YARN 2.4.0 version is used in the deployment of the Bwasw-Cloud. Uniform configurations of the computing nodes are considered in the deployments. Bakers yeast genomic database , consisting of approximately 250 million bp, is considered for evaluations. Overlapping of 10 k is considered. The query sequence segments are obtained from the Homo sapiens chromosome 1 segment reads. The length of the queries is varied (based on the read lengths) in terms of 1000 bp, 5000 bp, and 10000 bp. A total of 150 reads per length are considered. Generated log files obtained after the execution are studied to derive observations and results. The total makespan observed, that is, 𝒯, is shown in This section discusses public cloud deployments and performance analysis of BWASW-PMR against Bwasw-Cloud. Deployment of BWASW-PMR on the Amazon cloud and Azure cloud is possible. Here, deployment of BWASW-PMR on the Azure cloud is considered and presented. The deployed BWASW-PMR considers A3 VM instances. Each A3 VM instance consists of 4 computing cores, 7 GB of RAM, and 120 GB of local hard drive space. The deployment of Azure cloud consists of one master node and four worker nodes. HDInsight enables the deployment and provisioning of Apache Hadoop clusters on the Azure cloud platform . Apache 𝒯REDUCE when compared to Bwasw-Cloud. To prove efficiency in adopting a parallel map and reduce execution environment for the BWASW-PMR, makespans of the tasks executed at the map and reduce worker nodes are noted. The results obtained are shown in Detailed analysis of the experimental data reveals parallel execution of map function and SW optimization aid in reducing map makespans. The major alignment computations are carried out in the map phase, considering both BWASW-PMR and Bwasw-Cloud. An average reduction in the map phase makespan across all the experiments achieved by BWASW-PMR against Bwasw-Cloud is about 30%. The accumulation of the results, that is, aggregation of the alignment locations, is carried out in the reduce phase. No major computation is carried out in the reduce phase of BWASW-PMR and Bwasw-Cloud. The parallelization of the reduce function adopted in BWASW-PMR achieves an average reduction of 9.3% in The SW optimization considered in BWASW-PMR reduces execution time of sequences to be aligned based on the SW algorithm. The experimental study and the results obtained prove that BWASW-PMR is capable of aligning long sequences using the cloud computing platform.In this section, a comparison of BWASW-PMR with existing cloud aligners for long sequence reads is presented. Comparisons with Bwasw-Cloud , MapReduSequencing and analyzing of genomic data are important processes in bioinformatics. Rapid developments of the NGS technologies produce humongous amount of genomic data. For analysis of genomic data, alignment tools are used. The alignment tools are classified as short read type and long read type. The existing sequence aligners predominantly consider short read genomic sequences and lacking of support in cloud environment. These aligners exhibit deficiencies in the alignment of long sequence genomic data that are currently generated using NGS technologies. In NGS sequencers, users are required to wait for sufficient resources to become available in which the time needed to complete processing becomes unpredictable. More data is increasingly being generated which leads to serious issues in storing and processing. The existing long read aligners that adopt the cloud platform for computation suffer from drawbacks that are discussed in this paper. In this paper, we combined cloud infrastructure and MapReduce framework together as a solution to support long read sequence alignment. We proposed BWASW-PMR cloud platform to align long read sequences. The BWA-SW algorithm is adopted for long sequence alignment in BWASW-PMR cloud platform. Optimization of the SW algorithm and a Parallel MapReduce execution strategy are considered in BWASW-PMR. Parallel execution of the map and reduce functions is adopted to maximize resource utilization of the VM based cloud computing platform and reduce makespans. This paper highlights comparison of the proposed BWASW-PMR with the existing systems for long sequence alignments. The experiments presented have proven the efficiency of the optimized SW algorithm. Moreover, comparison with state-of-the-art Bwasw-Cloud for long sequence alignment is presented throughout the experimental study. The results obtained indicate significant improvement considering BWASW-PMR when compared to Bwasw-Cloud. Our proposed BWASW-PMR is of use to the genomic community to support the required computations for long sequence alignment efficiently. The parallel executions of map and reduce phases along with SW optimization are the main contributing factors to the speed-up observed in this study.In the future, we propose to undertake optimization of the BWA-SW algorithm due to its memory hungry nature and also accelerating BWA-MEM algorithm of Burrows Wheeler aligner  on diffe


Identifying large-scale structural variation in cancer genomes continues to be a challenge to researchers. Current methods rely on genome alignments based on a reference that can be a poor fit to highly variant and complex tumor genomes. To address this challenge we developed a method that uses available breakpoint information to generate models of structural variations. We use these models as references to align previously unmapped and discordant reads from a genome. By using these models to align unmapped reads, we show that our method can help to identify large-scale variations that have been previously missed. These mechanisms rely on the assumption that the reference is highly similar to the sample genome. As the reference being used is not an accurate representation of the cancer genome(s), alternative strategies that can use references that represent large-scale structural variation are needed. This paper introduces one such method that uses prior information about known characteristics of cancer genomes to inform a search strategy, which allows for a more efficient mapping of reads against alternative references.Cancer genomes are diverse, and often differ considerably from their germlines. This means that the standard One of the problems with cancer genomes is that they exhibit a high degree of structural variation from the germline. Genomic structural variation is defined as alterations to the genome sequence such as duplication, copy number variation, inversion or translocation . While tin situ hybridization (FISH), or spectral karyotyping (SKY), and associated with disease phenotypes. These large-scale chromosomal aberrations are rare in the population generally  and are often associated with severe disease phenotypes. However, the number and complexity of these large variants can be high in tumor genomes (Prior to the advent of high-throughput sequencing (HTS) technologies, microscopic methods enabled the identification of cancer structural variation at the level of chromosomal aberrations. Large insertions, deletions or translocations could be identified in a karyotype using Giemsa staining, fluorescence- genomes –4.In a number of cancers these microscopic levels of structural variation are clinically significant markers of tumor type and malignancy. Known variants can be used to stratify a patient's disease as in multiple myeloma with recurrent translocations between chromosomes 4, 11 and 14 , while oAs sequencing has become a common method of identifying individual variants in both clinical and research labs, identifying large-scale variants from HTS data alone is increasingly important. There are still a number of issues in identification of large-scale variants in short-read sequence data. The first issue is due to the small size of reads relative to the variation. The current generation of HTS technologies were developed to enable the rapid sequencing of entire genomes through the parallel sequencing of overlapping short-reads . In pairIn large-scale variations a breakpoint and recombination occurs at a potentially fragile location on the chromosome, altering the sequence. A read-pair generated from this genome can span the breakpoint , or result in a ‘split’ read where the beginning and end of the read align to different locations. However, due to computational limitations inherent in sequence alignment many reads that could identify these breaks may not be mapped to the reference ,14.Complicating the already difficult task of identifying large-scale structural variants in tumor samples is the high degree of genomic heterogeneity present. Samples taken from a solid tumor can include multiple sub-clonal cellular populations that do not share the same variants . The resreference based  cannot Local Assembly  are limited by the aligners and, as is shown by both the localassembly and reference free methods, alternative approaches are necessary to overcome the alignment issues.The analysis for SMUFIN also showed that there might be significantly more complex large-scale structural variation in tumor sequences than has been previously reported. This is due in large part to the fact that tumor genomes can include highly complex low frequency variations and the reference genome that alignment algorithms rely on cannot model these in mixed samples. The methods that rely on the de novo generation of multiple references. Our de novo method   . FinallyThe strategy we have used to enable more accurate alignment of cancer genomes from short reads is to limit the search space in which reads can align by altering the reference  cancer types were analyzed from the Mitelman CGAP database (http://www.ncbi.nlm.nih.gov/sky/skyweb.cgi). The majority of these were blood cancers .database  and 325 Cell line karyotypes. 84 were analyzed from the University of Cambridge CGP SKY/FISH of Epithelial Cell Lines (http://www.path.cam.ac.uk/∼pawefish/) and 67 from the NCI Fredrick National Laboratory NCI60 Cell Line Drug Discovery Panel (http://home.ncifcrf.gov/CCR/60SKY/new/demo1.asp). These were curated simply based on the tissue involved (e.g. ‘heart’ or ‘thymus’).Instead of aligning against a single reference, our method aligns against hundreds of smaller references. These smaller references model potential structural variations seen in cancer originating from fragile regions in the genome. The set of new references contain sequences from two different genomic regions thus simulating the result of a recombination event. These models are generated using prior knowledge of breakpoint frequencies in cancer based on karyotype data (e.g. breakpoints at cytogenetic bands). These frequencies were obtained from analysis of public karyotype data sets including patient karyotypes and cell lines:C320,2(51 040) possible combinations. While this would not seem to be too many combinations to test, and ideally testing against all of them would offer the most comprehensive view, there are computational limitations. First is disk space: the index for all simulated reference combinations requires 2.5 TB of hard disk space, and the subsequent alignment BAM files for a small set of reads (1.9 million) from a single genome would require more than 30 TB on disk. The second limitation is the alignment step itself. Aligning a small number of reads against many smaller references is an ideal situation for parallelization, however each single alignment (e.g. bwa mem –a –t 12) plus analysis computation still required 65 min on a single node in our local cluster. All 51 040 pairwise regions would require 840 000 compute hours (or 96 years) in order to align and analyze. Therefore even with access to a HPC cluster and a high degree of parallelization, this is a computationally intensive method.There are 320 major cytogenetic bands within the human genome , all of in silico references we use an informed search strategy. This informed approach is required to select regions that should be tested for breakpoints. As each breakpoint is not equally likely based on the karyotypes described, and to further decrease our search space and computational load, the frequencies calculated from karyotype data are used to generate a set of several hundred simulated references. This informed approach is outlined in the Optimisation of Reference Selection section below.Instead of using all possible The generated references now act as model regions for possible large-scale structural variation. Limiting the search space by creating smaller references also allows us to increase the number of possible alignments by including previously unmapped reads. A filtered set of reads from a patient sample that includes only those reads that were already aligned to different chromosomes (‘discordant’) or where one or both reads were unmapped are then aligned to these smaller references in parallel. This enables the method to rapidly compare multiple possible recombination regions. As we have limited the search space by using a smaller reference we are able to relax the search criteria to allow for more exhaustive searching and greater mismatches.in silico reference alignments. In all model regions, the distributions are consistently bimodal and non-symmetric, and each peak is reflective of these two different possible alignments  value are discarded. Each model region is then evaluated by analyzing the distribution of read-pair insert sizes in each new alignment. A bimodal distribution of the logged insert size between aligned read-pairs is observed across the in silico reference score is calculated in two parts. The first is based on the mixture model parameters for the second distribution as calculated by EM:P (n|z) is the conditional probability of the nth read belonging to each of the two distributions identified. The EMr reflects the proportion of reads that are found to have a higher mate pair distance, and is derived by finding the probability of the nth read belonging to the second distribution, then iterating over the set of N where N is the total number of reads aligned to this reference. The resulting value is a ratio based on the number of distributions found and the sum of the EMr for each is 1. As the first distribution describes ‘noise’ in the alignments we can use it to find a cutoff value for further analysis of the alignments described by the given in silico reference. All models where the second distribution have an EMr below the cutoff can therefore be discarded.The individual maxW is the cluster with the highest total count of reads from the second distribution, and bN is the total number of reads within the second distribution.The second part of the score is based on a sliding-window clustering approach to identify breakpoint locations based on alignment positions. Discordantly aligned reads from the second distribution are clustered by position if the read pairs also span both chromosomes represented by the simulated reference. This provides an estimation of windowed depth-of-coverage as discussed in  for a trTx scores and subsequent structural variant calling, we generated reads using the ART , a selection algorithm is introduced to generate populations of breakpoints. These populations are generated as individuals with full chromosomal complements and aberrant chromosomes.This selection uses a type of genetic algorithm known as differential evolution (DE)  with an The output of the selection algorithm is a population of pairs of chromosomal locations to be used in generating FASTA files. Each of these represents the sequence of the selected recombination. For example t, is defined as starting with 16q13 (56700001- 57400000) and ending with 8q24 (117700001- 46364022) creating a recombination point at 700 kb.We performed a test on simulated data to validate the method and identify suitable parameters for cancer variant selection. Then we applied the parameters we learned from the simulated validation tests to tumor/germline data sets from TCGA directly. For the patient data sets we compare our method to BreakDancer, as it continues to be the most commonly used tool for large-scale variant detection in tumors.Methods section. In the fully simulated set of 20 chromosome pairs we wanted to validate that the inserted breakpoint does result in a high Tx score when compared to a set of randomly selected references representing other potential breakpoint alignments. We applied k-means clustering to identify the set of regions with high Tx values. In order to keep the false positive rate (FPR) consistently below 10% in subsequent analyses we perform clustering using 4 centroids . Each set of unmapped and discordant reads from the genomes was compared against the same set of 278 regions selected by the optimization algorithm (see Supplementary Table S3). Based on the FPR rate calculated in the simulated data set above, we used a result selection from k-means clustering and took only the top cluster for analysis. We then filtered the germline hits from the tumor list in order to compare with BreakDancer. In each patient we identified regions that score highly for breakpoint inclusion. By using the germline samples to filter the results of the tumor samples we were able to remove regions that appeared to have significant unspecific read alignments as they were often found in both tumor and normal tissue samples.Two patients had been previously analyzed as part of a large cohort study for TCGA: the glioblastoma (GBM) patient  and one It is also worth noting that in the ovarian patient (OV) samples our method identified two bands that are found as part of multiple regions , which may suggest complex rearrangements. The 9q13 band is a known fragile site that is commonly involved in pericentric inversions in the germline linked with ovarian cancer , while 4Commonly identified regions. Structural variations that are found commonly across cancer types are highly likely to be due to biases in the detection method rather than a set of rearrangements that are common across cancer types. Both BreakDancer and our de novo method are going to find these due to the reporting of aligned reads. However, within the highest scoring regions, BreakDancer tends to find breakpoints in the same regions across multiple patients and cancers. Across all 32 regions, 26 were identified in more than one patient. For instance in the top regions with the highest scoring breakpoints, translocations in the 1p11–17p11 region were found in 6 of the 9 patients, while breakpoints in 1p11–11p11 are found in 4. In fact in the COAD-2 patient, where our method found no difference between tumor/germline, BreakDancer found only one region (3q27–6q15) where both bands involved were not identified in any other patient. Our method will also find regions that are common across cancers if the same set of regions are tested, however it is less likely with only 7 of the 29 regions being found in more than 1 patient, and the most common one (3q12–8p11) found in 4.Centromeres overrepresented. Across all 9 patients and cancers, centromeric regions are overrepresented in the top scoring breakpoints found by BreakDancer with 27 of the 32 regions including at least one centromere. Furthermore, in 7 of the patients all of the top 20 identified regions include at least one centromere. As centromeres (e.g. q11 and p11) make up only 15% of the major cytogenetic bands we would expect to see only 10 regions in 32 include a centromere in an unbiased sampling. As with the commonly identified regions, this is unlikely to be due to a common cancer event. Centromeres are poorly characterized across the chromosomes due to their highly repetitive sequences  two include alignments to the Y chromosome, however associated clinical data lists this patient as female. While this is not impossible, it would suggest issues with the alignment. As our method is not relying on a single reported alignment we did not find a similar inaccuracy.We compared these results to the BreakDancer analysis of tumor/germline pairs  are missing large numbers of structural variants due to their reliance on the alignment algorithm.The highly duplicative nature of the regions that are the most commonly found in translocations by BreakDancer suggests that only very common breakpoints are found, and that there are issues with alignment. Alignment algorithms typically report only a single ‘best’ alignment leaving BreakDancer, and other  defects , and it de novo references to model potential breakpoints prior to alignment, we find regions with variation that reference based methods such as BreakDancer cannot. Thus while reference based methods provide a good initial estimate for variations, concurrent use of reference free or de novo referencebased methods can provide a more complete view of the variation present in the tumor.As our method uses multiple de novo method may not result in fusion genes such as BCR/ABL but in altered regulation, or no detectable change.One of the difficulties present in the analysis of structural variation in cancer, is that in the absence of a directly gene related product (e.g. gene fusion) the effects may be subtle. For instance, if the break and recombination of a translocated segment occurs at intergenic regions the translocation may only affect the regulation or transcriptional enhancement of a gene rather than entirely inactivating the gene or increasing gene expression. Alternatively, as tumor genomes are often made of up highly heterogeneous cellular populations it is also likely that the effect of a fusion occurring in a sub-clonal population is at levels below our ability to detect in gene expression studies. It is therefore worth noting the regions identified by our In the second breast cancer patient (BRCA ) band 4qThe method introduced in this paper has a number of advantages and limitations. The main advantage is that our method is able to find structural variations that most commonly used tools would be unable to find. The reason these methods are unable to find these structural variations is that they lack the information required to identify them, as an exhaustive search during alignment is computationally inhibitive.The limitations of our method are due to the fact that the increase in the number of references increases both the noise and search space. Aligning to multiple references results in multiple alignments reported for a single read, increasing the potential for noise. While this is an issue with the method, it is also a necessary condition for the identification of structural variations in complex samples from a tumor. By applying EM and a clustering parameter the regions with the highest likelihood of variation can be selected with a reasonable FPR. Secondly, this is a computationally heavy approach in that there are more possible combinations than can be reasonably tested. However, using prior knowledge and the search optimization algorithm we can limit the search space for each genome tested. This means currently we may miss regions that have structural variation as we do not search all possible regions instead using HPC tools and optimizations to decrease the overall time and computational load. By limiting our search both to the set of unmapped and discordant reads, and using a set of model references based on prior information, we are able to evaluate many possible regions. Ultimately this increases the ability to identify low-frequency aberrations likely to be present in heterogeneous tumor samples.The identification of large-scale structural variation in cancer genomes continues to be difficult. Most of our current strategies have relied on alignment to a reference that is built on a ‘normal’ genome, assuming that the sample genome will align well enough for analysis. Unfortunately, due to the potential complexity of large-scale variations, the heterogeneity of a tumor sample itself, and the limitations of short-read sequencing, use of the standard reference can result in poor alignment for large-scale variant regions.reference-based methods cannot provide a complete view of the range and complexity of structural variation in tumor genomes. Thus, it is important to continue to explore alternative methods for large-scale variant detection in tumor samples. Our method approaches this issue by using multiple references to model potential breakpoints, decreasing the search space in which alignment algorithms function, and overcoming the problem of ‘best’ alignment mapping by allowing all alignments for each read and evaluating each region individually. This is an important step in tumor analysis due to the presence of clinically significant sub-clonal populations with complex chromosomal rearrangements.Alternative strategies that rely less on a reference genome, or skip alignment entirely, have provided evidence that the current De novo reference generation is implemented in Java using the Hadoop MapReduce v1.2.1 framework and HBase 0.94. This can be run on a standard desktop machine  with the standalone installation of Hadoop. It may also be run on a cluster which uses Hadoop locally or through Amazon EC2. A compiled version is available at http://sourceforge.net/projects/insilicogenome/files/releases/HBase-Genomes-1.2.jar the corresponding HBase database is available at http://sourceforge.net/projects/insilicogenome/files/Databases/GRCh37.tgzhttp://sourceforge.net/projects/insilicogenome/files/releases/denovo_analysis-1.2.tgzAnalysis of the resulting BAM files is performed in R v3.0.1, available at https://github.com/skillcoyne/IGCSAAll source code is available on Github (see README files in each module) at