Next-generation sequencing technologies generate a significant number of short reads that are utilized to address a variety of biological questions. However, quite often, sequencing reads tend to have low quality at the 3’ end and are generated from the repetitive regions of a genome. It is unclear how different alignment programs perform under these different cases. In order to investigate this question, we use both real data and simulated data with the above issues to evaluate the performance of four commonly used algorithms: SOAP2, Bowtie, BWA, and Novoalign.The performance of different alignment algorithms are measured in terms of concordance between any pair of aligners  and the accuracy of simulated read alignment.Our results show that, for sequencing data with reads that have relatively good quality or that have had low quality bases trimmed off, all four alignment programs perform similarly. We have also demonstrated that trimming off low quality ends markedly increases the number of aligned reads and improves the consistency among different aligners as well, especially for low quality data. However, Novoalign is more sensitive to the improvement of data quality. Trimming off low quality ends significantly increases the concordance between Novoalign and other aligners. As for aligning reads from repetitive regions, our simulation data show that reads from repetitive regions tend to be aligned incorrectly, and suppressing reads with multiple hits can improve alignment accuracy.This study provides a systematic comparison of commonly used alignment algorithms in the context of sequencing data with varying qualities and from repetitive regions. Our approach can be applied to different sequencing data sets generated from different platforms. It can also be utilized to study the performance of other alignment programs. The great demand for efficient, inexpensive, and accurate sequencing has driven the development of high-throughput sequencing technologies from automated Sanger sequencing to next-generation sequencing (NGS) over the past several years. Currently, NGS technologies are capable of producing low-cost data on a giga base-pair scale in a single run, which usually includes millions of sequencing reads. This ability makes the NGS technology a powerful platform for various biological applications, such as genetic variant detection by whole-genome or target region resequencing, mRNA and miRNA profiling, whole transcriptome sequencing, ChIP-seq, RIP-seq and DNA methylation studies. The first step of nearly all these applications is to align sequencing reads onto a reference genome. Thus, in order to obtain any further genetic information from sequencing data, the requirement of fast and accurate alignment tools has to be a priority .http://www.novocraft.com), SOAP, Bowtie, and BWA. Among all these five aligners, MAQ is the only one that indexes the reads, while all other aligners build indexes on a reference genome. In terms of the indexing algorithms they adopt, MAQ and Novoalign are two alignment programs that build an index with a hash table. To identify inexact matches in short-read alignments, MAQ uses a split strategy while Novoalign adopts an alignment scoring system based on the Needleman-Wunsch algorithm [In parallel with the rapid growth of new sequencing technologies, many alignment programs -20 have lgorithm . SOAP2 elgorithm  to buildNew challenges for alignments have arisen from applying sequencing technologies to address different biological questions. For example, how do reads with various sequencing qualities affect alignment results? How do they deal with the reads that can be mapped to multiple locations on a reference genome? In order to answer these questions, we select four commonly used aligners , and conduct a systematic analysis to evaluate the performance of these programs. First, we review and compare the algorithms these alignment programs employ as well as their advantages with respect to the major options they provide. Then, we use two sets of real Illumina sequencing data and two sets of simulated data to study how different alignment programs perform on sequencing data with varying quality and from repetitive regions. The performance is measured in terms of 1) concordance between any pair of the aligners, and 2) accuracy in simulated read alignment. We have demonstrated that, for sequencing data with reads that have relatively good quality or have had the low quality bases trimmed off, all four alignment programs perform similarly. Furthermore, we show that trimming off low quality ends markedly increases the number of aligned reads and improves the consistency among different aligners as well, especially for low quality data. However, Novoalign is more sensitive to the improvement of data quality. As for aligning reads from repetitive regions, our simulated data show that reads from repetitive regions tend to be aligned incorrectly, and suppressing reads with multiple hits can improve alignment accuracy.Hash table and suffix tree are two major indexing algorithms that current alignment programs use. Hash table indexing, which was first introduced into the field of alignment by BLAST , keeps t10(q), where q represents the probability of observing the read sequence given a specific alignment location. This score corresponds to the parameter setting “-t” at the command line when running Novoalign which finds the best alignment with the lower score and any other alignments with similar scores. Because of this alignment-score-based search algorithm, users cannot define the number of allowed mismatches in each alignment, but they can set up a threshold of alignment scores.To find inexact matches, alignment programs allow a certain number of mismatches using different strategies Table . SOAP2 um-long reads with 2% uniform base error rate have more than k mismatches, then the maximum number of allowed mismatches in these reads is set to be k. Thus, for our simulated data with 50-bp-long reads, k = 3. For the real NGS data with 68-bp-long reads, k = 4  to evaluate the bases quality. The plot of base qualities suggests that S1 has an overall better quality than S2. For example, S2 has more low quality bases at the 3’ end. In particular, the last 10 bases have average quality score less than 20  perform on real sequencing data with varying quality, we use two single-end Illumina sequencing data sets S1 and S2). S1 and S2 are sequenced from human colon cancer samples. For each of these two samples, about 3000 exons selected from cancer related genes are captured and sequenced by the Illumina sequencer, with 7,406,247 and 5,398,566 68-bp-long single-end reads generated respectively. We process the reads with ShortRead package inside of Bioconductor  138771 50-bp-long reads are generated from about 3000 exon regions from which the real data sets S1 and S2 are generated, and these 3000 exon regions do not have many repetitive regions; 2) 55018 50-bp-long reads are generated from 218 CpG islands. These 218 CpG islands are selected from 28226 CpG islands along the whole genome, by the criteria that each chosen CpG island must have at least 25% repetitive bases, and these repetitive bases must be at least 50 bp in length. Note that the purpose of this article is not to simulate reads from all repetitive regions in a genome. It is difficult to precisely define repetitive regions in a genome. Therefore, we simply choose to use some CpG islands that have some level of repetitive regions.In order to mimic the situation that one or two single bases are mismatches due to pure sequencing errors or true novel single nucleotide variants (SNV), we design the following four scenarios in our simulation data:1) randomly set one mismatch for each read and let all bases have high quality;2) randomly set two mismatches for each read and let all bases have high quality;3) randomly set one low quality mismatch for each read and let all other bases have high quality;4) randomly set two low quality mismatches for each read and let all other bases have high quality.10(p), where p is the base-calling error probability.1) and 2) are the cases that the mismatches are likely due to the existence of novel SNVs. 3) and 4) are the cases that the mismatches are likely due to pure sequencing errors. Low quality bases are the ones with Phred quality score ranging from 5-15; high quality bases are the ones with Phred quality score ranging from 30-40. Phred quality score is defined as -10*loghttp://www.novocraft.com), respectively. For the purpose of this study, we set the parameters in all four alignment programs as follows: 1) At most two mismatches are allowed in SOAP2, Bowtie, and BWA for each alignment. Due to the different alignment searching algorithm that Novoalign uses, we set the parameter t at 60 to allow approximately up to two mismatches, and then choose the alignment reads with no more than two mismatches using the NM tag in the output. 2) Randomly report one alignment for each read, or only report reads with unique alignments. For each alignment result we calculate the percentage of aligned reads. The performance of four alignment programs is measured in terms of concordance between any pair of aligners because no known truth for real sequencing data is available. In particular, for each pair of aligners, aligned reads are assigned into four classes as follows:In order to evaluate how different alignment programs perform in sequencing reads with low quality ends, we use the four alignment algorithms to map S1 and S2 before and after trimming off the low quality bases using BRAT trim , respectClass 1: a read is aligned to the same location by both aligners that we are comparing ;Class 2: a read is aligned to different locations by both aligners;Class 3: a read is only aligned by one of the two aligners ;Class 4: a read is only aligned by the other aligner in a comparison pair .If two alignment algorithms perform similarly, there should be a relatively small number of reads in class 2, 3 and 4 as shown in Figure In order to evaluate how different alignment algorithms perform on data containing reads generated from regions with more repetitive sequences, we use two simulated data sets. One data set is simulated from the 3000-exon regions that do not have a lot of repetitive bases and the other one is from 218 selected CpG islands that have many repetitive bases. For both simulated data sets, we align these reads using the four selected alignment programs. While aligning these simulated reads, all parameters are set the same as the ones used in real NGS data, except that we allow one mismatch for those data sets with only one mismatch simulated. Because we know the position from which each simulated read is generated, the performance of the four alignment tools is measured in terms of the accuracy of simulated read alignment. We define a true alignment as a situation when a read is aligned back into the same position from which it was generated. In addition, a false alignment is defined as a read that is aligned to other positions rather than the one from which it was generated.To assess the speed of index building and reads mapping in these four aligners, we use the non-trimmed S1 data, which has 7.4 million 68-bp-long single-end reads. We align these reads using the human genome 18 as a reference, with at most two mismatches allowed and one alignment randomly reported for each read Table . NovoaliFor the data set S1 that has relatively good quality, all four aligners generally show a good concordance, without trimming off low quality bases. A similar number of reads is aligned by each aligner Table . Over 95Performing trimming on NGS data not only cuts off the low quality bases from both ends, but also discards poor quality reads, and thus improving reads’ quality markedly. After trimming, 399,442 (5.4%) and 204,911 (3.8%) reads are discarded from S1 and S2 data, respectively. With slightly fewer reads available for alignment, however, the number of aligned reads is increased by 15-17% in the S1 data, and 34-42% in the S2 data, for all four-alignment programs. This apparent difference in the magnitude of increase indicates that trimming has a greater effect on the S2 data set than on the S1 data set. Another interesting observation is that, Novoalign aligns 42% more reads in trimmed S2 than non-trimmed S2, while this increment in the other three aligners is only about 35%, suggesting that data quality improvement has a larger effect on Novoalign.By trimming off the reads before alignment, we observe a substantial increase in the number of reads that fall into class 1 in all pair-wise comparisons, in both S2 and S1 data sets Tables , and 8. To evaluate these aligners in terms of their performance on reads with multiple alignments, we set the alignment parameters in two different ways: (1) randomly report one alignment for each read and (2) only report the read with a unique position . Compared to the former strategy, the latter discards around 4-10% of aligned reads from S1 and 2.5-8% in S2  in all comparison pairs, while the number of reads aligned to same positions (class 1) stays the same Tables , 6, 7, 8In order to study the four aligners’ performance on reads from repetitive regions, we use the two sets of simulated data as mentioned in the Data Set subsection. One data set is simulated from 3000 exon regions that do not have many repetitive bases. The other data set is from 218 selected CpG islands that have a lot of repetitive bases. In these two simulated data sets, no matter whether the mismatch positions are designed to have high or low quality, all four aligners show a lower false alignment rate in the data set generated from 3000 exon regions . To investigate this parameter setting in Bowtie, we both allow 2 mismatches and set the parameter “–e” at 20, 40, 60, and 80 respectively (data not shown). For our data sets, when the “-e” parameter is set at 40, 60 and 80, there are nearly identical results as compared to the output from only setting the number of allowed mismatches at 2 . But setting “–e” at 20 shows severe departures from other three aligners. In our data sets, most reads have moderate to good quality scores. However, setting “–e” at 20 only allows extremely low quality mismatched positions, and therefore, rules out the majority of reads with high quality mismatched positions.Like trimming off the reads, suppressing multiple alignments also improves the consistency among the three aligners Table . Out of In addition to the trimming and initial parameter setting of aligners, we also investigate the impact of filtering the alignments based on the mapping quality score provided in the output files of different aligners. Out of the four aligners, BWA and Novoalign both have a mapping quality score reported for each alignment. For BWA, this score is approximately a Phred-scaled probability of the alignment being incorrect, which takes the values of 37, 25, and any value between 23 and 0. In general, a score of 37 means the read is aligned to a unique position with less than 2 mismatches; a score of 25 means the read is aligned to a unique position with 2 mismatches; a score between 23 and 0 means the read is aligned to multiple locations, such that a lower score means that the mapped location is less accurate (based on BWA source code). For Novoalign, the mapping quality score is the probability of the alignment given the read and genome, which ranges from 0 to 150. Higher scores mean better alignment qualities. To explore the effect of quality score filtering, we checked the mapping quality scores in the untrimmed S1 and S2 data with one alignment reported randomly (Figure As for the mapability of those target regions that we used in our simulation data, we have checked the mapability using the “Duke uniqueness 35 bp” method provided by the UCSC genome browser for the 218 CpG islands and 3000 exon regions. This Duke method reports a mapability score between 0 and 1, with 1 representing a completely unique sequence. A score of 0.5, 0.3, 0.25, or 0 represents that the sequence occurs twice, three times, four times, or more than four times, respectively. For the 218 CpG islands, 80.09% are completely unique, which means all 35-bp sequences within these islands occur only once in the genome; while 19.91% are not completely unique, which means at least one 35-bp sequence within each of these islands occurs more than once in the genome. The median mapability score of all CpG islands is 1 and the mean is 0.9830. For the 3000 exon regions, 95.40% are completely unique and 4.60% are not completely unique. The median mapability score of all regions is 1 and the mean is 0.9930. Generally speaking, the 3000 exon simulation data has better mapabilty than the 218 CpG island data.There are different ways to evaluate the current available programs. For example, Ruffalo et al. developed a simulation and evaluation suite to compare a few available aligners only using simulated data . In thisIn order to study how alignment programs perform on data with varying quality and from repetitive regions, we have evaluated the performances of four commonly used alignment programs—SOAP2, Bowtie, BWA, and Novoalign—on two real NGS data sets and two simulated data sets. Our results show that, for sequencing data with reads that have relatively good quality or have had the low quality bases trimmed off, all four alignment programs perform similarly. We have also demonstrated that trimming off low quality ends markedly increases the number of aligned reads and improves the consistency among different aligners, especially for low quality data. However, Novoalign is more sensitive to the improvement of data quality. Trimming off low quality ends increases the concordance between Novoalign and the others significantly. Therefore, the quality of sequencing data has a great impact on alignment result, and we highly recommend assessing sequencing quality first and then trimming off low quality base if necessary. As for aligning reads from repetitive regions, our simulation data shows that reads from repetitive regions tend to be aligned incorrectly, and suppressing reads with multiple hits can improve alignment accuracy.The authors declare no competing interests.XY performed all statistical analyses and drafted the manuscript. KG, JW, MV, ZW, SM and MA were involved in the data collection and helped in preparation of the manuscript. SS provided suggestions on the project and revised the manuscript. All authors have read and approved the final document.


It is well known that the search for homologous RNAs is more effective if both sequence and structure information is incorporated into the search. However, current tools for searching with RNA sequence-structure patterns cannot fully handle mutations occurring on both these levels or are simply not fast enough for searching large sequence databases because of the high computational costs of the underlying sequence-structure alignment problem.We present new fast index-based and online algorithms for approximate matching of RNA sequence-structure patterns supporting a full set of edit operations on single bases and base pairs. Our methods efficiently compute semi-global alignments of structural RNA patterns and substrings of the target sequence whose costs satisfy a user-defined sequence-structure edit distance threshold. For this purpose, we introduce a new computing scheme to optimally reuse the entries of the required dynamic programming matrices for all substrings and combine it with a technique for avoiding the alignment computation of non-matching substrings. Our new index-based methods exploit suffix arrays preprocessed from the target database and achieve running times that are sublinear in the size of the searched sequences. To support the description of RNA molecules that fold into complex secondary structures with multiple ordered sequence-structure patterns, we use fast algorithms for the local or global chaining of approximate sequence-structure pattern matches. The chaining step removes spurious matches from the set of intermediate results, in particular of patterns with little specificity. In benchmark experiments on the Rfam database, our improved online algorithm is faster than the best previous method by up to factor 45. Our best new index-based algorithm achieves a speedup of factor 560.RaligNAtor a robust and well documented open-source software package implementing the algorithms presented in this manuscript. The RaligNAtor software is available athttp://www.zbh.uni-hamburg.de/ralignator.The presented methods achieve considerable speedups compared to the best previous method. This, together with the expected sublinear running time of the presented index-based algorithms, allows for the first time approximate matching of RNA sequence-structure patterns in large sequence databases. Beyond the algorithmic contributions, we provide with  Due to their participation in several important molecular-biological processes, ranging from passive carriers of genetic information (tRNAs) over regulatory functions (microRNAs) to protein-like catalytic activities (Riboswitsches), non-coding RNAs (ncRNAs) are of central research interest in molceular biology. NcRNAs,In this paper, we address the problem of searching nucleotide databases for occurrences of RNA family members. Since for this task it is not sufficient to rely on pure sequence alignment, we briefly review search methods that employ sequence and structure information.MARNA, 1 ≤ i ≤ n, denotes the base of S at position i. Let ε denote the empty sequence, the only sequence of length 0. Byn ≥ 0 overε is denoted byAn RNA S = S[1] S[2] … S[n] and 1 ≤ i ≤ j ≤ n, S[i..j] denotes the substring S[i] S[i + 1] … S[j] of S. For S = uv, u andu is a prefix of S, and v is a suffix of S. The k–th suffix of S starts at position k, while the k–th prefix of S ends at k. For 1 ≤ k ≤ n, Sk denotes the k–th suffix of S. For stating the space requirements of our index structures, we assume that |S| < 232, so that sequence positions and lengths can be stored in 4 bytes.For a sequence complementary and can form a base pair if and only ifa and b form a base pair we also say that there exists an arc between a and b. A non-crossing RNA structureof length m is a set of base pairs , 1 ≤ i < j ≤ m, stating that the base at position i pairs with the base at position j, such that for alli′ < i < j < j′ or i < j < i′ < j′ or i′ < j′ < i < j. A standard notation forstructure string R over the alphabet {.,} such that for each base pairR[i] = ( and R[j] = ), and R[r] = . for positions r, 1 ≤ r ≤ m, that do not occur in any base pair ofr ≠ i and r ≠ j for allThe secondary structure of an RNA molecule is formed by Watson-Crick pairing of complementary bases and also by the slightly weaker wobble pairs. We say that two basesφ(x) = {x} of exactly one element. A sequence pattern is a sequenceRNA sequence-structure pattern (RSSP)m is a pair of a sequence pattern P and a structure string R, both of length m. WithP[i..j],R[i..j]).Let Φ = {R, Y, M, K, W, S, B, D, H, V, N} be a set of characters. According to the IUPAC definition, each character in Φ denotes a specific character classS approximate matches of an RSSPS considering edit operations for unpaired bases and base pairs. That is, we compute semi-global alignments simultaneously obtaining the sequence-structure edit distance ofS.To find in a long RNA sequence S[p..q], 1 ≤ p ≤ q ≤ n, as set A = Amatch ⊎ Agap. The set Amatch ⊆ [1..m] × [p..q] of match edges satisfies that, for all different , ∈ Amatch, k > k′ implies l > l′. The set Agap of gap edges is defined asP[k] and S[l], 1 ≤ k ≤ m, p ≤ l ≤ q, are base mismatch (match), with cost ωm (zero), which occurs if there is an edge  ∈ Amatch and S[l] ∉ φ(P[k]) (S[l] ∈ φ(P[k])), and base deletion (insertion), with cost ωd, which occurs if  ∈ Agap  ∈ Agap). The possible edit operations on base pairs were first introduced by Jiang et al. and S[l2] are not complementary. An additional base mismatch cost ωm is caused if S[l1] ∉ φ(P[k1]) and another if S[l2] ∉ φ(P[k2]). To give an example, consider the semi-global alignment in FigureAmatch and  ∈ Amatch but S[11] = G and S[16] = G are not complementary. We note a difference between our definition and the definition of Jiang et al., where both aligned sequences are annotated with structure information. There, an arc breaking occurs if bases S[l1] and S[l2] are annotated as unpaired in addition to the condition of existing edges  ∈ Amatch and  ∈ Amatch. Hence, because in our case sequence S has no structure annotation, our definition is based on the complementarity of bases S[l1] and S[l2].• An curs if k,l1 ∈ Amaarc altering, with cost ωa, occurs if either (1)  ∈ Amatch and  ∈ Agap or (2)  ∈ Amatch and  ∈ Agap. Each case induces an additional base mismatch cost ωm if S[l1] ∉ φ(P[k1]) or S[l2] ∉ φ(P[k2]). As an example, observe in the alignment shown in FigureAgap and  ∈ Amatch.• An f either   arc removing, with cost ωr, occurs if  ∈ Agap and  ∈ Agap. As an example, observe in the alignment in FigureAgap and  ∈ Agap.• An curs if k,− ∈ AgapS[p..q] asWith this set of edit operations on the sequence and structure we can now define the cost of the alignment ofwhereA of minimum cost betweenS[p..q] is an optimal alignment ofS[p..q].An alignment S having a certain degree of similarity to a given RSSPS[p..q] with up to a user-defined sequence-structure edit distance and a limited number of allowed insertions and deletions (indels). More precisely: In practice, one is often interested in finding substrings of an RNA sequence • the costd.• the number of indels in the alignment should be at most S, given user-defined thresholdsd, is to report all intervals [p..q] such thatThus, the approximate search problem for finding occurrences of an RSSPS[p..q] satisfying Equation (3) a match ofS. In the subsequent sections we present algorithms for searching for matches of an RSSPS.We call every substring S consists of sliding a window of length m′ = m + d along S while computingp ≤ q ≤ n and q − p + 1 = m′. We note that, although the length of a match can vary in the range m − d to m + d, to find matches of all possible lengths it suffices to slide a window of length m′ along S corresponding to substrings S[p..q]. This holds because the alignment to a window of length m′ entails all possible alignments with up to d allowed indels. In the following we present a dynamic programming algorithm computingS[p..q]. Our recurrences are derived from the algorithm for global pairwise sequence-structure alignment of Jiang et al.. We begin the description of our algorithm by defining three functions required by the dynamic programming recurrences. Let i and j of the RSSPT, respectively, we define a function1. For computing base match and mismatch costs for positions i and j of T. Therefore, we define a function2. To determine whether an arc breaking operation can occur, we must also be able to check for base complementarity at positions 3. For determining the correct row (of the dynamic programming matrices introduced below) where certain operation costs must be stored we introduce a functionrow satisfies the following: (1) given the right index i of a base pair , it returns the left index i′ if  is preceded or followed by other structures; (2) given the left index i of a base pair , it returns 0 if the base at position i + 1 ofi, it returns i itself.Intuitively, function m′ + 1(m′ + 1) × (m′−k + 1) matrices DPk, for 1 ≤ k ≤ m′ + 1, such thatDPk is not defined for every subinterval [i..j]. While the recurrences of Jiang’s algorithm are divided in four main cases, we present a simplified recurrence relation with only two main cases. In addition, we observe that we use only three indices for a matrix entry instead of four. Our recurrences are as follows. Using these three functions, our algorithm determines the sequence-structure edit distancei = 0 or R[i] =. (unpaired base), then1. If R[i] ≠. (paired base), then2. If R[i] =) where i forms base pair(a) If R[i′−1] =. or R[i′−1] =), compute in addition to Equation (8)(b) If (a) holds and either DP matrices is top down, checking whether case 1, 2(a), or 2(b) applies, in this order. Due to the matrix dependencies in cases 2(a) and (b), the matrices need to be computed simultaneously.A natural way to compute these j, 1 ≤ j ≤ m′, clearlym′ beginning at position p are also computed in the computation ofNote that for all When sliding a window along S to computep ≤ q ≤ n, m′ = q − p + 1 = m + d, a window shift by one position to the right requires to compute only column m′−k + 1, i.e. the last column of matrices DPk, 1 ≤ k ≤ m′.T[1..m′] = S[p..q]. The computation ofm′ + 1 DP matrices, one for each suffix Tk of string T = T[1..m′], 1 ≤ k ≤ m′, and one for the empty sequence ε. As a result, it holds for every k thatTl+1[1..m′−l] = S[p + l..q], 0 ≤ l < m′, only differs by its last character from S[p + l + 1..q + 1] which are suffixes of the window substring shifted by one position to the right, the lemma holds. □Let DP matrices for every shifted window substring  in cases 1 and 2(a) takes O(1) time and in case 2(b) it takes O(m′) time. Now consider the two situations: Our S[1..m′], cases 1 and 2(a) require O(mm′2) time in total and case 2(b) requires O(mm′3) time in total. This leads to an overall time of O(mm′3).• For the first computed window substring O(mm′) time in total and case 2(b) requires O(mm′2) time in total, leading to an overall time of O(mm′2).• For one window shift, cases 1 and 2(a) require n − m′−1 window shifts, the computation for all shifted windows takes O(mm′2(n−m′)) = O(mm′2n) time. We observe that the time needed by ScanAlign to compute all window shifts reduces to O(mm′n) if recurrence case 2(b) is not required. This is the case if the structure ofSince there are S[p..q] of the searched RNA sequence, we can determine whether the cost of this alignment will exceed the cost thresholdScanAlign to skip the window, thus saving computation time and proceed with aligning the next window. The idea consists in checking, during the alignment computation, whether the cost of an already aligned region ofS[p..q] exceedsS[p..q] will also exceedOften, before completing the computation of the alignment between an RSSP• We decompose the RSSPS[p..q], progressively extending the alignment to other regions.• We compute the alignment of a given initial RSSP region and a substring of the current window • If the cost of aligning an RSSP region to a substring of the window exceeds cost thresholdx ≤ y ≤ m, satisfies exactly one of the following conditions. Formally, a valid RSSP region′ to 3′ direction, i.e. x = 1. Alternatively, it is an unpaired region of maximal length such that position x − 1 forms a base pairy′ of1.y is unpaired and there is at least one base pairx ≤ x′ < y′ < y. No extension of2. Position 3.y forms a base pairR[x′−1] =. or R[x′−1] =), 1 ≤ x ≤ x′−1. In addition, x = 1 ory′ > y. Examples of such RSSP regions are shown in red in Figure4. Note that regions can be embedded in other regions but cannot partially overlap another.S begins by considering only an in general small region ofT = T[1..m′] be a window substring of length m′ = m + d of S and d be the number of allowed indels. Pattern regions have the property that, for any regiony′ < x and x′ < y. Therefore, they can easily be sorted to indicate the order by which the rows of the DP matrices are computed. We observe that the top-down computation of the DP matrices, as described above, automatically sorts the regions and respects the dependency between rows. To obtain from the sorted regions the indices of the rows to be computed, we consider the condition satisfied by each region. The rows obtained according to each condition are computed according to one case of the recurrence. Given regionOur progressive alignment computation of an RSSPx..y] are computed by Equation (7).1. All rows in the interval  are computed by Equation (7).2. One scans the structure of regiony is computed by recurrence (a) of Equation (8).3. Row row(y) is computed by recurrence (b) of Equation (8).4. Row The sequential computation of the rows belonging to each region naturally leads to the computation of the entire alignment ofScanAlign algorithm is based on the following two observations. Our improvement of the m and n requires an (m + 1) × (n + 1) matrix. Let i and j be indices of each of the matrix dimensions and a diagonal v be those entries defined by i and j such that j − i = v. Given that the cost of each edit operation is a positive value, the cost of the entries along a diagonal of the matrix are always non-decreasing of the searched RNA sequence, a cost thresholdand number d of allowed indels. If for every d′, −d ≤ d′ ≤ min{d,x}, z ∈ {|d′|−d,−|d′|+d}, y + d′ ≤ m′, it holds thatthen, for every d′′, 0 ≤ d′′ ≤ d,e as those entries DPk (DPk(row(y),j)), 1 ≤ k ± d ≤ m′, such that j − i + offset = e, where offset = x − 1. Without loss of generality let d = 1. Assuming x − 1 > 0 and y + 1 ≤ m′, this means that an optimal alignment of patternT requiresIf the RSSP regionT[x..y], T[x..y − 1], or T[x..y + 1], requiring for all three alignments the computation ofz ∈ {0−1,0 + 1} = {−1,1};• T[x − 1..y − 1], requiring the computation ofz ∈ {|−1|−1,−|−1| + 1} = {0}; or• T[x + 1..y + 1], requiring the computation ofz ∈ {|1|−1,−|1| + 1} = {0}.• T[x..y], T[x..y + 1], and T[x..y−1] end in matrix DPx. The alignments with T[x−1..y−1] end in matrix DPx−1, and the alignments with T[x + 1..y + 1] end in matrix DPx+1. Every minimizing path obtained for the entire alignment ofT can only include the entries on the diagonals e, e + 1, and/or e − 1 for the alignments with T[x..y], T[x..y + 1], and T[x..y − 1], and can only include the entries on diagonal e for the alignments with T[x − 1..y − 1] and T[x + 1..y + 1] because these substrings already imply alignments with one indel. As the sum of the cost of the edit operations on the minimizing path increases monotonically and there cannot be other minimizing paths due to the limited number of indels d, the lemma holds. □The alignments with DP tables above, let d be the number of allowed indels, and T = T[1..m′] be a window substring of the searched RNA sequence. Applying Lemma 2, we modify algorithm ScanAlign to compute the alignment of each regiond ≤ d′ ≤ min{d,x}, y + d′ ≤ m′, and progressively extend the alignment to other RSSP regions and substrings of T as long asz ∈ {|d′|−d,−|d′| + d}, holds. That is, for each RSSP region, it determines the rows and recurrence case required for their computation according to conditions 1, 2, 3, or 4 above. Then, within each processed row i, it checks whether for at least one entry DPk on a possible minimizing path, i.e. on diagonals e′, e − d ≤ e′ ≤ e + d, DPk ≤ k. If no entry is belowDP matrices of an alignment computation whose entries on a possible minimizing path are highlighted in yellow.LetDP matrices entries are computed. Hence, a direct application of Lemma 1 is no longer possible. To overcome this, we define an array Z in the range 1 to z, where z is the number of RSSP regions, and associate each region with an index r, 1 ≤ r ≤ z. Let p be the starting position of the window substring S[p..q] in the RNA sequence. We set Z[r] = p whenever all DP matrices rows and columns belonging to region r are computed. This occurs when the cost of aligning this region does not exceed cost thresholdr to a different window substring S[p′..q′], p′ > p, computing all DP matrices columns requires to compute the last p′ − p columns. If p′ − p < m′ , this means that the two window substrings do not overlap and therefore no DP matrix column can be reused.When scanning the searched RNA sequence, a window can be shifted before all LScanAlign, in the worst case needs to process every RSSP region for every window shift. Hence, it has the same time complexity as algorithm ScanAlign. However, as in many cases only a few RSSP regions are evaluated, it is much faster in practice as will be shown later. ScanAlign and LScanAlign are the basis for further improvements presented in the subsequent sections.Our improved algorithm, hereinafter called S.Suffix trees and enhanced suffix arrays are powerful data structures for exact string matching and for solving other string processing problems,30. In tS is composed of the suffix array suf and the longest common prefix array lcp. Let $, called terminator symbol, be a symbol not insuf is an array of integers in the range 1 to n + 1 specifying the lexicographic order of the n + 1 suffixes of the string S$. That is, Ssuf[1],Ssuf[2],...,Ssufn+1][ is the sequence of suffixes of S in ascending lexicographic order. Table suf requires 4n bytes and can be constructed in O(n) time and space[lcp is a table in the range 1 to n + 1 such that lcp[1] = 0, and lcp[i] is the length of the longest common prefix between Ssufi−1][ and Ssufi][|} be the reading depth of suffix Ssufi] is represented by a suffix Ssufi][[1..pi] such that Ssufi][[1..pi] = S[p..q]. To matchS using a suffix array, we simulate a depth first traversal of the lcp interval tree..suf[i] + p′] as a matching interval ofS. That is,S[suf[i]..suf[i] + p′] beginning at position suf[i] of S.Consider an RSSPval tree of S on LESAAlign, builds on algorithms ScanAlign and LScanAlign. ScanAlign and LScanAlign exploit overlapping substrings of consecutive window substrings to avoid recomputation of DP matrices entries. LESAAlign exploits the enhanced suffix array in two different ways. First, for a single suffix Ssufi] between two consecutive suffixes Ssufi][ and Ssufi−1] − k + 1, of each matrix DPk. This means that, for lcp = min{pi,lcp[i]}, it avoids the recomputation ofSsufi] of S; and1. If pi ≤ lcp and2. If LESAAlign to benefit from the enhanced suffix array in a second important way. That is, it skips all suffixes Ssufi][ sharing a common prefix of at least length lcp with Ssufi−1][. To find the index j of the last suffix Ssufj],lcp[i+1],...,lcp[j]} ≥ lcp. If the first situation above holds, there are matches ofS at positions suf[i], suf[i +1 ],..., suf[j]. We note that suffixes can also be efficiently skipped using so-called skip-tables as described in[n additional bytes required to store such tables we do not use them here. Our algorithm continues the top-down traversal of the suffix array with suffix Ssufj+1][ and Ssufj+1][ to be considered in the processing of Ssufj+1][ is min{lcp[i],lcp[i+1],...,lcp[j],lcp[j+1]}.These situations allow ribed in. HoweverLScanAlign. This allows to skip suffixes Ssuf[i] as soon as it becomes clear that the sequence-structure edit distance of RSSPSsufi][[1..pi] be the current substring. When progressively aligning the regions ofT, we store the index r of the first region whose alignment cost exceedsr we store, it holds that for every d′, −d ≤ d′ ≤ min{d,x},l = y − x + 1, z ∈ {|d′|−d,−|d′|+d}, and y+d′ ≤ m + d (see Lemma 2). Then, when aligningSsufj][[1..pj], we must distinguish the regions ofpreviously computed from regions not computed. We also incorporate in our index-based algorithm the early-stop alignment computation scheme of algorithm Previously computed pattern regions are all regions whose index is strictly smaller than r. The alignment computation of these regions profits from the common prefix between Ssufi][[1..pi] and Ssufj][ [1..pj] by avoiding the recomputation of DP matrices columns as described above.• Non-computed pattern regions are all regions whose index is larger than or equal to r. In this case, all DP matrices columns of the respective pattern region need to be computed, even if Ssufi][[1..pi] and Ssufj][ [1..pj] share a common prefix.• T needed to assertT[x + l + d − 1] = T[x + y − x + 1 + d − 1] = T[y + d] as l = y − x + 1. Therefore, no suffix sharing prefix T[1..y + d] can matchS. Because in most cases y + d < pi, more suffixes are likely to share a prefix of length y + d than of length pi with Ssufi],Tx+d′..l+z)>K S, algorithm LESAAlign is very fast when it can Given an RSSPDP matrices columns due to a common prefix between suffixes of S; and• avoid recomputation of suf whose common prefix up to a required reading depth are known to match or not match• skip long ranges of suffixes of the suffix array LESAAlign exploits repetitions of substrings of S, i.e. substrings shared by different suffixes, and information of the lcp table to save computation time. However, the use of information of the lcp table alone does not necessarily lead to large speedups. Consider e.g. the DP matrices for the computation of the alignment ofSsuf[4][1..p4] = ACCCUCUU in FigureS is shown in FigureSsuf[4][1..p4] of length 8 shares a common prefix of length lcp[4] = 4 with the previously processed substring Ssuf[3][1..p3]. Despite this common prefix, still 182/252 ≈ 72% of the DP matrices entries need to be computed  in case no early-stop is possible, i.e. in case% of the DP matrices entries computed by the online algorithm LScanAlign for a window shift.Therefore, ring Ssuf[1..p4] =S that: Our next goal is to develop an algorithm traversing the enhanced suffix array of 1. can skip more suffixes; andDP matrices entries, reusing computed entries for as many suffixes as possible.2. improves the use of already computed d′, −1 ≤ d′ ≤ 1, z ∈ {|d′|−1,−|d′| + 1}, i.e. the alignment cost for this pattern region already exceeds the cost threshold of 1 (in accordance with Lemma 2). In other words,T[2..6] = CCCUC, T[3..6] = CCUC, T[3..7] = CCUCU, T[3..8] = CCUCUU, or T[4..8] = CUCUU with a cost lower than 1. Observe further that the alignment computation of regionT[2..8]=CCCUCUU from position 2 to 8 can matchT[1..8] as performed by LESAAlign. That is, we want to skip suffixes sharing a substring, not limited to a prefix, whose alignment cost to a pattern region exceeds cost thresholdTo address the first goal, we motivate our method by recalling the alignment computation example in FigureS be an arbitrary RNA sequence and T[x..y] = Ssufi][[x..y] contain all substrings whose alignment cost to a region of an RSSPT[x..y] from position x to y. (1) For any value of x, all suffixes sharing prefix T[1..y] can be skipped as performed by algorithm LESAAlign. (2) Now let x > 1. To find all suffixes of S sharing substring T[x..y] from position x to y, we first locate all suffixes sharing T[x..y] as a prefix. We begin by locating one such suffix, in particular the suffix of index suf[j] that contains all but the first x′ = x − 1 characters of Ssufi]] = i. It requires 4n bytes and can be computed via a single scan of suf in O(n) time. Given table suf−1, we can define the suffix link from T = Ssufi][ to T2 = Ssufi]+1[ as link = suf−1[suf[i] + 1], i.e. it holds that suf[link] = suf[i] + 1. Now, if x′ = 1, we already find that the index suf[j] of the suffix containing all but the first character of Ssufi][ is suf[j] = suf[link] becausej for any x′ ≥ 1. The obvious solution is to compute suffix links x′ successive times. Each suffix link skips the first character of the previously located suffix. For a more efficient solution, we generalize suffix links to point directly to the suffix without a prefix of any length x′ of the initial suffix. For this purpose we define a functionLet  |T| > 1. A pointj = link,x′ ≥ 1. All suffixes sharing T[x..y] as a prefix are all suffixes in the range jstart to jend where jstart is the smallest and jend is the largest index satisfying min{lcp[jstart + 1],...,lcp[j],...,lcp[jend]} ≥ y − x + 1. Finally, we find that all suffixes of S sharing substring T[x..y] from position x to y are allsuf, we mark their positions as true  in a bit array vtab of n bits. The suffix array traversal proceeds from position suf[i], but skips the marked suffixes when their positions are reached.Then, by letting d′ ≤ 1, z ∈ {|d′|−1,−|d′| + 1}, does not depend on character T[1] and regionDP tables. To decrease the chance that unnecessary computations occur, we sort the RSSP regions to begin aligning with the left-most RSSP regionx − d > 1.We remark that the described method for skipping suffixes can profit from a resorting according to the order by which RSSP regions are aligned. In the alignment computation example in FigureDP matrices entries for as many suffixes as possible. Recall that computing the sequence-structure edit distanceSsufi][[1..pi], 1 ≤ k ≤ m′, and one for the empty sequence ε. Observe that each suffix Tk, Tk ≠ T, also occurs itself as a prefix of a suffix in table suf, i.e. there exists a suffix Ssufj][ shorter than Ssufi][[1..pj]. Now note that if T′ is at a (nearly) contiguous position in suf to T, T′ and T are likely to share a common prefix due to their similar lexicographic ranking. This allows algorithm LESAAlign to avoid recomputation of DP matrices columns by using information from the lcp table. Unfortunately, T′ and T can be lexicographically ranked far away from each other in table suf, meaning that the DP matrices computed for T′ either: We now address the second goal, namely reusing computed T′ is lexicographically smaller than T, but were discarded to allow the processing of other suffixes until T was traversed; or• were already computed once because T′ occurs in table suf as a prefix of a suffix itself.• are computed for the first time otherwise, but will not be reused to also allow the processing of other suffixes until DP matrices by processing T′ directly after processing T for fixed k = 2, recalling that T = Ssufi][[1..pi] and T′ = Ssufj][[1..pj]. This value of k implies that Ssufj][ does not contain the first character of Ssufi][ and that we can locate Ssufj]. Therefore, to determineDP matrices required to computei and j are not necessarily contiguous positions in suf, we mark the processed suffix Ssufj][ in the bit array vtab so that it is only processed once. If no match to RSSPsuf[j], we also mark and skip every suffix sharing the substring with T′ whose alignment to a region ofT′ is processed and all possible suffixes are skipped, we recursively repeat this optimization scheme by setting T = T′ and processing the nextj′ = link. The recursion stops whenT′ is too short to matchsuf[j′] is already marked as processed in vtab. The suffix array traversal proceeds at position i + 1 repeating the entire scheme.In both cases, redundant computations occur. To avoid this, we optimize the use of computed LGSlinkAlign. For its pseudocode, see Section 1 of Additional fileLGSlinkAlign inherits all the improvements of the above presented algorithms. In summary, its improvements are as follows. We call our algorithm incorporating the presented improvements LGSlinkAlign traverses the enhanced suffix array of the searched sequence S, i.e. the suffix array suf enhanced with tables lcp and suf−1. During this traversal, it benefits from common prefixes shared among suffixes to (1) avoid the computation of DP matrix columns and to (2) skip ranges of suffixes known to match or not match RSSPLESAAlign.• DP matrices.• The suffix array traversal is predominantly top down, but non-contiguous suffixes are processed to optimize the use of computed LGSlinkAlign stops the alignment computation as early as the alignment cost of a region of RSSPLScanAlign.• m + d can be skipped, leading to larger ranges of skipped suffixes. The early-stop computation scheme also helps to identify and skip non-contiguous suffixes sharing a common substring which is not their prefix.• Due to the early-stop computation scheme, suffixes sharing common prefixes shorter than LGSlinkAlign with the following example. Consider the RSSPS whose enhanced suffix array is shown in Figureωd = ωm = ωb = ωa = 1 and ωr = 2. When traversing the enhanced suffix array of S, LGSlinkAlign always begins to alignS with regiond > 1. This means that the alignment computation of regionS aligningT = Ssuf[1][1..p1] = S14[1..8] from positions 4 − d = 3 and 6 + d = 7. For this, it computesd′ ≤ 1 and z ∈ {|d′|−1,−|d′|+1}. Observe thatT[1..6] = AACACC can matchT[3..6] = CACC from position 4−d = 3 to 5 + d = 6 can matchlcp[2] < 6, so we analyze case (2). The algorithm looks for suffixes sharing substring CACC from position 3 to 6. It begins by locating suffixes without the first two characters of T and containing CACC as a prefix. It follows the suffix link link = suf−1[suf[1]+2] = suf−1[16] = 7 and looks for the smallest jstart and largest jend satisfying min{lcp[jstart + 1],...,lcp[8],...,lcp[jend]} ≥ 4 = |CACC|. It finds that jstart = 5 and jend = 8, since min{lcp[5+1],lcp[7],lcp[8]} = min{4,5,5} ≥ 4 holds. The suffixes containing CACC from position 3 to 6 are Ssuf[5]−2 = S11, Ssuf[6]−2 = S7, and Ssuf[8]−2 = S14. S11and S7 are marked in the bit array vtab, whereas S14 = Ssuf[1] was already processed and does not need to be marked. We observe that Ssuf[7]−2 = S−1 is not a valid suffix. To reuse as many computed DP matrices entries as possible, the algorithm next processes the suffix Ssufj][ which does not contain the first character of Ssuf[1]. It determines j = link=suf−1[suf[1]+1] = 11 and sets T = Ssuf[12][1..p12] = S15[1..8]. The alignment to this substring T begins with its substrings from positions 3 to 7 andT cannot matchSsuf[12] = S15 was traversed via a suffix link, it is marked as processed in vtab. We now again analyze two cases of suffixes that cannot matchT[1..6] = CCACCC and (2) suffixes containing substring T[3..6] = ACCC from position 3 to 6. Satisfying case (1) are suffixes Ssuf[11] = S1 and Ssuf[10] = S8 since lcp[12] ≥ 6 and lcp[11] ≥ 6. These suffixes are marked in vtab. We now check if there are suffixes satisfying case (2). The algorithm begins by locating suffixes containing substring T[3..6] = ACCC as a prefix. For this, it follows the suffix link link = suf−1[suf[12] + 2] = 4 and determines jstart = 2 and jend = 4. The property min{lcp[2 + 1],lcp[4]} ≥ 4 is satisfied. The suffixes containing ACCC from position 3 to 6 are Ssuf[2]−2 = S8, Ssuf[3]−2 = S1, and Ssuf[4]−2 = S15. Since these were already marked in vtab, none of them needs to be marked. The algorithmic scheme of LGSlinkAlign to reuse as many computed DP matrices entries as possible continues processing other suffixes which are located by iteratively following the suffix links. It locates suffixes Ssuf[8], Ssuf[4], Ssuf[18], and Ssuf[19] because link = 8, link = 4, link = 18, and link = 19, respectively. These suffixes are processed analogously as above, one after the other, not resulting in matches toSsuf[20], since link = 20. However, |Ssuf[20]| < m−d, meaning that this suffix is too short to contain a match toSsuf[2]. After processing and skipping all possible suffixes, we note that LGSlinkAlign does not report any matches for the defined cost threshold and allowed number of indelsWe elucidate the ideas of algorithm RNAs with complex branching structures often cannot be adequately described by a single RSSP due to difficulties in balancing sensitivity, specificity, and reasonable running time of the used search algorithm. Although their description by a single short RSSP specifying an unbranched fragment of the molecule might be very sensitive, it is often too unspecific and likely to generate many spurious matches when searching for structural homologs in large sequence databases or complete genomes. In contrast, using a single long RSSP often requires a higher cost thresholdm matching substring T, wherebps denotes the number of base pairs inT. With this definition of a fragment’s weight, a positive weight is always guaranteed, thus satisfying a requirement for the chaining algorithm. Once the chaining of matches to the RSSPs is completed, the high-scoring chains are reported in descending order of their chain score. By restricting to high-scoring chains, spurious RSSP matches are effectively eliminated. Moreover, the relatively short RSSPs used in an SSD can be matched efficiently with the presented algorithms leading to short running times that even allow for the large scale application of approximate RSSP search.We solve this problem by applying the powerful concept of RNA secondary structure descriptors (SSDs for short) recently introduced in. The undLESAAlign and LGSlinkAlign, (2) the online algorithms LScanAlign, ScanAlign, both operating on the plain sequence, and (3) the efficient global and local chaining algorithms described in, and (3) d = 0 varyingLESAAlign and LGSlinkAlign with sufconstruct and their storage on disk required 372 seconds. In the following we refer to this dataset as RFAM10.1 for short. In this experiment we use the RSSP tRNA-pat of length m = 74 shown in FigureLGSlinkAlign and LESAAlign are the fastest algorithms. LGSlinkAlign is faster in particular for increasing values ofd, being only slower than LESAAlign for small values ofd and for fixed d = 0. The advantage of LGSlinkAlign over LESAAlign with higher values ofd is explained by the increased reading depth in the suffix array implicated byd and the fewer suffixes sharing a common prefix that can be skipped. This holds for both LGSlinkAlign and LESAAlign, however LGSlinkAlign counterbalances this effect by reusing computed DP matrices for non-contiguous suffixes of the suffix array. In a comparison to the two online algorithms considering only approximate matching, i.e.LGSlinkAlign over ScanAlign (LScanAlign) is in the range from 560 ford = 0 to 17 ford = 0 to 3 forLESAAlign achieves a speedup factor over ScanAlign (LScanAlign) in the range from 1,323 ford = 0 to 9 ford = 0 to 1.6 forLScanAlign is faster than ScanAlign by up to factor 45 forScanAlign profit from low values ofd reducing their search times. This is a consequence of the use of the early-stop alignment computation scheme. As shown in Figured influences the search time. For an additional experiment investigating the influence ofd on the search time required by the four algorithms, see Section 2 of Additional fileRaligNAtor and the widely used tool RNAMotif[tRNA-pat depicted in FigureIn a first benchmark experiment we measure the running times needed by the four algorithms to search with a single RSSP under different cost thresholdsterval 0,, (2)K=6. We set K=d vary. We set K=d vary. We set K=d varyScanAlign, LScanAlign, LESAAlign, and LGSlinkAlign scales on random subsets of RFAM10.1 of increasing size. The searched RSSPs flg1, flg2, and flg3 were derived from the three stem-loop substructures the members of family flg-Rhizobiales RNA motif (Acc.: RF01736)[d; see Figured are simply denoted cost and indels in the RaligNAtor RSSP syntax. The results are shown in FigureLGSlinkAlign and LESAAlign show a sublinear scaling behavior, whereas LScanAlign and ScanAlign scale linearly. The fastest algorithm is LGSlinkAlign, requiring only 11.68 (53.08) minutes to search for all three patterns in the smallest (full) subset. The second fastest algorithm is LESAAlign, followed by LScanAlign and ScanAlign, which require 32.27 (126.97), 40.47 (321.01), and 98.35 (754.66) minutes, respectively, to search for all the patterns in the smallest (full) subset. This corresponds to a speedup of 8.4 to 14.2 of LGSlinkAlign over ScanAlign on the smallest and the full subsets. Comparing the search time for pattern flg3 individually, the speedup of LGSlinkAlign over ScanAlign ranges from 22.6 to 38.8. We also observe that ScanAlign requires the longest time to match the longest pattern flg2 of length m = 37. The other algorithms profit from the early-stop computation approach to reduce the search time for this pattern on every database subset.In a second experiment we investigate how the search time of algorithms RF01736) fold intDP matrices using recurrences according to two main cases: either a row corresponds to an unpaired or to a paired base of the pattern. To analyze the influence of the used recurrence on the search time of each algorithm, we search RFAM10.1 for artificial stem-loop patterns. Therefore we vary the number of bases in the loop of patternDP recurrences in Equations (7) and (8), respectively. The cost threshold and the number of allowed indels are fixed atLGSlinkAlign and LESAAlign. This can be explained by the use of the early-stop alignment computation scheme in these algorithms. The reduction of the running time is explained by the fewer matches that need to be processed as the pattern gets longer and more specific. For an increasing number of base pairs in the stem, LGSlinkAlign is the least affected algorithm. We also observe that the linear increase in running time of the basic online algorithm ScanAlign, caused by an extension of the pattern by one base pair, is similar to the effect of adding two bases in the loop.When searching a database for matches of a given pattern, our algorithms compute the required % of the Rfam 10.1 families. This illustrates the importance of using both sequence and structure information encoded in the SSDs of this experiment. The SSD of family RF01736 comprises three RSSPs, denoted by flg1, flg2, and flg3 in Figureires1, ires2, ires3, ires4, and ires5 in Figure S5 of Additional fileires1 describes a moderately conserved strand occurring in these members. Observe also in Figuresd used per pattern, remembering that these are denoted cost and indels in the RaligNAtor RSSP syntax.In the next experiment we show the effectiveness of global chaining when searching with two SSDs built for Rfam families Cripavirus internal ribosome entry site (Acc.: RF00458) and flg-Rhizobiales RNA motif Acc.: RF01736). These t736. TheRFAM10.1 delivers 16,033,351 matches for ires1, 8,950,417 for ires2, 1,052 for ires3, 112 for ires4, and 1,222,639 for ires5. From these matches, RaligNAtor computes high-scoring chains of matches, eliminating spurious matches and resulting in exactly 17 chains. Each chain occurs in one of the 16 sequence members of the family in the full alignment except in sequence AF014388, where two chains with equal score occur. The highest (lowest) chain score is 171 (162). Using ScanAlign, LScanAlign, LESAAlign, and LGSlinkAlign, the search for all five RSSPs requires 688.32, 585.59, 186.88, and 92.25 minutes, respectively, whereas chaining requires 13.66 seconds. See Table S8 of Additional fileSearching with the SSD of family RF00458 in flg1, 68,024 for flg2, and 67 for flg3. Chaining the matches leads to 15 chains occurring each in one of the 15 sequence members of the family in the full alignment. The highest (lowest) chain score is 163 (156). Using ScanAlign, LScanAlign, LESAAlign, and LGSlinkAlign, the search for all three RSSPs requires 755.48, 336.69, 133.58, and 52.86 minutes, respectively, whereas chaining requires 0.03 seconds. The time required to match each pattern using each algorithm is reported in Table S9 of Additional fileThe same search is performed using the SSD of family RF01736. It results in 4,145 matches for RaligNAtor deteriorates sensitivity and specificity in the search for sequence members of families RF00458 and RF01736. For this, we report in Section 4 and Table S10 of Additional fileStructator tool[Structator is much faster but, in contrast to RaligNAtor, does not support all sequence-structure edit operations.We also show that the lack of the sequence-structure edit operations supported by tor tool. StructaRaligNAtor on 35 RNA families taken from Rfam 10.1 with different degrees of sequence identity and of different sizes. See TableRaligNAtor results obtained by using RSSPs derived from Rfam seed alignments with (2) results obtained for the same RSSPs ignoring base pairing information and (3) results obtained by blastn[P, we choose the IUPAC wildcard matching all symbols in the corresponding alignment column. As structure string R, we use the secondary structure consensus available in the Rfam seed alignment. From the resulting RSSPs we remove the maximum prefix and suffix containing neither sequence information (i.e. IUPAC symbol N) nor base pairing information. To obtain a query sequence for blastn, we compute the consensus sequence from the family’s seed alignment. Because blastn does not appropriately handle IUPAC wildcard characters in the query, we choose the most frequent symbol occurring in a column as representative symbol in the consensus sequence. For the RaligNAtor searches, we adjust the cost thresholdd such that we match the complete family. That is, we achieve a sensitivity of 100%. The used operation costs are ωd = ωm = 1, ωb = ωa = 2, and ωr = 3. For the Blast searches, we called blastn with parameters -m8 -b 250000 -v 250000 and a very relaxed E-value cutoff of 1000. From the two RaligNAtor and one blastn outputs we count the number of true positives (#TPs) and false positives (#FPs) and compute ROC curves on the basis of the RaligNAtor scoreblastn bit score. See TableTo assess the potential of using RSSPs for reliable RNA homology search on a broader scale and to investigate the effect of using base pairing information, we evaluated by blastn searchesRaligNAtor achieves a higher sensitivity with a reduced false positive rate compared to searches ignoring base pairing ” in Tableblastn only finds all members of those families whose sequence identity is at least 85%. This is due to the fact that blastn cannot appropriately handle IUPAC wildcard characters. Hence, by taking the most frequent symbol in an alignment column as consensus symbol, the heterogeneity of less conserved positions in the alignment cannot be adequately modeled. For the blastn searches, the average AUC value over all families is only 0.72.In addition, we show in FiguresRaligNAtor is an open-source software package for fast approximate matching of RNA sequence-structure patterns (RSSPs). It allows the user to search target RNA or DNA sequences choosing one of the new online or further accelerated index-based algorithms presented in this work. The index of the sequence to be searched can be easily constructed with program sufconstruct distributed with RaligNAtor.RaligNAtor. To ensure maximal flexibility, the user can also define the base pairing rules from an arbitrary subset ofSearched RSSPs can describe any  RNA secondary structure; see examples in FiguresFor describing a complex RNA with our concept of secondary structure descriptor (SSD), i.e. with multiple RSSPs, the user specifies all RSSPs in one text file. The order of the RSSPs in the file will then specify the order of the RSSP matches used to build high-scoring chains. The chain score directly depends on the score of each match occurring in the chain. This is inversely proportional to the sequence-structure edit distance of the RSSP and its matching substring in the target sequence. Hence, higher scores indicate sequences with a higher conservation which are probably more closely related to the sought RNA family.Chaining of matches discards spurious matches not occurring in any chain. An additional filtering option eliminates matches overlapping another with a higher score for the same RSSP. This is particularly useful when indels lead to almost identical matches that are only shifted by a few positions in the target sequence.RaligNAtor includes not only matching positions to single RSSPs and chains, but their sequence-structure alignment to the matched substrings as well. In the RaligNAtor software package, all programs for searching patterns support multithreading to take advantage of computer systems with multiple CPU cores. There are two modes of parallelism. At first, different patterns are searched using multiple threads. Additionally, the search space  is partitioned, processing each part using a different thread. Lastly, we remark that our software also provides an implementation of the original algorithm of Jiang et al. for global sequence-structure alignment[The output of lignment, easily RaligNAtor software, stand out from previous search tools based on motif descriptors by supporting a full set of edit operations on single bases and base pairs. See TableScanAlign. Our experiments show superior performance of the index-based algorithms LGSlinkAlign and LESAAlign, which employ the suffix array data structure and achieve running time sublinear in the length of the target database. When searching for approximate matches of biologically relevant patterns on the Rfam database, LGSlinkAlign (LESAAlign) was faster than ScanAlign and LScanAlign by a factor of up to 560  and 17 (29), respectively  and no allowed indels, but became considerably slower when the number of allowed indels was increased. In this scenario, LGSlinkAlign was faster than LESAAlign by up to 4 times. In regard to the two online algorithms, LScanAlign was faster than ScanAlign by up to factor 45. In summary, LGSlinkAlign is the best performing algorithm when searching with diverse thresholds, whereas LScanAlign is a very fast and space-efficient alternative. RaligNAtor also allows to use the powerful concept of RNA secondary descriptors[RaligNAtor integrates fast global and local chaining algorithms. We further performed experiments using RaligNAtor to search for members of RNA families based on information from the consensus secondary structure. In these experiments, RaligNAtor showed a high degree of sensitivity and specificity. Compared to searching with primary sequence only, the use of secondary structure information considerably improved the search sensitivity and specificity, in particular for families with a characteristic secondary structure but low degree of sequence conservation. We remark that, up to now, RaligNAtor uses a relatively simple scoring scheme. By incorporating more fine grained scoring schemes like RIBOSUM[RaligNAtor for RNA homology search can be further improved. Beyond the algorithmic contributions, we provide with the RaligNAtor software distribution, a robust, well-documented, and easy-to-use software package implementing the ideas and algorithms presented in this manuscript.We have presented new index-based and online algorithms for fast approximate matching of RNA sequence-structure patterns. Our algorithms, all implemented in the o 560 1,3 and 17 ( RIBOSUM or energ RIBOSUM, we beliRaligNAtor software package including documentation is available in binary format for different operating systems and architectures and as source code under the GNU General Public License Version 3. Seehttp://www.zbh.uni-hamburg.de/ralignator for details.The The authors declare that they have no competing interests.FM and MB developed the algorithms. FM implemented the algorithms. SK implemented the chaining algorithms. MB initiated the project and provided supervision and guidance. All three authors contributed to the manuscript. All authors read and approved the final manuscript.Supplemental material. Additional file 1 contains additional experiments, figures, and tables.Click here for file


We have developed the MC64-ClustalWP2 as a new implementation of the Clustal W algorithm, integrating a novel parallelization strategy and significantly increasing the performance when aligning long sequences in architectures with many cores. It must be stressed that in such a process, the detailed analysis of both the software and hardware features and peculiarities is of paramount importance to reveal key points to exploit and optimize the full potential of parallelism in many-core CPU systems. The new parallelization approach has focused into the most time-consuming stages of this algorithm. In particular, the so-called progressive alignment has drastically improved the performance, due to a fine-grained approach where the forward and backward loops were unrolled and parallelized. Another key approach has been the implementation of the new algorithm in a hybrid-computing system, integrating both an Intel Xeon multi-core CPU and a Tilera Tile64 many-core card. A comparison with other Clustal W implementations reveals the high-performance of the new algorithm and strategy in many-core CPU architectures, in a scenario where the sequences to align are relatively long (more than 10 kb) and, hence, a many-core GPU hardware cannot be used. Thus, the MC64-ClustalWP2 runs multiple alignments more than 18x than the original Clustal W algorithm, and more than 7x than the best x86 parallel implementation to date, being publicly available through a web service. Besides, these developments have been deployed in cost-effective personal computers and should be useful for life-science researchers, including the identification of identities and differences for mutation/polymorphism analyses, biodiversity and evolutionary studies and for the development of molecular markers for paternity testing, germplasm management and protection, to assist breeding, illegal traffic control, fraud prevention and for the protection of the intellectual property (identification/traceability), including the protected designation of origin, among other applications. The amount of genomic data is growing exponentially, due to the advances in technology and the evolution in the so-called “Next-Generation” Sequencing (NGS), including the latest second- and third-generation equipment. Thus, the former bioinformatics tools from the genic era are evolving to handle the current genomic data. These include alignment algorithms for sequence comparison like the Needleman-Wunsch Yet, these computational improvements in pairwise-alignment algorithms cannot properly handle long or very long sequences, like some partial or complete chromosomes and genomes. Thus, some heuristic strategies have been developed to overcome the large amount of memory and execution time required to align such sequences. The Fast Alignment Sequence Tools (FAST)-All (FASTA) Additionally, the “optimal” pairwise aligners can be ported to new computing architectures to overcome the previous limitations. Though an alternative to speedup algorithms is to use supercomputers, grid computing and clustering using several nodes At this point it is important to distinguish between multi-core and many-core microprocessors. Thus, in many-core architectures, the execution cores are small Central Processing Units (CPU) with fewer resources than a standard multi-core CPU; so in order to extract the full potential of the former, specific parallelism strategies for them should be developed. Besides, the many-core microprocessors can be classified into two different groups: many-core Graphical Processing Units (GPU), and many-core CPU. The former ones have thousands of Stream Processors Units (SPU), sometimes named cores, distributed in a hierarchical way: a GPU has several Thread Processing Clusters (TPC), where each one consists of an array of Stream Multiprocessor Units (SMU), having each one eight SPU. The resources are shared between the SMU in these models. On the contrary, the many-core CPU microprocessors have a matrix of uniform CPU with their own resources, usually interconnected via a high-throughput network. A graphical comparison of both architectures can be seen in There are experimental and commercial many-core CPU models. They include the Adapteva Epiphany IV with 64 cores Following this trend, Tilera has developed a many-core CPU architecture known as the Tile64 microprocessor, being a 90 nm Reduced Instruction Set Computing (RISC) System-on-Chip (SoC) microprocessor with 64 general-purpose CPU called tiles, each one being able of running an independent operative system (Linux). Each core runs at 500–866 MHz and can reach a global 0.166 Tera-Instructions Per Second (TIPS). The tiles are interconnected through a high-throughput network called intelligent Mesh (iMesh) with 31 Terabits per second (Tbps) bandwidth. As seen in Thus, thanks to the emerging many-core technologies, new approaches are available to address the increasing demand of computational power. This allows to tackle the bioinformatics analyses required for the exponentially increasing data generated by the new sequencing methodologies. That is the case of the previously mentioned Needleman-Wunsch (NW) Indeed, although some many-core CPU microprocessors with tens of cores have recently arisen, there are only a few bioinformatics developments deployed in such architectures, as the ones that we have previously published for the Tile64 microprocessor In this report, we have extended our work to the popular Multiple-Sequence Alignment (MSA) algorithm known as Clustal W. The new parallel version of Clustal W exploits the high-throughput parallelism in many-core CPU architectures, optimizing the most time-consuming stage of the original algorithm. This has been accomplished taking into account both the existing parallel versions of the Clustal W for multi-core and many-core GPGPU architectures, as well as our previous experience when developing the parallel MC64-NW/SW http://galactus.uma.es/manycore> web site Different benchmarks were run to test the new algorithm against the previous developments and to quantify its performance. Since the MC64-ClustalWP2 algorithm was developed to align relatively large sequences, for instance, from 10 kilobases (kb) to 300 kb, it was tested with a family of organisms whose genome lengths fall into such a range. It should be noted that this approach would require an excessive execution time in a multi-core system or would not even be allowed in a GPGPU architecture, due to the length of the sequences. The MC64-ClustalWP2 source code is available under the General Public License (GPL) license at the <The multiple-sequence alignments allows the comparison of two or more sequences, in contrast to the pairwise aligners like Smith-Waterman, which are limited to just a couple of them. Therefore, the former are particularly useful to identify identities (similarities) and divergences (differences) between many sources, allowing to build evolutionary phylogenetic trees (dendrograms). Thus, the identification of small and large variations, mutations or polymorphisms like base changes and insertions/deletions (indels), duplications or amplifications, recombinations and rearrangements like translocations can be exploited to develop molecular markers for identification, including germplasm management and protection, paternity testing, marker-assisted selection and breeding, illegal traffic control, fraud prevention and traceability. This can be applied to the Protected Designation of Origin (PDO), which is a label of food products from some geographical areas, showing particular organoleptic or otherwise desirable characteristics, conferring them a higher quality than similar products from other sites.However, generating a Dynamic-Program Matrix (DPM) in the sense of “optimal” pairwise aligners, but for more than two sequences, means a Nondeterministic Polynomial (NP) complete complexity problem in computation, and therefore no fast solution to it is known by definition. For this reason, several heuristic strategies were developed to simulate the behavior of the n-dimensional matrix, using a batch of pairwise alignments and simpler DPM. That is the case of MULTAN The first version of MULTAN was programmed in the FORTRAN language, but later on it was re-programmed in the C language, and a few new functionalities were added, like the generation of a dendrogram in the final stage Furthermore, as with pairwise aligners, quicker MSA heuristic methods which do not rely on a DPM have been also developed, as the Multi-LAGAN general anchoring-based method On the other hand, the Clustal W design is divided into three main stages: the distance matrix generation, the guide tree generation and the progressive alignment. It can be easily determined that the most time-consuming stages are the first and the last ones In order to properly optimize the Clustal W parallelization, each stage of the algorithm should be carefully analyzed. Traditionally, the Clustal W has been used to align short sequences of nucleic acids (DNA or RNA) and peptides, with high-performance implementations of Clustal W exploiting parallelism to obtain the best results for larger number of such sequences. However, as the MC64-ClustalWP2 focuses on longer DNA sequences , the parallelization and optimization strategies should be different.As previously noted, the Clustal W is divided into three main stages, shown at the top of The pairwise alignments of the first main stage are calculated using the local Myers-Miller algorithm The clustering stage builds a phylogenetic tree from the previously calculated distance matrix, returning an unrooted tree with the evolutionary distances between branches. This is the less time-consuming stage n in a well-balanced tree of n nodes for the best-case scenario. In a deeper level, the core of the aligner can be divided as well into two independent loops to be run separately, corresponding to the forward and backward loops Finally, the progressive-alignment stage globally aligns sequences in an iterative way, following the path described by the unrooted-guide-tree topology. This stage starts aligning the closest pairs of leaf sequences. Then, the resulting pairwise alignments must be aligned again, following the closest path given by the guide tree. This high-level alignment strategy is called profile-profile alignment, where a profile is obtained from a previous intermediate of the multiple alignment, following a path from the leaves to the top of the tree. Thus, a profile alignment presents a more advanced scoring system and is more computationally expensive than a common-pairwise alignment. Furthermore, dependencies in the tree do not allow parallelizing all alignments, as occurs in the first stage. Therefore, only independent parallel-branches of the tree can be simultaneously executed, and thus may be restricted by the tree shape. This low-parallelization factor is determined by the log In order to exploit the parallelism in the many-core architectures, the tasks to be executed must be spread among all the available cores, which should communicate between themselves at a high speed. Thus, the first step carried out in this work was considering Clustal as three main tasks to be independently parallelized. In addition, as we focus on aligning a few long sequences, the second stage can be ignored, because its execution time can be considered negligible. An exhaustive analysis of the Clustal W allowed us to identify the progressive alignment source-code that should be rewritten, in order to have a structure very close to that of the pairwise-alignments stage. A bird's eye Clustal W structure of such approach is shown at the bottom of This chart reveals the inherent similarity between the first and the last stage, where the alignment of a pair of sequences is replaced by the alignment of a pair of profiles. These profile alignments make use of a more-advanced scoring system, and they are far-more computational expensive. In particular, the score calculus function, prfscore, the hot spot of the aligner, is a time-consuming vector multiplication .In order to parallelize the first ClustalW stage in a previous work, we replaced it with an iterative call to the MC64-NW/SW pairwise aligner Following the Tile64 terminology, a geometry is a set of adjacent cores that forms a rectangular shape, behaving as an independent subset of the Tile64 chip from a functional point of view. The Tile64 can be partitioned into several geometries, but any tile can only belong to a particular geometry at a given time. In the present work, we propose a much more advanced and fine-grained approach, being a new phase in our efforts to improve the performance of Clustal W, in which the parallelism is exploited at two levels: i) several alignments are calculated simultaneously in different geometries of cores; and ii) inside each of them, the alignment is computed in parallel by all the cores contained in it.n workers calculate the full DPM for the forward stage (F-xx geometries from this point on), but geometries of only one worker perform the backtracking and alignment generation (B-xx geometries from this point on). Thus, several working geometries of these two classes collaborate to calculate all the alignments, taking the most out of every available tile. Meanwhile, the high-level controller schedules and manages all the pending jobs. Furthermore, the temporary grid-cache data must be shared between their geometries, in order to communicate a forward stage job with its backward stage counterpart.To achieve this new run-distribution requires a higher-level controller to schedule alignments among all groups of geometries. This new design unrolls the two main loops in the pairalign function, and replaces the original linear-space approach with a call to the MC64-NW/SW which, in turn, uses an advanced FastLSA strategy. With this unrolling method, each pairwise alignment is assigned to an available geometry, in the same way as our previous MC64-NW/SW works With MC64-ClustalWP2 we propose as well a new parallelization strategy for the progressive-alignment stage. As stated above, k value, the speedup will depend on the number of available processing resources, as long as the scalability factor is very close to that of our previous MC64-NW/SW However, two levels of parallelism cannot be achieved in this progressive-alignment stage, because each alignment profile depends on the previous ones; these dependencies are given by the guide tree. Our parallel approach transforms the prfalign function into a new parallel program, using the same FastLSA strategy to distribute a single alignment to all the available tiles in the many-core system. Thus, applying the wave-front parallel strategy and selecting an optimal In this design, a controller tile is in charge of the management of the grid cache and the distribution of jobs among the rest of available worker tiles. The profiles of the original Clustal W are still used, so the profile initialization is performed by the controller tile, whereas the workers call the original prfscore calculus-function profile. The new implementations of both the pairwise and progressive aligners, as well as their relative speedups, are discussed in the next sections.The main asset of the many-core CPU Tile64 System-on-Chip is the execution of parallel programs, due to its 64 cores/tiles running at 866 MHz in a TILExpress-20G card. The drawback is that inherently sequential code is poorly performed by a single tile. Considering that the Clustal W contains purely sequential blocks of code, yet being potentially parallelizable, we have based our implementation on a heterogeneous programming model. In such an approach, the code is commonly executed in the host CPU, using many-core GPU cards as code accelerators, effectively exploiting the best complementary potential of each kind of microprocessor. Different standards and programming methodologies have been proposed to unify and schedule the tasks in these systems, like the OpenCL The Clustal W 1.83 source code, written in the C programming language, has been carefully adapted to support the heterogeneous programming approach, generating a x86-Tile64 implementation of the MC64-ClustalWP2 parallel algorithm. Thus, the MC64-ClustalWP2 is composed by three executables: one runs in the host and two run in the Tile64. The main one is executed in the host; it processes the input parameters, sets the internal variables and orchestrates the three stages of the algorithm, eventually producing the multiple alignment. The three stages of the Clustal W are scheduled between the host CPU and the many-core CPU, as follows:The pairwise alignments are launched in the Tile64 environment, following the schema previously described. The pairwise-alignments overseer is the previously named “high-level controller”, a Perl script launched in the host to supervise the work distribution among the geometries.Once every pairwise alignment and its score are obtained, the host performs the clustering algorithm, whose execution time is negligible compared to the other stages, since the number of sequences is reduced. Therefore, the parallelization is not worth the effort in this stage. The result is the phylogenetic tree.Again, each profile alignment of the progressive-alignment stage is executed in the Tile64 microprocessor, following the job distribution previously stated. Every intermediary result is processed by the host, in order to calculate the next one, until the final multiple alignment is obtained.Therefore, this hybrid work distribution exploits both architecture strengths, thus running the sequential code in the PC host and the parallel code in the Tile64 microprocessor. The complete workflow can be seen in the The MC64-ClustalWP2 pairwise aligner design requires two different job classes to be run in the many-core platform; they deal with the forward and backward stages, respectively. In order to balance the developing effort versus the performance gain, the starting point was our local MC64-NW/SW pairwise aligner F-xx geometry, whereas the backward stage is launched on a B-xx one. This requires to share data between the two controllers of the same alignment (now decoupled), by temporarily storing the grid cache and the internal variables in the TILExpress-20G card 8 GB SSD file system. On the other hand, the original MC64-NW/SW used 6 bytes per cell, since gap penalties are never greater than 128 k values calculated in the MC64-NW/SW parallelization are invalid and must be recalculated.Both stages were decoupled, but keeping the same controller-worker strategy. So, in order to run any job, at least two tiles are needed in each stage: one acting as the stage controller and the other executing the code of the worker stage. With this approach, the same executable runs a forward or a backward stage, depending on a flag parameter: the forward stage is launched on a Taking into consideration that four cores are reserved for the internal management of the chip and communications with the host, the 60 remaining available cores are distributed into a number of static geometries. Different empirical tests have demonstrated that the optimal geometries to use are four forward stage geometries of 7×2 tiles and two backward stage geometries of 2×1 tiles. The rationale of such result is that the forward stage is far more computationally expensive than the backward one. Nonetheless, when the number of pairwise alignments is between five and eight, it is better to use eight 7×1 F-xx geometries, because this optimizes the workload by avoiding a second batch of alignments with idle geometries.To achieve two levels of parallelism, a “high-level” controller is needed to manage several alignments simultaneously; this controller appears in Although other kinds of communications would be possible, like a telnet access, a client-server approach using the available Ethernet ports or a serial console, the communication between the Perl script and the Tile64 is performed via the more straightforward tile-monitor command line tool: when the Perl script starts a new tile-monitor session, it takes input/output communication channels through which the commands and results are sent. The Perl script manages pending jobs and available geometries, in order to schedule workload. Finally, when every pairwise alignment has been calculated, the MC64-ClustalWP2 pairalign function gets all of them from the Perl script. Starting from them, the scoring matrix is computed as the original Clustal W does.The implementation of the progressive aligner requires as well communicating the MC64-ClustalWP2 program, run in the host, with the parallel code executed in the Tile64. However, an intermediate script is not needed this time, as long as the algorithm has only one level of parallelism .This stage is controlled by the host main executable, which launches each profile alignment on the Tile64 environment. To do this, many internal variables must be shared between these both hardware environments. In total, 37 different variables and structures are needed, in order to rebuild the profiles and run the algorithm in the Tile64; other five are updated and returned to the host. To perform these communications and, at the same time, to avoid spreading dependencies throughout the code, we have built an intermediate layer library, called MC64-NWProfile_ParameterManagement, that encapsulates data structures and communication functions. This layer carries out communications using properly-formatted plain-text files. This facilitates uploading the required data into the Tile64, as well as download from it when required. In addition, the MC64_ParameterManagement provides the implementation of the profile-alignment algorithm, which is executed in the Tile64 microprocessor, as seen in The profile alignment algorithm for the Tile64 microprocessor has been developed starting from the MC64-NW/SW code, and applying several major changes to align profiles instead of sequences. For internal purposes, we have called MC64-NWProfile this new sub-algorithm, due to its global behavior. Each profile is a bi-dimensional structure of 35×m, where the height (35) is determined by all the possible residues plus gaps, whereas the width (m) is determined by the profile length; i.e., the length of the globally aligned sequences in the block. The progressive alignment is based on the alignment of these structures, with the prfscore function determining the score for position i,j of the DPM from the vector multiplication of the profile1[i] row and the profile2[j] row.k values for each pair of profiles are completely different from those calculated in the pairwise-alignments stage. Thus, new optimal k values for significant profiles lengths must be pre-calculated, and, when a pair of profiles is to be aligned, the actual k value is interpolated from these ones.Firstly, the MC64-NWProfile computes the profiles using the internal variables passed to it. Once calculated, the algorithm follows the same controller-workers approach than the MC64-NW/SW, using a similar wave-front parallelism. However, the prfscore is now a vector-multiplication operation instead of an indirect-access operation. Additionally, the cells are 12 bytes instead of 6 bytes, so the job-partition optimal k values. Indeed, both the grid cache and the profile copies coexist in the available 8 GB . As a consequence, the available memory to store the grid cache decreases using this approach, and then, a low k value may produce a memory overflow.To calculate a cell's content in this parallel approach, each worker requires accessing to the profile structures to compute the scores and alignment. However, for relatively long sequences, each profile can take several megabytes , so a new problem arises: the location of these two profiles in memory. A direct approach to overcome such limitation is to store a single copy of the profiles in the shared memory, being accessible by both the controller and every worker. Yet, unfortunately, the shared memory access is very slow (can consume tens of CPU cycles) in comparison to the local memory access. Therefore, a better approach is to copy the profiles into the local memory of each tile. This simple modification represents an 8x speedup in execution time, though it requires much more memory , and thus restricts the range of valid k values. Therefore, an intermediate approach is to only copy the necessary fragments for the current job to the worker local memory. Hence, when a worker receives a new job, it brings these fragments of the profiles to the local memory and when finished, it frees the memory. This approach opens the door to use small k value sizes, though the handicap is that the same fragments are eventually copied in the same worker several times for each alignment. In spite of this, using low k values instead of bigger ones, allows obtaining an additional 1.5x overall speedup, as it has been empirically demonstrated (data not shown). Finally, when the alignment is obtained, the local Clustal W variables are modified and returned to the host, which generates the intermediary multiple alignment and continues the process, following the guide-tree topology to progressively generate the next alignments.In contrast, the MC64-NWProfile obtains a better performance with lower Once the algorithm was developed and implemented using the approach stated above, the MC64-ClustalWP2 was tested to benchmark its performance against the original algorithm and other parallel implementations, using different architectures and approaches. At first, we launched stress tests with ten sequences of different sizes and measured the speedup of the MC64-ClustalWP2 against other Clustal W implementations. A further test was carried out to analyze families of long sequences, which are aligned in much less time than with other general global MSA strategies, as discussed below.In order to test the relative performance of the new MC64-ClustalWP2 parallel strategy implementation with the many-core Tile64 microprocessor, the original Clustal W algorithm and other parallel implementations were compared using a set of sequences with different sizes. In particular, the tested algorithms were the Clustal W 1.83  All of the algorithms tested were run in the same environment (Intel Xeon Quad Core 2.0 GHz PC with 8 GB of quad-channel DDR2 memory), providing the same results but with rather different execution times. The MC64-ClustalWP2 used the Tile64 microprocessor on-boarded in a TilExpress-20G card with 8 GB RAM. The execution time and minimum-gain ratio of the MC64-ClustalWP2 algorithm against the other ones is shown for each set of sequences in The results reveal the best performance of the MC64-ClustalWP2 against every other implementation. This tendency increases with the length of the sequences, because a higher parallelization factor is obtained. Thus, the MC64-ClustalWP2 reaches a speedup of more than 7x when compared to the best multi-core implementation . In particular, both multi-core approaches present very similar behaviors with differences only appreciable when aligning 300 kb length sequences. On the other hand, the MC64-ClustalWP2 obtains a speedup factor of more than 18x when compared to the original algorithm. The Herpes simplex virus type 1; HHV-1) genomes, publicly available at GenBank <http://www.ncbi.nlm.nih.gov/genbank>, were selected to be aligned. The complete genome of the HHV-1 is about 152 kilobase pairs (kbp), and the different strains should a priori present several mutation areas, due to the high mutation-rates of this kind of DNA viruses. Therefore, such alignments can be carried out with dynamic-programming algorithms like Clustal W.As a final test, the MC64-ClustalWP2 was used to run an experiment whose execution time is near prohibitive when non-parallel or less-efficient parallel MSA methods are applied. In this stress test, 37 different human herpesvirus 1 . The final phylogenetic tree is shown in The quality of the alignments provided by any MSA algorithm, including Clustal W, depends on many factors, being an issue open for discussion This alignment was also performed with other global MSA algorithms, and even with other Clustal W implementations , but the quickest among them  required 274,149 seconds (3.17 days) to complete the alignment, being 7.59 times slower than our approach which, hence, becomes the only one capable to deal with this type of alignments in a reasonable time. On the other hand, some of the MSA algorithms do not generate an evolutionary tree of the genomic sequences, as they are oriented to find only the similar regions. Besides, they lack accuracy when the aligned sequences have highly polymorphic regions , which is typical for most viruses We have developed a new parallel strategy for Clustal W, which is one of the most relevant algorithms in bioinformatics for multiple-sequence alignments and dendrogram generation. This algorithm uses the pairwise-alignment operation as its core. Our work has focused on large sequences, a field where the resources of a GPGPU architecture have proven insufficient but, in contrast, our previous developments in many-core technologies for bioinformatics, like the MC64-NW/SW, have shown a high performance The MC64-ClustalWP2 has been implemented for the heterogeneous standalone system x86-Tile64, using the Tile64 many-core microprocessor. Thus, within a single computer, the MC64-ClustalWP2 offers an impressive performance, being scalable to larger many-core systems using the same parallel strategy. This represents also a proof of concept that new parallel strategies can be exploited to harness the new developments in the many-core microprocessors to analyze the increasing amount of data generated by the second- and third-generation sequencing platforms. Thus, the Clustal W dynamic-programming approach that we have developed  can align genomes with highly polymorphic regions These developments allow to identify identities and differences to generate dendrograms for biodiversity and evolutionary studies, and to develop molecular markers like the ones based, for instance, on Single Nucleotide Polymorphisms (SNP), as well as on microsatellites, also known as Short Tandem Repeats (STR) in animals and as Simple Sequence Repeats (SSR) in plants, for germplasm management, breeding, identification and protection of the intellectual property, PDO, illegal traffic control and fraud prevention, as we have described for the olive oil Additionally, the MC64-ClustalWP2 strategy can be deployed for any many-core system with very little effort, like the current and future Intel Xeon Phi and the Tilera microprocessors. In particular, we have already developed a threaded version of MC64-NW/SW for the Intel i7 processor, and we are planning to use it as starting point to take full advantage of every Intel Xeon Phi core, where several threads must be executed to achieve an instruction-per-cycle performance. Furthermore, the approach can be escalated, both in processor and in memory resources, interconnecting several TilExpress-20G cards through their 10GBase-CX4 connectors to build a powerful cluster of Tile64 microprocessors, with thousands of cores, increasing both the number and the length of the sequences to align. We have already designed such a cluster, which is currently being evaluated On the other hand, a possible optimization in order to improve even further the parallelism of the first stage is to get rid of the intermediate Perl script and to transfer its high-level controller functionalities to a core of the many-core system. This allows using a single geometry where each worker is independent , receiving in each case the data from the input sequences to be aligned. This new approach would maximize the core usage in the first stage, but would require more data transfers and memory resources in the overall system, which is strongly limited to 8 GB of RAM in the case of the Tile64 hardware. As an additional optimization for the alignment of thousands of sequences, the clustering stage could be parallelized as stated in On the other hand, the MC64-ClustalWP2 should be considered as a complementary tool to heuristic algorithms like the MGA http://galactus.uma.es/manycore>. In the same way, the MC64-ClustalWP2 project is available under the GNU General Public License (GPL), and its source code can be downloaded from the same website.Lastly, the MC64-ClustalWP2 algorithm is freely accessible to the scientific community and can be remotely executed on our server (a system that integrates the TilExpress-20G card) by means of a web service at <