Partitioning the human immunoglobulin variable region into variable (V), diversity (D), and joining (J) segments is a common sequence analysis step. We introduce a novel approximate dynamic programming method that uses conserved immunoglobulin gene motifs to improve performance of aligning V-segments of rearranged immunoglobulin (Ig) genes. Our new algorithm enhances the former JOINSOLVER algorithm by processing sequences with insertions and/or deletions (indels) and improves the efficiency for large datasets provided by high throughput sequencing.In our simulations, which include rearrangements with indels, the V-matching success rate improved from 61% for partial alignments of sequences with indels in the original algorithm to over 99% in the approximate algorithm. An improvement in the alignment of human VDJ rearrangements over the initial JOINSOLVER algorithm was also seen when compared to the Stanford.S22 human Ig dataset with an online VDJ partitioning software evaluation tool.HTJoinSolver can rapidly identify V- and J-segments with indels to high accuracy for mutated sequences when the mutation probability is around 30% and 20% respectively. The D-segment is much harder to fit even at 20% mutation probability. For all segments, the probability of correctly matching V, D, and J increases with our alignment score. Immunoglobulins (Ig) are a family of proteins that identify and bind foreign pathogens, e.g., bacteria and viruses. Diversity in the antigen-binding region of Ig provides an appropriate immune response to the wide array of pathogens confronting individuals. This diversity is generated by VDJ recombination, which joins a Variable (V) gene segment, a Diversity (D) gene segment, and a Joining (J) gene segment from distant regions of DNA to potentially create about 10 billion different antibodies, each of which binds to a distinct epitope.During recombination, nucleotide excision of the germline gene termini and the addition of nontemplated N nucleotides by terminal deoxynucleotidyl transferase (TdT) at the V to D and D to J junctions provide additional diversification. Furthermore, during germinal center reactions B cell receptors undergo further changes including somatic hypermutation as well as nucleotide insertions and/or deletions (indels) of various length which creates a larger potential repertoire of antibodies.Previously the JOINSOLVER  algorithThis paper addresses the challenge of both accurately aligning heavily mutated Ig rearrangements, potentially with indels, to the nearest matching V, D, and J germline gene and identifying junctional N nucleotides. We introduce a sequence alignment algorithm that approximates the results of a dynamic programming (DP) algorithm, which can save up to 98% of the computational time.i, j) is the best alignment up to the ith base of the first sequence and the jth base of the second sequence. Dynamic programming algorithms have rules that define how to initialize the matrix, how to fill matrix elements after initialization, and where to find the highest score. A traceback is kept to mark the path of the best alignment through the matrix starting from the highest score matrix element. Most DP algorithms rules allow for insertions, deletions, mismatches, and matches. A match or mutated base is a one base step forward in both sequences, corresponding to a diagonal step in the DP matrix. Insertions and deletions increment one sequence, but not the other, corresponding to a right or downward step, respectively. Typically DP algorithms have poor performance, of order O(NM) where N and M are the lengths of the two sequences being matched [Dynamic programming algorithms have been used to align sequences since 1970 . Typical matched . Durban  matched  providesPrevious work suggests that banding, or working along a diagonal band in the DP matrix, improves the performance of DP algorithms . In the The alignment of the V-segment accounts for most of the computational work. The amount of work is related to the length of the segments and the number of sequences being compared against the segment. In germline database used by JOINSOLVER, there are many more V-germline genes (289) than D- or J-germline genes (84 and 12 respectively), and the V-germline genes have the longest germline sequences (~285 nucleotides) used in the analysis. The algorithm balances the need for aligning these irregular V-segments with the need to analyze large numbers of sequences provided by next generation sequencing. A new desktop application, HTJoinSolver, is provided as an implementation of the new partitioning method.Similar to the original JOINSOLVER algorithm, conserved motifs initiate the alignment process . In prepNext the partitioning method identifies the JH segment using a highly conserved J motif in a process similar to the VH alignment Figure c. The reUnlike the DP algorithms commonly used for sequence alignment, the approximate backwards algorithm starts from a known point and works backwards to the start of the matrix formed by the germline gene and the unknown or query sequence. In this study, the scoring rules are +5 for a match and −4 for a mismatch; gap opening (indel formation) is penalized −30, and gap extension (continuation of indel) is penalized −1. The numbers in the matrix elements are the scores of the best alignment up to the bases corresponding to the column and row. The algorithm starts at the location of the 5’ T of the conserved V-motif in both the germline sequence and the query Ig sequence. The score is initialized to zero. The algorithm steps one base backwards, toward the 5’ end in both sequences. A traceback, which points to the previous matrix element, is kept to mark the alignment. As long as the bases match, the algorithm continues stepping in the 5’ direction.If the bases don’t match, the algorithm considers the possibility of a mutation or an indel. The algorithm steps forward 10 bases (toward the 3’ end) and calculates a small 40 × 40 DP matrix using an algorithm similar to the Needleman-Wunsch algorithm ,10 to fiFigure After completing the alignment, sequences with indels are re-aligned from 5’ end of the alignment to the 3’ end of the V-motif. The two alignments are compared for consistency. This quality control step catches suboptimal alignments caused by local maxima.After aligning the 5’ end of the V-segment, we align the 3’ of the V-segment starting at the conserved VH-motif. The V-end alignment algorithm, which is a type of overlap DP algorithm , begins The start of the J-segment is aligned using a DP algorithm similar to a Needleman-Wunsch algorithm; however, the maximum score can be anywhere in the DP matrix, and the algorithm is run backwards from the J-motif. The maximum score corresponds to the start of the J-segment. The alignment is determined by tracing back from the highest scoring matrix entry.The D alignment uses a local alignment algorithm similar to a Smith-Waterman algorithm. A match is given a score of +1, however a mismatch sets the score back to 0. When the score = 3, the algorithm looks back 4 steps to check for a potential mutation, if the score before the match start is greater than 3, the current score is increased by the previous score. This allows a mismatch to be recognized as a mutation in the D-segment if and only if there are at least 3 consecutively matching bases on both sides of the mutation. The probability of randomly matching 3 bases is around 1.5% or 3% on either side of the D-segment, and 3 was chose to keep the probability of random matches below 5%. The termination condition is the maximum score anywhere in the matrix. These modifications to the Smith-Waterman algorithm maintain the basis of the original JOINSOLVER, matching D-segments based on consecutive matching bases.In the event, the V-motif is not found in the sequence, the algorithm looks for the motif with one mutation. If the motif with a single mutation is found, the approximate backwards algorithm is run using the mutated motif location. Because the J-motif is shorter, mutations are not allowed in the J motif.When the location of the V-motif is wrong, the alignment is very poor resulting in a low score. Setting a score threshold of 3.1/base effectively catches an incorrect motif location. Before relinquishing to a slower algorithm, one last attempt is made to identify the location of the motif. The sequence is aligned to the first germline sequence in our database (IGHV1-18*01). The motif location from this alignment is used to align the query sequence to all other germline genes to determine the best match or alignment. If the highest scoring alignment is still below 3.1/base, we run an overlapping sequence DP algorithm for the V or a Smith-Waterman algorithm for the J alignments. Our only modification is that the end of the V- or J-segment occurs at the maximum score anywhere in the matrix, not just in the last row or column.P(n) = cne-3(n-1), where n is the number of indels, and c = (1-e3)2 is a normalization constant. In our simulator, the indel length is selected from a Poisson distribution. If a length of zero is selected, the distribution is re-sampled. Table Artificial VDJ rearrangements were generated by randomly recombining a VH-, D-, and JH-germline gene from the JOINSOLVER germline database. A random number of terminal nucleotides from the 3’ V, 5’ & 3’ D and 5’ J were removed to mimic exonuclease activity. Various random numbers of nucleotides were added to the V-D and D-J junctions to mimic TdT activity. In our simulation, each base had a fixed probability of being mutated (referred to as the mutation probability). The mutation probability should not be confused with the mutation frequency, which is the number of mutations is a sequence divided by the sequence length. The number of indels in our simulation was randomly selected using a distribution that heavily favors one indel per sequence, = cne-3n-, where nTwo sets of simulations were performed. In the first simulation, 10,000 rearrangements were generated with mutation probabilities ranging from 0% to 95% in increments of 5% throughout the VDJ sequence. Using these rearrangements, the approximate backwards algorithm was compared against a complete DP algorithm, to calculate the success rate. The second set of simulations with 1,000 rearrangements engineered with a mutation probability of 3.5% was performed to compare HTJoinSolver with the original JOINSOLVER using the original JOINSOLVER germline database. The results of an alignment are considered a successful match to a germline if the algorithm returns the correct gene; the exact allele is not required.http://www.emi.unsw.edu.au/~ihmmune/IGHUtilityEval/evalForm.html) compares the performance of VDJ partitioning tools on the Stanford_S22 dataset. As a performance metric, the tool reports the percentage of V, D, or J assignments to germlines that are not present in the predetermined S22 genotype.Additionally 13,153 human sequences from the Stanford.S22 dataset  results.www.imgt.org/download/GENE-DB/IMGTGENEDB-ReferenceSequences.fasta-nt-WithGaps-F+ORF+inframeP, downloaded July 30, 2014). HTJoinSolver provides methods to download and reformat germline genes from IMGT. These methods allow the user to maintain an up-to-date library of germline genes usable by HTJoinSolver. When comparing HTJoinSolver with JOINSOLVER, we used the JOINSOLVER germline database, which was derived from IMGT and included pseudogenes.The germline genes used in HTJoinSolver are IMGT reference sequences ,19, www.https://dcb.cit.nih.gov/HTJoinSolver.HTJoinSolver is available for public use, and can be downloaded from The approximate backward algorithm was designed to quickly estimate the alignment score of a complete DP algorithm for overlapping sequences without sacrificing accuracy. Figure The results of the HTJoinSolver approximate algorithm were then compared to the original JOINSOLVER V-alignment algorithm. Although the JOINSOLVER algorithm was not designed to identify indels, large portions of the rearrangement can align correctly up to the position of the indel, but the remaining V-segment was mismatched. Unfortunately, the results were not usable for mutation analysis because the offset alignment appeared as a region of high mutation instead of a single indel. Adding V-segment indels to our simulations hurt the overall performance of the original JOINSOLVER. With a 3.5% mutation probability and simulated indels, the original algorithm selected the correct V-germline 61% of the time, whereas the new algorithm selected the correct V-germline over 99% of the time. The percent of success for the old vs. new algorithms were 73% vs. 91% and 98% vs. 99% for the D- and J-germlines, respectively. These simulations show that the approximate backwards algorithm performs as well or better than the original JOINSOLVER algorithm.As the mutation probability increases, sequence alignment becomes more difficult. The effect of the mutation probability on the success rate can be seen in Figure  The D-segment, which is relatively short, is the most difficult segment to identify and align. The D-segment nucleotides must be found within the CDR3, which is the most complex region of the VDJ rearrangement. In order to locate the CDR3 5’ and 3’ boundaries the algorithm must first correctly identify the 3’ end of the V-segment and the 5’ end of the J-segment whose nucleotide sequences may differ from the germline gene if nucleotide excision and/or mutation has occurred. N addition may be present or absent and varies in the number of nucleotides flanking the D-segment. Furthermore, D-segment mutations decrease the number of consecutively matching D-nucleotides. Thus, the D alignment success rate drops off almost immediately as the mutation probability increases. The curve in Figure In our simulations, the alignment scores tend to decrease as the mutation probabilities increase. However, for most biologically relevant situations, the success rate remains high. Figure The second column in Figure The “Evaluation of IGH partitioning using inferred genotypes tool”  was usedHigh-throughput sequencing of B-cell receptor VDJ rearrangements produces vast numbers of sequences and requires extremely fast algorithms for VDJ alignment. The original JOINSOLVER algorithm was fast and accurate, but was not designed to handle indels. One of the main reasons for creating HTJoinSolver was to handle alignments with indels, while maintaining or improving the speed. Aligning a sequence to approximately 300 VH-germlines takes about 1.7 second on a 2.93 GHz Intel Core i7 iMac. Our approximate algorithm takes around 207 ms. Falling back to standard methods can take over 7x longer. One million lightly mutated sequences  and may be incorrectly aligned. Usually, the score threshold will catch the incorrect alignment, and fall back to the overlap DP algorithm. If it is known beforehand that long stretches of indels or mutation will occur, increasing the block size may improve the performance. The larger the block size, the less likely the algorithm will fall into a local maximum requiring a fall back algorithm. However, the price for a large block size is increased computation time.We were not surprised by the comparison between the original JOINSOLVER and HTJoinSolver. We wanted to show that the HTJoinSolver can perform complete alignments of VDJ rearrangements containing indels. Therefore, all of our simulations included indels, which hindered JOINSOLVER’s performance. The increased success rate of the improved algorithm is due to the ability to identify the size and position of indels. The D and J results were very similar because the DP algorithms used to align the D and J regions resemble the original JOINSOLVER algorithms, which were designed to prevent random D matching at the 95% confidence level .The ability to successfully match a sequence to a germline is heavily affected by the length of the V-, D-, or J-segment and the mutation probability. Matching V-segments is easiest because longer germline sequences are available for comparison. As the mutation probability increases, the success rate drops. Both of these cases can be explained by the loss of information from the original germlines by truncation and mutation. Additional studies on whether the success rate can be explained by a noisy-channel model could give additional insights into the theoretical performance of any alignment algorithm.Germline sequence similarity presents an additional challenge for HTJoinSolver, especially for aligning heavily processed sequences. The problem is compounded by the fact that the D-germline sequences can be similar to the 5’ end of the J-germline sequences. A small fraction of our simulations show a VH-JH distance of zero, even though we know a D-germline is present along with VD and DJ junctions. This ambiguity cannot be resolved without additional information.When compared against the original JOINSOLVER algorithm using the Evaluation of IGH partitioning using inferred genotypes tool, we see an improvement in the V, D, J, and overall results. However, the improvement is small because the sequences are minimally mutated. The median mutation frequency for the S22 dataset is 0% and the mean is 0.9%.The results of the evaluation tool are highly dependent on the set of germlines used in the comparison. When compared against other partitioning tools using the original JOINSOLVER database, the results show that HTJoinSolver makes fewer V, D, and J assignment to germlines and alleles not present in the S22 genotype than the tools in the study by . When thThe development of an evaluation tool with a standard dataset is an important first step, but without a better method of assessing the quality of results, and a standard library of germlines, comparison between partitioning tools remains difficult. We have shown that for the original JOINSOLVER® database, always including the highest scoring D alignment, even when we would not recommend using the alignment, improves the partitioning performance according to the evaluation tool. Furthermore, providing a D-segment that occurs in the S22 genome will always improve the performance as measured by the evaluation tool, even if the D-segment is very likely to be wrong. Relaxing the criteria for including a match  should not have improved results of the evaluation. After updating the germline database, the performance for the V-segment dropped. With more allelic variants to choose from, the system is more likely to match previously unknown allelic variants causing the evaluated performance to decrease. Improvements in the D alignment can be explained by the removal of the reverse germlines from the database when we switched from the JOINSOLVER germline database to the IMGT reference database for HTJoinSolver. Interestingly, always including the highest scoring D-germline, regardless of whether the score is high enough, does not improve the D-segment performance using the updated database. Evaluation results for the same algorithm changed because of differences in the germline database and not on algorithmic differences.These results suggest that evaluating a partitioning tool based solely on whether or not a germline occurs in an assumed genotype is not sufficient. Consider a partitioning algorithm that only assigned IGHV1-3*01 to all query sequences. Since IGHV1-3*01 is in the S22 genome, 0% of the sequences would come from outside the S22 genome. The partitioning algorithm would be evaluated as perfect. However, this algorithm would not make a useful tool. Whereas the evaluation tool is a good idea, currently the tool is too sensitive to similarity between the S22 genome and the germlines present in the partitioning tool’s database.All of the simulations and the comparison with the partitioning tool evaluation software show that HTJoinSolver can identify V-, D-, and J-segment to very high accuracy for reasonably mutated sequences. As the mutation probability increases, alignment becomes more difficult but problematic only for unnaturally highly mutated or truncated sequences. The V-segment is identified very accurately for most biologically relevant cases. The J-segment is fairly well identified from 0-20% mutation probability but above 20% the success rate falls below 90%. At the 20% mutation probability, the D-segment has about a 55% chance of being correct. Regardless of the partitioning software, biologists should be skeptical of short D-matches to highly mutated sequences, even if mutations are not seen in the D.


Sequence alignment has become an indispensable tool in modern molecular biology research, and probabilistic sequence alignment models have been shown to provide an effective framework for building accurate sequence alignment tools. One such example is the pair hidden Markov model (pair-HMM), which has been especially popular in comparative sequence analysis for several reasons, including their effectiveness in modeling and detecting sequence homology, model simplicity, and the existence of efficient algorithms for applying the model to sequence alignment problems. However, despite these advantages, pair-HMMs also have a number of practical limitations that may degrade their alignment performance or render them unsuitable for certain alignment tasks.In this work, we propose a novel scheme for comparing and aligning biological sequences that can effectively address the shortcomings of the traditional pair-HMMs. The proposed scheme is based on a simple message-passing approach, where messages are exchanged between neighboring symbol pairs that may be potentially aligned in the optimal sequence alignment. The message-passing process yields probabilistic symbol alignment confidence scores, which may be used for predicting the optimal alignment that maximizes the expected number of correctly aligned symbol pairs.Extensive performance evaluation on protein alignment benchmark datasets shows that the proposed message-passing scheme clearly outperforms the traditional pair-HMM-based approach, in terms of both alignment accuracy and computational efficiency. Furthermore, the proposed scheme is numerically robust and amenable to massive parallelization. Sequence alignment has become an indispensable tool in modern molecular biology research, as it provides an effective and intuitive way of comparing and analyzing biological sequences. Given a set of biological sequences, the primary objective of sequence alignment is to predict the best overall mapping between the sequences, which accurately aligns the homologous regions that are embedded in them. This provides an effective means for detecting conserved sequence regions with potentially important functional roles. The concept of sequence alignment has had diverse applications in biomedical research –7, whichad hoc scoring schemes have been proposed to obtain intuitive and biologically meaningful sequence alignment results. As an alternative to heuristic scoring schemes, there have been also research efforts to develop probabilistic models for sequence alignment that can be used to evaluate and compare potential alignments and to estimate the symbol-to-symbol alignment probabilities.Typically, sequence alignment is carried out by formulating and solving an optimization problem - either implicitly or explicitly - where the goal is to maximize an objective function that measures the overall quality of the sequence alignment. For example, one simple way of aligning a sequence pair would be to score each potential alignment by assigning a "substitution score" to every aligned symbol pair and penalty scores for gaps and then find the optimal alignment that maximizes the overall score through dynamic programming . In the ad hoc score. This is typically referred to as the maximum expected accuracy (MEA) alignment  is used to balance the contribution from the neighbors and that from the joint probability of  in estimating the alignment confidence score. A large λ gives more weight to the "messages" received from the neighbors in estimating the scores, which tends to penalize gaps more heavily, and it generally leads to longer aligned regions with fewer gaps. On the contrary, a small λ gives more weight to the joint symbol occurrence probability P  while giving less weight to the messages received from the neighbors, which tends to be more lenient to gaps. Once the symbol alignment confidence score cxy is updated for all i = 1, ⋯ , L and j = 1, ⋯ , M, we normalize the scores to keep them within a proper numerical range, as shown in STEP-3. For example, a simple way would be to divide the score matrix C =  by its matrix norm to normalize the confidence scores. After normalization, the updated scores are compared to the scores in the last iteration, and the algorithm terminates if the specified convergence criterion has been met. Otherwise, the algorithm goes back to STEP-2 and repeats the message-passing process.In STEP-1, we first initialize the alignment confidence score In order to evaluate the performance of the proposed message-passing scheme, we carried out pairwise sequence alignment experiments based on the BAliBASE 3.0 protein alignment benchmark . BAliBASλ  to investigate the effect of λ on the overall sequence alignment performance. For the joint symbol occurrence probability P , we used the joint emission probability (at state M) of the pair-HMM that was used in [C by the matrix 2-norm: C←C/||C||2. We terminated the message-passing process if cxy is the current score and cxy  ∝ P , we used the confidence score cxy  to find the MEA alignment through dynamic programming. The predicted alignment was compared to the benchmark alignment in BAliBASE 3.0 to compute the sensitivity TP is the number of correctly aligned symbol pairs, FP is the number of incorrectly aligned pairs, and FN is the number of symbol pairs that are aligned in the benchmark alignment but not aligned in the predicted alignment. For comparison, we repeated similar experiments using the pair-HMM with the same set of parameters as the one used in [For every sequence family in BAliBASE 3.0, we performed pairwise sequence alignment for all possible sequence pairs in the given family. The pairwise alignment was performed in the following manner. First, we estimated the probabilistic symbol alignment confidence score using the proposed message-passing scheme. In our experiments, we used three different values of  used in . At the  used in .Table λ. For example, the message-passing scheme achieved up to 0.23 higher SN and 0.09 higher PPV for λ = 0.25, and up to 0.37 higher SN and 0.19 higher PPV for λ = 0.75. Our experiments showed that a larger λ tends to yield more accurate alignments, while a smaller λ tends to make the algorithm converge faster, hence computationally more efficient. For example, when the weight parameter was set to λ = 0.25, the message-passing scheme was around 2.3 ~ 2.5 times faster than the pair-HMM, while still yielding much more accurate alignments.From Table SNMP (the sensitivity of the message-passing scheme) and SNHMM (the sensitivity of the pair-HMM-based approach) for every pairwise sequence alignment that we have performed in our experiments. Similarly, we calculated the difference between PPVMP (the PPV of the message-passing scheme) and PPVHMM (the sensitivity of the pair-HMM approach) for all sequence pairs in BAliBASE 3.0. Figure SNMP- SNHMM and PPVMP- PPVHMM for all sequence pairs. To avoid any bias from unsuccessful alignments, sequence pairs for which neither method yielded an alignment with at least one correct symbol alignment were excluded. The plots in the left column of Figure SNMP- SNHMM, and those in the right column show the distributions of PPVMP- PPVHMM. The results obtained from the same reference set are shown in the same row, where the first row shows the results on RV11 and the last row shows the results on RV50. As we can see in Figure The results in Table P  of the entire sequence pair) to be estimated, and the exchanged messages  are normalized after each iteration, which ensures that they lie within a reasonable numerical range. Finally, the simple iterative estimation process - in which the neighboring symbol pairs only exchange "local" messages - makes the proposed message passing scheme amenable to massive parallelization through the utilization of modern GPU (graphics processing unit) architecture. These characteristics open up the possibility of applying the proposed message-passing scheme to accurate probabilistic alignment of genome-scale sequences, which has not been possible using traditional pair-HMMs.In this paper, we proposed a novel method for sequence alignment based on an efficient message-passing approach. Given two biological sequences, the proposed method estimates the symbol alignment confidence scores for all possible symbol pairs. These scores are iteratively computed by exchanging messages between neighboring symbol pairs, where empirical evidence shows that these scores quickly converge within several iterations. The proposed message-passing scheme effectively addresses a number of limitations of the traditional pair-HMM-based approach, and extensive performance assessment based on BAliBASE 3.0 shows that the proposed scheme consistently outperforms the pair-HMM approach, both in terms of alignment accuracy and computational efficiency. Considering that pair-HMMs have been widely adopted by many modern multiple sequence alignment algorithms –11, the cxy in the proposed message-passing algorithm bears conceptual similarity to the eigenvalue equation used by the network alignment algorithm called IsoRank [cxy to further improve the performance of the proposed message-passing scheme, and we are currently in the process of investigating several different implementations.Finally, it is worth noting that the formula that is used to update  IsoRank  for esti IsoRank , 19, tec IsoRank , and it


In this work, the development of a parallel algorithm for multiple sequence similarities calculation using the k-mers counting method is proposed. Tests show that the algorithm presents a very good scalability and a nearly linear speedup. For 14 nodes was obtained 12x speedup. This algorithm can be used in the parallelization of some multiple sequence alignment tools, such as MAFFT and MUSCLE.With the advance of genomic researches, the number of sequences involved in comparative methods has grown immensely. Among them, there are methods for similarities calculation, which are used by many bioinformatics applications. Due the huge amount of data, the union of low complexity methods with the use of parallel computing is becoming desirable. The  The use of sequence comparison methods has been remarkably growing in recent years, in response to the data expansion of genomic research. Consequently, methods that reduce the execution time are fundamental to the progress of this area. Many efforts have been made concerning their optimization , 2.With the increase in number of multiple sequences alignments problems, the development of methods which have a lower computational complexity also increases. Nevertheless, just the creation of low complexity methods was not enough to work with high volume of data , 4. The In general, the sequence comparison starts with the similarity calculation between pairs of sequences. It occurs through methods which require or not a prior alignment, varying the algorithm complexity and the level of biological accuracy. The alignment free methods correspond to a class of low complexity methods, which have a low temporal and spatial complexity in relation to the methods which requires a prior alignment. Moreover, they also keep a high level of biological accuracy for divergent sequences. For this reason, they are becoming increasingly important in computational biology. k-mers counting method, which was proposed by Katoh et al.  = F F are, therefore, stored in a triangular matrix M.All obtained values of  k-mers counting method. For this reason, we propose a parallel algorithm that calculates the similarities of multiple pairs of sequences using the k-mers counting method.In an application, the amount of sequences in comparison can be huge, making its implementation unfeasible in a single machine, even with the use of low computational complexity methods, as theThis algorithm dynamically divides the computation among existing processors through a master-slave approach. This parallelism is performed distributing the similarity calculation of sequence pairs to available slaves. It is possible because each similarity pair calculation is independent of other calculations of similarity pair.Initially, the master sends all sequences by broadcasting to all slaves. Using broadcast the master can reduce the overhead with the messages exchange, decreasing the communication time between processors.p-step loop, where p is the number of processors. In The tasks distribution is based on the processor identifier, assigning to each slave the calculation of specific lines of the triangular matrix of similarities. The way this matrix is obtained is exemplified in SimVect) which is sent to master. The master receives all slave vectors and, from these vectors and the identifier the slave has sent, it builds the similarities matrix. From the previous example, the obtained similarities matrix is shown in During the matrix lines calculation, each slave stores the results of all lines in a single vector (In http://www.ncbi.nlm.nih.gov/). For each performed test, we describe specific information of the dataset used, as the number of sequences and the average and maximum length of the sequences.Many tests were performed to measure the performance of the proposed algorithms. The datasets used in these experiments have differences only on the number of sequences to be aligned and their lengths. The sequences used were extracted from the NCBI database , (b) and (c)) illustrated in The graphic (d), also in Comparing the minimum system (2 nodes) with the maximum system (15 nodes) we have an increase of 14 nodes and, approximately, a 12x speedup. Therefore, it can be noticed that the parallel algorithm presents excellent results with a nearly linear speedup. k-mers couting method, a low computational complexity method with good biological results. This calculation is used, for example, in multiple sequence alignment tools. The proposed parallel algorithm has been implemented for distributed memory systems, due to the wide use of Beowulf clusters in genomic research laboratories.In this work, we presented a parallel strategy to calculate similarities between multiple pairs of sequences using theThe tests performed show that the algorithm presents a good scalability and a nearly linear speedup. With the use of 14 processing nodes (slaves), the system achieved a 12x speedup. This speedup is justified by the total independence of the scheduled tasks and by the good load balancing obtained with the triangular matrix lines distribution.Additionally, the communication cost is minimized because the computed data in slaves are sent to master in a single message. Considering the 12x speedup achieved and using Amdahl's law  we can e


Motivation: Alignment-free methods for sequence comparison are increasingly used for genome analysis and phylogeny reconstruction; they circumvent various difficulties of traditional alignment-based approaches. In particular, alignment-free methods are much faster than pairwise or multiple alignments. They are, however, less accurate than methods based on sequence alignment. Most alignment-free approaches work by comparing the word composition of sequences. A well-known problem with these methods is that neighbouring word matches are far from independent.Results: To reduce the statistical dependency between adjacent word matches, we propose to use ‘spaced words’, defined by patterns of ‘match’ and ‘don’t care’ positions, for alignment-free sequence comparison. We describe a fast implementation of this approach using recursive hashing and bit operations, and we show that further improvements can be achieved by using multiple patterns instead of single patterns. To evaluate our approach, we use spaced-word frequencies as a basis for fast phylogeny reconstruction. Using real-world and simulated sequence data, we demonstrate that our multiple-pattern approach produces better phylogenies than approaches relying on contiguous words.Availability and implementation: Our program is freely available at http://spaced.gobics.de/.Contact:chris.leimeister@stud.uni-goettingen.deSupplementary information:Supplementary data are available at Bioinformatics online. Alignment-free methods became popular in recent years, as their runtime is usually proportional to the total sequence length; see Traditional methods for comparative sequence analysis and phylogeny reconstruction rely on pairwise and multiple sequence alignments . A majorguide trees for progressive multiple alignment  word-frequency vector for each of the input sequences. Various distance measures on vector spaces can be used to calculate a pairwise distance matrix from these word-frequency vectors  and, similar as in other alignment-free approaches, each sequence iS is represented by the distanceiS and jS as the distance between these frequency vectors, using some standard distance metric on vector spaces.To define a setP [similarly, average of the distances defined by the patterns This approach can be generalized by considering a whole multiple-pattern version of our spaced-word approach.We call this extension the spaced words in a sequence with respect to a pattern P, we implemented a hash function that maps each spaced word to an integer in k and define the i-th word of sequence S as h for which the value h(iw) by removing the value of the i-th character and adding the value of the new character at i + k. Such hash functions are called recursive or rolling  is calculated with this formula, the values Once h(iw) can be calculated for a sequence S of length n in k. The advantage of using bit operations—compared with recursive hashing with multiplications and divisions—is that rotating bits is much faster than these algebraic operations.Note that, with this recursion, all hash values contiguous words. The above recursive formula spaced words as long as the number of don’t care positions in the underlying pattern is small. To do this, we first calculate the hash value for words of length ℓ, and we then remove the terms corresponding to the don’t care positions in the underlying pattern. This corresponds to removing single characters from a word when a sliding window is moved to the next position. For example, if there is a single don’t care at position p in the pattern, we calculate i. Consequently, the time complexity of this approach is k is the weight of the pattern P.So far, we only considered don’t care positions in P is greater than the number k of match positions, i.e. if naive non-recursive approach, which calculates the hash values based on the match positions in the underlying pattern. In this case, the hash value h(iw) must be explicitly calculated for each spaced word iw leading to an However, if the number of spaced words in a sequence, we implemented a simple hash table. To determine an appropriate size of our hash table, we first calculate the maximum number of distinct (spaced) words. Clearly, for a sequence S and a pattern P of weight k and length ℓ, the maximum number of distinct spaced words in S is given by n is the length of S. According to this observation, we choose as hash table size the smallest integer b such that b most significant bits of our 64-bit hash value, which is achieved by shifting the bits b bits are used as index in the hash table, resulting in a complexity of open addressing. This method led to a better performance in our tests than an alternative approach using linked lists.To store the frequencies of all Once the word frequencies are determined for our input sequences, we can easily compare them for different sequences, as a basis to calculate pairwise distances values. To do so, we iterate over both hash tables and for each key we search the equivalent key in the other hash table, which can be accomplished in multiple-pattern approach, we need to calculate spaced-word frequencies for a large number of patterns. To do this efficiently, we implemented multithreading in our program to increase the speed. Both steps, counting word frequencies and calculating pairwise distances are easily parallelizable. We are using threads to determine the word frequencies as well as for the computation of pairwise distances.In our reference tree that we consider to be reliable.To evaluate our approach and to compare it with other sequence-comparison methods, we used benchmark sequence sets from different sources: for DNA and protein sequence comparison, respectively, we used real-world as well as simulated sequence sets. Each sequence set consists of a number of evolutionarily related sequences together with a contiguous as well as spaced words using the above-described single-pattern and multiple-pattern approaches. To obtain distance matrices for these sequence sets, we applied the Jensen–Shannon (JS) distance metric  is the Kullback–Leibler divergence (P and M defined asFeature Frequency Profile (FFP) approach. In addition, we applied the Euclidean distance to the relative-frequency vectors obtained with our multiple-pattern approach.For all sequences, we calculated the relative frequencies of e metric  to the overgence  between NeighbourJoining program (PHYLIP package (Average Common Substring (ACS)  metric  metric .Malvidae clade plus the grape vine genome as outgroup. As a reference tree for these 14 genomes, we used a tree based on a multiple sequence alignment of manually assembled CAP and Arp2/3 protein sequences as published by BAliBASE 3.0, a standard benchmark database for multiple alignment . For practically all values of ℓ, the multiple-pattern approach with the JS distance led to better results than the single-pattern approach with the same pattern length ℓ. For some values of ℓ, the RF distance to the reference tree was 0, so here the tree topology reconstructed by our approach precisely coincides with the reference topology. For values ℓ < 19, the multiple-pattern approach with the Euclidean distance performed worse than with the JS distance and even worse than the single-pattern approach. For longer patterns, however, the Euclidean distance performed better; for ℓ > 20, multiple patterns with the Euclidean distance produced perfect tree topologies, i.e. the RF distance was 0. The established alignment-free approaches rK and ACS performed worse than spaced words with single or multiple patterns for all ℓ > 10. FFP performed better than these two approaches, but was outperformed by multiple spaced words with the Euclidean distance for all pattern lengths ℓ > 18 and by multiple spaced words with the JS distance for most values ℓ > 13.For the primate mitochondrial genomes, the approach with contiguous words produced best results with a word length of The test results for our plant genomes are shown in k = 8 gave the best results, with an average RF distance of 50 to the respective reference trees. We therefore generated patterns with weight k = 8 and with length ℓ between 8 and 38. The results of these test runs are summarized in single patterns and one or several don’t care positions (ℓ > k) performed better than the usual approach with contiguous words (k = 0), as long as the number of don’t care positions is small. The relative improvement in the quality of trees is modest, however, and if the number of don’t care positions is increased, the tree quality deteriorates. By contrast, a substantial improvement could be achieved by using our multiple-pattern approach. Here, increasing the pattern length ℓ—i.e. increasing the number of don’t care positions—further improved the resulting trees, leading to average RF distances of around 16 between the constructed trees and the reference trees. On these sequences, the difference between JS and Euclidean distance was small. The three competing alignment-free methods were clearly outperformed by our multiple-pattern approach. A classical approach using multiple sequence alignment and maximum likelihood led to slightly better results than our multiple-pattern program.For our simulated DNA sequences, we found that contiguous words with length BAliBASE, a word length of k = 4 produced best results for contiguous-word frequencies. Thus, we used patterns with weight k = 4 and lengths ℓ between 4 and 34. single patterns gives better results than with contiguous words, but only for word lengths ℓ up to 7; a further increase in word length by using more don’t care positions led to deteriorated results, with RF distances to the reference trees larger than for contiguous-word frequencies. By contrast, the results of our multiple spaced-word approach led to better trees, and the quality of the trees further improved when the number of don’t care positions was increased. On these benchmark data, the results of our multiple-pattern approach were best when patterns of varying length were combined.Overall, our test results on proteins were similar to the results on DNA. On single patterns improved the quality of the resulting trees compared with distances based on contiguous-word frequencies, but the improvement was relatively small and could only be achieved for small numbers of don’t care positions. Using distance measures combining different patterns led to far better results. As with the real-world protein sequence sets, the multiple-pattern approach with varying pattern lengths gave the best results. On our simulated protein families, this approach led to even better results than the classical approach based on multiple alignment and likelihood.Finally, the results of our test runs on simulated protein families are shown in m = 100 randomly selected patterns (m = 60 for the plant genomes) or used all possible patterns where fewer than m patterns are possible. The multiple-pattern option improved the performance of our method but is more expensive in terms of program runtime compared with the single-pattern version.In the above test runs, we applied the multiple-pattern version of our approach with sets of m of patterns on the resulting trees, we used a set of 50 simulated DNA sequences of length 16 000 nt and a set of 125 simulated protein sequences of length 300 aa. We applied our multiple-pattern method to these datasets using different values of k and ℓ and pattern sets m. The results of these tests are shown in m of patterns is increased to 60 or 70, but a further increase in m does not lead to a significant improvement of tree quality.To study the influence of the number Clustal Ω , and lower for the multiple-pattern option than for the single-pattern option.As mentioned, a main advantage of spaced-word frequencies is that occurrences of spaced words at different sequence positions are statistically less dependent on each other. Distance measures using spaced words can therefore be expected to be more stable. To study this point in detail, we generated 100 pairs of DNA sequences of length 16 000 nt with product of their lengths , spaced-word matches at neighbouring sequence positions are less dependent on each other if a non-periodic pattern P is used.In database searching, word matches have been replaced by so-called recursive hashing and fast bitwise operations. While in general, algorithms using suffix-trees have the same linear time complexity as hashing algorithms, tree-based pattern searching is more difficult in our approach where patterns contain don’t care positions. Furthermore, the bitwise operations that we are using are much faster than operations on characters. The efficiency of our new approach enabled us to introduce a multi-pattern version where spaced-word frequencies with respect to an entire set P.In the first version of this spaced-word approach, we used a tree structure to find matching spaced words . There is no guarantee, however, that this is always the case, so further research is necessary to find suitable weights for k depending on the input sequences.A crucial parameter in our approach is the m of the set of m generally increases the quality of the resulting trees. For m > 60 or 70, however, no significant further improvement could be achieved in our test examples. It would be desirable to have a general rule to find a suitable number m of patterns in the multiple-pattern approach, depending on the input sequences. Finally, it is not clear which distance measure on the spaced-word frequency vectors is most suitable in our approach. In this study, we used the JS and Euclidean distance measures, but other distances may be more suitable to estimate phylogenetic distances between DNA or protein sequences based on their spaced-word frequencies.Another open question is which size spaced seeds in database searching (To answer these questions, the statistical properties of spaced-word frequencies need to be analysed in detail, as has been done for the hit probabilities of


Identifying the germline genes involved in immunoglobulin rearrangements is an essential first step in the analysis of antibody repertoires. Based on our prior work in analysing diverse recombinant viruses, we present IgSCUEAL (Immunoglobulin Subtype Classification Using Evolutionary ALgorithms), a phylogenetic approach to assign V and J regions of immunoglobulin sequences to their corresponding germline alleles, with D regions assigned using a simple pairwise alignment algorithm. We also develop an interactive web application for viewing the results, allowing the user to explore the frequency distribution of sequence assignments and CDR3 region length statistics, which is useful for summarizing repertoires, as well as a detailed viewer of rearrangements and region alignments for individual query sequences. We demonstrate the accuracy and utility of our method compared with sequence similarity-based approaches and other non-phylogenetic model-based approaches, using both simulated data and a set of evaluation datasets of human immunoglobulin heavy chain sequences. IgSCUEAL demonstrates the highest accuracy of V and J assignment amongst existing approaches, even when the reassorted sequence is highly mutated, and can successfully cluster sequences on the basis of shared V/J germline alleles. Antibodies, which can block infection through binding , are genWith the advent of high-throughput sequencing platforms, insights can be gained into the microevolutionary events that shape antibody repertoires, and into the underlying mechanisms . This inDiversity in the immunoglobulin heavy chain (IGH) repertoire is generated by four processes: combinatorial choice of V, D and J regions; deletions in the V, D and J regions; addition of palindromic (‘P’-) and non-templated (‘N’-) nucleotides at the junctions; and somatic hypermutation . As a reMacaca mulatta G.G’ regular-expression motif in the J region, or—should the motif be absent—to the position preceding the conserved tryptophan in the MSA of the J region.(d)κ L chain transcripts in mice exposed to a B-cell superantigen , scaled so that all model weights add to unity, which is then interpreted as evidence in support of this model. The V–J combination with the highest total model weight is reported as the inferred rearrangement, and all other rearrangements with at least 0.01 Akaike weight are reported as alternatives. Thirdly, the D allele which has the highest nucleotide alignment score  with the junction region of the query is reported as the inferred allele; in the case where multiple D alleles yield the same alignment score, they are all reported as equally likely.Our algorithm is an adaptation of the previously published SCUEAL (Subtype Classification Using Evolutionary ALgorithms) method , originarantigen ; we haverantigen  based on(e)Inferred rearrangements are often used to help operationally to define a clonotype ,25. For (f)http://github.com/cmccoy/ighutil), a simple Smith–Waterman alignment-based program previously used in an evolutionary analysis of immunoglobulin repertoires ,,http://w(iii)AF262145–AF262201, extracted from PopSet 8810007 )et al.  stated:Sdividual , which idividual , or thatdividual  and sampdividual . While m


Sequence alignment algorithms are a key component of many bioinformatics applications.Though various fast Smith-Waterman local sequence alignment implementations have been developed for x86 CPUs, most are embedded into larger database search tools. In addition, fast implementations of Needleman-Wunsch global sequence alignment and its semi-global variants are not as widespread. This article presents the first software library for local, global, and semi-global pairwise intra-sequence alignments and improves the performance of previous intra-sequence implementations.https://github.com/jeffdaily/parasailunder the Battelle BSD-style license.A faster intra-sequence local pairwise alignment implementation is described and benchmarked, including new global and semi-global variants. Using a 375 residue query sequence a speed of 136 billion cell updates per second (GCUPS) was achieved on a dual Intel Xeon E5-2670 24-core processor system, the highest reported for an implementation based on Farrar’s ‘striped’ approach. Rognes’s SWIPE optimal database search application is still generally the fastest available at 1.2 to at best 2.4 times faster than Parasail for sequences shorter than 500 amino acids. However, Parasail was faster for longer sequences. For global alignments, Parasail’s prefix scan implementation is generally the fastest, faster even than Farrar’s ‘striped’ approach, however the opal library is faster for single-threaded applications. The software library is designed for 64 bit Linux, OS X, or Windows on processors with SSE2, SSE41, or AVX2. Source code is available from Applications that require optimal alignment scores could benefit from the improved performance. For the first time, SIMD global, semi-global, and local alignments are available in a stand-alone C library.The online version of this article (doi:10.1186/s12859-016-0930-z) contains supplementary material, which is available to authorized users. Sequence alignment is an order-preserving way to map characters between two DNA or amino-acid (protein) sequences. It is a pervasive operation in bioinformatics workflows used to identify regions of high similarity between sequences. Similarity is generally measured by assigning a positive score to matches and a negative score to mismatches. For proteins, a substitution matrix, such as BLOSUM  or PAM , is usedglobal alignment causes the alignment to span the entire length of each sequence and is used when the two sequences are similar in length and presumed to be related. A local alignment identifies highly conserved regions or subsequences though the rest of the sequence may be divergent. A semi-global alignment does not penalize beginning or end gaps in a global alignment such that the resulting alignment will tend to overlap one end of a sequence with an end of the other sequence.There are three primary classes of sequence alignment, namely global, semi-global, and local. A s1[ 1…m] and s2[ 1…n], three recurrences are defined as follows: Let Si,j denote the optimal score for aligning the prefixes s1[ 1…i] and s2[ 1…j] such that the alignment ends by substituting s1[ i] with s2[j]. Di,j denotes the optimal score for aligning the same two prefixes such that the alignment ends in a deletion, i.e., aligning s1[ i] with a gap character. Similarly, Ii,j denotes the optimal score for aligning the prefixes such that the alignment ends in an insertion, i.e., aligning s2[j] with a gap character. Given the above three ways to end an alignment, the optimal score for aligning the prefixes corresponding to the subproblem {i,j} is given by: Ti,j=max. The dependencies for the individual dynamic programming recurrences are as follows: Si,j derives its value from the solution computed for the subproblem {i−1,j−1}, while Di,j and Ii,j derive their values from the solutions computed for subproblems {i−1,j} and {i,j−1}, respectively.Sequence alignments are computed using dynamic programming because it is guaranteed to find an optimal alignment given a particular scoring function. Regardless of the class of alignment being computed, a dynamic programming recurrence of the following form is computed. Given two sequences i,j) in the table stores three values Si,j, Di,j, and Ii,j, corresponding to the subproblem {i,j}. Given the dependencies of the entries at a cell, the dynamic programming algorithms for all three sequence alignment classes can be represented using the pseudocode outlined in Algorithm 1. The algorithm has a time complexity of A typical implementation of this dynamic programming algorithm builds a table of size The three classes of sequence alignment initialize the first row and column differently (lines 1 and 2 in Algorithm 1). SW and SG alignments initialize the first row and column of the table to zero, while NW alignments initialize the first row and column based on the gap function. The table values for SW alignments are not allowed to become negative, while NW and SG allow for negative scores.Hereafter, for convenience according to common practice, we call the sequence with characters along the rows of the table the “query” sequence and the sequence with characters along the columns of the table the “database” sequence.m and n requires Aligning two sequences of lengths as BLAST , howeverThere have been numerous efforts to parallelize optimal sequence alignments using vector instructions –12. Howehttps://github.com/jeffdaily/parasail under the Battelle BSD License. The same files are included in a gzipped tar archive as Additional file Parasail is a SIMD C (C99) library containing implementations of the Smith-Waterman , Needleman-Wunsch , and semi-global pairwise sequence alignment algorithms. Here, semi-global means insertions before the start or after the end of either the query or target sequence are not penalized. Parasail implements most known algorithms for vectorized pairwise sequence alignment, including diagonal , blockedparasail Required. The function prefix to avoid clashing with other libraries. All functions begin with this prefix.  Required. The class of alignment; global, semi-global, or local, respectively, stats Optional. Return alignment statistics.  Optional. Return the entire dynamic programming table or return the last row and column of the dynamic programming table, respectively.  Optional. The vectorized approach. Striped is almost always the best choice. profile Vector approaches only, specifically only prefix scan and striped, but optional in that case. The prefix scan and striped vector implementations must first compute a query profile. That step can optionally be performed ahead of time and the query profile reused to avoid costly recomputation of the query profile.  Vector approaches only, but optional in that case. The instruction set and vector width. If not given, the default will use CPU dispatching to select the best available instruction set on the host system.  Vector approaches only, but required in that case. The integer width of the solution. This also affects performance; a smaller integer width will increase the number of lanes used by the vectors and thus improve performance. If “sat” is used, the function defaults to the 8-bit implementation and, if overflow of the score is detected, it will retry using the 16-bit implementation.There are over one thousand functions within the Parasail library. To make it easier to select a particular function, the function names follow a naming convention. The following will use parentheses “” to indicate a selection from a set must be made. If a portion of the function name is optional, the description will indicate as such. Underscores “_” separate each function name component. The components of the function names are listed in the order they should appear in the constructed function names. parasail_sw As simple as possible, this is the non-vectorized reference implementation of Smith-Waterman local alignment. It will return the alignment score as well as the ending locations of the alignment along the query and database sequences. parasail_sg_stats_scan_avx2_256_sat The prefix scan vector implementation of semi-global alignment. It will use 8-bit integers during the computation and recompute using 16-bit integers if overflow is detected (‘sat’ is for ‘saturation’). AVX2 is selected for the CPU instruction set. Additional alignment statistics are also computed and returned.The two examples to follow should help illustrate how function names should be constructed. Parasail supports the SSE2, SSE4.1, AVX2, and KNC (Xeon Phi) instruction sets. In many cases, a compiler can compile source code for an instruction set that is not supported by the host CPU. The code is still compiled, however, Parasail uses a technique called CPU dispatching to avoid running code that uses instructions that the host CPU does not support. CPU dispatching tests for features of the CPU at runtime which is different than testing whether a compiler can compile certain instructions at compile-time. The results of the runtime CPU tests are used to correctly select the appropriate Parasail implementation for the highest level of instruction set supported on the host platform. This allows Parasail to be compiled and distributed by a maintainer for the best available system while still allowing the same distribution to run with a lesser CPU.Parasail is optimized for the SSE4.1 level of instruction set. Some SSE4.1 instructions are missing for SSE2 and were emulated using other available SSE2 instructions. Some AVX2 instructions, e.g. bit packing, lane shifting, also needed to be emulated using other AVX2 instructions because AVX2 is effectively two SSE 128-bit lanes operating independently and some operations do not cross the lane boundaries.lazy F evaluation loop. This improvement was duplicated in the SSW [Farrar’s striped method  remains  the SSW  implemenMore recently, the striped method was evaluated with respect to the number of vector lanes utilized within the calculation. For example, SSE uses a 128-bit vector that can be divided into 64-, 32-, 16-, or 8-bit integers which corresponds to two, four, eight, or sixteen vector lanes, respectively. A feature of the striped method is the recalculation of a column within the dynamic programming table until the column converges. Evaluating SSW showed that the number of corrective passes increased as the number of vector lanes was increased .This problem is improved within Parasail using a technique of shifting the origin of the calculation towards the smallest representable integer. This takes advantage of the fact Smith-Waterman does not allow for negative values. For example, for 8-bit integers −127 is treated as 0, allowing the entire range of 0 through 255 to be utilized. Further, using saturation arithmetic keeps the calculation from underflowing. This is not unlike previous SSE2 implementations that mitigated the lack of a signed 8-bit vector maximum instruction and instead used unsigned integers [Unfortunately, similar improvements were not possible for Needleman-Wunsch or semi-global alignment because the table values are allowed to be negative. As a result, in some cases the prefix scan implementation of those algorithms will perform better than the striped implementation .One last improvement was made to the implementation of Zhao et al. . In theiThe Parasail library uses a code generation step in addition to some compiler preprocessor directives to create the thousands of functions implemented within the library. This takes advantage of the observation that the basic algorithms for the three classes of alignments remain the same no matter which instruction set and integer width is used. End users do not need to worry about this code generation step, though for code development this has a number of benefits including faster time to solution when modifying the code as well as enforcing consistency between the myriad implementations possible.The Parasail library is rigorously tested using cross-verification to verify correct implementations of the various library functions. Given a test dataset in FASTA format, the result of the reference implementation for a given class of alignment is compared against all other implementations for the same class of alignment. This is done both for the resulting alignment scores as well as for the byte by byte comparisons of the entire dynamic programming tables involved. The software is not released until all tests pass for reasonably sized test datasets.parasail_aligner application. The aligner application will take a FASTA- or FASTQ-formatted query file as input and align all of the query sequences against a set of database sequences found in a second FASTA- FASTQ-formatted database file. Alternatively, if only one file is supplied, all of the sequences in the file will be compared against themselves. Since the Parasail library implements many different alignment routines, the name of the function to use must be specified to the aligner. Optionally, a filter can be applied to skip the alignment of any two sequences that don’t contain an exact-matching seed of the given length. This filter uses an enhanced suffix array data structure [In addition to the Parasail library, the software also provides the tructure , 18 thatThe following benchmarks repeat those performed by Rognes . The reaGreat care was taken to compare the various software libraries and applications fairly. All tests were run three times and the average GCUPS performance was recorded. The loading times for the database sequences were excluded from this study in order to focus on the algorithmic performance of the alignment routines. The following software reported the GCUPS performance directly for the alignments performed: Parasail, opal, SWIPE, and SSW. For the remaining software including BLAST, BLAST+, and libssa, a wall clock timer was patched into the code that reported on the alignment times only, and the wall clock time was used to calculate the GCUPS result based on the known amount of work for each test case performed.Table parasail_sw_striped_profile_sse41_128_sat,parasail_sw_striped_profile_avx2_256_sat,parasail_nw_striped_profile_sse41_128_16,parasail_nw_striped_profile_avx2_256_16,parasail_nw_scan_profile_sse41_128_16,parasail_nw_scan_profile_avx2_256_16,parasail_sg_striped_profile_sse41_128_16,parasail_sg_striped_profile_avx2_256_16,parasail_sg_scan_profile_sse41_128_16, andparasail_sg_scan_profile_avx2_256_16.The parasail_aligner can be run using any one of the many alignment routines the Parasail library provides. The initial benchmarks compare other local alignment implementations against Parasail’s local alignment implementation. The latter benchmarks compare Parasail’s local alignment performance against its global and semi-global performance for the striped and scan vectorized approaches. The following Parasail functions were evaluated. Results were taken on compute nodes of the Constance cluster, part of Pacific Northwest National Laboratory’s Institutional Computing. Each Constance node contains dual Intel Haswell E5-2670 CPUs (2.3 Ghz) giving 24 cores and 64 GB 2133Mhz DDR4 memory per node. Intel’s Haswell CPUs support the AVX2 instruction set. Although these processors are capable of hyper-threading, which would have given each node 48 logical cores, it was not used for these experiments. Hyper-threading is intentionally disabled as a matter of policy because Constance is a general use cluster and hyper-threading benefits are strongly application dependent . BecauseThe UniProt Knowledgebase Release 11.0  : Platform independentProgramming language: C, with Python language bindingsOther requirements: SSE2, SSE4.1, and/or AVX2 compiler intrinsics preferredLicense: Battelle BSD-styleAny restrictions to use by non-academics: N/A


Phylogenetic inference is generally performed on the basis of multiple sequence alignments (MSA). Because errors in an alignment can lead to errors in tree estimation, there is a strong interest in identifying and removing unreliable parts of the alignment. In recent years several automated filtering approaches have been proposed, but despite their popularity, a systematic and comprehensive comparison of different alignment filtering methods on real data has been lacking. Here, we extend and apply recently introduced phylogenetic tests of alignment accuracy on a large number of gene families and contrast the performance of unfiltered versus filtered alignments in the context of single-gene phylogeny reconstruction. Based on multiple genome-wide empirical and simulated data sets, we show that the trees obtained from filtered MSAs are on average worse than those obtained from unfiltered MSAs. Furthermore, alignment filtering often leads to an increase in the proportion of well-supported branches that are actually wrong. We confirm that our findings hold for a wide range of parameters and methods. Although our results suggest that light filtering  has little impact on tree accuracy and may save some computation time, contrary to widespread practice, we do not generally recommend the use of current alignment filtering methods for phylogenetic inference. By providing a way to rigorously and systematically measure the impact of filtering on alignments, the methodology set forth here will guide the development of better filtering algorithms. Phylogenetic reconstruction pervades computational and evolutionary biology; it is thus important to be able to compute accurate phylogenetic trees. While it is possible to infer phylogenetic trees directly from sequence data , the mosGiven a set of sequences, an ideal MSA identifies homologous characters, that is, characters having common ancestry. However, computing such an MSA can be challenging. While most alignment programs will correctly identify and align highly conserved regions, regions containing a large number of insertions and/or deletions are typically less reliable. Such unreliable sections and erroneously aligned residues can negatively affect downstream analyses, such as tree inference .Filtering—that is, removing unreliable columns before tree reconstruction—has been promoted as a way to increase the signal to noise ratio of MSAs . DespiteIn recent years, a number of software packages aimed at detecting unreliable alignment columns have been published. They take a wide variety of different approaches, from ad hoc rules based on substitution patterns  to more Recently, We first review the filtering methods considered in this study. Next, we introduce the four main assessment methods used to support our main conclusions . The remainder of this section provides a detailed exposition of the empirical and simulated data underpinning our analyses, the alignment and tree methods, and the alignment accuracy measures used in the study.Filtering methods take a variety of mathematical and heuristic approaches. Those considered here all have in common that they are fully automated and they remove entire columns of the alignment. Our (non-exhaustive) selection includes a broad range of software packages that distinguish themselves by popularity or originality . One of gappyout, strict, and automated1. The gappyout heuristic sets the gap threshold parameter—the minimum proportion of sites in a column—at the point where the variation of the proportion of alignment removed is greatest . The strict heuristic uses the same approach but trims the alignment further based on an automatically selected similarity score threshold. The automated1 option chooses between gappyout and strict based on a decision tree optimized on a benchmark  tries to infer columns that are phylogenetically uninformative by assessing the degree of homoplastic sites compared to random columns . Rather Aliscore v.1.0  assessesBlock Mapping and Gathering with Entropy (BMGE)  attemptsZorro . The downside of this approach is that findings based on such six-sequence alignments might not generalize well to larger alignments. To test larger alignments, one could use larger trusted topologies, but larger topologies can be more difficult to defend. Instead, we have developed a new test variant, which we call the “enriched species tree discordance test” . In thisTo assess the impact of branch support on the analyses, we modified the enriched species discordance test as follows. In the tree inference step , step 4,et al. emphasize in their article, some resulting reconciled trees contain very poorly supported duplication nodes : these nodes are followed by differential gene losses, as opposed to strongly supported duplications, where both duplicated genes remain in most of the subtrees. To gauge the impact of alignment filtering on the reconciled gene trees generated by the Ensembl pipeline, we use the average number of losses per branch over all trees as indicator. If filtering leads to the average number of losses per branch significantly decreasing, the parsimony principle suggests that gene tree inference and reconciliation has improved and therefore alignment filtering is on the whole beneficial. If the average number of losses per branch increases, the opposite holds. The criterion used in this test is conceptually similar to the minimum duplication test, but the two were implemented independently and differ in several minor points: contrary to the minimum duplication test, this one only considers a single-gene tree rooting (inferred in the tree reconciliation), counts losses instead of duplications (both are correlated but not exactly so), and uses a reference species tree in the reconciliation step.The third kind of filtering assessment method used in this study is based on the Ensembl Compara data and pipeline, which infers gene trees reconciled with a reference species tree  . As VileFourth, in addition to these various sets of empirical data, we also evaluated the effect of alignment filtering on simulated data. We used the species tree discordance test, using the true tree as reference. Because the correct alignment is known with certainty in simulated data, we were also able to directly measure the effect of filtering on the alignment itself.Besides these four approaches, we note that other classes of alignment benchmarking techniques could conceivably be used, such as benchmarks based on protein structure conservation; however, these other approaches also have their assumptions and shortcomings. We refer the interested reader to a recent review on this topic .http://dx.doi.org/10.5061/dryad.pc5j0). Additional homologs were automatically collected from the SwissProt database via NCBI BLAST using the script Mafft-Homologs  database release  relying Homologs , with a For the minimum duplication test, we retrieved groups of up to 60 homologs (∼36 on average) from eukaryotic and fungal genomes. Both the species tree discordance test and the minimum duplication test were run once on amino acid and once on nucleotide data.http://dx.doi.org/10.5061/dryad.pc5j0). The enrichment led, for all three phyla, on average to shorter sequences than the original 6 orthologous sequences (by about 100 amino acids). This is explained by the local matching of BLAST . The average percent identity decreased slightly for fungi and eukaryotes , and increased for bacteria by about 2%. Alignments and trees for all data sets are available as raw data on Dryad at http://dx.doi.org/10.5061/dryad.pc5j0.The setup resulted in highly diverse conditions, which have been characterized in terms of percentage sequence identity, sequence length, and number of sequences . Since the tests are based on variations in rooted tree topology (see above), we considered clusters with at least four genes and two supporting species . Of these, 11,231 could be processed with all filtering methods and parameters such that there was at least one column in the resulting alignment. The resulting data set covered 898,138 genes (92.6%) of the initial gene set.The tests based on Ensembl data use version 66 (February 2012), which contains 19,491 homologous clusters (and their corresponding MSAs/trees) containing 969,577 protein-coding genes from 57 species. The distribution of the cluster sizes is uneven , which suggests that the majority of these dubious nodes are due to limitations in the inference process and not bona fide differential gene losses that would be overly penalized by the parsimony criterion.Of note, dubious nodes in the Ensembl data set are not concentrated in places with documented whole-genome duplications and/or bursts of segmental duplications, but rather in places with rapid species radiation : “small MSAs” cover fewer than ns/2 species and contain fewer than ns/2 genes; “medium MSAs” cover at least ns/2 species and contain between ns/2 and 3ns/2 genes; “large MSAs” cover at least ns/2 species and contain more than 3ns/2 genes. These categories are disjoint and collectively cover 97.4% of all the MSAs we could process (Supplementary Table 1 available on Dryad at http://dx.doi.org/10.5061/dryad.pc5j0).To investigate the differences with respect to cluster size , we defined subsets depending on cluster size with respect to the total number of species In each analysis reported here, to ensure that results are comparable across all methods tested, we only considered alignments that lead to trees for all filtering methods considered .Two sets of 500 30-sequence MSAs were simulated using Artificial Life Framework (ALF) . In bothmatrices , and ins–retree 2”. For Prank, the option “-F” was used. ClustalW and T-Coffee were used with their default parameters.auto and M-coffee is called to combine alignments from Mafft, Muscle  with theesoft/;  and WAG eesoft/; , as welleesoft/;  over dn,To assess the accuracy of alignments performed on simulated data, for which we know the true alignment, we computed the precision and recall of alignments using the sum-of-pair measure . For prelignment .For recall, we compute the fraction of residue pairs in the true alignment that are present in the filtered alignment:lignment .Alternatively, we also used precision and recall based on the more the simple “true column” measure as described in We first present an in-depth assessment of filtering methods based on the enriched species discordance test. The analyses consistently indicate that on average and across all data sets, trees reconstructed from filtered alignments are generally worse than those obtained from unfiltered alignments. Next, we show that our main findings hold when controlling for numerous potential confounding factors , and when considering the minimum duplication test. Finally, we show that our conclusions are also corroborated by analyses based on the Ensembl Compara pipeline and data, and by analyses on simulated data.In aggregate, the main analyses reported here required computing over 540,000 alignments of up to 60 sequences and over 11.1M phylogenetic trees, for a total computational cost of over 1.26M CPU hours. For the analyses on Ensembl Compara data, 900,000 trees of up to 400 sequences were computed for a total runtime of 150,000 CPU hours.Methods” section, and that controls using other alignment programs, testing strategies, and data follow after we have presented our main results.We first compared the filtering methods on the enriched species tree discordance test , based oOverall, we found that tree inference does not generally improve after alignment filtering . With amongly so , top. Weongly so ; one couignments , bottom.colCutoff parameter to 0.2 substantially reduced the error rate of reconstructed trees relative to the default parameter ; two-sided Wilcoxon test, http://dx.doi.org/10.5061/dryad.pc5j0); two-sides Wilcoxon test, http://dx.doi.org/10.5061/dryad.pc5j0), http://dx.doi.org/10.5061/dryad.pc5j0); The poor performance of some of the filtering methods can be partly explained by inadequate default parameter values. For instance, consistent with previous observations , the mucEven with optimized parameters, most filtering methods did not yield better trees than unfiltered alignments, especially with amino acid alignments  at the cost of increasing the proportion of unresolved branches (which are merely uninformative). These types of errors can be thought of as false positive and false negative, respectively (see section “Methods”).http://dx.doi.org/10.5061/dryad.pc5j0).We decomposed the effect of the various filtering methods on the false positive and false negative rates, using the enriched species discordance test and taking the approximate Bayesian posterior  as the mhttp://dx.doi.org/10.5061/dryad.pc5j0). For DNA alignments, we observed more variation in the effect of filtering among aligners, but the general trends are unchanged: in the overwhelming majority of the combinations, filtering worsens the trees obtained . The single instance of a significant improvement was obtained with T-Coffee . As the version of T-Coffee used in this study is several releases behind the latest version, we also repeated this particular analysis with a more recent version (v.10.00). The results we obtained were consistent with our previous observations . Based on that, we decided not to recompute all other results with that newer version of T-Coffee.The findings above were confirmed with a broad range of different controls. First, we assessed the influence of the alignment software on our analyses by repeating them with Mafft , Clustalequences  and henchttp://dx.doi.org/10.5061/dryad.pc5j0) and nucleotide data , the effect of alignment filtering with distance trees remained overwhelmingly detrimental on average, though less so than with maximum likelihood trees.Second, to assess the influence of the tree building method on our analyses, we compared the results using maximum likelihood trees with those using weighted least-squares distance trees . For bothttp://dx.doi.org/10.5061/dryad.pc5j0).Third, to ensure that our observations hold for a broad range of sequence length, sequence divergence, and “gappiness”, we repeated the analyses on data partitioned into quartiles according to these aspects. Average length could conceivably affect the analysis, because under typical models of sequence evolution, confidence intervals may be expected to tighten with the square root of the alignment length. Thus, removing for instance half of the sites can be expected to introduce more error in trees inferred from shorter alignments than in trees inferred from longer alignments. However, our empirical results suggest that filtering remains detrimental on average even for the longest quartile .Likewise, one could imagine a different impact of filtering for different average sequence divergence, because highly variable sites can be signs of alignment error or sequence artifacts. Yet for all three phyla, filtered alignment resulted in poorer trees for all observed evolutionary ranges . Even in the data quartiles with highest gap proportion, we found many instances where filtering led to significantly poorer trees.Furthermore, to investigate whether filtering showed differential performance depending on the “gappiness” of the alignments, we partitioned the data according to the proportion of gaps in the original alignments. Once again, filtering proved detrimental across all bins .Fourth, to control for our choice of branch support measure in http://dx.doi.org/10.5061/dryad.pc5j0). In the majority of cases, the topology remained unchanged. When it changed, in 25 out of the 30 method data set combinations, changes were overwhelmingly for the worse. In four cases there was practically no difference between the number of topologies improved or worsened. In only one combination—Noisy on the bacteria data set—a slight majority of the changes were toward better topologies.So far, all results report performance in terms of the mean Robinson–Foulds (RF) distance between reconstructed and reference tree. As alternative evaluation measures, we also computed the fraction of trees where the topology improved, remained constant, and worsened in terms of RF distance , which indicates that this choice of enrichment strategy has a negligible, if any, effect on our assessments.Sixth, to ensure that the results are not confounded by the enrichment approach, which adds portions of homologous sequences based on local alignments (http://dx.doi.org/10.5061/dryad.pc5j0 for a list of species and key statistics of the two data sets). Recall that if a filtering method reduces the number of implied duplication nodes on average, it is considered to be beneficial. Conversely, if a filter increases the number of duplications on average, it is considered to be detrimental. Under this test, the difference between unfiltered and filtered alignment showed in all cases an increase in gene duplication and therefore a worsening of the trees upon filtering; in about half of the cases, this was statistically significant .As a seventh set of controls, we repeated the analyses using a different criterion of alignment quality altogether: the minimum duplication test, described above, which assesses the number of gene duplications in a gene family . The minIn light of the overall negative effect of alignment filtering on phylogenetic inference under all considered conditions, and to minimize the risk of systematic errors in our data sets or methods, we replicated the investigation in an independent study. This was done on a separate data set, using a different criterion of accuracy, and implemented in separate computer programs. We assessed the impact of alignment filtering on the Ensembl Compara pipeline , which iWe investigated the effect of alignment filtering on the average number of gene losses per branch using selected filtering methods .As http://dx.doi.org/10.5061/dryad.pc5j0), we also sought to determine the impact of problem size on the effectiveness of alignment filtering. We separately reassessed the impact of filtering for small, medium, and large data sets (see section “Methods” for exact definitions). Although our main conclusions still hold, we observed a few cases for which filtering yielded a small but significant improvement . Consistent with our previous observations, light filtering with Guidance yielded a marginal improvement. The other few significant cases of improvement were concentrated in the small data sets, which generally showed irregular results  and very high levels of tree discordance. In contrast, filtering on medium and large data sets led to stronger and consistent worsening of the trees when 20% or more of sites were removed.Because of the broad distribution of gene tree sizes  in Ensembl .One advantage of simulated data is that the true alignments are known with certainty. We used this knowledge to directly assess the impact of filtering on alignment quality, in terms of precision  and recall . Ideally, filtering should increase precision while maintaining recall, but instead with most filtering methods, precision only moderately improved while recall substantially dropped , right. http://dx.doi.org/10.5061/dryad.pc5j0).One concern with simulation is whether the choice of parameter yields data sets that are representative of real data. Our simulated data depart in many ways from real data but we nevertheless note that they result in 90% of pairwise residues correctly aligned , right, Altogether, the picture that emerges from the combination of the different empirical tests and data sets coherently indicates that current alignment filtering methods do not generally lead to better trees. On the contrary, there were many instances where filtering worsened the trees significantly. Furthermore, the few combinations of methods, data, and parameters for which we observed an improvement revealed no clear patterns, making it difficult to predict when existing filtering methods can be effective. Even for these positive cases, the improvement was modest.The empirical tests used in this study are based on assumptions that might be violated in individual problem instances but these are unlikely to have a strong influence on the aggregate results. For instance, what happens to the species discordance test if some of the input sequences are not true orthologs? In instances where orthology inference is erroneous, an improvement due to filtering can potentially be misreported as worsening. Conversely, a worsening can potentially be misreported as an improvement. However, as long as errors in ortholog calling are not correlated with the performance of individual filtering strategies, the overall ranking of the methods will remain unaffected. Likewise, while the parsimony assumption might not hold for gene families with rampant duplications and losses, the minimum duplication test gives meaningful results as long as the errors thereby introduced are not biased in favor of a particular filtering method.Our results on empirical data are also backed by simulated data. The strong agreement between the two types of analysis was not a given, because simulated data tend to be easier to model than real data and therefore filtering on the former could have been expected to fare better than on the latter. It nevertheless indicates that our findings hold across quite different data sets.Filtered alignments led to poorer trees for different combinations of data sets, aligners, tree reconstruction methods, and evaluation criteria. Furthermore, these results held for a considerable range of sequence lengths, divergences, and gap proportions. In particular, it is perhaps surprising that on highly diverged—thus hard to align—sequences, filtering does not appear to do particularly well. We hypothesize that tree inference might be robust to alignment errors between highly divergent sequences (or subsequences) but at the same time sensitive to loss of any genuine phylogenetic information incurred by excessive filtering. More work is needed to test the validity of this explanation.One reason for the poor performance of current filtering methods appears to be that they remove columns too aggressively; in particular, this is the case for Gblocks and BMGE with default parameters, which appear to have an excessively low tolerance for columns containing gaps. Generally, the methods that removed more columns performed worse. Correspondingly, the best parameters values we found for Noisy and Zorro were those resulting in the fewest removed alignment columns. In part, this may be explained by the substantial phylogenetic signal that may be contained in sparse (gap-rich) regions, previously discussed in the context of both multilocus  and singSome solace lies in the observation that modest amounts of filtering have little impact on the reconstructed trees but decrease computational time, which typically scales linearly with alignment length. For instance, in the Ensembl pipeline, a ClustalW minimum column score of 3 decreases the computation burden by 25% with no apparent worsening of the trees. In other contexts, appropriate cutoff values need to be carefully established as they will depend on the particularities of individual data sets. Our results suggest that filtering up to 20% of alignment positions is relatively safe.http://dx.doi.org/10.5061/dryad.pc5j0) does not bode well for filtering concatenated alignments. Second, this study focused on tree topology and ignored the problem of inferring accurate branch lengths. This is mainly due to the limitations of our empirical tests, which can only assess topology. However, there are also inherent complications in assessing branch length accuracy when part of the data gets removed. For instance, if the fast evolving sites are preferentially filtered out, the true branch length (in average number of substitution per site) gets shorter, thus making the true length a moving target. Third, the number of sequences contained in most alignments considered here was relatively low by current standards, with about 30 sequences on average in the species discordance and minimum duplication tests. However, the Ensembl Compara data set contained larger alignments, with approximately 20% of alignments containing more than 100 sequences. The filtering performance on these larger alignments was below average , but due to the risk of latent confounders in these larger gene families , we cannot draw strong conclusions from this particular result. Fourth, the automated nature of our test pipelines is not fully representative of typical phylogenetic studies, which are often restricted to relatively well-established markers and often involve manual intervention (http://dx.doi.org/10.5061/dryad.pc5j0), and that ClustalW performed “surprisingly well” compared with other aligners, are indicative that the data sets used for the species discordance test are flawed; (iii) the parsimony criterion underlying the minimum duplication test and the Ensembl analyses is questionable.We finish this section by discussing caveats. First, our empirical tests were limited to single-locus alignments. As such, we could not investigate the effect of filtering concatenated alignments. However, the lack of positive trend with respect to sequence length when filtering single-gene alignments (Supplementary Fig. 17–18 available on Dryad at rvention . Fifth, For filtering to be worthwhile, the decrease in phylogenetic noise achieved should exceed the loss of phylogenetic signal incurred. Our results indicate that with current filtering methods, this is generally not the case. However, this does not necessarily imply that filtering is inherently a bad idea. By providing new methods to systematically and rigorously test the impact of filtering on tree reconstruction, we hope that this study will help lay the methodological foundations to guide the development of better filtering methods.Whether effective filtering methods will eventually come to fruition is yet to be seen. But from the methodology standpoint, filtering is a “band-aid” solution to the deeper problem of handling uncertainty in the alignment. More statistically sound approaches have been developed, which ascribe probabilities to alignment columns  or compuhttp://dx.doi.org/10.5061/dryad.pc5j0.Data available from the Dryad Digital Repository: