Notwithstanding these bounds, WFA’s Pairwise sequence alignment remains a fundamental problem in computational biology and bioinformatics. Recent advances in genomics and sequencing technologies demand faster and scalable algorithms that can cope with the ever-increasing sequence lengths. Classical pairwise alignment algorithms based on dynamic programming are strongly limited by quadratic requirements in time and memory. The recently proposed wavefront alignment algorithm (WFA) introduced an efficient algorithm to perform exact gap-affine alignment in In this article, we present the bidirectional WFA algorithm, the first gap-affine algorithm capable of computing optimal alignments in https://github.com/smarco/BiWFA-paper.All code is publicly available at Bioinformatics online.Supplementary data are available at  Pairwise sequence alignment provides a parsimonious transformation of one string into another. From this transformation, we can understand the relationship between pairs of sequences. Because similarities and differences between biosequences  relate to variation in function and evolutionary history of living things, pairwise sequence alignment algorithms are a core part of many essential bioinformatics methods in read mapping . Its impdynamic programming (DP) techniques. These methods require computing a matrix whose dimensions correspond to the lengths of the query q and target t sequences. Using DP recurrence relations, these methods compute the optimal alignment score for progressively longer prefixes of q and t, which correspond to the cells of the DP matrix. Thus, an optimal alignment can then be read out by tracing the recurrence back through the matrix.Classical approaches to derive alignments involve the application of gaps with a long-tailed length distribution. This motivated the development of gap-affine models in which the penalty of starting a new gap is larger than that of extending a gap    to O(s),The rest of the article is structured as follows. Section 2 presents the definitions, algorithms and formal proofs supporting BiWFA. Section 3 shows the experimental evaluation of our method, comparing it against other state-of-the-art tools and libraries. Lastly, Section 4 presents a discussion on the BiWFA method and summarizes the contributions and impact of this work.n and m, respectively. Likewise, let v from the ith to the jth character. We will use x, and a gap of length l costs Let the query s reaches coordinate s is the minimal alignment score. Moreover, it can derive the optimal alignment by tracing back the partial alignments that led to score s at Basically, WFA computes partial optimal alignments of increasing score until an alignment with score k in the DP-matrix to the farthest-reaching (f.r.) cell that has score s and ends with a match, mismatch, insertion or deletion, respectively. In general, we denote by wavefront the tuple of offsets for a given score components, and we associate a corresponding sentinel value to specify each component: Let v and w. The base case for this recursion is given by In p as the wavefront scope or, in other words, the maximum score increase between partial alignments. Moreover, note that s wavefronts of increasing length, totalling LCP must be computed once for each cell. However, within a diagonal, the total number of offset increments cannot exceed the length of the sequences. Hence, WFA requires LCP, dominates over the LCP computations often finish after performing only a few character comparisons, except along the optimal alignment in which In the worst case, WFA requires computing p wavefronts in memory. This is insufficient to perform a full traceback. However, when they ‘meet’ in the middle, we can infer a breakpoint in the alignment that divides the optimal score roughly in half. Then, we can apply the same procedure on the two sides of the breakpoint recursively. We will show that this results in only a constant-factor slowdown. This technique was previously employed to a similar end with the Myers The core idea of the BiWFA algorithm is to perform WFA simultaneously in both directions on the strings: from start to end (i.e. forward) and from end to start (i.e. reverse). Each direction will only retain lgorithm .v and w. Note that the same argument used in First, let us define the WFA equations for the forward and reverse alignment directions. The recursions for the forward direction are equivalent to those of the standard WFA presented above . However similar , using XWF_NEXT and WF_EXTEND from the standard WFA [see Algorithm 1 presents the BiWFA algorithm to compute a breakpoint in the optimal alignment at p. Accordingly, we will demand a weaker notion of balance. If The first technical detail involved in finding an alignment breakpoint between the two directions is that it is often not possible to split an alignment into an equally scoring prefix and suffix. In general, two prefixes of the optimal alignment that differ by one character can have scores that differ by as much as o at the beginning of gaps, but the reverse incurs it at the end of gaps . Thus, if the two directions meet in a gap, then we have The second technical detail is that the optimal score is not always the sum of the two scores. This occurs because the forward iteration incurs the gap open penalty The final technical detail is that offsets of the two directions may not precisely meet. WFA proceeds by greedily taking matches in both directions. This makes it possible for the two directions to shoot past each other without actually meeting. It turns out that it is sufficient to detect that such an overshoot has occurred, as will be shown in Section 2.4.k in wavefront o. Hence, the score from overlaps at indel components has to be decreased by o.In Algorithm 2, we reconcile these three difficulties. Without loss of generality, we assume that a forward wavefront WF_OVERLAP. An efficient implementation can keep track of the farthest reached antidiagonal by each wavefront. If the most advanced antidiagonal reached by the forward and reverse wavefronts do not overlap (ANTIDIAG on Algorithm 1), it follows that no offsets from any diagonal can overlap, rendering the call to WF_OVERLAP unnecessary.In practice, Algorithm 1 can avoid most calls to Lemma 2.1.The optimal alignment score if and only if there exist and k such that and at least one of the following is true:and and and and further, (resp. ) is included in the traceback of an alignment with score at most s if the first  condition is true.Proof.See p score values on each iteration.Algorithm 2 checks a window of Algorithm 1 iterates through alternatingly increasing values of After finding some This lemma implies that the minimum value s for which the ‘only if’ condition holds is the optimal score. Moreover, if the first of the three conditions is found to hold for some values The correctness of the Algorithm 1 stems from the following lemma.Algorithm 3 shows how to use BiWFA to recursively split alignments into smaller subproblems until the remaining alignment can be trivially solved.M  at https://github.com/smarco/WFA2-lib.We implement the BiWFA algorithm described in this work in C. The code and the scripts required to reproduce the experimental results presented in this section are publicly available and can be found at https://github.com/smarco/WFA2-lib). Also, we select the efficient wfalm  nt wfalm , as the nt wfalm  and BitPWe considered including other popular methods like those implemented in the Parasail , SeqAn  librarieAll the presented methods have been configured to generate global alignments. These algorithms are grouped in two categories: ‘Gap-affine Exact’ for exact algorithms that use gap-affine penalties , and ‘Others’ for methods that use simpler penalty models or can only compute the alignment score .e = 10% and 20%) randomly generated. Regarding the evaluation with real datasets, we use a first set of sequences generated by the Human Pangenome Reference Consortium ( kbp (maximum ∼306 kbp). In addition, we use a second dataset comprising ONT MinION reads from  kbp long, for a total number of 48 sequence pairs of average length equal to 630 kbp (maximum ∼1 Mbp).For the evaluation, we use simulated and real datasets. For the simulated datasets, we simulate several datasets of various sequence lengths  and different error rate .All the executions are performed single-thread on a node running CentOS Linux (release 8.1.1911) equipped with an AMD EPYC 7742 CPU and 1 TB of DRAM  and error rates  on At the same time, BiWFA proves to be one of the fastest implementations aligning long sequences. Using ultra-long sequences, our method is  Kbps. In this scenario, BiWFA demonstrates to be one of the fastest implementations, requiring less than 10 MB to execute.For completeness, As long sequencing technologies improve and high-quality sequence assembly decreases in cost, we anticipate that the importance of pairwise alignment algorithm will continue to increase. To keep up with upcoming improvements in sequencing and genomics, pairwise alignment algorithms need to face crucial challenges in reducing execution time and memory consumption. In this work, we have presented the BiWFA, a gap-affine pairwise alignment algorithm that requires Most notably, BiWFA execution times are very similar, or even better, than those of the original WFA  by modifying the initial conditions and termination criterion. At the same time, the BiWFA retains the strengths of the original WFA: no restrictions on the sequences’ alphabet, preprocessing steps, nor prior estimation of the alignment error.Due to the simplicity of the WFA’s computational pattern, BiWFA’s core functions can be easily vectorized to fully exploit the capabilities of modern SIMD multicore processors. Our implementation, relies on the automatic vectorization capabilities of modern compilers. As a result, the BiWFA implementation can exploit the SIMD capabilities of any processor supported by modern compilers, without rewriting any part of the source code.Genomics and bioinformatics methods will continue to rely on sequence alignment as a core and critical component. BiWFA paves the way for the development of faster and more accurate tools that can scale with longer and noisy sequences using a minimal amount of memory. In this way, we expect BiWFA to enable efficient sequence alignment at genome-scale in years to come.btad074_Supplementary_DataClick here for additional data file.


Multiple sequence alignments (MSAs) of homologous sequences contain information on structural and functional constraints and their evolutionary histories. Despite their importance for many downstream tasks, such as structure prediction, MSA generation is often treated as a separate pre-processing step, without any guidance from the application it will be used for.Here, we implement a smooth and differentiable version of the Smith–Waterman pairwise alignment algorithm that enables jointly learning an MSA and a downstream machine learning system in an end-to-end fashion. To demonstrate its utility, we introduce SMURF , a new method that jointly learns an alignment and the parameters of a Markov Random Field for unsupervised contact prediction. We find that SMURF learns MSAs that mildly improve contact prediction on a diverse set of protein and RNA families. As a proof of concept, we demonstrate that by connecting our differentiable alignment module to AlphaFold2 and maximizing predicted confidence, we can learn MSAs that improve structure predictions over the initial MSAs. Interestingly, the alignments that improve AlphaFold predictions are self-inconsistent and can be viewed as adversarial. This work highlights the potential of differentiable dynamic programming to improve neural network pipelines that rely on an alignment and the potential dangers of optimizing predictions of protein sequences with methods that are not fully understood.https://github.com/spetti/SMURF.Our code and examples are available at: Bioinformatics online. Multiple sequence alignments (MSAs) are commonly used in biology to model evolutionary relationships and the structural/functional constraints within families of proteins and RNA. MSAs are a critical component of the latest contact  and ratiLearned Alignment Module (LAM), which is a fully differentiable module for constructing MSAs and hence can be trained in conjunction with another differentiable downstream model. Building upon the generalized framework for differentiable dynamic programming developed in Here, we introduce the We demonstrate the utility of LAM with two differentiable pipelines. First, we design an unsupervised contact prediction method that jointly learns an alignment and the parameters of a Markov Random Field (MRF) for RNA and protein, which we use to infer better structure-based contact maps. Next, we connect our differentiable alignment method to AlphaFold2 [here referred to as AlphaFold, as in We implemented a smooth and differentiable version of the Smith–Waterman algorithm for local pairwise alignment in JAX  implementation to these similarity matrices to align each sequence to the reference, yielding an MSA , takes as input unaligned sequences and jointly learns an MSA (via LAM) and MRF parameters. These parameters can then be used for contact prediction.We used contact prediction as a case study to demonstrate that joint learning with the LAM can recover alignments that have similar (and sometimes better) performance on contact prediction over traditional methods that input an MSA, establishing that our module works as designed. Our method, Finally, we applied the LAM to reveal unexpected behavior of AlphaFold: some low-quality inconsistent alignments yield better structure predictions than sensible alignments of the same sequences. We modify AlphaFold, replacing the input MSA with the output of LAM. For a given set of unaligned, related protein sequences, we backprop through AlphaFold to update the parameters of LAM, maximizing AlphaFold’s predicted confidence. Doing so results in learned MSAs that improve the structure prediction over our initial input MSA for three out of four structures. Despite the improved structure predictions, we find that the MSAs learned by the LAM may be adversarial as indicated by their self-inconsistency. This finding raises questions about how AlphaFold uses the input MSA to make its predictions.Differentiable dynamic programming algorithms are needed in order to model combinatorial structures in a way that allows backpropagation of gradients , grammaret al. implemented a differentiable version of the Needleman–Wunsch algorithm for global pairwise alignment  welignment  and . The value ija in the similarity matrix that we input to Smith–Waterman is the dot product of the corresponding vectors, B other sequences, the LAM constructs a similarity matrix between each sequence and the reference, applies differentiable Smith–Waterman to each similarity matrix, and outputs an alignment of each sequence to the reference (which can be viewed as an MSA)  see . Since tWe confirmed that the similarity scores learned by LAM are much more expressive than BLOSUM scores. Smooth Markov Unaligned Random Field or SMURF.GREMLIN is a probabilistic model of protein variation that uses the MSA of a protein family to estimate parameters of an MRF see , which iSMURF takes as input a family of unaligned sequences and learns both (i) the LAM convolutions and (ii) the parameters of the MRF that are, in turn, used to predict contacts. SMURF has two phases, each beginning with the LAM. First, BasicAlign learns LAM convolutions by minimizing the squared difference between each aligned sequence and the corresponding averaged MSA . This obWe compare SMURF to GREMLIN trained with MLM (MLM-GREMLIN) as in t predicted contacts that are correct for t equals 1 up to L, where L is the length of the protein. We trained and evaluated our model on a diverse set of protein families, as described in Next, we sought to compare qualities of the MSAs learned through SMURF and MSAs fed into GREMLIN, which were generated with HHblits  of all pNext, we applied SMURF to 17 non-coding RNA families from Rfam . Due to In Finally, we performed an ablation study on SMURF . We founNext, we tested whether jointly learning an alignment with AlphaFold could improve structure prediction. While our experiment found this to be possible, the more interesting takeaway was our finding that AlphaFold sometimes makes better predictions from strikingly low-quality alignments as compared to sensible alignments of the same sequences. For our experiment, we selected four CASP14 domains where the structure prediction quality from AlphaFold was especially sensitive to how the MSA was constructed see .1. We reFor each of the four selected CASP targets, separate LAM parameters were fit to maximize AlphaFold’s predicted confidence metrics see . We repeNext, we compared the learned MSAs that led to better structure predictions to the MMSeqs2 MSAs. Evaluating the learned MSAs by eye, we found our learned MSAs to be strikingly low quality. We saw many examples of inconsistently aligned motifs and even pairs of nearly identical sequences exhibiting completely different alignments with the query. We explored a simple explanation for how low-quality alignments could yield improved structure predictions; perhaps AF uses its axial-like attention to consider only a subset of sequences, and the poor alignments by the other sequences isn’t important or could further disqualify those sequences from being attended to. To investigate this, we evaluated how sensitive the AF predictions are to the inclusion of each individual sequence  and 4. SE-value threshold of 10, which may introduce sequences in the MSA that are not true homologs. For targets T1064-D1 and T1070-D1, we removed all sequences with an E-value smaller than E-values 1.4 and 0.16) that almost certainly are not homologs of the query.  While removing either individually does not substantially change the accuracy of the prediction, removing both worsens the prediction with the MMSeqs2 MSA significantly (RMSD 3.46 to 12.11) and worsens the prediction with our learned MSA mildly (RMSD 1.47 to 2.48). In T1070-D1 we realized the opposite outcome; removing the sequences with E-value at least E-value E-value greater than Next, we considered the effect of removing subsets of more distant sequences. The MMSeqs2 MSAs were constructed with a lenient Finally, we repeated our optimization experiment after removing the distant sequences . We founIn this work, we explored the composition of alignment in a pipeline that can be trained end-to-end without usage of any existing alignment software or ground-truth alignments. With SMURF, we trained alignments jointly with a well-understood MRF contact prediction approach and found mild improvement in accuracy using learned MSAs that were consistent and reasonable. When we instead optimized with AlphaFold’s confidence metrics, we found low-quality MSAs that yielded improved structure predictions for three out of four examples. Our result establishes that in some cases AlphaFold can make accurate structure predictions from very low-quality alignments. Therefore, the task of optimizing AlphaFold structure predictions does not force the LAM to learn high-quality alignments. Perhaps by changing our objective function to also penalize self-inconsistent alignments, we could learn more reasonable MSAs while still improving AlphaFold predictions. Our work both establishes the feasibility of pipelines which jointly learn alignments in conjunction with downstream machine learning systems and highlights the possibility of unexpectedly learning odd alignments when it is not well-understood how exactly the downstream task uses alignments.While our findings that low-quality, self-inconsistent MSAs can yield improved AlphaFold predictions and that AlphaFold predictions may be quite sensitive to the inclusion of particular sequences may seem paradoxical, these observations reflect behaviors found across deep learning systems. It is well-known that deep neural networks are not robust to adversarial noise . StudyinOur SSW implementation is designed to be usable and efficient, and we hope it will enable experimentation with alignment modules in other applications of machine learning to biological sequences. There is ample opportunity for future work to systematically compare architectures for the scoring function in SSW. The use of convolutions led to relatively simple training dynamics, but other inductive biases induced by recurrent networks, attention mechanisms, or hand-crafted architectures could capture other signal important for alignment scoring. Moreover, training one network across protein families (rather than training a separate network for each family) to produce vector encodings of residues and their contexts could be a promising strategy for aligning arbitrary pairs of protein sequences. We also hope that the use of these more powerful and general scoring functions enables applications in remote homology search, structure prediction, or studies of protein evolution.Besides MSAs, there are numerous other discrete structures essential to analysis of biological sequences. These include Probabilistic Context Free Grammars used to model RNA Secondary Structure  and Phylbtac724_Supplementary_DataClick here for additional data file.


The continuous development of sequencing technologies has enabled researchers to obtain large amounts of biological sequence data, and this has resulted in increasing demands for software that can perform sequence alignment fast and accurately. A number of algorithms and tools for sequence alignment have been designed to meet the various needs of biologists. Here, the ideas that prevail in the research of sequence alignment and some quality estimation methods for multiple sequence alignment tools are summarized. The developments in sequencing technologies have enabled unprecedentedly fast sequencing speeds and large-scale sequencing capabilities. The increasing number of sequences are challenging the automated sequence analysis procedures ,2. SequeSequence alignment software usually inserts gaps between the nucleotides or amino acid residues in the sequences, so that as many similar sites as possible can be aligned. Finally, a character matrix with the same number of columns and rows that correspond to the number of the sequences is obtained. In this review, the pairwise sequence alignment algorithms and the corresponding scoring system, heuristic algorithms for multiple sequence alignment and their defects, and quality estimation methods used to test multiple sequence alignment software are reviewed. There have been several reviews for multiple sequence alignment, such as Refs ,9. In orO(l1l2), where l1 and l2 are the lengths of the two sequences to be aligned. Such overheads are acceptable for short sequences but not for sequences with more than several thousand sites. As a space-saving strategy of the dynamic programming algorithm, the Hirschberg algorithm [O(l) without any sacrifice of quality.Pairwise sequence alignment is the basis of multiple sequence alignment and mainly divided into local alignment and global alignment. The former is to find and align the similar local region, and the latter is end-to-end alignment. A commonly used global alignment algorithm is the Needleman–Wunsch algorithm , which hlgorithm  is able An optimal solution for the pairwise sequence alignment of very long sequences is usually impossible to find in practice. Heuristic algorithms can be used to reduce the time and space cost incurred by dynamic programming. For this, the most widely applied method is to limit the state transition and conduct the alignment in a smaller search space. Although heuristic algorithms do not guarantee that there will be no poor results, ideal alignments can be achieved in many types of software because the sequences to be aligned are usually quite similar.In addition, the hidden Markov model (HMM) is also widely utilized in sequence alignment tools, such as HHalign , which cOne of the heuristic methods is based on divide and conquer. In such methods, homologous segments (or seeds) are found and used as the “anchors” for the alignment. Each anchor point can divide the dynamic programming matrix into four sub-matrices located at the four corners. Backtracking always goes toward the upper left direction, and these anchors are regarded as the waypoints that the optimal path must pass; therefore, the sub-matrices located at the lower left and upper right are useless and naturally disregarded. When more anchor points distributed throughout the sequences are found, the scale of the dynamic programming matrix can be greatly reduced, thereby reducing the time and space complexity A.Many alignment algorithms are designed to find such seeds to accelerate alignment. FASTA ,22 and BThe concept of divide and conquer is also adopted in some multiple sequence alignment software to scale to large datasets, such as FAME  and FMAlBounded dynamic programming  is basedSeveral methods are available for determining the strip range. A basic idea is to use the shape-based division, but this does not fully consider the biological significance and is rarely used. A simple improvement of this method is to set a threshold to filter the states that could be ignored. If the score of one state in the dynamic programming plus the score for the transition from this state to the final state is greater than the threshold, then transitions from this state are allowed. However, this approach requires transition scores to be estimated and a threshold to be set. The transition scores can be conservatively estimated by its upper bound, and the threshold can be generally determined using the iterative method proposed in Ref .The most critical factor for the quality of a pairwise sequence alignment is the scoring system. It is the basis of the sequence alignment, including multiple sequence alignment, because it determines the direction of the alignment and reflects its quality. Most types of the sequence alignment software aim to obtain good alignment by defining an explicit or implicit objective function for scoring and improving their ability to achieve high score by adjusting alignment strategy. The higher score an alignment can achieve, the higher we think its accuracy will be in the corresponding scoring system. As an example, a model and the corresponding scoring system for pairwise alignment of nucleotide sequences containing frameshifts and stop codons comprise the main feature of MACSE, a multiple sequence alignment tool that is specific to coding sequences and takes into account frameshifts and stop codons .Generally, the score of a pairwise sequence alignment is the sum of the scores of all aligned pairs. For alignment of two protein sequences, for example, each pair of aligned sites is scored depending on whether a gap is involved, or, if no gaps are involved, whether the two aligned residues are matched or mismatched. When a gap is involved, a gap penalty, which is usually a negative score, is given. Additionally, the score for matched and mismatched amino acid residues is generally determined using a substitution matrix.The most basic substitution matrix is based on whether the two residues are matched or not. More complex matrices also take into consideration the attributes of nucleotides or amino acid residues. For example, the score of conversion–transversion matrix reflects the difference between conversion and transversion frequency in natural mutations . ProteinIn addition to substitution matrices, gap penalty is also an important part of the scoring system. A simple rule is to assign a fixed negative score when a nucleotide or amino acid residue aligns with a gap. However, this scoring method has some intrinsic limitations, mainly because insertions and deletions (indels) are small-probability events, especially in nucleic acid sequences where indels can cause frameshifts and disrupt all subsequent codons. Once a gap is inserted in an alignment, adjacent gaps are more likely to occur compared with gaps inserted at a distance from the first gap. Therefore, almost all sequence alignment algorithms now use the gap penalty rule based on the number of the gaps successively inserted, and the most typical one is the affine penalty. No optimal solution is universally applicable for the gap penalty, which is referred to as a “black art” requiring constant trial of errors .Optimal alignment results can be obtained under a certain scoring system with the multi-dimensional dynamic programming by extending the classic Needleman–Wunsch algorithm. However, the time and space requirements grow exponentially with the number of sequences, which makes it impossible to complete an alignment in an acceptable time or space using this method even for a small sequence data set. To overcome this problem, some heuristic algorithm frameworks that seek quasi-optimal alignment solutions within reasonable time and space have been developed. There are also important algorithms that cannot be categorized as one of the algorithm frameworks described below, such as MAHDS , which ia, b, c, and pairwise alignments A, A, if there is ai ~ ck in A and bj ~ ck in A, then ai should be aligned with bj, where c is regarded as the center sequence, and ai ~ bj means that the ith residue of a is aligned with the jth residue of b.The star alignment strategy  is a heuThe star alignment algorithm uses the star topology to simulate the relationship between sequences, and the selection of the center sequence determines the accuracy of the alignment to some extent. A brute-force choice is the sequence that has the shortest average distance from the other ones, but the time required is proportional to the square of the number of the sequences. Therefore, it is usually difficult to apply this scheme to data sets with large number of sequences. Generally, the longer the sequence is, the more likely the information lost by other sequences due to mutations will be retained. On the basis of this assumption, the clustering software CD-HIT  selects This system has a limited ability to simulate the relationship between sequences, so it is less effective for data sets with complex relationships; but good alignment results can often be obtained for data sets with high similarities. Moreover, thanks to its extremely low time complexity, it can handle tasks that are almost impossible to be completed by other algorithms within a finite time, so that an approximate solution can be quickly obtained. The pairwise sequence alignments involved in the star alignment strategy are independent of each other, making it easy to parallelize programs based on the star alignment strategy, for example MASC  and I-CSThe progressive alignment strategy  is anothO(n2) time and space complexity is still a problem, where n represents the number of the sequences to be aligned. To avoid the calculation of the distance matrix, some multiple sequence alignment software adopts heuristic methods, such as PartTree [O(l2), which is too time consuming. Therefore, the string similarity based on k-mers [The tree topology is used in the progressive alignment algorithm to simulate the relationships between sequences, and therefore, the alignment quality depends on the quality of the guide tree. Several established hierarchical clustering algorithms for the construction of guide trees are available. Among them, unweighted pair group method with arithmetic mean (UPGMA)  has beenPartTree  used in PartTree  and mBedPartTree  used in PartTree  and KaliPartTree . On the n k-mers  has becon k-mers  and redun k-mers ,62,63.In addition, PRANK  and PAGAn pairwise alignments. However, the pairwise alignments in the latter involve profile-to-profile alignment, which requires additional computation. Furthermore, the selection of the center sequence in star alignment could be very simple, while the construction of a guide tree in progressive alignment requires a hierarchical clustering, which is complicated and thus takes much more time. These make progressive alignment more time consuming than star alignment, especially when the number of sequences is over tens of thousands.Compared with the star alignment, the tree topology used in the progressive alignment fits far more closely with the natural relationships among sequences; therefore, progressive alignment can often process more complex data  and has The star alignment and the progressive alignment strategies have two main defects. The “once a gap, always a gap” cardinal rule  is one oTwo methods can be applied to handle these problems : one is a, b, and c, for which the pairwise sequence alignments are A, A, and A. If there is ai ~ bj in A and bj ~ ck in A, then the score of ai ~ ck should be increased in the multiple sequence alignment of these three sequences. Match scores of every possible pair are re-estimated in this way. Gap penalties are already considered in the pairwise sequence alignment. The matching score of the two residues obtained by re-estimation with the consistency objective function is the score after the gaps have been penalized. Therefore, gap penalties need not be considered in the subsequent multiple sequence alignment process. It takes O(n2l2) time to obtain all the possible pairwise alignments and O(n3l) time to re-estimate the match scores of all the pairs indirectly aligned [Consistency objective function provides another scheme for scoring two aligned residues. For example, consider three sequences,  aligned , which i aligned ,72. Prob aligned  goes fur aligned .The iterative refinement method ,75 proceProgressive alignment relies on the guide tree, but the heuristic distance estimation or hierarchical clustering do not necessarily produce the optimal tree for alignment. Therefore, in some iterative refinement algorithms, the completed alignment results are used to recalculate the sequence distance matrix to construct a more solid guide tree, which can be used to improve the alignment performance in an additional round of alignments. This refinement method is seldom used as the core of iterative refinement because of its excessive time overhead. Nevertheless, SATé ,77 uses Currently, an iterative refinement method based on division and re-alignment is widely used. In this method, the results of the old completed alignment are divided horizontally into two parts, and the columns that contain only gaps are deleted to form a valid alignment for each part. Then, the two parts are re-aligned to get new completed alignment. Finally, the old and new completed alignments are compared, and the better one is selected as the basis for the next iteration. The iteration will continue until it meets a certain condition, which is usually provided as an argument for users to determine, depending on how much time they can afford to improve the accuracy of the alignment. A commonly used horizontal segmentation method is to cut an edge of the guide tree to divide all sequences into two parts, which has been shown to produce the best trade-off results between time consumption and accuracy ,79. MoreA reliable scoring method is needed to quickly and effectively compare the two completed alignments. The SP (sum of pairs) ,83 scoreAnother important kind of multiple sequence alignment methodology is stochastic and mainly based on evolutionary algorithms, which are intrinsically iterative. It starts with a set of possible alignments and performs genetic operators on them to yield a new generation of alignments, which act as the initial alignments of the next iteration. This kind of algorithm has become a promising alternative field of multiple sequence alignment research . Two of Multiple sequence alignment software uses different algorithms, and therefore, the alignment quality can vary. To score the alignment algorithms and to provide references for the design and development of sequence alignment software, different quality estimation methods have been proposed. Several kinds of classical quality estimation methods , as wellStructured benchmarks for protein alignment software include BAliBASE , which iAnother method scores multiple sequence alignment software using generated data sets. These methods simulate the evolution of sequences  using a probabilistic model and generate the evolved sequences and the reference alignment based on the evolution. Because the generated mutations are completely determined by the evolution model, the accuracy of such benchmarks is restricted by the degree to which the adopted model represents the natural evolution. In some conditions, the probabilistic model can even bias the estimation if it is similar to the sequence relationship model the tested software adopts, which also poses a challenge to the establishment and selection of the probabilistic model.p-value, which indicates the likelihood that the performance difference between different methods is due to chance.These two types of benchmarks provide preset reference alignment against which the performance of newly developed software can be tested. Therefore, a scoring method is needed to measure the degree to which the alignment of the tested software is close to the reference. Two commonly used scoring methods are the sum of pairs (SP) and total column or true column (TC) scores , which eAnother kind of quality estimation method does not rely on reference alignment but the commonality among the alignments obtained by different multiple sequence alignment strategy. It is based on the idea that if several different pieces of software consistently align two residues, then, most likely, these two sites are correctly aligned. However, an important defect of this method is that if different pieces of software consistently but wrongly align two residues, then this error will be deemed correct in the scoring. Unlike the estimation methods based on reference alignment, the commonality-based methods use some special scoring methods, such as the multiple overlap score  and the The explosive growth of biological sequence data has brought unprecedented challenges to the automated biological sequence analysis software. Sequence alignment, as the basis of sequence analysis, restricts subsequent analysis in applicability and accuracy. In this review, pairwise sequence alignment and its scoring system, main algorithms for multiple sequence alignment, as well as their advantages and disadvantages, and the quality estimation methods for multiple sequence alignment software, are presented and discussed. The collation of this information  might he


Detailed and accurate analysis of mobile genetic elements (MGEs) in bacteria is essential to deal with the current threat of multiresistant microbes. The overwhelming use of draft, contig-based genomes hinder the delineation of the genetic structure of these plastic and variable genomic stretches, as in the case of pipolins, a superfamily of MGEs that spans diverse integrative and plasmidic elements, characterized by the presence of a primer-independent DNA polymerase.ExplorePipolin is a Python-based pipeline that screens for the presence of the element and performs its reconstruction and annotation. The pipeline can be used on virtually any genome from diverse organisms and of diverse quality, obtaining the highest-scored possible structure and reconstructed out of different contigs if necessary. Then, predicted pipolin boundaries and pipolin encoded genes are subsequently annotated using a custom database, returning the standard file formats suitable for comparative genomics of this mobile element.github.com/pipolinlab/ExplorePipolin.All code is available and can be accessed here: Bioinformatics Advances online. Infectious diseases caused by multidrug-resistant pathogens pose a major threat to global health, food security and development today, a situation aggravated by the insufficient production of new drugs . This relates pipolins to other elements commonly referred to as ‘self-synthesizing’ by the presence of a PolB, the eukaryotic polintons and casposons, detected in archaea and some bacteria , overlapping with a tRNA gene. We previously took advantage of that common structural features to delineate their structure out of contig-scale E.coli draft genomes using a preliminary customized Python pipeline . This strategy is also implemented in other tools, as Phaster , as the element can be often split into two or more contigs. Scaffold of pipolins might be improved using raw reads or comparison with related elements, as they maintain a certain degree of gene synteny . Additionally, when alternative orientations of the piPolB-containing fragment are possible, both reconstructions are generated and denoted as v0 for that with the piPolB gene pointing toward the attR , which iatt direct repeats are also included in the output files and labeled as ‘pipolin conserved’ or ‘de novo’, according to their relationship to known att-like sequences of E.coli pipolins or identified as a novel direct repeat fragment.The pipolin reconstructed structure(s) are subsequently annotated using a custom Prokka-based pipeline that includes previously known pipolin-related genes. The output includes files in standard formats GBK and GFF generated for wide compatibility for downstream analysis of pipolins genetic structure. An additional file, appended with the suffix ‘single_record’, contains a final reconstructed pipolin comprising individual pipolin fragments from different contigs joined by an assembly gap feature type. The annotated att, tyrosine recombinases, restriction–modification systems, uracil DNA glycosylase…) for straightforward representation with EasyFig  takes approximately 30–60 s per genome using multiple threads for annotation (option –cpus). Further optimization may be achieved using additional parallelization strategies, like GNU Parallel . MoreoveE.coli-harboring pipolins collection obtaining nearly identical pipolin structures than in the original analysis (Notwithstanding a throughout modification of the original pipeline, ExplorePipolin analyzed the genomes of our laboratory analysis .att sequences or any direct repeat at all.Furthermore, the new reconstruction strategy allows successful reconstruction and analysis of pipolins from a wide range of distant organisms, spanning different bacteria phyla and Fungi genomes . The usaExplorePipolin is an example of a comprehensive pipeline that not only looks for MGE presence by marker screening but also reconstructs the whole pipolin element structure and attempt to provide precise boundaries, making further analysis more accurate and straightforward. Importantly, it can be widely used to screen for pipolins on virtually any genome of a wide range of organisms and diverse assembly qualities and it returns putative reconstructed pipolins with homogenous annotation suitable for comparative genomics of this mobile element.Furthermore, the modular structure of the pipeline will facilitate its future update for bulk analysis of other elements with one or few universal hallmark features, particularly those related with pipolins, such as casposons or polintons.vbac056_Supplementary_DataClick here for additional data file.


The alignment of large numbers of protein sequences is a challenging task and its importance grows rapidly along with the size of biological datasets. State-of-the-art algorithms have a tendency to produce less accurate alignments with an increasing number of sequences. This is a fundamental problem since many downstream tasks rely on accurate alignments.a posteriori objective and decode an alignment. We rely on automatic differentiation of the log-likelihood, and thus, our approach is different from existing HMM training algorithms like Baum–Welch. Our method does not involve progressive, regressive, or divide-and-conquer heuristics. We use uniform batch sampling to adapt to large datasets in linear time without the requirement of a tree. When tested on ultra-large protein families with up to 3.5 million sequences, learnMSA is both more accurate and faster than state-of-the-art tools. On the established benchmarks HomFam and BaliFam with smaller sequence sets, it matches state-of-the-art performance. All experiments were done on a standard workstation with a GPU.We present learnMSA, a novel statistical learning approach of profile hidden Markov models (pHMMs) based on batch gradient descent. Fundamentally different from popular aligners, we fit a custom recurrent neural network architecture for (p)HMMs to potentially millions of sequences with respect to a maximum Our results show that learnMSA does not share the counterintuitive drawback of many popular heuristic aligners, which can substantially lose accuracy when many additional homologs are input. LearnMSA is a future-proof framework for large alignments with many opportunities for further improvements. Apart from the model architecture being problem dependent, another common issue is that algorithms may get stuck at local optima in the parameter space. Simulated annealing [Profile hidden Markov models (pHMMs) are probabilistic models for protein families. One of their applications is remote homology search in large databases , 2. Typinnealing  and partnnealing , 7 couldnnealing  are a hinnealing .Established tools that construct MSAs are either unfit for large numbers of sequences or their accuracy decreases when the number of aligned sequences grows large , 11. ThiOne way to revert incorrect gaps is iterative refinement, where intermediate alignments guide the construction of subsequent ones . Althougdissimilar ones first and merges subalignments by using an overlapping sequence [Divide-and-conquer strategies like PASTA  and MAGUsequence . Divide-sequence ,15. HoweLast, UPP  is relatAll mentioned MSA algorithms rely on accurate guide trees, and tree construction often becomes the computational bottleneck. Clustal Omega  uses theTo date, deep learning is not commonly used for multiple sequence alignment, and if it is, its function is usually supplementary, for example, by optimizing the order of progressive alignment with reinforcement learning  or emploOur proposed aligner learnMSA is based on automated statistical learning of a pHMM with gradient descent. It does not require a tree and has a linear asymptotic runtime in the number of sequences, which is faster than most tree algorithms. No progressive, regressive, or divide-and-conquer heuristic is used. Therefore, we avoid heuristic-based errors when merging subalignments or progressively adding sequences. We provide a more robust framework for (ultra-)large MSAs without the counterintuitive drawback of losing accuracy when many additional homologs are input.We begin with the description of the underlying model and a batchwise variant of the forward algorithm that plays a central role during parameter training. We empirically show the suitability of learnMSA by testing it on ultra-large protein families from Pfam  with up Profile hidden Markov models are well-known probabilistic models of sequence consensus. When used to model a protein family, the aim is to define a probability distribution over the space of all possible protein sequences such that member sequences of the family have large probabilities. The resulting statistical model can be used for database searches  and MSA In a pHMM, a linear chain of match states represents the consensus sequence of the family in question. Insertions and deletions with respect to the consensus are modeled by position-specific states and transitions. See Fig. tabula rasa. We have special states for the left (L) and right (R) flank of the model. Initialization and regularization of the flanking states differ from ordinary insertion states iI (see section “Training”). Moreover, the augmented model allows multihit alignments . The state C models any unannotated region between 2 domain hits and must be visited to jump from the end state E back to the start state S. The model further handles sequence length heterogeneity (fragmentary sequences) through entry and exit probabilities from S into the consensus and, respectively, from the consensus to E. Note that since version 2, HMMER uses a trick to achieve a uniform distribution over all possible pairs of entry and exit points into the core model [ad hoc.In addition to the standard pHMM architecture, we deploy an augmented model following HMMER’s “Plan7” , 30 ora. The HMML, R, and C are tied to prevent a bias toward one of the sides. Delete states (as well as the domain start and end states S and E) are silent .The set of all transition and emission parameters is learned from data with careful initialization and under the use of Dirichlet priors (see section “Training”). In general, we have 1 trainable parameter for each possible state transition and, in case of the emissions, 1 parameter per match state and amino acid. There are exceptions: insertion and flanking states use a fixed background emission distribution that is not optimized. The self-loop (and respectively exit) probabilities for the flanking states Q be the set of all states and A be the stochastic |Q| × |Q| matrix of state transitions. Observe that for pHMMs, this matrix is very sparse. We call the number of match states in a model its length l. Let B be the |Q′| × 25 emission matrix, which is constructed by concatenating l learnable emission distributions of the match states with background distributions for all insertions and the flanks. The second dimension of B corresponds to the 20 standard amino acids, plus selenocysteine, pyrrolysine, and the ambiguous codes X, B, and Z. The terminal symbol (26th letter) has an implicit probability of 0 at all states except T.A pHMM can be parameterized by 2 probability matrices for transitions and emissions and an initial state distribution. Let A and θB and enforce the probabilistic constraints that the rows of A and B sum up to 1 with a softmax function defined on a real vector: In order to apply gradient descent, we parameterize the model by unconstrained kernels θiM is computed by ith row of θB. The matrix B is constructed from the kernel θB by using softmaxes to compute the match distributions over the amino acid alphabet.As seen in Fig. A is a collection of parameter vectors corresponding to different transition types that share the same initialization and prior. For example, we have l − 1 parameters for the match-to-match transitions. The total number of allowed transitions in the model as shown in Fig. l. The probability distribution of transitioning from, for example, match iM to one of the 4 adjacent states Mi + 1, iI, Di + 1, or E is calculated by constructing the vector A  in sparse matrix representation where illegal transitions are implicitly zero.The kernel θP0, we use a simple parametrization by introducing a scalar θinit that controls the probability of starting in the left flank. To this end, we define initp = σ(θinit), where σ is the sigmoid function. The initial distribution is P0(L) = initp and P0(S) = 1 − initp and P0(q) = 0 for q ≠ L, S.For the initial state distribution init, θA, θB) denote the complete set of learnable parameters for the (augmented) pHMM.In the following, let θ =  = ∑πP, which is related to HMMER’s forward score [The likelihood of a sequence is the sum of the joint probabilities over all possible hidden paths: rd score . Intuitird score . We presn − 1) isq = P(s0∣q)P0(q).The forward probabilities are b. Let the b × 25 matrix Si) are b × |Q| batches of scaled forward variables, ○ denotes element-wise multiplication , and the matrix multiplication that involves A uses an efficient implementation that exploits the sparse representation. Observe that Equation  lends itP(S) = ∑qα(n − 1)q. However, we prevent numerical underflow by equivalently using the partial log-likelihood values in Equation (The likelihood (for a single sequence) can eventually be computed as Equation :(3)\docWhen we decode an alignment, we are interested in the hidden path of a sequence with maximum probability  [iM → Di + 1 → …Dj − 1 → jM for j > i + 1 is replaced by an edge with probabilityConventionally, the forward recursion for pHMMs is implemented in linear time per step by explicitly handling silent states (the delete states state E) . This rei) in parallel given α(i − 1). In the batchwise forward algorithm, the bottleneck is the matrix multiplication with the transition matrix, which should use an efficient implementation that exploits sparseness.This changes the asymptotic runtime of the forward algorithm, because the number of possible transitions from each match state is not constant anymore. However, we can now implement Equation  by takinimplA replacing A from section “Model.” Note that implA is still very sparse. Transitions over the start state S and the end state E  are handled analogously. Also note that empty, infinite silent loops through the model are not possible, because the unannotated segment state C is an insertion that emits at least 1 amino acid and cannot be skipped.Folding all edges adjacent to silent states is referred to as the implicit model, represented by a transition matrix b sequences isDuring training, learnMSA uses a recurrent neural network architecture with a pHMM cell that scans a batch of sequences column-of-residues-wise and successively applies Equation . This arThe general goal while successively observing random batches is to adjust θ such that Existing optimization algorithms like Baum–Welch  or simulfamilies . Traditifamilies . Typicalm is low. Our experiments will mainly focus on cases where m is large . However, we can still have overfitting problems. Domain motifs of subfamilies might be underrepresented in the sequence set, leading to a skewed model. Moreover, we might end up with a result that fits the data well but is not biologically plausible . A maximum a posteriori estimate attempts to fit the data while at the same time penalizing implausible models [Models found by maximizing e models . In thisb < m and consequently perform stochastic gradient descent. This allows us to rapidly train models even on millions of homologous sequences. We use random uniform batch sampling. The second term is the prior density . We normalize by The loss ℓ has a foundation in Bayesian statistics. The first term is the log-likelihood per sequence averaged over a batch of sequences. Usually, we choose tandards , we use tandards , 35 overTo reduce the total number of hyperparameters that have to be set by hand, we salvaged as much general-purpose information as possible from Pfam HMMs. For the core model probabilities, we took over 3 million example transition distributions and maximized the likelihoods of 3 Dirichlets: one for matches, insertions, and deletions, respectively , which we trained on the match emission distributions of Pfam HMMs, but found that for large sequence counts, a single Dirichlet density  is enough. The expectation of this Dirichlet distribution is also used to initialize the match emissions as well as the (fixed) insertion emissions and the flanks.flank, αsingle, and αglobal and (currently only roughly) searched for suitable values based on the quality of the produced alignments. These hyperparameters have a probabilistic foundation as parameters of Dirichlet priors over specific Bernoulli distributions that were defined to favor the probability p = 1 for particular, carefully defined events. That means the prior can be maximized by maximizing p, but this choice has to be balanced with the likelihood. For each possible choice of p and α, the logarithmic prior densities are (α − 1)ln p + (α′ − 1)ln (1 − p), where we set α′ = 1. The motivation behind this ad hoc choice was to keep the set of hyperparameters for the method simple while maintaining the probabilistic interpretation of the regularization term.As described earlier, we optimize the transition probabilities for flanking states, domain multihits, and the entry and exit probabilities jointly with the core model. We found that these transitions require strict regularization. We defined a simplified set of hyperparameters αflank controls the pressure to align to the core model (rather than using the flanking states), that is, increasing αflank will result in longer insertions at the flanks and between repeated domain segments. The parameter αflank regularizes the self-loop probabilities of all flanking states, as well as P0(L) and P(R∣E). Furthermore, we introduce αsingle to penalize core model repeats favoring large values for the probability 1 − P(C∣E) = P(R∣E) + P(T∣E). Last, αglobal penalizes local alignments that use entry and exit transitions other than S → M1 and lM → E. The probabilities regularized by αglobal were chosen such that all choices of start and end points into the consensus S → iM → … → jM → E for 1 ≤ i ≤ j ≤ l,  ≠  are penalized uniformly. More precisely, we favor large probabilities 1 − P(iM∣S)P(E∣jM) for 1 ≤ i ≤ j ≤ l,  ≠ . The values used for this article are αflank = 7,000, αsingle = 1e9, and αglobal = 1e4.In particular, αl by taking the median of the sequence lengths and scaling it by a constant c. We found that c = 0.8 works well. It is easier to find a rough initial consensus if the number of match states is limited, which forces the model to restrict itself to the more relevant parts of the sequences. The median is more robust against fragmentary sequences than the average.First, we guess an initial model length tabula rasa training with an universal initial parameter set independent of the input sequences. We chose an ad hoc position independent initialization that reflects the prior distributions. Intuitively, we want the initial model to focus its probability mass on paths that use all match states. We do this by having larger probability for the initial match–match transitions. We took care to initialize the entry probabilities dependent on the model length such that P(M1∣S) is always roughly E → C with a very small probability, and for the flanking states L, R, and C, we initialize such that the self-loops are more likely than the exits.The initialization of θ could in principle use prior knowledge about the protein family at hand. However, we are interested in model surgery [After training, we might observe rarely used match states or overused insertion states. We can discard or expand those positions and adapt the model length, which is known as  surgery .init, the kernel for the transition distribution of the end state E, and the evolutionary times τ of the ancestral probability layer  are reset to default and the model is trained again. This is repeated at most 4 times, which we found is a good compromise between speed and accuracy. Per default, we train 5 independent models and optimize them with model surgery. Eventually, we choose the model with parameters θ that maximizes Given a trained model, we discard match positions that are used by less than 50% of the sequences. Likewise, we expand positions where more than 50% of all sequences have an insertion by a number of new match states equal to the average insertion length. If a match position is discarded, all incident edges are removed and new edges with default initialization are carefully inserted to close the holes (there is a hole for each consecutive segment of discarded positions). If an insertion is expanded, edges at the position of interest that connect left and right model parts are removed. Eventually, all edges incident to a new match state are default initialized. After each surgery iteration, the flanking states, θqth quantile while keeping a minimum of k sequences. Therefore, initial parameter updates are always on sequences that have roughly full length. Short fragmentary sequences may disturb early training epochs. It is easier to incorporate them, if a rough consensus is established and the matter simplifies to fine-tuning the entry, exit, and repeat probabilities. We found that k = 10,000 work well. This is in line with other large-scale MSA methods, where a common denominator is a strong preliminary focus on putative full-length sequences . For example, MAFFT-Sparsecore only considers sequences with lengths above the median for its core alignment and the regressive strategy favors the longest sequence as representatives of subtrees .If the number of surgery iterations is >1, we found it beneficial (both performance- and accuracy-wise) to restrict training in all but the last iterations to sequences with lengths above the C that represents the sequence set we wish to align. Homologous sequences iS may be closely or distantly related to C . Model-wise, we introduce evolutionary times τi to estimate the distance of iS to C. The process is conventionally described by the general time-reversible substitution model parameterized by a 20 × 20 matrix Q of instantaneous substitution rates from 1 amino acid to any other [Q models prior biological knowledge on the relative expected frequencies of amino acid substitutions. From Q, the amino acid mutation probabilities after time τ given an initial amino acid can be derived as follows:ath row of this matrix, P(τ)a, corresponds to the expected amino acid distribution after time τ when starting with amino acid a. As the model is time-reversible, it is also the distribution of amino acids τ time units ago at a site where amino acid a is observed now.We naturally assume the existence of a single whole-protein consensus sequence ny other ,37. LikeS ≤ 2.5, where the maximal value of 2.5 corresponds to the PAM250 matrix and zero is the identity. The vector τ is learned jointly with the HMM parameters θ. Put differently, we learn the branch lengths of a starlike tree jointly with the sequence model. For each batch of sequences, the correct subset of τ is gathered. The ancestral probabilities with the final values τ are also used during Viterbi decoding of the alignment. More precisely, we replace all likelihoods We initialize τ with zeros and optimize it under the constraints 0 ≤ τi are related to sequence weights, but they are learned from data and do not require a tree or any other pairwise sequence comparison. Assume that for some suitable distance metric, one sequence iS has a large total distance to all other sequences. In a sequence weighting scheme, iS would typically have a larger weight than sequences with many close relatives to account for the underrepresentation. Choosing a large τi can increase P by smearing iS toward the consensus. But this increase is independent of all other sequences and involves no change of θ.The τWe use TensorFlow  to automWe tested learnMSA on HomFam , BaliFamTo test the ability of our method to align under high sequence length heterogeneity, we constructed a fragmentary variant of BaliFam10000 by following the procedure that was used to test UPP before . We chosFinally, we experimented with 10 ultra-large datasets that were acquired from Pfam by selecting the largest families  and downloading the respective UniProt datasets that were generated by searching the UniProtKB database using the Pfam family HMM. We also downloaded the corresponding seed alignments to use them as a reference. For the training datasets, we added the seed sequences to the UniProt datasets if not already present and removed all gaps. The families are Zinc finger C2H2 type (PF00096), WD domain G-beta repeat (PF00400), ABC transporter (PF00005), Protein kinase domain (PF00069), Ankyrin repeats (PF12796), Major Facilitator Superfamily (PF07690), Leucine rich repeat PF13855), Fibronectin type III domain (PF00041), Response regulator receiver domain (PF00072), and Immunoglobulin I-set domain (PF07679). All have known 3-dimensional structure. ABC transporter is the largest dataset with about 3.5 million sequences. See Table 855, FibrWe compared learnMSA to the following aligners: Clustal Omega (version 1.2.4), regressive T-Coffee (version 13.45.0.4846264), MAGUS (git hash f9a3676 from 2022-01-21), UPP (version 4.5.2), and MAFFT-Sparsecore (MAFFT Version 7.490). To the best of our knowledge, there is no mature deep learning–based tool for large multiple alignment of proteins available for comparison.The command lines to align HomFam and BaliFam were (input/output and CPU arguments omitted):    MsaHmm.py    clustalo -t protein --outfmt=fa    mafft-sparsecore.rb    run_upp.py -M -1 -m amino    magus.py -t clustal --recursive false    t_coffee -reg -nseq 100 -tree mbed             -method mafftginsi_msaand for the ultra-large datasets :    mafft --parttree    magus.py -t random --recurse True             --recurseguidetree clustal    t_coffee -reg -nseq 1000 -tree parttree             -method mafftfftnsi_msaWe ran learnMSA as well as UPP on all datasets (including ultra-large) in default mode without manual parameter adjustments. We did not attempt to align the ultra-large files with Clustal Omega, because we already observed a severe drop in accuracy on sequences in the thousands. MAFFT-Sparsecore refused to align the ultra-large datasets. We used MAFFT with the parttree option instead. For MAGUS, we enabled recursion for the ultra-large datasets, set the guide tree for the highest recursion level to “random” due to very long runtimes with other choices, and used clustal trees for all other recursion levels. To use T-Coffee regressive on the ultra-large datasets, we increased the maximum number of sequences in the subalignments to 1,000 in the hope that we could avoid very long MSAs due to concatenated independent gaps during the merging steps. For a speedup, we also ran T-Coffee with parttree and MAFFT FFT-NS-i. All parameter changes in order to align the ultra-large datasets were done reactively after testing the slower and more accurate settings used for HomFam and BaliFam first.Our method was run using 8 CPU cores, 100 GB of RAM, and a NVIDIA GeForce RTX 3090 GPU for all datasets, including the ultra-large ones. All other aligners did not utilize a GPU and were run using 8 CPU cores and 100 GB of RAM for HomFam and BaliFam and 16 cores and 500 GB of RAM for the ultra-large datasets. We chose all memory numbers as a safe upper limit and did no further experiments to evaluate tight requirements. We used a wall clock limit of 3 days for each individual ultra-large alignment.aln_compare option. The reference sequences are not known to the aligning method.Sum-of-pairs (SP) score and total column (TC) score were computed by comparing the subalignments induced by the reference sequences to a structure-based alignment  or the Pfam seed alignment (in case of the ultra-large datasets). We used T-Coffee with the On the ultra-large datasets, learnMSA is most accurate and fastest in almost all cases . For the same, Clustal Omega took 3.5 hours, MAFFT-Sparsecore 24 hours, UPP 19 hours, T-Coffee regressive 9 hours, and MAGUS 48 hours.We also evaluated how increasing the number of homologs that are aligned together with the reference sequences affects alignment accuracy. To create a biologically realistic test setting, we took the 10 Pfam datasets from Table As seen in Fig. For the high-fragmentation collection BaliFrag, learnMSA can compete with MAFFT-Sparsecore, UPP, and MAGUS Fig. . All relWe have proposed learnMSA, a novel unsupervised learning approach for the alignment of large protein families. In contrast to state-of-the-art aligners, learnMSA does not require a tree, which eliminates a crucial performance bottleneck and makes learnMSA asymptotically fast—linear in the number of sequences. It is interesting to see that state-of-the-art performance on large sequence numbers can be reached without a tree by uniform batch sampling. Our method does not rely on progressive, regressive, or divide-and-conquer heuristics. We showed empirically that learnMSA, when aligning millions of sequences, is both more accurate and faster . Moreover, when aligning Pfam families, additional homologs decrease the accuracy of traditional, heuristic methods , whereas learnMSA is more robust. Whether this statement also applies to established benchmarks like HomFam remains an open question that can be answered if more homologs are gathered for these datasets in the future. A similar scaling experiment, which was done for T-Coffee regressive  based ontabula rasa configuration, which greatly reduces the required hand-holding. This is also beneficial, if a suitable alignment mode for a dataset is unknown. LearnMSA is designed in a way that minimizes the assumptions a user has to make. Note that for all tested datasets, including dramatically varying sequence numbers and levels of fragmentation, we used learnMSA’s default configuration of hyperparameters. It should be pointed out that learnMSA is particularly accurate compared to other methods when aligning families that contain multihits. This is clearly visible in Fig. LearnMSA generalizes and automatizes earlier pHMM training approaches for protein families. It does this by taking HMMER’s “Plan7” model but avoids the manual adjustment of the “alignment mode” . Instead, the extra states and transitions . Fig. Note that in principle, learnMSA could also align DNA/RNA sequences, but this feature is not implemented yet. Machine learning methods can likely play out their advantages more for proteins due to the relative complexity of parameter space and priors. Further, learnMSA is currently best suited for short- or medium-length sequences.Our proposed approach constitutes a probabilistically grounded framework for a large MSA that has potential for further improvements in several directions. Further development might be straightforward because of the extensible nature of our method.A natural extension of the work presented here are ensembles of pHMMs. They are used in UPP, where a subset of the sequences is aligned and subsequently represented by an ensemble. Recently, MAGUS combined with an HMM ensemble has shown improved accuracy as well . On the When benchmarking learnMSA, we observed decreasing relative performance when reducing the number of sequences to align. The behavior of state-of-the-art tools is usually complementary: they are more accurate for lower sequence numbers. Moreover, Fig. In contrast to traditional learning algorithms for HMMs, gradient-based learning can, in principle, be a module of a larger machine learning model that is trained end-to-end. By design, learnMSA can incorporate any type of sequence context encoded into the HMM alphabet. For instance, single-sequence secondary structure predictions can be incorporated. Secondary structure is more conserved than primary sequence, and this approach has been shown to increase accuracy in the presence of low sequence identity . There aProject name: learnMSAhttps://github.com/Gaius-Augustus/learnMSAProject homepage: Operating system(s): Platform independentProgramming language: Python3Other requirements: Python packages tensorflow, optional for visualization: networkx, logomakerLicense: MITRRID: SCR_022572biotoolsID: learnMSAhttps://github.com/felbecker/MSA-HMM-Analysis. An archival copy of code and data is also available via the GigaScience database GigaDB [The datasets supporting the results of this article are available in the repository e GigaDB .GPU: graphics processing unit;MSA: multiple sequence alignment;(p)HMM: (profile) hidden Markov model.The authors declare that they have no competing interests.F.B. designed and implemented learnMSA, prepared the data, ran all software, and wrote the manuscript. M.S. conceived the idea and designed and implemented an initial version of a recurrent machine learning layer for HMMs and provided prototype code for the usage of ancestral probabilities. Both authors approved the final manuscript.giac104_GIGA-D-22-00144_Original_SubmissionClick here for additional data file.giac104_GIGA-D-22-00144_Revision_1Click here for additional data file.giac104_GIGA-D-22-00144_Revision_2Click here for additional data file.giac104_Response_to_Reviewer_Comments_Original_SubmissionClick here for additional data file.giac104_Response_to_Reviewer_Comments_Revision_1Click here for additional data file.giac104_Reviewer_1_Report_Original_SubmissionJulie Thompson -- 7/10/2022 ReviewedClick here for additional data file.giac104_Reviewer_1_Report_Revision_1Julie Thompson -- 9/6/2022 ReviewedClick here for additional data file.giac104_Reviewer_2_Report_Original_SubmissionJames Procter -- 7/13/2022 ReviewedClick here for additional data file.giac104_Reviewer_2_Report_Revision_1James Procter -- 9/6/2022 ReviewedClick here for additional data file.


As the size of reference sequence databases and high-throughput sequencing datasets continue to grow, it is becoming computationally infeasible to use traditional alignment to large genome databases for taxonomic classification of metagenomic reads. Exact matching approaches can rapidly assign taxonomy and summarize the composition of microbial communities, but they sacrifice accuracy and can lead to false positives. Full alignment tools provide higher confidence assignments and can assign sequences from genomes that diverge from reference sequences; however, full alignment tools are computationally intensive. To address this, we designed MTSv specifically for alignment-based taxonomic assignment in metagenomic analysis. This tool implements an FM-index assisted q-gram filter and SIMD accelerated Smith-Waterman algorithm to find alignments. However, unlike traditional aligners, MTSv will not attempt to make additional alignments to a TaxID once an alignment of sufficient quality has been found. This improves efficiency when many reference sequences are available per taxon. MTSv was designed to be flexible and can be modified to run on either memory or processor constrained systems. Although MTSv cannot compete with the speeds of exact k-mer matching approaches, it is reasonably fast and has higher precision than popular exact matching approaches. Because MTSv performs a full alignment it can classify reads even when the genomes share low similarity with reference sequences and provides a tool for high confidence pathogen detection with low off-target assignments to near neighbor species. Advancements in sequencing technologies over the last few decades have transformed the way that we study complex microbial communities, which in turn has had a large impact on human health, ecology, and industry. In the past, characterization of microbial communities was restricted to sequencing the few individual members that could be successfully isolated and cultured. Fortunately, due to dramatic improvements in the efficiency and cost-effectiveness of sequencing technologies, we can now use shotgun metagenomic sequencing to study the composition of entire microbial communities in a culture-independent manner.e.g., Bowtie2) use various heuristics to rapidly search for good  alignment locations within a single reference genome. Although there may be many potential alignment locations within the genome, aligners will typically perform a prescribed level of search effort and then report only the best alignment found   . This isBLAST and its many variations  were dese.g., species that diverge from the reference sequences and low abundance organisms) and Metalign only includes a single representative genome per taxon which may reduce assignments. Metalign uses Minimap2 . Our goal in developing MTSv was to use efficient data structures and algorithms, like those used by traditional read alignment tools , in a way that was specifically designed to handle metagenomic data. As described above, traditional read aligners typically report only the top alignments for each read which causes them to miss assignments to multiple taxa in mixed reference databases. However, when forced to report all valid alignments, they are very slow because once they find a valid alignment for a taxon, they continue the exhaustive search across all reference sequences for the same taxon . For taxonomic assignment however, the location and number of potential alignments is not relevant; it is sufficient to know that there is at least one high-quality alignment to a given taxon. With this in mind, we designed our algorithm to bypass further alignment attempts once an alignment of sufficient quality is found resulting in highly accurate assignments with significant speedups. Using both simulated and real datasets, we demonstrate that our full alignment approach (1) offers unbiased assignments and is faster than traditional read aligners (2) drastically reduces false positive assignments compared to metagenomic tools that do not provide full alignments (3) increases the number of assignments for taxa that share low similarity with reference sequences, and (4) provides high confidence detection of pathogens while avoiding off-target assignments to near-neighbor species.Here we introduce a new alignment-based taxonomic assignment algorithm called MTSv  and customized to fit the project. Because MTSv was designed to be highly parallelizable, we recommend building multiple indices from smaller chunks of the reference sequences. This helps reduce the memory requirements and allows for faster processing for both index building and assignment. A tool for splitting up the reference files is included with MTSv.MTSv implements a custom metagenomic index (MG-index) based on the FM-index data structure . To builDuring the index build, MTSv concatenates the sequences in the FASTA file while recording the location of sequence boundaries and the TaxID associated with each sequence. MTSv then builds a suffix array , BurrowsWhen classifying metagenomic sequences, MTSv uses the MG-Index to narrow down reference sequence locations that are most likely to contain a good alignment. The binning step begins by extracting overlapping substrings of the same size (seed-size) using a user-provided offset  from each read and its reverse complement . It thenk-mer matches  that can be expected between an n-length read and a reference sequence with at most e mismatches. In the worst case, where all mismatches are evenly spaced across the alignment, the minimum number of matching k-mers is: lth overlapping k-mer is used, the minimum number of matching k-mers is expected to be e mismatches but will still find many alignments with more than e mismatches up to the user provided maximum edit distance cutoff.The filtering step is based on a q-gram algorithm which defines the minimum number of exact ne+1⌋>k) . If onlyAfter the filtering step, the remaining candidate positions are sorted by the number of seed hits in descending order so that the most promising regions are evaluated first. For each candidate region, MTSv extracts the corresponding range from the reference sequence and looks up the TaxID associated with the reference in the MG-index. If the current read has already been successfully aligned to the TaxID associated with the candidate region, no additional alignment is attempted, and the next candidate region is checked. Otherwise, a Single Instruction Multiple Data (SIMD)-accelerated Smith-Waterman alignment  is perfoe.g., “R1:1392=2,1396=3”). The edit distance provided for a TaxID is recorded from the first successful alignment (because no more are attempted afterward) and therefore it is not guaranteed to be the minimum. However, because the alignments start with the most promising candidates first, the edit distance is likely to be near the minimum for each TaxID. Once all output files from the binning step are completed, the files are merged to combine read assignments across multiple MG-Indices.Reads are binned against each MG-Index producing separate output files. Each line of the output file contains the read name and a comma separated list of the TaxIDs it aligned to with the edit distance . TaxIDs were assigned to each sequence using the accession to TaxID mapping file provided with the download  and there was a total of 4,785 unique TaxIDs . To reduce the memory requirements for building and loading the MG-index, we split the reference file into 13 roughly 10 GB chunks and we built individual MG-indices for each chunk. Centrifuge and Kraken2 do not have an option to split up their data structures, so we built them with a single sequence file using default parameters (Kraken2 k-mer length was 35 bp). For comparison purposes, the Centrifuge index was built without reference sequence compression and to avoid variation due to masking methods, low complexity regions were not masked for any of the tools.To compare the taxonomic assignment tools, reference indices were built from bacterial/archaeal genome assemblies from RefSeq  at the CBacillus genus and was dominated by Bacillus anthracis genomes  challenge . The sime.g., when multiple assignments are classified at a higher common taxonomic level than the one being evaluated). Sample purity (or precision) is the sum of all true positive sequences in the sample divided by the total number of assigned sequences.Amber is a tool that helps standardize comparative assessment of metagenomic binning tools by providing performance metrics evaluated against the ground truth assignments for each read. Amber defines true positives (TP) as sequences that are assigned to the correct taxon at a given taxonomic level and false positives (FP) as those assigned to an incorrect taxon. False negatives (FN) include unassigned sequences and those that are not assigned at the current taxonomic level .The average completeness is the sensitivity of each taxonomic bin averaged across all bins. Completeness is calculated as the number of true positive sequences in the bin divided by the true size of the bin. Again, the average completeness weighs all bins equally, regardless of their size.Faecalibacterium prausnitzii (GCA_003312465.1) at rates ranging from 0–15% with indels occurring at 1/10 the rate of the SNPs. From these genome sequences we simulated 1 million paired reads using Mason. These reads were then added to the CAMI2 gastrointestinal tract reads which we confirmed did not contain any sequences from F. prausnitzii. For benchmarking the taxonomic assignment tools, 10 million reads were simulated from the reference sequences in the bacterial RefSeq database.We used Mason v2.0.9  to simulvia qPCR , SAMN31055687 (Y. pestis mixture), SAMN31055688 (F. tularensis mixture), and SAMN31055689 (soil background).Using real sequencing data, we tested the performance of MTSv, Centrifuge, and Kraken2 at detecting pathogen DNA spiked into a complex soil background. We extracted and quantified, via qPCR , DNA frolcR gene , for F. rotocols , and forBacillus genus  resulting in biased alignments. Minimap2 performed better than Bowtie2 because it reported some alternative alignments by default. But in either case, when using a multi-species reference index, the level of search effort did not extend to cover all taxa evenly resulting in biases and missed assignments.The standard search effort and reporting modes of traditional read alignment tools result in misclassification when used for taxonomic assignment. Bowtie2 and Minimap2 missed thousands of alignments particularly for species that were underrepresented in the reference index  or when us genus  and was ound 75% . DespiteStaphylococcus index compared to just a couple of minutes in standard mode (~300 times longer). Minimap2 was faster than Bowtie2 although it did not report as many alignments (855 M vs 986 M), and it failed to find any alignments for over 5,000 reads. Minimap2 took over an hour compared to ~18 min in standard mode (~4 times longer). Minimap2 also required 111 GB of memory when searching for all alignments compared to 7 GB in standard mode. MTSv took on average 12 min to run (~35 times faster than Bowtie2 and 5 times faster than Minimap2) and required 9.5 GB of memory (only 8% of what Minimap2 required) to achieve the same or better results (in terms of correct taxonomic assignments). MTSv does an exhaustive search of all candidate alignments but it is taxon-aware so it will stop attempting alignments against reference sequences for a TaxID that already has a valid match. Consequently, MTSv was able to skip over 1 billion (93%) redundant and computationally expensive alignments while still correctly assigning the Staphylococcus reads.When Bowtie2 and Minimap2 were forced to find and report all alignments, the number of missed assignments was reduced but the runtime and memory requirements increased . Bowtie2MTSv had nearly perfect sample purity and much higher average purity compared to Centrifuge and Kraken2  it was still able to assign a large portion of the reads at the species level  and relaxing the edit distance cutoff to 20%, MTSv performed even better. With these settings, MTSv assigned 84% of In microbial forensics and clinical applications, it is important to have a high level of confidence that a detected organism is truly present in the sample. The presence of many false positive assignments can erode this confidence and may require additional time consuming and expensive validation tests. Ideally, adding a single organism to a complex background should not result in many additional false positive assignments. However, although each of the tools was able to detect the correct species in our pathogen spike-in experiments, Centrifuge and Kraken2 introduced many additional false positive assignments to near neighbor species.B. anthracis, F. tularensis, and Y. pestis spike-ins, respectively and the median hit length for Centrifuge was 24 bp for all three samples. In contrast, all the assignments made by MTSv had 20 or fewer mismatches from the reference sequences. Relaxing the mismatch cutoff to 30 increased the number of assigned reads to 706,182, 680,015, and 604,819, respectively, while still requiring an 80% match with the reference sequence.The total number of reads assigned by each tool was low which is expected when performing taxonomic assignments of a complex soil mixture against a relatively limited collection of reference sequences . Both CeB. anthracis and Y. pestis, fewer calls could be made at the species level, likely due to high genetic similarity between these and near neighbor species. For example, Bacillus anthracis shares >99% average nucleotide identity with its closest relatives in the Bacillus cereus group . The higher number of  assignments for Bacillus species in the background sample masked the increase due to the spike-in prior to filtering them out. MTSv had only three Bacillus species with greater than a 2-fold increase with a maximum of 5-fold.A comparison to the soil background sample  shows thMTSv offers a lot of flexibility when it comes to managing the resources required for taxonomic assignment against large databases. This begins with the index building step. Unlike Centrifuge, Kraken2, and other tools, MTSv allows the indices to be split up into different sized chunks to accommodate resource limitations. For example, because the Centrifuge index must be built all at once and requires a large amount of RAM that scales with the size of the reference genome, we were restricted to using a smaller reference database in this article to allow direct comparison with Centrifuge. MTSv indices can be built using any reference database or even multiple reference databases because it breaks them into smaller chunks that require less memory to build and run. MTSv can easily combine the assignment results across multiple indices and additional indices can be added without needing to rebuild the entire database.In this article, we chose to split the reference sequences among 13 different MG-indices for MTSv. Each index required 251 GB of memory and an average of 1 h 23 min to build. Alternatively for more memory limited situations, the reference sequences could be split into 66 ~2 GB files which would each require 49 GB of memory and about 18 min each to build . Both apBecause MTSv performs full alignments, it cannot compete with the speeds of exact matching tools, but it is reasonably fast compared to traditional alignment tools. For taxonomic assignment, the average time per index was about 30 min for MTSv and each required about 34 GB of memory . If eache.g., Metalign . Here wWe compared MTSv to two traditional alignment tools to illustrate the issues that arise from off-label use for taxonomic assignment against multi-species reference databases. Bowtie2 is one of the most popular rapid read alignment tools and one of the first to use an FM-index data structure to rapidly look up seeds to extend for full alignments . MinimapOne approach to overcome the bias of traditional alignment tools is to increase their search effort to include all valid alignments. Unfortunately, this made the tools very inefficient in terms of speed and, for Minimap2, memory usage, even on a small toy dataset. Attempts to scale traditional read alignment tools to realistic sized metagenomic samples with larger more complex reference databases are sometimes unsuccessful  and incrWe compared MTSv to two taxonomic assignment tools, Centrifuge and Kraken2, to demonstrate the situations where MTSv may provide superior performance. Using data with known read assignments, MTSv had high precision and sensitivity at each taxonomic level. Consequently, MTSv reduced the number of false positive species identified by thousands. The false bins introduced with Centrifuge and Kraken2 were larger and therefore are more likely to be treated as a positive signal. Noisy results interfere with true positive signals and make it difficult to detect low abundance taxa with confidence. MTSv reports only high-quality alignments (with >87% similarity with the reference) and reduces the number of false positives thereby increasing overall confidence in the assignment results.e.g., ancient DNA; e.g., reducing the fixed k-mer size in Kraken2) often sacrifices resolution. By performing a full alignment, MTSv can account for mismatches and assign reads that would be missed by other tools and the alignment sensitivity can be modified easily to capture more divergent sequences. With sensitive parameters, MTSv found 80% of assignments for reads that diverged up to 15% from reference sequences, whereas Kraken2 and Centrifuge found less than 20%. Increasing sensitivity also increases runtime but this can be mitigated by running sensitive mode only on reads that could not be assigned using the default mode.Exact match approaches underperform when sequences diverge from reference genomes. Differences between a read and a reference sequence can occur due to sequencing errors, DNA degradation (The assignments missed due to sequence divergence are likely to influence downstream species abundance estimates. Both Centrifuge and Kraken2 (with the help of Bracken ) can proBacillus anthracis, shares >99% average nucleotide identity with its closest relatives in the Bacillus cereus group that includes ubiquitous soil bacteria like Bacillus cereus and Bacillus thuringiensis .K-mer based approaches like Kraken2 must strike the right balance between sensitivity and precision when picking a k-mer size. With longer k-mers, fewer exact matches are likely due to sequencing errors or biological variation. However, as k-mer size decreases, there are fewer unique k-mers per taxon and the resolution suffers . This isngiensis . This hi subways , 2015b. akenUniq  which we present . We are Our goal when developing MTSv was to provide an alignment algorithm that was specifically designed for analyzing metagenomic data. Although traditional sequence alignment tools are very efficient, they must perform exhaustive searches to avoid missing alignments using multi-species reference databases which become computationally expensive at scale. In metagenomics, it is less important to identify all the possible alignments and more important to know that an alignment of sufficient quality was found among the reference sequences for a given taxon. MTSv performs a taxon-aware exhaustive search but skips references for TaxIDs where a valid alignment has been found. This allows MTSv to provide full alignment-based taxonomic assignments at speeds that are feasible for metagenomic data. MTSv is more flexible than other existing tools when it comes to modifying the pipeline to work within different computational restrictions. For example, with Centrifuge, the FM-index must be built all at once which requires large amounts of memory that scales with the size of the reference database (we did not have the resources to build anything larger than the bacteria-only complete genomes database). The Centrifuge developers do provide prebuilt indices but that limits control of updates and customization. MTSv indices, on the other hand, can be built in smaller units that suit the user’s computing resources and additional reference sequences can be added without needing to rebuild the complete data structure. Although MTSv cannot compete with the assignment speeds of the exact matching approaches, it offers several key advantages while still being faster than other traditional alignment tools.In conclusion, MTSv provides a novel and relatively fast algorithm for taxonomic classification based on full alignment of nucleotide sequences. It offers greater precision, and confidence in assignments than exact-matching tools which is required in some situations. It offers flexibility to fit different computational needs and can easily expand to accommodate rapidly growing reference databases.10.7717/peerj.14292/supp-1Supplemental Information 1Bacillus genus: 116 genomes from B. anthracis (287 sequences), and one genome each from B. cereus, B. subtilis, B. thuringiensis. The second index contained 961 sequences from Staphylococcus aureus  and one genome from S. epidermidis. The highlighted genomes were used to simulate the reads.The first index included sequences from the Click here for additional data file.10.7717/peerj.14292/supp-2Supplemental Information 2Click here for additional data file.