With the arrival of telomere-to-telomere (T2T) assemblies of the human genome comes the computational challenge of efficiently and accurately constructing multiple genome alignments at an unprecedented scale. By identifying nucleotides across genomes which share a common ancestor, multiple genome alignments commonly serve as the bedrock for comparative genomics studies. In this review, we provide an overview of the algorithmic template that most multiple genome alignment methods follow. We also discuss prospective areas of improvement of multiple genome alignment for keeping up with continuously arriving high-quality T2T assembled genomes and for unlocking clinically-relevant insights.The online version contains supplementary material available at (10.1186/s13059-022-02735-6). The advent of new sequencing technologies has made available hundreds of high resolution vertebrate genome assemblies and along with them the possibility of answering a number of biological questions . As a shcolinear, i.e., each row read from left to right, ignoring the empty columns, must be the original input sequence. As a result, MSA can only capture small indels and point mutations. Finding alignments that maximize homology is a computationally difficult task; maximizing the sum of pairwise matches is NP-Hard in the number of sequences [A useful starting point for discussing multiple genome alignment (MGA) is to first define multiple sequence alignment (MSA). MSA is the method of assigning homology relationships to nucleotides across 3 or more sequences , where a set of nucleotides are homologous if they descend from the same common ancestor. These alignments are typically represented by a 2-dimensional array where each row represents an input sequence and each column represents a group of homologous nucleotides. MSA has the strict constraint that the alignment is equences –10. Furtequences . Despiteequences ; by one equences .Following the sequencing of the first full genome in 1995 , early mTwo recent reviews have given a broad background on MGA and described a number methods available for it , 25. OurAfter billions of years of evolution, insertions, deletions, substitutions, and rearrangements are ubiquitous to the genomes comprising the Tree of Life . The accA slight variation of the problem of MGA involves aligning only the “core-genome.” The core-genome of a set of sequences consists of orthologous segments which are shared between all of the sequences , 35. CorDetermining whether two homologs are orthologs or paralogs is a challenging problem which remains actively studied . AdditioAs MGA consists of finding and aligning homologous regions of the input genomes; a natural first step is identifying such regions. Pairwise local sequence alignment methods have been the prevailing technique to find and track homologous segments between genomes. First, all pairs of genomes are searched for high scoring local alignments i.e. highly similar segments. These pairwise local alignments are then merged together to identify regions of high similarity across subsets of the input genomes, also known as multiple local alignments.Pairwise local alignment tools form the foundation for the most widely used global genome alignment methods , 41. HowHandling local sequence similarity within global sequence alignment was previously coined “glocal” alignment  , combiniOver time, it has become apparent that some local alignment algorithms are better suited for application to homology-finding in MGA than others. While using local pairwise matches between all pairs of genomes to construct alignments that span multiple genomes is very sensitive, the number of pairwise alignments that must be run grows quadratically with the number of genomes, so an efficient local pairwise aligner is paramount. Alternatively, very specific local alignment methods, such as finding sequences shared across all input genomes, are extremely efficient but only reasonable for highly similar sequences. As we will see, the problem of MGA is expanding in all aspects, therefore necessitating the development of new heuristics for large-scale local alignment.Decades of striking technological development have driven huge gains in DNA sequencing throughput with parallel advances in the efficiency of algorithms to assemble the outputs into full-length genomes. That has led to an exponential growth in public genome databases and biologists studying all kingdoms of life are increasingly comparing ever larger numbers of genomes, which means the problem of MGA has changed greatly in character over that time. That being said, the development of algorithms and software for MGA has notably not kept pace Fig. .Fig. 2TThe “thousand genome era” is here with respect to vertebrate assemblies, but with nearly 700,000 high quality bacterial genomes available in the European Nucleotide Archive  and in eanchors and then use these anchors to identify larger rearrangement-free regions across input genomes known as locally colinear blocks (LCBs).Many of the challenges that hinder MGA approaches are also found in MSA and even local alignment Table  and thusThe canonical problem statement of MSA, maximizing the number of pairwise-matching sites, quickly becomes an intractable problem when dealing with more than a few sequences  and indeOne heuristic method often employed is progressive alignment , which uFor MGA, the problem of scalability in the number and diversity of genomes is exacerbated by the absence of the simplifying assumptions from MSA that restrict the type of mutation events under consideration. It also makes presenting the output more challenging, as a simple row-by-row representation cannot easily display transpositions, inversions or duplications. Methods have emerged for MSA that deal more aptly with large numbers of more diverse sequences, for example by using a divide-and-conquer strategy , but mosAs noted earlier, the extreme sequence length and presence of genomic rearrangements, gains and losses mean that nearly all MGA programs need to break the problem down first, and can thus be separated into three parts: (1) anchoring, (2) locally colinear block (LCB) identification, and (3) alignment, and in some cases extension, of the LCBs Fig. .Fig. 4Tanchors  and (2). In many cases, these steps are modular. For example, a recent study took the popular MGA program Mugsy  and replMost anchoring recruitment algorithms can be classified in two distinct ways: pairwise versus multiple anchors, and exact versus approximate. The first classification is selfexplanatory, whether the recruitment identifies anchors that link regions in two genomes at a time or anchors that jointly link at least two of the input genomes directly. The trade-off is that while pairwise anchoring is more sensitive, computing all pairwise alignments scales quadratically with the number of genomes. The second classification is whether exact matching or something more approximate like gapped alignment is used for the anchoring. Again, the trade-off between the two approaches is speed versus sensitivity.Importantly though, sensitivity is a double-edged sword in MGA. Any anchoring algorithm that emphasizes multiple-genome anchors that span large subsets of the input set of genomes is going to lose sensitivity as the number of genomes and their diversity increases. On the other hand, the increased sensitivity of pairwise alignment is often accompanied by more spurious alignments.mummer tool from the suite works by first finding all maximal exact matches (MEMs) between a reference genome and a query genome longer than some minimum length l by using a suffix array [The MUMmer suite is a set of tools for pairwise genome alignment , 106–108ix array . MEMs whnucmer, which uses the intuition that matches which are close together and appear in the same order in both genomes often make up a rearrangement-free homology. First, all matches within distance g of each other are clustered together. From each cluster, the maximum length colinear block of matches is extracted. If the total number of nucleotides from all matches within an extracted colinear block does not add up to some parameter c, the block is discarded. For each remaining colinear block, the regions between matches as well as the flanking regions of the block are aligned using the Smith-Waterman alignment algorithm [The MUMmer suite also contains a pairwise aligner, lgorithm .Another widely accepted methodology for computing pairwise genome alignments is the BlastZ method, introduced by PipMaker  in 2002 Parsnp discovers anchors for MGA by using a compressed suffix graph (CSG) to obtain maximal unique matches (MUMs) that are present in all input genomes . Parsnp Following the filtering step, Parsnp constructs a CSG of the reference, a data structure which is able to perform the same MUM finding capabilities of a suffix tree but in a significantly more compressed representation. It should be noted that excluding the filtering step, the selection of the reference will have no effect on the output produced. Each input genome is then streamed through the CSG, resulting in a candidate set of pairwise MUMs. The final set of MUMs used as anchors is the intersection of MUMs obtained from each input genome.w is incurred. Upon termination of an extension, it is linked to nearby matches. This way, when a less frequent match is popped from the queue, it can avoid computing extensions of linked neighbor regions, as they have already been aligned.Using the seed-and-extend strategy, ProcrastAligner produces local multiple alignments in a manner which aims to find anchors which span as many of the input genomes as possible. First, ProcrastAligner indexes the location of a set of palindromic spaced seed  patternsMany alignment algorithms represent the anchors and the sequences between them in graph data structures. Common alignment errors such as short spurious alignments are often signaled by motifs in the graph such as short cycles. By constructing the graph data structure for an alignment, algorithms can identify these motifs and attempt to correct them. Additionally, graph representations provide a straightforward way to both visualize alignments as well as compute them in parallel. In order to describe the locally colinear block construction methods, it is necessary to briefly describe the different graph data-structures that these algorithms operate on. We refer the reader to  for a moblock-edge and for each pair of adjacent segments there is a directed adjacency-edge. A block is a block-edge connected component in the alignment graph. These graphs often serve as the basis for construction of other types of genomic graph data structures.In the basic alignment graph , each noIn 2004, Pevzner et al. presented the concept of an A-Bruijn graph , which mG= be an A-Bruijn graph and G′= be the corresponding Enredo graph. The definition of V′ is straightforward, we split every vertex in the A-Bruijn graph into a head and tail vertex: V′={vh,vt|v∈V}. For each pair of head and tail vertices for a vertex from the A-Bruijn graph, we create a directed block-edge , and the set of adjacency-edges for the Enredo graph is Ea′={|∈E}. By adding these head and tail vertices, Enredo graphs are able to provide a sense of orientation to the local alignments from the A-Bruijn graph.Enredo graphs were introduced as part of the Enredo-Pecan aligner  and are nets, which contain the adjacencies between the connected edges. A fundamental cycle in the cactus graph is therefore a series of consecutive blocks, or local alignments. Additionally, by recursively constructing cactus graphs from the adjacencies within nodes, cactus graphs can easily represent different resolutions of genome alignment.The Cactus aligner  uses a mg and some integer k, we denote de-Bruijn graph [G. The nodes of G are all k-mers in g, and for every pair of vertices u and v, there is a directed edge u→v if the k−1 length suffix of u is the same as the k−1 length prefix of v. Therefore, edges in the graph represent (k+1)-mers. A de-Bruijn graph of multiple sequences is the union of the de-Bruijn graphs for each individual sequence.For a genome jn graph  as G, i.e. regions which share a common ordering of anchors. While the initial set of anchors are sufficient to construct LCBs, they are often noisy and may contain artifacts of micro-rearrangements and assembly error. Consequently, the focus of this step is removing anchors in order to obtain longer, more reliable LCBs. Often times, once the LCBs are obtained methods will repeat the anchoring and LCB construction steps on the regions in-between colinear anchors within an LCB, as these regions are likely to be homologous as well given the high confidence of their flanking regions. In the this section, we provide examples of LCB construction algorithms for different anchor representations.Given a set of anchors represented as a graph, the next step is to identify ProgressiveMauve  uses MUMMugsy  represenL are removed from the graph and any necessary adjacency edges are introduced in their place. Any LCB in the graph containing two anchors a and b separated by more than G from the same genome are disconnected. Such a disconnection is made by using a and b as source and sink nodes in a min-cut instance [In order to extend the LCBs, Mugsy attempts to remove breakpoints that are artifacts of micro-rearrangements or the initial greedy selection of anchors. To address the former issue, LCBs shorter than some threshold instance  on the gThe Enredo algorithm for creating LCBs in the Enredo graph contains three major steps . In the In the second step, Enredo attempts to remove small spurious matches which may result in a scenarios where two potential disjoint LCBs intersect at a spurious anchor. By looking for local multiple alignments whose partitioning into two disjoint alignments results in longer LCBs, the occurrence of such false homologies can be reduced. As a final step, small cycles in the graph, often a result a result of assembly error and duplication, are identified and short anchors from the cycle whose removal results in a larger chain are deleted.α alignment columns in total [α.Given a cactus graph, the goal is once again to remove a subset of local alignments in order to improve the obtained LCBs. Cactus provides a mathematical formulation of this problem in terms of the cactus graph. The Maximum weight Cactus subgraph with large chains (MSLC) problem is an optimization problem which seeks to remove as few local alignments as possible while ensuring that all chains in the cactus graph contain at least in total . The cacWithin a net of the cactus graph, it is likely that there are still rearrangements, albeit smaller, among the adjacencies. In order to resolve these rearrangements, the Cactus aligner seeks to construct another cactus graph from the set of adjacencies within a net. To do this, Cactus uses the Base-level Alignment Refinement algorithm (BAR). BAR uses a modification of the Pecan aligner  to alignk-mers, and therefore carrying paths should be able to represent LCBs in the de-Bruijn graph. These walks start at random edges and are terminated when either the score of the path falls below some threshold or there are no valid steps left. The score of a carrying path in SibeliaZ reflects how well the carrying path aligns with the segments it represents, and a carrying path represents a segments if it overlaps with at least m nucleotides from that segment.SibeliaZ  takes a The significant increase in quality and quantity of assemblies has direct consequences for the characteristics of MGA. Long-read sequencing has enabled the closing of countless gaps as well as the correct assembly of repeated regions, which logically leads to more accurate genome alignments. Where short read assembly had over-collapsed and under-collapsed regions, MGA methods would subsequently make erroneous calls of deletions and duplications, respectively. Historically, many methods addressed such errors by identifying and removing small local alignments that correspond to micro-rearrangements. In addition to resolving repeats, complete genomes will also provide adjacency edges along with their sequence labels to alignment graphs which were previously absent or unlabeled. The era of complete genomes also unleashes the possibility of correctly aligning sequences which often elude current methods of analysis. These alignments, both between humans as well as between humans and model-species, can potentially yield new insights into structural variation and human disease .With genomes now being complete and repeat regions resolved correctly, MGA algorithms can focus more on finding accurate local alignments and untangling the resulting net of anchors and less on adjusting for sequencing error. As MGA is a broad field encompassing repeat-masking, local alignment, graph theory, LCB construction, and MSA, it is no surprise that there are a number of avenues for future improvements. In this section, we focus on five main areas that we believe will be key to unlocking future advances in MGA.In order to improve the performance of MGA methods, it is critical to develop standardized validation methods which accurately model the problem at hand. The lack of a comprehensive benchmark for MGA methods spurred the Alignathon  in 2014.Further development of a broader benchmark and a more comprehensive set of evaluation metrics would be useful as new tools are developed for the volume of data now common in MGA. With the new era of T2T genomes comes much higher quality assemblies. Indeed, it may now be worth regenerating the alignathon simulation datasets from the now completed T2T human genome , althougN50 metric in genome assembly, which is informative about a method’s output but is not properly a higher-is-better measure of accuracy . For MGA, given a biological benchmark of appropriate size such as the over 130 high-quality assemblies already available in the Vertebrate Genomes Project [Finally, a direction for validation and benchmarking that may be appropriate is to depart from the paradigm of evaluating by measuring quantitative accuracy with respect to a ground truth. An analogy is the  Project , some meDue to the quadratic complexity of all-pairs anchoring methods, a priority aim of many MGA tools is to reduce the computational cost of anchoring while still maintaining a high level of sensitivity. The utilization of a progressive alignment in Cactus has proved to do just that, enabling an alignment of over 600 amniotes consisting of over 1 trillion base pairs. While the progressive approach of Progressive Cactus lends itself to a linear time complexity with respect to the number of input genomes, it is unclear how much computational resources this alignment required. In fact, the authors of SibeliaZ showed that Progressive Cactus was unable to complete an alignment on 8 mouse genomes from a recent dataset  in 1 weeWith the advent of extremely powerful GPUs in recent decades has also come GPU adaptations to many alignment algorithms –130. NotWorthy of its own review, the seed-and-extend approach to local alignment has been shown to be heavily influenced by the type of seed used. LAST , a pairwBoth MashMap , 137 andAs seeding is an approach that is used as a tool for more specific applications, it is necessary to benchmark the seed types for different contexts. For example, Edgar  substituIt could be argued that the method of representing the graph genome, modifying it, and identifying LCBs in the graph is the core novelty of most modern MGA approaches. Given the prominence of genomic rearrangements and repeats in most genomes, the problem of finding homologous regions boils down to untangling a set of more clear and conserved anchoring regions. There are two likely ways forward in order to increase MGA alignment quality given a set of anchors. The first is to come up with new representations of sequences and their anchors. This has been an approach which has yielded significant advances in MGA, with each of the graph types discussed in this review contributing to major milestones of MGA.Additionally, LCB construction can be improved by improving algorithms for existing representations of genomes and their anchors. Often, methods for removing anchors to construct better LCBs are posed as mathematical optimization problems. In some cases, such as Mugsy , these pProgressive approaches to MSA and MGA have been largely successful in enabling the alignment of larger sets of sequences. That being said, the progressive method it is not without faults of its own. Sophisticated progressive methods which account for global data are still susceptible to propagating early errors up the tree . AdditioOne intersection between the problem of MGA and MSA is the representation of internal nodes. While using ancestral reconstruction algorithms to replace internal nodes by single sequences is simpler and often more efficient, it has been shown that different methods of MSA can lead to biases in ancestral reconstruction . AnotherFuture methods for MGA will likely follow the path laid out for them by years of work on progressive and divide-and-conquer approaches to MSA. By horizontally splitting the input genomes up, different regions may be treated with different phylogenies. More sophisticated representations which allow for gene flow may also contribute to correctly identifying and aligning shared regions of distal genomes. Furthermore, a divide and conquer approach similar to SATe  may aim There is a stark contrast between the number of MGA approaches developed in the first decade of the 2000s as opposed to the last 10 years. Indeed, there have only been a few methods published since the Alignathon that comprehensively tackle MGA, notably SibeliaZ and Cactus. It is likely that the increasing arrival rate of high quality assemblies will rejuvenate the effort towards advances in MGA, whether it be improvements to existing methods, new representations of anchors, or novel approaches to the problem as a whole. It is entirely possible the computational methods that are in large part responsible for the deluge of T2T assemblies will help to inspire a new generation of MGA approaches to decipher their evolutionary histories. Regardless, future improvements will be required to help advance MGA and allow it to scale up with the vast number of genomes being ushered in by the T2T era.Additional file 1 Review history.


In bioinformatics, alignment is an essential technique for finding similarities between biological sequences. Usually, the alignment is performed with the Smith-Waterman (SW) algorithm, a well-known sequence alignment technique of high-level precision based on dynamic programming. However, given the massive data volume in biological databases and their continuous exponential increase, high-speed data processing is necessary. Therefore, this work proposes a parallel hardware design for the SW algorithm with a systolic array structure to accelerate the forward and backtracking steps. For this purpose, the architecture calculates and stores the paths in the forward stage for pre-organizing the alignment, which reduces the complexity of the backtracking stage. The backtracking starts from the maximum score position in the matrix and generates the optimal SW sequence alignment path. The architecture was validated on Field-Programmable Gate Array (FPGA), and synthesis analyses have shown that the proposed design reaches up to 79.5 Giga Cell Updates per Second (GCPUS). In Bioinformatic, the analysis can be divided into three parts called primary, secondary, and tertiary analysis , 2. The A large amount of raw data has been generated in recent years due to the replacement of Sanger sequencing by Next-Generation Sequencing (NGS), also called High-Throughput Sequencing (HTS) , 4, wherAfter sequencing reads, alignment methods can be performed to map and determine the evolutionary line of the targeted organism, such as its phylogeny. As a result, it is possible to understand the sample’s action mechanics by comparing them with cataloged samples in existing databases , 9. The However, different from the meta-heuristics, deterministic alignment methods offer the optimal alignment for a given input sequence instead of an approximate solution. The main deterministic methods are the Needleman-Wunsch (NW) and Smith-Waterman (SW) algorithms for global and local alignment, respectively , 12. NonThus, it is essential that the processing of algorithms associated with the bioinformatics area cover three critical requirements: high processing speed (high-throughput), ultra-low-latency, and low-power –15. BioiGenomic analysis solutions associated with the High-Performance Computing (HPC), which is first (CPUs) and second (GPUs) categories of computational infrastructure, use systems based only on software that can be implemented using only CPUs and GPUs. According to results presented in –24, thesTherefore, this work presents a parallel FPGA design with a systolic array structure to accelerate both the forward and backtracking stages of the SW algorithm. The main contributions are high-speed data processing implementation and low memory usage that allows for high scalability. According to , the sysTo overcome the low-speed processing bottleneck and maintain the optimal alignment of deterministic algorithms, parallel hardware implementations for the SW algorithm have been proposed in the literature. The main platforms used are Field Programmable Gate Arrays (FPGAs), Central Processing Units (CPUs), and GPUs. FPGAs are widely known for their flexibility for parallelization and low-power consumption. An FPGA is a matrix of logic blocks that allows designing different circuits, such as processors, logic circuits, and even algorithm development . FPGA plThis subsection briefly discusses hardware-based approaches for the SW algorithm available in the literature. Usually, the SW is used for protein and DNA sequence alignment, but their parallel strategies are quite different. Many hardware implementations targeting protein alignment were developed on supercomputers, CPUs, GPUs, and FPGAs, as can be seen in –40. Due GPUs are well-known for their high degree of parallelism and computing intensity. However, high-performance GPUs have a high cost, and, broadly speaking, GPUs have significant computing latency, and low energy efficiency compared to custom hardware, but with more energy efficiency than CPUs, as seen in the results , 23, 40.Unlike the conventional platforms previously mentioned, the SW algorithm has also been implemented on ResCAMs, as can be seen in . ResCAMsIn , a hardwIn , a systoIn , an FPGAIn , an ASICMeantime, in  is preseIn , an FPGAIn , a heterSimilarly, more FPGA approaches using systolic arrays for the NW and SW with backtracking sequencing techniques have been proposed, such as , 57. In In , the SW Therefore, it can be noted from the literature that the key points for a high-performance SW implementation on FPGA are the operating frequency and number of PEs, which in turn are associated with the hardware capacity and design critical path. Thus, we present an FPGA implementation for the SW algorithm using systolic arrays, as in , 44, 57.Our approach performs both the forward and backtracking stages of the algorithm. Unlike the approaches in the literature, we obtain the alignment path distances during the forward stage processing and the maximum score, reducing the complexity of the backtracking stage processing. Memories are used to propagate the distances and maximum score, allowing the backtracking step to follow the path directly. Thus, our architecture achieves good performance , reduced memory usage and, theoretically, high scalability, and prevents memory access overlap latency, even implementing the two stages of the SW algorithm. It is essential to mention that our proposed hardware architecture can perform the alignment of any sequence length. However, the resources available in the target FPGA are a limiting factor for the size of the score matrix, as shown in , 47, 58.q, and dataset sequence s. The query sequence can be expressed byqj is the j-th nucleotide or amino-acid protein and N is the length of the query sequence. The dataset sequence can be expressed bysi is the i-th nucleotide or amino-acid protein, and M is the dataset sequence length. Therefore, the SW algorithm is calculated iteratively for two dimensions, and it has a computational complexity of O(M × N).Smith and Waterman originally proposed the SW algorithm in 1981 to performs local sequence alignment of nucleotides and proteins in the biological field . The seqH, where H is a two-dimensional array that can only take values greater than or equal to 0  is the maximum alignment score of two sub-sequences s and q. The initialization condition isThe forward stage calculates the scoring matrix, ented in , 36, 59,H in the forward stage is the last sequence that will be aligned. To determine the relationship, the previous neighborhood values of the analyzed element are required, i.e., the values on the diagonal, horizontal, and vertical positions, as illustrated in w can be found based on its neighborhood , which is H, H, H, respectively. This windowing step occurs throughout the process of determining all scores in H.The maximum score value of x, y and v, must necessarily be known to determine the value of w ). For this purpose, those values are defined based on the sequences q and s. Thereby, the w score is determined asγ, α, and β represent the linear gap, a match, and a mismatch, respectively. A gap is a penalty that causes an empty element in the sequence (represented by a dash symbol), while the other sequence continues. It can result from the query or database sequence. The x+ (α∨β) = H+ P, y+ γ = F and v+ γ = E. Finally, when fully populated, the H matrix contains the score and path information.As shown in H matrix, i.e., calculating the score of all cells H. Hence, the backtracking begins at the cell with the highest value in the H matrix (maximum score) and trace-back the next position based on the highest neighborhood value, according to s and q sequences, respectively.The backtracking stage starts after determining all the scores in the N PEs is shown in The hardware architecture for the SW algorithm proposed in this work was developed using systolic arrays to input two DNA sequences and increase the processing speed of the local sequence alignment. An overview of the systolic array structure of the proposal for q and s, defined according to Eqs A, G, T, or C . The design proposed supports any sequence set as in any SW algorithm, but for an efficient alignment, it is necessary to adopt a suitable scoring matrix that models each possible symbol’s frequencies that can occur in the sequences.The labeled signals shown in q and s signals. As seen in k-th element of the q and s sequences are shifted to each MSM output to shorten and stabilize the critical path, as well as allowing the computation of scores synchronously, preserving the systolic array structure. Afterward, the MSM computes the score according to Initially, the circuit starts when the MSM modules propagate the H matrix. Also, the last MSM enables the backtracking process. Consequently, the MM switches to the read mode, and the BS reads the data computed by its respective MSM. The alignment starts from the calculations performed in the MSM. Then, from the respective defined PE in the forward stage, the process starts and ends according to the definitions of the SW algorithm.The forward stage is completed after fully computing the scores of the q and s, the MSM also receives an enable signal, en. After computing the score between each k-th element of the two sequences , the MSM outputs to the next PE the following signals: the calculated score, Scj; the maximum score, MaxVal, and its position, AddrRAMi∧j; the PE index; along with the input signals q, s, and en, shifted in time. In addition, the MSM also outputs signals to the MM, which are the calculated path direction, Direction, and the storage address of that path wAddrDir.H matrix and, consequently, the D matrix, the forward stage is finished enabling the Traceback signal, which in turn begins the BS. Firstly, the BS sets signal BTStart to 1, indicating the start of the backtracking process. Therefore, the mRAMi∧j are propagated back until it reaches the BS with maximum score, which is identified by the signal index. From this location match, the btcontrol signal is changed to allow the reading of the memory by MM. Thus, the BS receives the path value from the MM at signal dj when sending the memory address rAddrDirj signal. The dj value allows the BS to calculate the next requested address and propagate it to the next module through the path(j) signal, representing the memory address of the request path in MM. Lastly, the alignment value enters valDir, and the process continues until it reaches the complete alignment. All modules are detailed in the following subsections. All signals present in this Section are shown in Subsequently, after fully populating the D matrix. The D matrix is not widely used in the SW literature. However, it is important to achieve a solution at lower-level programming. Besides, a matrix with two different types of information, such as the H matrix, increases the hardware design complexity. Matrix D needs to store only 4 levels of values which are: 0, 1, 2 and 3. Each element of the matrix D needs 2 bits to be expressed, delivering a more economical storage process compared to H, which can certainly need more than 2 bits to represent each element.Firstly, based on the principles “divide and conquer” for solving computational problems, we propose a matrix used to store only the values of the recursive path, called the q and s, respectively. Also, there can have different sizes, represented by N and M, which define the size of the matrices H and D, respectively. The Matrix Score Module (MSM) calculates the scores and distances in columns of matrices H and D in parallel.As previously mentioned, the alignment process is performed based on the query and dataset input sequences, N PEs. Therefore, for each j-th element in q, there is a j-th PE. It is based on dividing the construction of the H score matrix expressed byD matrix returns the correct sequence alignment, which in turn is equivalent to the directional flags that determined the alignment path. Moreover, for each PEj (which represents a column of the matrix H) there is i-th s(i) that varies from 0 to M − 1, according to the followingThe systolic array structure developed for the matrices is composed of q, i.e., H is formed by N columns, according to The number of MSMs submodules corresponds to the number of elements in en(k) signal, which enables the memory components in the MSM and MM modules to allocate the two sequences q(k) and s(k). The en(k) is a sequence of pulses of value 1 with size equal to the s sequence. Thus, the sequences are transmitted at each sampling time to the forward module. The signals are received in MSM, and the respective q(k) is allocated according to its position, while s(k) is propagated to the MSM based on the internal counter within each module. The counters within each MSM module are activated with each pulse of the en(k) signal.The SW algorithm in this work is initialized by the k-th q element is compared to all elements in s, iteratively, i.e., the s is traversing, going from the first PE to the last one. If the values are equal, a value from the Match constant is propagated; otherwise, the value of Mismatch is propagated. Match corresponds to a reward for similarity, while Mismatch is a penalty for inequality between values. Afterward, the addition block sum the values according toα and β are arbitrary values that correspond to the match value and mismatch values, respectively.Each Scj−1(i − 1), and correspondence value, α ∧ β, are added to define a portion of gj(i). The Scj−1(i − 1) value is equivalent to the H(i − 1)(j − 1) value ). The values of Scj−(i), MaxValue(−1), AddrRAMi∧j(−1) and index(−1) are initialized with 0. At the same time, the Scj−1(i), which is the score value of the previous block, it is received and operated with the value of Gap. In addition, the value of the scoring operation of this block in the previous time, Scj(i−1), is also operated with the Gap. Thus completing the computation of gj(i) that can be seen in the Eqs Subsequently, the score value, M elements. Thereby, the process of choosing the maximum value among the calculated scores is carried out based on γ is an arbitrary value that represents the chosen linear gap value. This expression is equivalent to opr, are propagated to the next submodule for choosing the maximum score and distance path, as shown in opr signals, as seen in The output of the pink blocks, called q and s, respectively, as described in Section 2. The encoding process of directions is performed in the forward step, as illustrated in H score matrix are needed, i.e., the oprj−1(i−1), oprj−1(i) and oprj(i − 1), as seen in Selecting path distances is based on a simple encoding of three levels representing the alignment action to be adopted: 2, 1, and 3. Therefore, the levels 2, 1, and 3 represent a match, a gap in the target sequence H score matrix. The variables x = H, y = H and v = H are known values, while w is a score to be computed. Starting from w = H as the observed cell for determining a generic path and x, y and v as the neighborhood. An integer value is associated with the dj corresponding to the address of w, according to the maximum value determined in the neighborhood, these values are assigned according to the expressionx+ (α∨β)) = oprj−1(i−1), (y−γ) = oprj(i−1) and (v−γ) = oprj−1(i). Besides, (α∨β) = α for a match and (α∨β) = β for a mismatch.Then, for demonstrating the realization of the path coding process is done, the information in Algorithm 1: SW foward stage pseudo-code based in structure this proposalInput: query sequence qInput: dataset sequence sOutput: distance path matrix DOutput: row position of maximum score posMiOutput: column position of maximum score posMjN, length dataset sequence M, match value α, mismatch value β and linear gap value γ;//length query sequence forfor k = 0 to M × N step 1 dodoH and D with zeros; Initialize the DP matrix end//Forward Stage;forfor j = 0 to N − 1 doforfor i = 0 to M − 1 do ifq(j) = s(i) then  α;   sel ← else  β;   sel ← end  H computation;  //x = H + sel; y = H + γ; v = H + γ;  score ← 0, direction ← 0;  ifx > y ⋀ x > vthen  score ← x; direction ← 2;   else  ify > vthen   score ← y; direction ← 1;    else   score ← v; direction ← 3;    end   end    // Stores the score in matrix H and the direction in matrix D calculated;H = score; D = direction;    // checking which is the highest calculated score;ifmaxVal < scorethen  maxVal ← score; posMi ← i + 1; posMj ← j + 1;   end  end endD, posMi, posMj;return cntR, to determines the number of times that the selection of the score and direction is carried out, i.e., the H matrix row that the process is on. This is necessary to determine the AddrRAMi address. At the beginning of MSM processing, index(j − 1) is added to 1, just once for each MSM, becoming index(j) and determining the address of this MSM. For the determination of Maxval, it is seen whether the previous value is less than the current computed score value, then the calculated current score value becomes the Maxval, AddrRAMj = index(j) and respective row process value is AddrRAMi. It is noted the AddrRAMi∧j signal are corresponding to the location of the maximum score value.After the process of selection the score and direction, it has the choice of the maximum score based on a logic of multiplexers and relational blocks. There is a counter, called Direction of the submodule is prepared in set with the value wAddrDir, which comes from the H matrix row calculated at that moment, allowing to write in order in RAM memory according to the respective positions of H matrix .In parallel with the process of determining the maximum score value, there is the process of storing the directions. Thus, the output q(k), s(k), and en(k) are shifted in time, that is, q(k − 1), s(k − 1), and en(k − 1), to match the calculation structure of the H matrix, as seen in the Sc(i), MaxVal, AddrRAMi∧j and index are also propagated to the next MSM to preserve the scores calculating structure. This process repeats until the last element of s is calculated with the last element of q; a counter in is used to determine that moment since the values of the sequences are previously informed to all PEs. The forward stage finishes with the calculation of the last element of the matrix, i.e., H(M − 1)(N − 1). Consequently, the signal Traceback is enabled, indicating the end of the process in all MSM, and the addresses AddrRAMi∧j corresponding to the maximum score value is sent to the next step .Finally, according to the systolic structure, after the MSM processing is over, the signals are parallelly sent to the next MSM. Thereupon, H and D. The input is the signals q and s, which is Eqs N element used, as seen in En to allow the calculation of each element of s in each PE. The first conditional structure is the multiplexer for making choices in the MSM, as seen in x, y and v. The outputs are D matrix stored in MM and the position of the maximum values defined in MSM.Algorithm 1 presents the SW pseudo-code for forward stage and storage process structures. The Algorithm 1, is prepared to perform the calculation of scores and storage of matrices s sequence; also, there is a flag to indicate that the memory is in write mode while computing the H matrix and, subsequently, in fetch mode, in the backtracking process.The MM communicates with both the MSM and the BS, as shown in Direction, obtained in the MSM that is thereafter needed in the BS module. Hence, the RAMs are in write mode throughout the forward stage and reading mode during backtracking. The RAM input ports are the address and data busses and write enable mode. Besides, the memory size of each memory is defined based on the size of the sequence s, which in turn, the amount of RAM memories is equal to the number of PEs in the systolic array.The MM consists of Random Access Memories (RAMs) used to store the path directions, en, is used as write enable for each RAM in the MM. Therefore, en = 1 defines the write mode, while en = 0 the read mode. In addition, the btcontrol signal selects which module controls the RAM address bus. Hence, for btcontrol = 0 the memory addresses are defined by the MSM module through wAddrDir signal, while btcontrol = 1 selects the BS module to define the addresses via rAddrDir signal.The enable signal, en = 1 and btcontrol = 0) the wAddrDir signal defines the address of the RAMs where the Direction value is stored by the MSM. Subsequently, after the H matrix is fully calculated, the Traceback is enabled to indicate the end of the forward stage, and the MM goes into reading mode (en = 0 and btcontrol = 1). Accordingly, the rAddrDir signal defines the address space the BS fetches the data corresponding to the value reported by the trace-back.Thus, in write mode  and AddrRAMj(N − 1), are sent to the respective BS. Also, the values of AddrRAMi(N − 1) and AddrRAMji(N − 1) are assigned to mRAMi(N − 1) and mRAMj(N − 1), respectively, by the BT Enable submodule. It is important to emphasize that if the mRAMj(N − 1) value ) is not already in the BS PE, it will trace-back by checking the Memory Index submodule. This process happens until it reaches the PE corresponding to the maximum score location. Afterward, the Memory Index submodule assigns mRAMi value to rAddrDir to read the memories in the MM, which in turn, returns the d(i) value to the Direction Process submodule, as can be seen in Firstly, after dj(i) is used as the multiplexer selector to perform the dj(i) = 3, BS remains in the same memory position and moves back one BS module, i.e., horizontal displacement. While for dj(i) = 1, only the memory position decreases by 1, and BS is verified by the Direction Process and Continue Processing submodules . Meanwhile, for dj(i) = 2, the memory position also decreases by 1, and it moves to the previous module with the displacement in the memory position. The circuit after the first multiplexer prevents negative addresses in the memory.Secondly, the alignment process starts. The circuits used to build the alignment submodule are shown in rAddrDirj and dj(i) signals to define the path to be followed by the next BS, as seen in dj(i) = 1, the data processing is held in the current BS module, and for dj(i)≠1, the signal BTNext is enabled, indicating the end of data processing in the current PE to start in the next one.Given that the path to align the first element is found, the Alignment Block submodule receives the mRAMi and mRAMj signs finish their function. Thus, from the determination of the BS with the maximum score, the path(j) sign is used as a guide for locating the alignment of each module. Then, the data in MM is requested and the dj value is returned for verification and establishment of alignment. The verification and establishment of the alignment path is done by the Memory Index, Direction Process, and Continue Processing submodules. Decisions related to dj value are made in Alignment Block submodule, as illustrated in After finding the module for the maximum score, the valDir output of each submodule is used to construct the alignment path, along with the maximum score position values. The trace-back continues until it reaches PE0 or finds a path direction with a value of 0.Finally, the Finish Processing and Continue Processing submodules finish the data processing in the module. Thereby, the D, starting from the positions of the maximum score, as seen in this Section. Inputs for this step are provided by Algorithm 1. The loop for this step represents all backtracking stage modules from N − 1 to 0. The conditional structure of Algorithm 2 is the representation of submodule Alignment Block, valDir, in RAM memory.Algorithm 2 presents the SW pseudo-code for backtracking stage this proposal. The backtracking stage, algorithm 2, is ready to perform the alignment in a list using the path informed in Algorithm 2: SW backracking stage pseudo-code based in structure this proposalInput: query sequence qInput: dataset sequence sInput: distance path matrix DInput: row position of maximum score posMiInput: column position of maximum score posMjOutput: alignment sequences list AOutput: alignment path sequences list path//Backtracking Stage;auxi = posMi; auxj = posMj,; aux ← D;A ← ;path ← concat;whileaux > 0 doA ← concat; s(auxi − 1)]); ifaux = 2 then auxi = auxi − 1, auxj = auxj − 1;  else ifaux = 3 then  auxj = auxj − 1;   else  ifaux = 1 then   auxi = auxi − 1;    else       break;end   end  end aux ← D; path ← concat; endA, path;return This section presents the synthesis results for the architecture described in the previous section and analyses it regarding the following key points: critical path, operation frequency, number of PEs, and performance. The performance measures the time to calculate an element of the scoring matrix.The development of the algorithm was carried out using the development platform provided by the FPGA manufacturer, in this case, Xilinx . This plThe performance for hardware implementations of the SW algorithm is usually measured in Giga Cell Update Per Second (GCPUS), which in turn is defined asq and s were randomly generated and varying the match, mismatch, and linear gap values. Initially, the analysis was carried out for 8 PEs and by varying the size of the sequences q and s from 8 to 32. The number of PEs also varied according to the size of q. Our architecture works with sequences of varying lengths, only requiring that the length of s is greater than or equal to the q length.To validate the architecture proposed in this work, the sequences H and D was verified by monitoring the MSM outputs, such as Sc and Direction, as described in Section 2. Secondly, it was verified if the D matrix elements were stored in the correct memory positions in the MM. Lastly, the operation of the BS modules was also verified by monitoring the path(j + 1) bus and the Memory Index submodule.Firstly, the correctness of the matrices path(j + 1) value, that is, according to Following, the Alignment Block and Direction Process are observed to check if the memory accesses are in accordance with the total − PEs × α. Meanwhile, the sequence counters for s is logs − size.The data bit-width was defined by the maximum size of the input sequences, limited by FPGA memory capacity. Hence, the input sequence bit-width was set to 3 while constants were defined according to its value. Besides, the bit-width for the MSM buses that perform mathematical operations was defined as logs sequence, while the x axis refers to the q sequence. To increase the resolution of the image, only the parts of the sequences that are aligned are used, where the position at which the alignment starts and the maximum score value are shown in the title of the illustration. The value of Row refers to the position in the s, whereas Column is related to the element of the q. The amount of sequence alignment performed is represented by Number of Alignments.match = 5, mismatch = −5, gap = 1, and 128 PEs. Hence, the size of the sequence q is also 128. Meanwhile, the size of the sequence s was set to 8, 192, resulting in a total of 1, 048, 576 calculated cells. Sequence q is loaded into memory at each iteration, where it can vary between 4 different 128 nucleotide sequences in the demonstration. The demo is available at [The architecture parameters for the demo were set to lable at , and thelable at .manager circuit to control the data flow. Therefore, the q and s sequences were transferred to the FPGA (via Ethernet protocol) and stored in the buffer, and, subsequently, fed into the SW architecture to perform the alignment by the manager circuit.The SW architecture was developed using the Xilinx System Generator on Matlab, and the traffic of data between the host PC and the FPGA was accomplished via Ethernet protocol. Moreover, we added on the FPGA a buffer to store data and developed a Analysis of the synthesis results for the SW hardware implementation were carried out for two FPGAs: Virtex-6 XC6VLX240T and Virtex-7 XC7VX485T. q, the number of parallel computations will also increase, thus, improving the performance. Therefore, according to the resources available in the target hardware, our architecture can operate with a number significantly bigger than 512 PEs.The critical path of the design was ≈8.34ns and ≈6.44ns for the Virtex-6 and Virtex-7, respectively. Therefore, the maximum clock frequency was 120MHz for the Virtex-6 and 155MHz for the Virtex-7. Regarding the FPGA area occupation, increasing the number of PEs also increases the hardware resources used. For 512 PEs in the Virtex-6, a total of 68% of the Slice Look-Up Tables (LUTs) were used in contrast to only 7% for 64 PEs. Concerning the frequency, a slight decrease is observed as the number of PE increases due to an increase in the critical path. Concerning the Virtex-7, there are unused FPGA resources as less than 35% of Slices LUTs were used. Therefore, it can be used to increase the number of PEs and, thus, the performance. Note that, increasing the number of PEs and, consequently, the size of the sequence Comparisons with state-of-the-art works were also performed. The performance of systolic array-based implementations increases with the number of PEs. Hence, the comparisons were carried out for the maximum number of PE in each proposal. We compared our design to the most relevant and recent works similar to our proposal, i.e., the SW algorithm has to be implemented using a systolic-array structure, deploy the backtracking step, and provide the parameters concerning processing time and area occupation. Given that, we discuss a direct comparison to the architecture presented in , 41, 47.The works presented in Furthermore, a comparison with  was alsoAs shown in Regarding the operation frequency, our proposal can reach up to 155 MHz. So, it is observed that the proposals with the best performances have a similar structure, even with different approaches to the solution. Our proposal and  achievinD and H, resulting in linear computational complexity. On the other hand, the architecture proposed by [Therefore, our work uses fewer hardware resources to perform the alignment process due to the chosen backtracking approach. As the backtracking stage results in high computational complexity, we simplified the process using the path mapping through the maximum value in posed by  uses conThe SW proposed by  is—to ouThe hardware implementation of the alignment process through our approach, developed based on a chain of directions and the maximum score address, is a key contribution for the low use of memories. Thus, as we did not carry out tests with real biological datasets, theoretically speaking, it is possible to achieve high hardware scalability. Besides, the sequence of any size can be aligned with our approach limited by the hardware resources available. In addition, the proposed method can compress the data, using only 3 bits in a fixed-point implementation.This paper presented a parallel FPGA platform design to accelerate both the forward and backtracking stages of the SW algorithm. The main contributions were the high-speed data processing implementation and low memory usage that theoretically allows high scalability. The hardware resources available on the FPGA are a limiting factor to the size of the score matrix but not to the size of sequences to be aligned. Therefore, satisfying the high-throughput, ultra-low-latency and low-power requirements and to alleviate the raw data processing problem in bioinformatics. From the strategy of storing alignment path distances and maximum score position during forward stage processing, it was possible to reduce the complexity of backtracking stage processing which allowed to follow the path directly. The proposal architecture achieved a satisfactory critical path, reduced memory usage and, theoretically, a high scalability for two-step SW algorithm. Synthesis results showed that the proposed method could support up to 1, 024 PEs in only one FPGA, using the Xilinx Virtex-7 XC7VX485T. The main advantage is the low hardware resource usage and high performance of 79.5 GCUPS, with an operating frequency of up to 155MHz, without using external resources.


Phylogenetic placement is the task of placing a query sequence of unknown taxonomic origin into a given phylogenetic tree of a set of reference sequences. A major field of application of such methods is, for example, the taxonomic identification of reads in metabarcoding or metagenomic studies. Several approaches to phylogenetic placement have been proposed in recent years. The most accurate of them requires a multiple sequence alignment of the references as input. However, calculating multiple alignments is not only time-consuming but also limits the applicability of these approaches.Alignment-free phylogenetic placement algorithm based on Spaced-word Matches (App-SpaM), an efficient algorithm for the phylogenetic placement of short sequencing reads on a tree of a set of reference sequences. App-SpaM produces results of high quality that are on a par with the best available approaches to phylogenetic placement, while our software is two orders of magnitude faster than these existing methods. Our approach neither requires a multiple alignment of the reference sequences nor alignments of the queries to the references. This enables App-SpaM to perform phylogenetic placement on a broad variety of datasets.Herein, we propose App-SpaM is freely available on Github at https://github.com/matthiasblanke/App-SpaM together with detailed instructions for installation and settings. App-SpaM is furthermore available as a Conda-package on the Bioconda channel. The source code of matthias.blanke@biologie.uni-goettingen.deBioinformatics Advances online. If this probability is above a chosen threshold, the k-mer w is called a phylo-k-mer. RAPPAS creates a database with all phylo-k-mers and their associated probabilities to occur at a branch in the reference tree. Once this database is constructed, new query reads can be rapidly placed based on their k-mers that are also present in the database.In comparison to APPLES . Similar to APPLES, App-SpaM also performs distance-based phylogenetic placement; it neither needs a multiple alignment of the reference sequences nor alignments of the query reads to the reference sequences. Thus, it skips the time intensive alignment procedures that are needed for most other software tools. Additionally, in contrast to existing placement methods, App-SpaM can be applied to datasets where no multiple sequence alignment of the references can be created.In this paper, we present a new approach to phylogenetic placement that we call App-SpaM is based on an approach that was originally implemented in the program Filtered spaced-word Matches  First, we find so-called spaced-word matches between every query and every reference sequence; (ii) then, we estimate the phylogenetic distance between every query and reference sequence based on a ‘filtered’ subset of the identified spaced-word matches; (iii) at last, we determine a placement position for each query sequence in the reference tree T.As input, our approach takes a set of characters called the alphabet, a sequence over Σ is an ordered list of elements of Σ. For a sequence S, its length is denoted by i-th element of S is denoted by n over Σ is denoted by patterns—over the nucleotide alphabet extended nucleotide alphabet wildcard character.For a set Σ of j in the pattern is called a match position if don’t care position if P is called the weight of P. A spaced word W with respect to P is defined as a sequence of length i is a match position of P. We say that a spaced word W occurs in a sequence S over i—or that  is a spaced-word occurrence in S—if j of P.A spaced word is defined with respect to a given binary pattern S1 and S2 and positions i1 and i2 in S1 and S2, respectively, we say that there is a spaced-word match (SpaM) between S1 and S2 at j of P. In other words, there is a SpaM at , if there is a spaced word W, such that S1 and S2. A spaced-word match with respect to P can, thus, be seen as a local gap-free alignment of length P and possible mismatches at the don’t care positions, see score of a spaced-word match as the sum of all substitution scores of nucleotide pairs aligned to each other at the don’t care positions of P. Spaced-word matches—called spaced seeds in this context—have been originally introduced in sequence-database searching  between the query reads and the reference sequences. More precisely, for each pattern P in the queries and references are stored in two lists L1 (references) and L2 (queries) that are both sorted in lexicographic order. For the sorting procedure, only the nucleotides at the match positions are considered, the don’tcare positions are ignored. For every spaced-word occurrence , we also store all nucleotides of the don’t care positions. Thus, for each possible spaced word W, all occurrences of W in the read and reference sequences appear as consecutive blocks in the lists L1 and L2, respectively.In a first step, we determine a fixed set L1 and L2 have been established, they can be traversed simultaneously, such that for each spaced word W, the blocks with the occurrences of W in L1 and L2 are processed at the same time. Each pair of occurrences L1 and L2 corresponds to a spaced-word match at  between a reference sequence and a query read. For each such spaced-word match, we calculate its score and we discard all SpaMs with a score smaller or equal to a pre-set threshold t. The SpaMs with a score below t are considered to be background or random spaced-word matches. All remaining, high-scoring, SpaMs are referred to as filtered spaced-word matches. These filtered spaced-word matches are regarded as putative homologous matches. As in FSWM, we use the HOXD70 nucleotide substitution matrix  of SpaMs between Q and S with score larger than t. Additionally, we calculate the proportion of mismatches at the don’t care positions of all filtered SpaMs between Q and S, and we estimate the phylogenetic distance d between Q and S using the well-known Jukes–Cantor formula  denote the length (‘weight’) of e. For each query Q, we first select an edge Qe in T and insert a new internal node into this edge, thereby splitting Qe into two new edges e1 and e2 with Q, together with a new edge In the following, we propose five heuristics to find a suitable position in our reference tree Qe for a query sequence Q and to assign lengths to the newly generated edges, we are using either the phylogenetic distances d or the number of spaced-word matches s with scores larger than t between Q and all reference sequences S. A detailed description how we determine the edge lengths for e1, e2, and the newly inserted edge T.To find a suitable edge S that minimizes the distance d over all reference sequences, and we define Qe to be the edge in T that is adjacent to the leaf labelled with S. If multiple references have the same smallest distance to Q, one of them is chosen randomly.In this approach, we first select the reference sequence S that minimizes the distance to Q, we select the reference S that maximizes the number s of spaced-word matches with score S and Q.This works like the previous approach, but instead of selecting the reference sequence two reference sequences S1 and S2 with the lowest distances Q. Let v be the lowest common ancestor in T of the two leaves that are labelled with S1 and S2, respectively. The edge Qe is then defined as the edge in T that connects v with its parental node.Here, we identify the S1 and S2 minimizing the distance with Q, we select the two references S1 and S2 with the maximal number Q with scores larger than t.This is similar to the previous approach, but instead of using reference sequences SpaM-COUNT and LCA-COUNT approaches: For S1 and S2 as in LCA-COUNT and a given X, we evaluate whetherSpaM-COUNT, otherwise, the query is placed according to LCA-COUNT. As a result, when Q is placed at the branch directly above S1. App-SpaM uses SpaM-X on default with X = 4 (referred to as SpaM-4 in the following).The first four approaches will yield either only placement locations at branches directly above the leaves, or only placement locations at inner branches, respectively. Thus, in this approach, we combine the SpaM+APPLES: As a sixth approach, in addition to five versions of App-SpaM, we used the distance values d as input for the program APPLES   implemented in PEWO determines the placement accuracy of an evaluated method as follows: First, a randomly chosen subtree is removed (‘pruned’) from the reference tree. All sequences at the leaves of the chosen subtree are removed from the reference MSA as well. Next, artificial reads are generated by splitting the removed sequences into segments of a given length; these reads are used as query sequences. An algorithm under evaluation is then used to place the queries onto the pruned reference tree.We primarily used the recently developed PEWO uses the so-called node distance (ND): For each query sequence Q, the distance between the proposed placement position of Q and the edge where the subtree was pruned is measured by counting the number of nodes on the corresponding path. The average of these distances over all query sequences is then a measure of accuracy for one pruning event. This procedure is repeated with randomly pruned subtrees and PEWO uses the average accuracy over all pruning events as the overall accuracy of the evaluated method. PEWO also provides a resources evaluation (RES) workflow to measure the runtime and memory usage of programs. This includes the alignment of queries against the MSA of references for those methods that are based on sequence alignments and the construction of the phylo-k-mer database in RAPPAS.To measure the placement accuracy of a method, PEWO PAC workflow to evaluate the placement accuracy of the five versions of App-SpaM that we outlined above. In addition, we evaluated the combination of SpaM and APPLES. Both programs were run with default parameter values. We also ran our program with non-default parameter values for the pattern weight w and the number of patterns by varying We used the App-SpaM to all programs that are currently supported by PEWO with the PAC workflow. At present, the programs pplacer, EPA, EPA-ng, RAPPAS and APPLES are included in the PEWO package. The datasets used for this evaluation vary with respect to the number and length of the reference sequences, with respect to the degree of similarity between the reference sequences, and with respect to the sequence locus. A short overview of the reference datasets is given in APPLES, we used the updated version 2.0.1 that automatically re-estimates the branch lengths of the reference tree appropriately to match the evolutionary model of query-reference distances. For each reference dataset, we performed 100 pruning runs with randomly sampled subtrees and recorded the average ND. As default, we used a length of 150 base pairs (bp) for the simulated query reads. We also performed additional test runs with longer read lengths for three datasets .We compared the accuracy of PEWO’s RES workflow on two datasets of differing sizes (CPU-652 and CPU-512). We recorded the average runtime and memory usage over five repeats for each program. For CPU-652, we placed 100 000 query sequences, and for CPU-512, we placed 10 000 query sequences on the reference tree. In addition to these evaluations, we performed an additional runtime test with App-SpaM on the tara-3748 dataset, similarly to the runtime study conducted in EPA-ng: We recorded the runtime of App-SpaM to place up to 37 480 000 query reads with parallel execution on 30 cores. All of these tests were carried out on Intel(R) Xeon(R) E7-4850 CPUs with 2 GHz.To measure the runtime and memory requirements of the evaluated methods, we used hiv-104 dataset that consists of complete HIV genomes, and wol-43, a dataset of complete microbial genomes of 43 different Wolbachia strains. Because PEWO does not support unassembled reference sequences, we used a simple leave-one-out procedure to assess the accuracy of App-SpaM in this scenario. For this, we simulated reads with a length of 150 bp and values for the sequencing coverage of 4, 2, 1, 0.5, 0.25, 0.125, 0.0625 and 0.03125 for all sequences. These ‘bags of reads’ constitute the reference sequences for the evaluation. Then, for a given coverage, a leaf is pruned from the reference tree, its corresponding reads are used as the query sequences, and the average ND across all reads is measured. This is repeated one by one for all references, and subsequently for all coverages. To assess the accuracy of App-SpaM in these tests, we used three control methods: First, we compared the placement accuracy to the accuracy achieved on contiguous (‘assembled’) reference sequences; this shows the decline in accuracy depending on the sequencing coverage. Second, we compared the achieved results to the accuracy when placing all query reads at the root of the tree; and third, when placing all query reads at the midpoint of the tree.Lastly, we also performed test runs using simulated sets of unassembled sequencing reads as references, instead of contiguous reference sequences. In these experiments, we used the App-SpaM described above, together with the combination of SpaM+APPLES, on the bac-150 dataset and a query read length of 150 bp. On average, SpaM-4 performed best among the five methods while MIN-DIST was the least accurate. These results are consistent across different values of w and across different datasets  and 3–5 times faster than the placement step of RAPPAS. On CPU-652, App-SpaM has the lowest memory footprint and on CPU-512 the second lowest behind APPLES  can introduce a certain bias in our approach: If a local homology contains an indel, it is possible that an SpaM correctly aligns homologous sequence positions to each other on one side of the indel, but then aligns non-homologous positions on the other side. Since the score of such an SpaM may still be above the filtering threshold, such ‘partial homologies’ can pass our filtering procedure and would then be used to estimate the phylogenetic distance between the two sequences. As a result, the average number of substitutions per position—i.e. the Jukes–Cantor distance of the sequences—would be over-estimated. We investigated the influence of indels on the accuracy of the estimated distances based on simulated sequence data , where App-SpaM performed not as accurate as the competing programs that we evaluated. In contrast, App-SpaM achieved the best placement accuracy out of all programs that we tested on two other datasets: a large set of 16S sequences (tara-3748) and a set of fungal 5.8S sequences (epa-628). The runtime of App-SpaM on these datasets was considerably faster than for the alternative methods.Our extensive evaluation shows that App-SpaM in this paper. Two of them performed best across all datasets, namely LCA-Count and SpaM-4. There are situations, however, where SpaM-4 is superior to LCA-Count. If a leaf labelled with some reference sequence iS is far away from the other leaves in the reference tree, but the distance between a query read q and iS is small, then LCA-based methods would place q at the proximal branch of the node connecting iS and its neighbour in the tree. The query would, thus, be placed far away from iS. In contrast, our heuristic SpaM-4 would be able to correctly place q near the leaf labelled with iS. Therefore, SpaM-4 is the default version of our program; we advise the user to always use App-SpaM with the default settings.We also presented five different placement heuristics for APPLES, App-SpaM does not rely on a multiple sequence alignment of the reference sequences. This gives App-SpaM distinct advantages over existing, alignment-based programs: Calculating a reference MSA in a pre-processing step is time-consuming, and errors in the reference alignment can be a source of errors in the placement results. Moreover, if the homologies between the reference sequences are not co-linear, due to evolutionary events such as genome rearrangements, it is not possible to find a meaningful multiple alignment for them in the first place. This advantage of App-SpaM also implies that it does not require assembled reference sequences as input, but can be applied to taxa for which only unassembled reads are available. Genome assembly is still a non-trivial task (In comparison to existing programs, besides ial task , and wroial task . For thiSpaM approach can be caused by low-complexity regions in the input sequences, e.g. by long runs of ‘ATATATATAT’. If such runs would be present in both compared sequences, a large number of SpaMs with 100% matches at the don’t care positions would be found. For the viral and bacterial sequences used in this paper, low-complexity regions are not an issue. However, we would recommend to filter out low-complexity regions prior to running our software on eukaryotic sequences by using a software such as RepeatMasker (In this context, another certain difficulty for our atMasker .App-SpaM, as well as APPLES, compared to other programs, is their inability to describe the uncertainty of inferred placement locations. While App-SpaM only reports a single placement location for every query sequence, other programs can specify multiple placement locations for a single query read and assign weights to these locations according to their likelihood, so-called likelihood-weight ratios.Another drawback of PEWO for our evaluations to ensure clearly defined and reproducible evaluation workflows. A certain draw-back of PEWO is the fact that it simulates query reads by simply splitting the pruned reference sequences into segments of the desired read length; sequencing errors are not modelled. To evaluate our method under more realistic conditions, we performed additional test runs with simulated query reads obtained with the simulation software ART (PEWO framework. Unlike PEWO, ART models sequencing errors. A conspicuous result of these test runs is that our method outperformed all other approaches substantially when we used reads simulated by the ART software (see We used the benchmarking system ware see .App-SpaM. The placement heuristics that we implemented so far depend either on the number of spaced-word matches or on the estimated phylogenetic distances between a query read and the references. However, both sources of information may complement one another, so improved placement results might be obtained by combining both measures of distance and similarity, respectively. Such a placement method could also be used to express placement uncertainty similar to the likelihood-weight ratios used by ML-based methods. Hence, we are continuing to work on additional placement heuristics that use all available information to fully utilize the spaced-word matches approach for phylogenetic placement. Moreover, while the runtime of App-SpaM is already fast, the current implementation is not yet optimized for speed and memory efficiency and multiple strategies to further decrease the runtime are possible: All spaced words are held in main memory in a non-optimized data format instead of referencing the corresponding positions in the input sequences. This significantly increases the runtime and memory usage and can be improved by more efficient data handling. Given the test results shown in this study, App-SpaM should already be a useful tool for performing phylogenetic placement on large datasets and efforts should be made to further improve its accuracy and efficiency.There are several possibilities to further improve vbab027_Supplementary_DataClick here for additional data file.


How reliable are alignments of amino acid sequences of proteins, and what inferences about protein relationships can be drawn? Using techniques not previously applied to these questions, by weighting By analyzing the sequences and structures of 1 million protein domain pairs, we report the variation of the expected distance between sequence-based and structure-based alignments, as a function of (Markov time of) sequence divergence. Our results clearly demarcate the ‘daylight’, ‘twilight’ and ‘midnight’ zones for interpreting residue–residue correspondences from sequence information alone.Bioinformatics online. Evolution introduces mutations, insertions and deletions at the level of individual protein domains. These changes flow down, in varying degrees, causing perturbations in observed three-dimensional structures. The effect that these perturbations have on the biological functions of proteins influences the fitness and evolutionary selection of these changes. This mechanism of heritable changes is a driver of the Darwinian descent of species through modification .alignment helps scientists understand the cumulative evolutionary events on extant proteins and gain insights into the patterns of conservation and divergences of sequences and their effect on structure and function. Therefore, it is common to see protein alignments as a first step supporting subsequent downstream analyses in biology  atomic coordinates of their corresponding three-dimensional structures.An alignment is simply a hypothesis of a residue–residue correspondence between protein sequences. The corresponding amino acid residues can imply they likely descended from the same locus in the genome of their common ancestor  and corresponding gap parameters . Such parameter-tinkering has significant effects on the reliability of sequence alignments, as clearly demonstrated by comparative surveys , when the reliability of sequence alignments gives diminishing returns compared to structures (the twilight zone) and hence handled with caution for subsequent downstream studies, and finally when the evidence of residue–residue relationships become untenable and sequence-based alignments are best ignored (the midnight zone).f of amino acids strictly conserved within the core, that generalizes to domains of different structural types: not consider the relationship between divergence of sequence and structure at a residue level. Nevertheless, A more thorough assessment of the consistency of residue–residue relationships derived from sequences and structures requires modeling the deteriorating quality of sequence alignments as proteins diverge. Further, the utility of any systematic assessment is enhanced if it stands on a sound mathematical foundation and is based on effective and consistent statistical models, whose parameters are not prespecified or frozen, but rather automatically inferred as a part of the statistical framework of comparison.expectation of the fraction of amino acids that can change under that model of relationship. Further, it provides a powerful method to decipher distant sequence relationships between proteins when no singular alignment relationship is significant, by computing the marginal probability over all possible alignments between the sequence pair, to adjudicate their relationship (see Section 2).This issue motivates our current work, which attempts to study more systematically how sequences and structures diverge in evolution using, to the best of our knowledge, techniques not previously explored. We build on top of our recently developed Bayesian framework for protein sequence comparison . This frbiased sequence alignment against its structural counterpart. Instead, we compute the exact mathematical expectation of the distance between sequence and structure alignments, by integrating over all possible sequence alignments for any given protein pair, weighted by their individual marginal probabilities.Building on this framework, the current work introduces a natural and intuitive distance measure between any two given alignments for any protein pair. Here, we avoid using this measure of distance to compare a single Note that despite the number of total possible alignments between any pair of sequences growing factorially large as a function of their sequence lengths, we provide an exact quadratic-time algorithm (in sequence length) to calculate this mathematical expectation of the inter-alignment distance between sequence and structure alignments, using probabilities estimated by our Bayesian and MML framework.Further, to validate the above exact computation, we also compute an empirical expectation of the distance to compare against. This empirical expectation is achieved by randomly sampling not a single, but thousands of alignments over the marginal probability matrices that are generated as a part of the MML framework.1 million pairs of protein domains and analyze their lossless compression statistics, the inferred (Markov) time parameter of sequence divergence, the expected fraction of amino acid changes and the expected and empirical inter-alignment distance with respect to their reference structure alignments. Our analysis reveals the distinct ranges of amino acid divergences where pairwise sequence alignments could be trusted, the ranges where they should be handled with caution, and those where they should be ignored.Finally, we evaluate the sequence and structure alignments of We briefly summarize the main features of the protein sequence comparison framework , the lossless encoding length of which is denoted by The MML criterion  is a BayI is an information-theoretic restatement of the product axiom of probability, E with a probability bits.From the Bayesian point of view, the two-part message length  content , where tH and D, since the difference in their message lengths of lossless encoding, The MML criterion can therefore be used to compare any two competing hypotheses, H, a best hypothesis minimum value for Therefore, over the space of all possible hypotheses This defines the MML criterion for hypothesis selection.null hypothesis test. This test relies on the null model message, i.e. the efficient lossless encoding of the data D without the aid of any hypothesis, the length of which is denoted by H does not beat the null model message length, i.e. if H is rejected, else it is accepted.Further, implicit within the MML framework is a natural S and T with the aid of the alignment relationship Recall, from the general introduction to MML above, that the two-part message length of any alignment is the negative logarithm of the joint probability of the alignment and the sequencesmarginal probability of the two sequences being related. Importantly, this estimation of marginal probability provides a reliable way to identify a distant relationship between sequences even when any single alignment not statistically significant. That is, in the MML framework, this implies Further, the framework described in The marginal probability is estimated by summing the joint probabilities Note that the number of all possible alignments Despite this growth, the estimation of marginal probability can be implemented over a two-dimensional dynamic programming algorithm that runs in time proportional to biased Markov time parameter single optimal alignment. When estimating the marginal probability of the two sequences being related, the program also reports the amount of compression gained (>0) or lost . The most basic measure of distance between two alignments can be obtained by counting the fraction of columns in the two alignments that are concordant with each other, i.e. the fraction of identically aligned columns (including insertions and deletions) between them. However, this distance measure is not very informative as it only provides a binary yes  or no  measure of distance, which can render very closely related alignments as fully distant from each other.Considering a very large data set of homologous protein domain pairs with known sequences and structures, our aim is to quantify the qualitative differences between sequence alignments and their structural counterparts. At a fundamental level, this requires some reliable measure of mathematical expectation of the distance between the sequence and structure alignments over all possible alignment relationships between the two amino acid sequences, weighted by their posterior probabilities.To overcome this issue, we define below a better and more intuitive way to measure the distance between any two alignments for any protein pair. We then use this measure to compute the i, j), any alignment path traverses either horizontally to the right  cuts across two (not one) skew diagonals. Hence, both widths, one for the skew-diagonal index i + j, need accounting for in the dynamic program.Finally, the ndex see  in the ci, j) can be evaluated in O(1) time, it is straightforward to see that the time and space complexity of the computation of this mathematical expectation is The above dynamic program can be computed bottom-up starting from the trivial subproblem where the expected inter-alignment distance at the source is 0: Based on the method of computation presented here, seqMMLigner (v2.5) has been extended for users to be able to compute the expected distance between sequence and any given reference alignment.1 million distinct pairs of protein domains across four major structural classes of SCOPe (v2.07): all-α, all-β, α/β and α + β. To restrict the analysis to pairs of homologous proteins, the domains in each sampled pair were drawn from either the same ‘superfamily’ (yielding 541 162 domain pairs) or same ‘family’ (yielding 458 838 domain pairs) as defined by their SCOP classification , (ii) TM-align  (structure) MMLigner  , VTML  against the exact mathematical expectation computed by Next, we verify the computation of the mathematical expectation of distance, using an empirical (sampling) approach that can be used to generate highly probable alignments using the marginal probability matrices generated by seqMMligner. For each of the million domain pairs, 1000 alignments were sampled from the marginal probability matrix of each pair. The distance between each sampled alignment for a pair was compared to its corresponding reference structure alignment using daylight, twilight and midnight zones, we sampled >200 000 distinct domain pairs related at varying levels of SCOPe (v2.07). This yields five distinct groups of domains pairs with their pairwise SCOP relationship similar up to the level of (i) same ‘family’ , (ii) same-‘superfamily’ , (iii) same ‘fold’ , (iv) same ‘class’  and (v) ‘decoy’ —see Supplementary Table Further, to better understand the correlations between the SCOP classification of domains and the identified ranges for Figure S6 in  shows thNext, the twilight zone accounts for Furthermore, the midnight zone accounts for These numbers reveal a broad agreement with the classification criterion used in SCOP, where related proteins with similar sequences fall under the same family and proteins with weak sequence similarity signal, but detectable functional and structural conservation is classified under same superfamily. Similar superfamilies without compelling evidence of common evolutionary origin are grouped into folds, which are further arranged into classes mainly encompassing the recurrent secondary structural features and their architecture. As a future exercise, it would be useful to examine the domain pairs that fall in the midnight zone of sequence relationship, despite being related at the same-family or same-superfamily levels.Chironomus erythrocruorin (1ECA chain A), (iv) bacterial hemoglobin (4VHB chain A), (v) paramecium truncated hemoglobin (1DLW chain A) and (vi) red-alga phycocyanin (2BV8 chain A).Finally, we present a qualitative case study comparing the sequence of human hemoglobin (1HHO chain A) with the sequences of six other related proteins, at varying levels of sequence divergence, but all classified within the same SCOP Globin-like SCOP fold: (i) chicken hemoglobin (1HBR chain A), (ii) sperm whale myoglobin (1MBD), (iii) The rows of this table appear in the increasing order of the resultant expected distance statistics for each pair. This distance correlates with increasing sequence divergence between the respective pairs, as can be verified by their estimated patterns .In sum, we have performed a comprehensive analysis of the compatibility of alignments derived from sequences and structures of pairs of proteins. This analysis identifies the extent of sequence divergence between pairs beyond which the alignments agree closely with each other (the daylight zone of sequence relationship), the extent of sequence divergence where the alignments start to diverge and where any given residue-level alignment should be handled with caution (the twilight zone), and finally the extent of divergence beyond which it is impossible to detect any sequence relationship (the midnight zone) where any alignment generated from sequences alone should be discarded all together.In contrast to previous attempts which deal with a single optimal sequence alignment computed using ad hoc choices for alignment parameters, the work presented here explicitly models the deteriorating quality of sequence relationships by employing marginal probabilities computed rigorously using time-parameterized statistical models. Another key difference of this work compared to the work of As future work, it would be interesting to examine, using the developed framework of sequence comparison, probabilistic models of amino acid relationships and measures of inter-alignment distance, the specified relationships and classification in popular databases such as SCOP (Financial Support: none declared.Conflict of Interest: none declared. btac247_Supplementary_DataClick here for additional data file.