Multiple whole-genome alignment is a challenging problem in bioinformatics. Despite many successes, current methods are not able to keep up with the growing number, length, and complexity of assembled genomes, especially when computational resources are limited. Approaches based on compacted de Bruijn graphs to identify and extend anchors into locally collinear blocks have potential for scalability, but current methods do not scale to mammalian genomes. We present an algorithm, SibeliaZ-LCB, for identifying collinear blocks in closely related genomes based on analysis of the de Bruijn graph. We further incorporate this into a multiple whole-genome alignment pipeline called SibeliaZ. SibeliaZ shows run-time improvements over other methods while maintaining accuracy. On sixteen recently-assembled strains of mice, SibeliaZ runs in under 16 hours on a single machine, while other tools did not run to completion for eight mice within a week. SibeliaZ makes a significant step towards improving scalability of multiple whole-genome alignment and collinear block reconstruction algorithms on a single machine. Multiple whole-genome alignment is a challenging problem in bioinformatics, especially when computational resources are limited. Here the authors present SibeliaZ, an algorithm and software based on analysis of de Bruijn graphs, which provides improved computational efficiency and scalability. The multiple whole-genome alignment problem is also closely related to the synteny reconstruction problem and to the questions of how to best represent pan-genomes.Multiple whole-genome alignments are the problem of identifying all the high-quality multiple local alignments within a collection of assembled genome sequences. It is a fundamental problem in bioinformatics and forms the starting point for most comparative genomics studies, such as rearrangement analysis, phylogeny reconstruction, and the investigation of evolutionary processes. Unfortunately, the presence of high-copy repeats and the sheer size of the input make multiple whole-genome alignment extremely difficult. While current approaches have been successfully applied in many studies, they are not able to keep up with the growing number and size of assembled genomes2. The first one is based on finding pairwise local alignments7 and then extending them into multiple local alignments11. While this strategy is known for its high accuracy, a competitive assessment of multiple whole-genome alignment methods1 highlighted several limitations. First, many algorithms either do not handle repeats by design or scale poorly in their presence, since the number of pairwise local alignments grows quadratically as a function of a repeat’s copy number. In addition, many algorithms use a repeat database to mask high-frequency repeats. However, these databases are usually incomplete and even a small amount of unmasked repeats may severely degrade alignment performance. Second, the number of pairwise alignments is quadratic in the number of genomes, and only a few existing approaches could handle more than ten fruit fly genomes1. Therefore, these approaches are ill-suited for large numbers of long and complex genomes, such as mammalian genomes in general and the recently assembled 16 strains of mice12 in particular.There are two common strategies to tackle the whole-genome alignment problem13. These are blocks that are free from nonlinear rearrangements, such as inversions or transpositions. Once such blocks are identified, they can independently be global aligned17. The problem of constructing blocks from anchors is known as the chaining problem which had been extensively studied in the past20. All of the methods applicable to datasets consisting of multiple genomes are heuristic since the exact algorithms depend exponentially on the number of genomes. Such strategies are generally better at scaling to handle repeats and multiple genomes since they do not rely on the computationally expensive pairwise alignment.Alternatively, anchor-based strategies can be applied to decompose genomes into locally collinear blocks23 widely used in genome assembly. Though these approaches do not work well for divergent genomes, they remain fairly accurate for closely related genomes. For example, Sibelia23 can handle repeats and works for many bacterial genomes; unfortunately, it does not scale to longer genomes. However, the last three years has seen a breakthrough in the efficiency of de Bruijn graph construction algorithms27. The latest methods can construct the graph for tens of mammalian genomes in minutes rather than weeks. We therefore believe the de Bruijn graph approach holds the most potential for enabling scalable multiple whole-genome alignments of closely related genomes.A promising strategy to find collinear blocks is based on the compacted de Bruijn graph28. This allows it to maintain a simple, static, data structure, which scales easily and allows simple parallelization. Thus, SibeliaZ-LCB overcomes a bottleneck of previous state-of-the-art de Bruijn graph-based approaches22, which relied on a dynamic data structure which was expensive to update. Further, we extend SibeliaZ-LCB into a multiple whole-genome aligners called SibeliaZ. SibeliaZ works by first constructing the compacted de Bruijn graph using our previously published TwoPaCo tool27, then finding locally collinear blocks using SibeliaZ-LCB, and finally, running a multiple-sequence aligner spoa29 on each of the foundation blocks. To demonstrate the scalability and accuracy of our method, we compute the multiple whole-genome alignment for a collection of recently assembled strains of mice. We also test how our method works under different conditions, including various levels of divergence between genomes and different parameter settings. Our software is freely available at https://github.com/medvedevgroup/SibeliaZ/.In this paper, we describe an algorithm SibeliaZ-LCB for identifying collinear blocks in closely related genomes. SibeliaZ-LCB is suitable for detecting homologous sequences that have an evolutionary distance to the most recent common ancestor (MRCA) of at most 0.09 substitutions per site. SibeliaZ-LCB is based on the analysis of the compacted de Bruijn graph and uses a graph model of collinear blocks similar to the "most frequent paths” introduced byAs described in the introduction, the major algorithmic innovation of this paper is the SibeliaZ-LCB algorithm. SibeliaZ-LCB takes as input a de Bruijn graph built on a collection of assembled genomes. An assembled genome is itself a set of contig sequences. SibeliaZ-LCB identifies and outputs all nonoverlapping blocks of homologous subsequences of the input genomes. A block can be composed of two or more sequences from one or more genomes. In this subsection, we will give a high level overview of SibeliaZ-LCB, leaving the more formal and detailed version for the “Methods”.k-mers (substrings of fixed length k) of the input. A k-mer that appears multiple times in the input is represented using just one node. Then, k-mers that appear consecutively in some input sequence are connected by an edge from the left one to the right one  = 1 − ∣H⧹A∣/∣H∣ and precision(A) = 1 − ∣A⧹H∣/∣A∣, where ⧹ denotes set difference.To measure accuracy, we used several ground-truth alignments (to be described) and employed the metrics of precision and recall used in the Alignathon and implemented by the mafTools package31. The alignment contains both orthologous and paralogous genes, though most of the paralogous pairs come from the well-annotated mouse reference genome. We removed any pairs of paralogous genes with overlapping coordinates, as these were likely mis-annotations, as confirmed by Ensembl helpdesk32. We made these filtered alignments as well as the alignments produced by SibeliaZ available for public download from our GitHub repository (see Section “Data availability” for the links).To evaluate recall, we compared our results against annotations of protein-coding genes. We retrieved all pairs of homologous protein-coding gene sequences from Ensembl and then computed pairwise global alignments between them using LAGANWe define the nucleotide identity of an alignment as the number of matched nucleotides divided by the length of an alignment, including gaps. The distribution of nucleotide identities, as well as the coverage of the annotation, is shown in Supplementary Fig. 6, a reliable local aligner known for its accuracy. We only computed alignments between chromosomes of different genomes, i.e., did not include self-alignments, which excludes duplications such as paralogous genes from the alignment. We used default settings of LASTZ except that we made it output alignments of nucleotide identity at least 90%. We then evaluated the recall and precision of our alignments but restricted our alignments to the sequences of chromosome 1. We then treated the LASTZ alignments as the ground truth. The LASTZ alignments are available for download from our repository’s supplemental data section. Note that because the alignment is represented as a set of positions pairs, it is possible to evaluate a multiple whole-genome alignments using pairwise local alignments.Since protein-coding genes only compromise a small portion of the genome, we also computed all-against-all pairwise local alignments between chromosomes 1 of genomes 1–2 and 1–4 using LASTZ33. In an alignment of highly similar genomes that has high precision, we expect these numbers to below (close to 0) for most of the columns in the alignment. Otherwise, it would suggest the presence of unreliable poorly aligned blocks in the alignment. Formally, given a column c of a multiple whole-genome alignments with ci being its ith element, average number of nucleotide differences is given by I[ci ≠ cj] is equal to 1 if both ci and cj are different valid DNA characters and 0 otherwise; ∣c∣ is the number of rows in column c.To measure precision, we use the LASTZ alignments on chromosome 1. However, it is computationally prohibitive to compute such alignments with LASTZ for the whole genome. We therefore also use an indirect way to assess precision for the whole genome. For each column in the alignment, we calculate the average number of nucleotide differences34, an aligner based on analysis of the Cactus graphs35 built from pairwise alignments. We also attempted to run Sibelia23 (a predecessor of SibeliaZ) and MultiZ + TBA8, but these could run to completion within a week on even a single mouse genome. Other multiple aligners16 benchmarked in the Alignathon could not handle a dataset of 20 flies and hence are unlikely to scale to a mammalian dataset. We also chose to not run Mercator14 since it requires a set of gene exons as input and hence solves a different problem: in this paper we focus on computing the whole-genome alignment directly from the nucleotide sequences without using external information. Further details about parameters, versions, and hardware are in Supplementary Note We benchmarked the performance of SibeliaZ against Progressive CactusThe running times of SibeliaZ and Cactus are shown in Fig. 34 as a part of its pipeline. SibeliaZ’s recall decreases only slightly going up to the whole 16 mice dataset, indicating that the recall scales with the number of genomes.In Tables We also measured coverage, which is the percent of the genome sequence that is included in the alignment. The coverage of both tools is roughly the same, but SibeliaZ has only about half the blocks. The different amounts of blocks produced by the tools are likely to be a result of the different approaches to the formatting of the output. Representation of multiple whole-genome alignments is ambiguous and the same alignment can be formatted in different but mathematically equivalent forms varying by the number blocks.12 using an alignment produced by Cactus which was further processed by annotation software CAT36. This fact might give Cactus a slight advantage in this comparison and explain why Cactus has a slightly better recall. Recall on orthologous gene pairs remains consistent in both two- and four-mice datasets for both datasets.We further investigate how the recall behaved as a function of nucleotide identity, for the two- and four-mice dataset Fig. . As expeAt the same time, we observed a much less consistent picture for paralogous pairs of genes. For example, SibeliaZ was able to recover nearly 90% of the paralogous basepairs belonging to gene pairs of nucleotide identity of 90% but found less than 45% of the base pairs of gene pairs of 80% identity.The results of the precision and recall measured with respect to LASTZ alignments are shown in Supplementary Table π(c) ≤ 0.1, which is what we would expect from aligning closely related genomes. Cactus has a slightly lower percentage of highly similar columns, which may simply indicate that it finds more blocks with higher divergence.Finally, since we could not evaluate genome-wide precision, we use the proxy of the average number of nucleotide differences for the alignment columns  and actually has a relatively high recall.We wanted to further understand SibeliaZ’s ability to recall homologous nucleotides from large gene families. Aligning genes having many copies is a challenging task since they generate a tangled de Bruijn graph. To investigate, we took each pair of genes in the two-mice dataset that have greater than 90% nucleotide identity. We then identify any other homologous genes that had a nucleotide identity of at least 90% to one of the genes in the pair. We refer to the number of such genes as the inferred family size of the gene pair, which roughly corresponds to the gene family size in the biological sense. Supplementary Fig. This experiment shows that finding all copies of even very similar homologous sequences within long genomes can be a challenging task. Moreover, the high variance we observe indicates that this challenge cannot be reduced to a single factor like family size. A manual inspection of false negatives suggests that the drop in recall may be due to complex substructures of unannotated repeats forming tangled graph structures.k-mer  and the maximum allowed length of a bubble b. For a given sequence divergence, the distance between shared k-mers forming bubbles in homologous regions increases with k. At the same time, the maximum allowed length of a bubble is b. If the distance exceeds b, then SibeliaZ may fail to uncover such regions and result in a lower recall. To avoid this situation, we can either decrease k or increase b. Decreasing k is desirable up to a point, but when k becomes too low, the de Bruijn graph becomes convoluted and our algorithm becomes more time and memory consuming. Increasing b can also be done but simultaneously increases the allowable gap length, leading to decreased precision.SibeliaZ-LCB has four primary parameters that affect its performance. The most critical dependence is on the size of a 37. In our case, one can control over-alignment by looking at the π(c) scores, as we have done in our analysis , we recommend k = 15, since it yields the highest recall. This value is impractical for complex organisms  due to runtime, so we recommend setting k = 25 in those cases as it provides a reasonable trade-off between accuracy and required computational resources (we used this for our mice datasets). For the value of b, we observed that increasing b lowers the precision at only higher values. Hence, we recommend b = 200 as the default in all cases, as it led to high recall across all tested ranges of k on our simulated data without lowering precision.To investigate this complex interplay between k = 15 or 25 and b = 200 and plotted the precision vs. recall curve as a function of the root-to-leaf divergence of the dataset  to construct the collinear blocks. This is most useful in applications stemming from studies of genome rearrangements, which can be applied to study breakpoint reuseThere are several remaining open questions of interest. A formal analysis of SibeliaZ-LCB’s runtime is relevant, but doing it in a useful way is a challenge. The worst-case running time does not reflect the actual one; moreover, we observed that the actual one depends on multi-thread synchronization, which is challenging to model. However, it would be interesting if such a time analysis can be performed parametrized by the crucial properties of the structure of the input. We also did not investigate how close to an optimal solution our greedy heuristic gets. One way to do this would be to find an optimal carrying path using exhaustive enumeration, but the search space even for a small realistic example is too big. We suspect that a polynomial time-optimal solution is not possible, but the computational complexity of our problem is open.48, currently holds 18 assembled maize genomes, with more to come in the recent future. Similarly, the Solanaceae Genomics Network has recently released the genomes of 13 diverse tomato accessions (https://solgenomics.net/projects/tomato13/). Analysis of such datasets is likely to be carried out in single-lab settings with limited compute resources, rather than at large computing centers like EMBL or NCBI. SibeliaZ makes a significant leap toward enabling such studies.SibeliaZ is the first multiple whole-genome aligners that can run on a single machine in a reasonable time on a dataset such as the 16 mouse genomes analyzed in this paper. With ongoing initiatives like the Vertebrate Genomes Project and the insect5k, thousands of species will soon have a reference genome available, and the sequencing and assembly of various subspecies and strains will be the next logical step for many comparative genomics studies. For examplek and a string s, we define a multigraph G as the de Bruijn graph of s. The vertex set consists of all substrings of s of length k, called k-mers. For each substring x of length k + 1 in s, we add a directed edge from u to v, where u is the prefix of x of length k and v the suffix of x of length k.First, we will define the de Bruijn graph and related objects. Given a positive integer k + 1)-mer yields a unique edge, and every edge corresponds to a unique location in the input. Two edges are parallel if they are oriented in the same direction and have the same endpoints. Note that two edges are parallel if and only if they were generated by the same (k + 1)-mer. This way, we use the notion of parallel edges to refer to a set of identical (k + 1)-mers in the input strings. Parallel edges are not considered identical. The de Bruijn graph can also be constructed from a set of sequences S = {s1, …, sn}. This graph is the union of the graphs constructed from the individual strings: G = ⋃i≤n1≤G. See Fig. Each occurrence of a k + -mer yielG is denoted by E(G). We write  to denote an edge from vertex u to v. A walkp is a sequence of edges , , …, ) where each edge  belongs to E(G). The length of the walk p, denoted by ∣p∣, is the number of edges it contains. The last edge of a walk p is denoted by the end(p). A path is a walk that visits each vertex at most once.The set of all edges in a graph x was generated by a (k + 1)-mer starting at some position j of some string si. To retrieve the position j of the (k + 1)-mer that generated edge x, we define function pos(x) = j. We use the function pos to map edges of the graph back to positions of the k-mers that generated them. For an edge x, its successor, denoted by next(x), is an edge y such that both x and y are generated by the same string and pos(y) = pos(x) + 1. Note that a successor does not always exist.In a de Bruijn graph, a given edge p =  is genomic if next(xi) = xi+1 for 1 ≤ i ≤ ∣p∣ − 1. In other words, a walk is genomic if it was generated by a substring in the input. The b-extension of a genomic walk p is the longest genomic walk q =  such that y1 = next(end(p)) and ∣q∣ ≤ b. The b-extension of a walk p is uniquely defined and usually has length b, unless p was generated by a substring close to an end of an input string. Intuitively, b-extension defines edges lying ahead of a walk. As our algorithm works in the seed-and-extend manner, we later use b-extensions to find the most appropriate extension of the current block. The concatenation of two walks x and y is a walk (if it exists) xy consisting of edges of x followed by edges of y. We use the concatenation operation to extend the genomic walks constituting locally collinear blocks using appropriate b-extensions.A walk collinear block is a set of edge-disjoint genomic walks with length at least m, where m is a parameter. We call walks constituting a collinear block collinear walks. In order to quantify how well collinear walks correspond to homologous sequences, we will define a collinearity score of a collinear block. Our problem will then be to find a set of collinear blocks that are pairwise edge-disjoint and have the largest score.In this section, we will define the collinear block reconstruction problem in de Bruijn graphs. A bubble is a subgraph corresponding to a possible mutation flanked by equal sequences. Formally, a pair of walks x and y form a bubble  if all of the following holds: (1) x and y have common starting and ending vertices; (2) x and y have no common vertices except the starting and ending ones; and (3) ∣x∣ ≤ b and ∣y∣ ≤ b, where b is a parameter. A chainc = , , …, ) is a sequence of bubbles such that x = x1x2…xn and y = y1y2…yn are walks in a de Bruijn graph. In other words, a chain is a series of bubbles where each bubble is a proper continuation of the previous one. Note that two parallel edges form a bubble and a chain arising from equal sequences corresponds to a series of such bubbles. This way, a chain models a pair of sequences that potentially have point mutations and indels. For an example of a bubble and a chain, see Fig. We capture the pattern of two homologous collinear walks through the concept of chains and bubbles. A g holds:  x and y 53, mostly in the context of analyzing assembly graphs. We decided to follow a different approach designed specifically for capturing locally collinear blocks.The subgraph resulting from more than two collinear walks can be complex, and there are several ways of capturing it. We note that there are previous studies generalizing the idea of bubbles, seesa, they should have many common k-mers and there should be a path pa = G through the graph forming a long chain with each walk p in the collinear block. We call such a path a carrying one. We require the chains to be longer than m to avoid confusing spuriously similar sequences with true homologs. At the same time, a collinear walk may only partially form a chain with the carrying path, leaving hanging ends at the ends of the carrying path, which is undesirable since it implies that these graphs originate from dissimilar sequences. We note that compared to the previous work on chaining anchors20 our definition of the block will be more relaxed. Namely, we will not require common k-mers to be present in all copies of a block. In addition, many alignment methods use phylogenetic information for scoring purposes. We decided to not use it since our method targets closely related sequences, such as strains of the same species, where the phylogeny is often unknown.Our approach is to give a definition that naturally leads itself to an algorithm. As homologous sequences all originate from some common ancestral sequence pa and a genomic walk p, let q2 be the longest subpath of pa that forms a chain with p. Then, we can write pa = q1q2q3. Recall that m is the parameter denoting the minimum length of a collinear block, and b is the maximum bubble size We define the score f as∞) where the hanging ends are too long, and the first case ignores walks  that weave through pa but are too short. The second case gives a score that is proportional to the length of the part of pa that forms a chain with p. At the same time, it reduces the score if the collinear walks leave hanging ends q1 and q3—the parts of pa not participating in the chain. The penalty induced by these ends is squared to remove spuriously similar sequences from from the collinear block. This form of scoring function was chosen because it performed well during the initial stages of development. We do not penalize for the discrepancy between p and q2 for the sake of simplicity of the scoring function and avoiding extra computation needed to calculate it. Figure We formalize this intuition by introducing a scoring function quantifying how well a carrying path describes a collection of the collinear walks. The function rewards long chains formed by the carrying path and a collinear walk and penalizes the hanging ends. Given a carrying path collinearity score of a collinear block is given bypa can be any path (not necessarily genomic). In other words, we are looking for a path forming longest chains with the collinear walks and thus maximizes the score. The collinear blocks reconstruction problem is to find a set of collinear blocks The mentclass2pt{minimPbest. If the block has a positive score, then Pbest is added to our collection of collinear blocks Our algorithm’s main pseudocode is shown in Box P, we first initialize the collinear block with a walk for each unused edge parallel to the seed (including the seed) (lines 7 and 8). These parallel edges represent the different occurrences of the seed string in the input and, hence, form the initial collinear block. We then proceed in phases, where each phase is an iteration of the while loop (lines 9–19). During each phase, the carrying path pa is extended using a walk r of length at most b (lines 10–14). Next, we try to extend each of the collinear walks in a way that forms chains with the extended pa (lines 15–19). The extension of a seed into a collinear block is also a greedy process since we only change pa and the walks in P by extending them and never by changing any edges. Finally, we check that the collinearity score for our extended block is still positive—if it is, we iterate to extend it further, otherwise, we abandon our attempts at further extending the block. We then recall the highest-scoring block that was achieved for this seed and save it into our final result To extend a seed into a collinear block r by which to extend pa, we use a greedy heuristic (lines 10–14). First, we pick the vertex t which we want to extend to reach (lines 10–12). We limit our search to those vertices that can be reached by a genomic walk from the end of pa and greedily chose the one that is most often visited by the b-extensions of the collinear walks in P. Intuitively, we hope to maximize the number of collinear walks that will form longer chains with pa after its extension and thereby boost the collinearity score. We then extend pa using the shortest b-extension of the walks in P to reach t. We chose this particular heuristic because it showed superior performance compared to other possible strategies.To pick the walk r by which to extend pa, we must select the extensions to our collinear walks P that will form chains with par. This is done by the function Update-collinear-blocks . For each location, we check if it can be reached by a b-extension from an existing p ∈ P. If yes, then we extend p, so as to lengthen the chain that it forms with pa. If there are multiple collinear walks that reach w, we take the nearest one. If no, then we start a new collinear walk using just x. Figure Once we have selected the genomic walk ocks Box . We exteOur description here only considers extending the initial seed to the right, i.e., using out-going edges in the graph. However, we also run the procedure to extend the initial seed to the left, using the incoming edges. The case is symmetric and we, therefore, omit the details.Box 1 Algorithm Find-collinear-blocksS, integers k, b and mInput: strings G representing collinear blocksOutput: a set of edge-disjoint subgraphs of 1: G ← G         ⊳ Construct the multigraph2: for all distinct pairs  ∈ E(G) do        ⊳ Check possible seeds3: pa with 4:     Initialize the current-carrying path pa5:     pa6:     for edges x ∈ E(G) parallel to  not marked as used do7:     P a new collinear walk consisting of x8:        Add to while f(P) ≥ 0 do        ⊳ Extend the carrying path as far as possible9:     Q ← {q ∣ q is the b − extension of a p ∈ P}10:        w0 ← last vertex in pa11:        t ← a vertex, reachable from w0 via a genomic walk, that is visited by the most walks of Q.12:        r ∈ Q be the shortest walk from w0 to t13:        Let r as w0, w1, …, wr∣∣, with wr∣∣ = t14:        Denote the vertices of fori ← 1 to ∣r∣ do15:        wi−1, wi) to the carrying path pa16: Append 17: if f(P) > f(Pbest) then18: Pbest ← P19: if f(Pbest) > 0 then20:     21:        Pbest as used22:        Mark edges visited by walks of return 23: Box 2 Update-collinear-walksP, a vertex wInput: A sorted set of collinear walks POutput: Updated set for edges x ∈ E(G) ending at w not marked as used do1: p ∈ P be a walk such that its b-extension q contains x and pos(end(p)) is maximized         ⊳ Find a walk extendable with x2:      Let if such p exists then3:      q so that end(q) = x4:         Truncate p with q         ⊳ Lengthen the chain that p forms with pa5:         Append else6:     x to P7:        Add a new walk consisting of the edge return P8: ℓ = v1, …, vn be the list of k-mers corresponding to junctions, in the order, they appear in the underlying string s. The edge set of the compacted graph consists of edges {v1 → v2, v2 → v3, …, vn−1 → vn}. We efficiently construct the compacted graph using our previously published algorithm TwoPaCo27.For simplicity of presentation, we have described the algorithm in terms of the ordinary de Bruijn graph; however, it is crucial for running time and memory usage that the graph is compacted first. Informally, the compacted de Bruijn graph replaces each non-branching path with a single edge. Formally, the vertex set of the compacted graph consists of vertices of the regular de Bruijn graph that have at least two outgoing (or ingoing) edges pointing at (incoming from) different vertices. Such vertices are called junctions. Let k + 1)-mer one by one. Our previous description of the algorithm remains valid, except that the data structures operate with vertices and edges from the compacted graph instead of the ordinary one. The only necessary change is that when we look for an edge y parallel to x, we must also check that y and x spell the same sequence. This is always true in an ordinary graph but not necessarily in a compacted graph.This transformation maintains all the information while greatly reducing the number of edges and vertices in the graph. This makes the data structures smaller and allows the algorithm to fast-forward through non-branching paths, instead of considering each (k + 1)-mers, which can clog up our data structures. To handle this, we modify the algorithm by skipping over any junctions that correspond to k-mers occurring more than a times; we call a the abundance pruning parameter. Specifically, prior to constructing the edge set of the compacted de Bruijn graph, we remove all high abundance junctions from the vertex set. The edge set is constructed as before, but using this restricted list of junctions as the starting point. This strategy offers a way to handle high-frequency repeats at the expense of limiting our ability to detect homologous blocks that occur more than a times.An important challenge of mammalian genomes is that they contain high-frequency (P are stored as a dynamic sorted set, implemented as a binary search tree. The search key is the genome/position for the end of each walk. This allows performing a binary search in line 2 of Algorithm Update-collinear-walks.The organization of our data in memory is instrumental to achieving high performance. To represent the graph, we use a standard adjacency list representation, annotated with position information, and other relevant data. We also maintain a list of the junctions in the paragraph above in the order they appear in the input sequences, thereby supporting next queries. The walks in the collinear block s is a string, then let 27. Our algorithm and corresponding data structures can be modified to work with the comprehensive graph with a few minor changes which we omit here.Another aspect that we have ignored up until now is that DNA is double-stranded and collinear walks can be reverse-complements of each other. If Our implementation is parallelized by exploring multiple seeds simultaneously, i.e., parallelizing the for a loop at line 3 of Algorithm Find-collinear-blocks. This loop is not embarrassingly parallelizable, since two threads can start exploring two seeds belonging to the same carrying path. In such a case, there will be a collision on the data structure used to store used marks. To address this issue, we process the seeds in batches of fixed size. All the seeds within a batch are explored in parallel and the results are saved without modifying the “used” marks. Once the batch is processed, a single arbiter thread checks if there is any overlap in the used marks of the different threads. If there is, it identifies the sources of the conflict and reruns the algorithm at the conflicting seeds serially. Since most seeds do not yield valid carrying paths, such conflicts are rare. Once there is no conflict, the arbiter updates the used main data structures with the results of the batch. This design allows the computation result to be deterministic and independent of the number of threads used.Further information on research design is available in the Supplementary InformationReporting Summary


Bioinformatic workflows frequently make use of automated genome assembly and protein clustering tools. At the core of most of these tools, a significant portion of execution time is spent in determining optimal local alignment between two sequences. This task is performed with the Smith-Waterman algorithm, which is a dynamic programming based method. With the advent of modern sequencing technologies and increasing size of both genome and protein databases, a need for faster Smith-Waterman implementations has emerged. Multiple SIMD strategies for the Smith-Waterman algorithm are available for CPUs. However, with the move of HPC facilities towards accelerator based architectures, a need for an efficient GPU accelerated strategy has emerged. Existing GPU based strategies have either been optimized for a specific type of characters (Nucleotides or Amino Acids) or for only a handful of application use-cases.In this paper, we present ADEPT, a new sequence alignment strategy for GPU architectures that is domain independent, supporting alignment of sequences from both genomes and proteins. Our proposed strategy uses GPU specific optimizations that do not rely on the nature of sequence. We demonstrate the feasibility of this strategy by implementing the Smith-Waterman algorithm and comparing it to similar CPU strategies as well as the fastest known GPU methods for each domain. ADEPT’s driver enables it to scale across multiple GPUs and allows easy integration into software pipelines which utilize large scale computational systems. We have shown that the ADEPT based Smith-Waterman algorithm demonstrates a peak performance of 360 GCUPS and 497 GCUPs for protein based and DNA based datasets respectively on a single GPU node (8 GPUs) of the Cori Supercomputer. Overall ADEPT shows 10x faster performance in a node-to-node comparison against a corresponding SIMD CPU implementation.ADEPT demonstrates a performance that is either comparable or better than existing GPU strategies. We demonstrated the efficacy of ADEPT in supporting existing bionformatics software pipelines by integrating ADEPT in MetaHipMer a high-performance denovo metagenome assembler and PASTIS a high-performance protein similarity graph construction pipeline. Our results show 10% and 30% boost of performance in MetaHipMer and PASTIS respectively. Sequence alignment lies at the core of most bioinformatics applications. Aligning two sequences determines a degree of similarity which may yield homology of the proteins or genes and their functional information. Local sequence alignment has been used in de novo sequence assembly to determine how different regions of a genome are connected , 2 and fO(nm) where n and m represent the lengths of two sequences to be aligned. Its quadratic time complexity makes performing large number of alignments or aligning long sequences time consuming. As a solution, heuristic based strategies were presented in the form of BLAST and Gapped BLAST which speed up the process considerably with the trade-off being an approximate solution [Smith-Waterman is a sequence alignment algorithm that scores all possible local alignments between two sequences using a dynamic programming method and outputs the optimal alignment . Gotoh msolution , 8.In this paper we present ADEPT, a novel domain independent sequence alignment strategy for GPU architectures and demonstrate it by implementing a GPU-accelerated complete Smith-Waterman algorithm for the use case of pairwise sequence alignments. ADEPT derives its performance from architecture specific optimizations and is performant regardless of the type of sequence. Our analysis shows that ADEPT out-performs similar CPU approaches and either closely matches or out-performs domain specific existing GPU approaches. ADEPT provides an added advantage of built in capability of scaling across multiple GPUs with minimal effort from the developer. It can effectively be used as a drop-in replacement for CPU libraries. For the rest of this paper, the acronym ADEPT will be used interchangeably for the ADEPT-based implementation of the Smith-Waterman algorithm and the proposed strategy in general.With the introduction of multi-core and GPU devices, multiple parallel strategies for exploiting modern architectures were introduced. Parallelizing the Smith-Waterman algorithm is particularly challenging because of the inter-cell dependencies in the dynamic programming matrix . ComputaInter-task parallelism strategies translate to embarrassingly parallel approaches that compute multiple alignments in parallel. One such implementation, which has been discussed by Rognes , involveNM total alignments, where N and M are sizes of query and reference sets respectively. One-to-one alignment deals with aligning only those sequences which are present at same indices in two sets of sequences, i.e. given that total sequences in query and reference sets are N, then total alignments would also be N.A very common scenario in bioinformatics applications requires pairwise sequence alignments where one-to-one alignments are performed between two given sets of sequences , 16. ThiUse cases of one-to-one pairwise alignments are quite common in short-read DNA mapping , 18 and Similarly, one-to-one pairwise alignments play an important role in inferring homologous proteins. The detection of homologous proteins is fundamental to several applications such as functional annotation (assigning functions to unknown proteins), gene localization , or identifying protein families (the proteins that descend from a common ancestor). For example, a common method for the identification of protein families is (i) to first perform a similarity search , 22 in aHowever, the available GPU implementations employ methods which are either domain specific, such as query profile construction for protein database search  and use This leaves a gap for a parallel strategy that leverages a GPU’s hardware to derive performance rather than relying on application specific approaches. Such a method could enable offloading of sequence alignments to the GPU regardless of the type of sequence (Protein or DNA). Such libraries are widely available for CPUs and have enabled the development of numerous tools due to their generic nature. In an attempt to mitigate this gap, in this paper we introduce ADEPT, a parallelization strategy that can exploit a GPU’s architecture for performance, to provide a consolidated GPU-accelerated sequence-alignment library.Q and R to be aligned; Q is a query sequence represented as Q={q1,q2,q3,,,qn} and R is a reference sequence represented as R={r1,r2,r3,,,rm}, where n=|Q| and m=|R|. In this paper we consider the case of one-to-one alignments where, given two sets of sequences, A and B, each sequence in set A will be aligned to one sequence in set B located at the same index. The total number of sequences in set A is equal to the number of sequences in set B.Consider two sequences Q and R, Smith-Waterman with Gotoh scoring computes three scoring tables E, F and H, following the equations below:Given the sequences E and F are used for keeping track of gap insertions in reference and query sequences. A gap insertion in query sequence can be seen as a deletion in reference sequence or vice versa. Gap insertion/deletion or collectively known as indels enable accurate alignment of two sequences even if they are not of equal lengths. Without matrices E and F it is not possible to penalize gap insertion separately. Matrix H keeps track of alignment extensions. Ginit is the gap initialize penalty, Gext is the gap extend penalty and S is the match or mismatch score based on how closely characters a and b match. The algorithm starts by initializing the first row and column of tables E, F and H with zeroes. The first phase involves computing each cell of table H with the help of tables E and F, using the above equations. While scoring the table H, information is maintained to keep track of the highest scoring cell  were introduced as dedicated graphics processing devices, but with the development of advanced programming tools and improvement in GPU hardware, they have rapidly emerged as accelerators of choice across the High-Performance Computing community. A typical CPU-GPU computing setup involves selecting a computationally intensive portion of an application and offloading it to GPU. This involves offloading the data to the GPU’s Global Memory, launching a kernel to run on the GPU, and then moving the results back from the GPU to the CPU. CPU and GPU communication happens via a PCI express connection.GPU hardware consists of multiple Streaming Multiprocessors (SM), where each SM contains multiple Floating-Point units, Integer operation units and in more recent devices Tensor operation units. All the cores collectively provide GPUs with their massively parallel nature. For instance, NVIDIA’s V100 GPU contains a total of 80 SMs and 64 FP32 units giving a total of 5,376 computing cores. GPUs traditionally have an on-chip memory resource, termed Shared Memory, and off-chip Memory or Global Memory . TypicalCUDA is a parallel programming platform which enables the use of CUDA-enabled GPUs for general purpose computing. CUDA provides lower level software access to the computational elements of GPUs and enables a programmer to write kernels for offloading computational load to GPUs. The CUDA programming model provides two levels of parallelism in the form of a grid of CUDA blocks, where each CUDA block consists of multiple threads. CUDA Threads are the basic computational unit; each thread can be identified by a unique thread id and a block id representing its parent block. Inter-thread communication can take place either via the Shared Memory or using register-to-register data transfers. Inter-block communication can happen only via the Global Memory. In NVIDIA hardware, the threads of a block are scheduled on to the SMs in groups of 32 known as warps. Depending upon the resource availability, multiple warps may be scheduled on the same SM.Our ADEPT implementation has two parts: a driver and a kernel. The driver initializes the GPU memory, packs the sequences into batches, and once enough sequences are available to saturate the GPU global memory, transfers all this data to the GPU. Additionally, the driver also detects different GPUs available on the node and balances the amount of work across all the available GPUs as shown in Fig. Batched sequences are stored in two arrays, one for query sequences and the other for reference sequences. The number of sequences in the query and reference arrays are the same, and sequences located at the same indices are aligned with each other. For instance, if there are N query sequences and N reference sequences, a total of N alignments will be performed. Each alignment is mapped to a unique CUDA block. Then, inside each CUDA block a more fine-grained approach is implemented. From here on, all the implementation details are per-block and the same algorithm is replicated across each CUDA block.Hij of the dynamic programming table H, cells Hi−1,j, Hi,j−1, and Hi−1,j−1 need to have been computed. Because of this aspect of the algorithm, parallelism is restricted only along the ant-diagonal of the matrix as shown in Fig. As discussed in the H, we start by assigning one CUDA thread per column  for masking out threads in each iteration for which dependencies are not ready. The BMA has a length b, where b is equal to 3∗|Q| and BMA is initialized as:To calculate the scoring table xi is the ith element of BMA. The number of threads that need to track their state is equal to the size of query sequence. Since each thread needs to track its state in three different phases of algorithm, the length of BMA is fixed to three times the size of query sequence.In the above equation, C is used to keep track of the region which the algorithm is in.Figure I is the iteration number, which also corresponds to the diagonal being computed. It can be observed in Fig. C becomes false and the array stops shifting until the algorithm enters the red region. Here again the array starts shifting to the right  bytes. Where N is the size of a batch. This yields a total global memory requirement of several hundred GBs for a million alignments, and even top of the line GPUs have global memory of only a few GBs.To compute the highest scoring cell a pass over the complete table H is required. Since the total number of cells that are computed in the table H are n−1) bytes apart, which can be more than 200 bytes apart if n is larger than 100.Apart from the storage size of the dynamic programming table, another challenge that occurs often on GPUs is that of non-coalesced global memory accesses. The Global memory accessed by threads of a CUDA warp is bundled into the minimum number of cache loads, where L1 cache line size is 128 bytes. Thus if the two threads are accessing a memory location that is more than 128 bytes apart, the memory accesses will be un-coalesced. It can be observed in Fig. It can be seen in Fig. Once all the cells have been computed we use CUDA’s warp shuffle intrinsics to implement a block-wide reduction for obtaining the highest scoring cell as shown in Algorithm 1. Our implementation of block-wide reduction has been adopted from NVIDIA’s own reduction method  with modHi,j it requires values from cells Hi−1,j, Hi,j−1 and Hi−1,j−1. In the figure it can be observed that the cell Hi−1,j is computed by thread j while the other two cells are computed by thread j−1. For this inter-thread transfer we explored two methods of data sharing between threads i.e. communication using shared memory and register-to-register memory transfer.Figure CUDA’s warp shuffle intrinsics allow threads to perform direct register-to-register data exchange without performing any memory loads and stores, while use of shared memory involves going through the on-chip shared memory. Due to much faster performance we opted for the register-to-register data exchange method.q)−1 and 32∗q (where q>0) would not be possible through register-to-register transfers because these do not belong to same warp. For instance, in Figure However, register-to-register transfers are only allowed among the non-predicated threads of the same warp. This introduces several edge cases, for example in the CUDA platform where a warp is 32 threads wide, a communication between thread , which can be several hundred GBs when N is close to a million alignments and unlike the scoring phase we cannot discard parts of the traceback matrices because that may lead to missing optimal alignments. The second challenge is that of un-coalesced memory accesses, as mentioned before. The write-back to the traceback matrices occurs along the anti-diagonals and since the matrices are laid down in the global memory in row-major indexing, this leads to un-coalesced memory accesses as shown in Fig. The third phase of the Smith-Waterman algorithm involves performing a traceback starting from the highest scoring cell and ending when the score drops to zero or the top left end of the matrix is reached. This requires maintaining the traceback pointers, which can be stored in the form of two matrices, one for storing the indices of the query sequences and the other for storing the indices of the reference sequences. However, storing these matrices yields two sets of challenges. First, the amount of memory required to store traceback matrices equals 2∗However, in most of the practical Smith-Waterman applications, complete alignment details are rarely required. The majority of the applications only require the optimal alignment score and the optimal alignment start and end indices , 13, 27.To obtain the start positions of the alignment we make use of the symmetric nature of the optimal alignment. An optimal alignment is symmetrical i.e. scoring two sequences forward or with their directions reversed yields the same optimal alignment.For reverse scoring, we make use of this property as previously done in  and compUsing the indices of the highest scoring cell in the reverse scoring phase, we can compute the start index of alignment. Using the reverse scoring phase enables us to avoid storing traceback matrices and helps free up GBs of space. The reverse scoring kernel follows the same implementation as the forward scoring kernel that has been shown in Algorithm 1, hence we re-use that implementation by providing flipped sequences at the input. It must be noted that the reverse scoring matrix in most of the cases ends up having less total work because of known end positions.Thus far, ADEPT has not required any domain specific optimizations, and the Smith-Waterman implementation discussed above has been oblivious of the types of sequence.The difference between aligning protein sequences and DNA sequences is between the scoring methods. When aligning DNA sequences, if two of the same nucleotide bases align, that is considered a match and a fixed match score is used for computing the total score; similarly if the bases do not match, a mismatch score is used instead. When aligning protein sequences, two aligning amino acids need to be scored based on their chemical similarity. Similarity scores for all possible comparison of amino acids are characterized and available in the form of a scoring matrix . InsteadSince a scoring matrix needs to be accessed very frequently, in our implementation we move the static scoring matrix to the GPU’s shared memory to reduce the overhead associated with multiple accesses. For simplifying the scoring matrix lookups and minimizing shared memory usage, we use a decoding matrix to index into the scoring matrix. Typically, the scoring matrix is indexed by the amino acid characters, which leads to large amount of memory being reserved for the matrix. In this implementation, we first index a with the ASCII code associated with the amino acid character to retrieve an encoded index, which is then used to access the scoring matrix.Underlying kernel for protein and DNA alignment still remains the same, for protein alignment the only difference is that instead of a match/mis-match score and similarity score is obtained from the scoring matrix, everything else remains the same as in Algorithm 1.In order to make the switch between protein kernel and DNA kernel easy for the user, we provide two different kernels for protein alignment and DNA alignment. The DNA kernel accepts match, mismatch, gap open and gap extend scores at input while the protein kernel accepts a scoring matrix along with gap open and gap extend scores at compile time.driver component which manages all the communication, load balancing and batch size determination for the GPU kernels while keeping the developer oblivious of these intricacies.In a typical CPU-GPU setup, the CPU prepares a batch of data that is offloaded to a GPU and launches a GPU kernel to process that data; once the data is processed, the results are moved back to CPU. However, with the evolution of GPU technology, a widespread adoption of GPUs has taken place, and instead of having one GPU per node, a typical GPU system has several GPUs on each node. For instance, the Summit supercomputer  has six ADEPT’s driver gathers hardware information about all the GPUs installed on a node and then divides the work equally among them. A separate context is created for each GPU where a unique CPU thread is assigned to a particular GPU. This CPU thread divides the total computational load into smaller batch sizes depending on the memory capacity of the GPU assigned to it. Each batch is then prepared and packed into a data structure which is then passed to a GPU kernel call. Using CUDA streams, the GPU kernel call and the data packing stage are overlapped so that CPU and GPU work can be carried out in parallel. An overview of ADEPT’s design can be seen in Fig. ADEPT’s driver makes it easier for the developers to integrate ADEPT in high performance bioinformatics software pipelines by reducing the complexities of dealing with multiple GPUs, and requiring just one call to the driver function. Effectively, making ADEPT a drop-in replacement for existing CPU libraries, whereas existing GPU libraries require significant amount of work in order for them to be included in an existing software pipeline.We evaluate the performance of ADEPT against two of the popular CPU libraries which can perform both Protein and DNA alignments. Among existing GPU implementations we chose GASAL2 for comparison because it is the fastest known GPU library for performing DNA pairwise alignments . The onlSSW-library Striped Smith Waterman or SSW-Library [-Library  is an im-Library  and is oSeqan library Seqan Library is a widely used CPU sequence alignment library [align_bench_par program as it performs pairwise alignments. We used the build option for AVX2 instructions with scoring range set at 16 bit for optimized performance. library . For thi library . For theGASAL2 GASAL2 is a recently developed GPU implementation for short read DNA analysis that performs pairwise alignments. The authors of GASAL2 have comprehensively demonstrated that GASAL2 is the fastest GPU implementation for pairwise alignments [ignments . Hence, NVBIO NVBIO is a GPU based library developed by NVIDIA developers which provides multiple algorithms implemented for accelerating bioinformatics pipelines. For this paper we used the proteinsw program provided with the NVBIO library. Here, it needs to be considered that NVBIO has not been maintained for some time now and we found out that with the same scoring conditions, using the same dataset, NVBIO’s protein alignment output does not match with the well known Smith-Waterman implementations. An issue has been opened at their github page regarding this. The NVBIO-based experiments in this paper were performed assuming that the library still computes the complete scoring matrix, but because of a bug in the protein scoring function the results do not match with other libraries.To evaluate the performance of ADEPT against existing methods we identified use cases in genomics and proteomics that require performing large numbers of pairwise alignments. For each of these use-cases we obtained real world datasets, which were then processed to form three curated sets of Query and Reference sequences. Below we discuss in detail the data generation process for DNA and Protein evaluation datasets.For the alignment of DNA sequences, we used a set of 29 million FASTQ reads, of lengths from 150 to 300, from the SYNTH64 dataset  as the qFor the alignment of protein sequences, we use a curated  dataset called SCOPe  . The curFor all the pairwise alignments we assumed the longer sequence is the reference and the shorter sequence is the query. These sequences were then binned based on the length of the query sequences into three different datasets: Protein-1, Protein-2 and Protein-3. Details are provided in Table For each of the above discussed datasets we performed three experiments to evaluate ADEPT’s performance against existing approaches. First we compare the Giga Cell Updates Per Seconds (GCUPS) for each approach, which was done by running all the methods in only forward scoring phase to obtain only the highest score. Then we compute the total cells (of the DP table) that were processed for that dataset and divide that by the total runtime, which is given by:In the second experiment, we turn on the reverse scoring phase for all the algorithms  and evaluate the total execution time for obtaining the score, start and end positions of the optimal alignment.Finally, we repeat the above experiments by running all of the algorithms on complete CPU and GPU nodes of the Cori Supercomputer  to evaluFor all the CPU runs we made use of the Cori Supercomputer’s Haswell nodes , each ofFor DNA alignments we used the same scores for all algorithms, i.e. match-score of 6, mismatch penalty of 4, gap open penalty of 4 and gap extension penalty of 1. For Protein alignments we used the Blosum 62 matrix , with a We first compared the total GCUPS for all algorithms for DNA alignment datasets. As discussed before, this was done by turning off the reverse scoring feature and only using one socket of a Haswell Node (16 CPU cores) for CPU libraries and one V100 GPU for GPU libraries. It can be observed in Fig. To evalute the high-performance computational capability for large scale systems, we repeated the above runs, but this time for the CPU libraries we used all the available CPU cores on a Cori Haswell node (32 CPU cores). And for GPU codes, we performed analysis for 2 GPUs, 4 GPUs and full node runs with all 8 GPUs. GASAL2 does not have support for multiple GPUs, hence it was not included in these experiments. It can be observed in Fig. To compare overall performance, we turned on the reverse scoring feature for all algorithms so that each can compute the start position of the alignment as well. The Seqan benchmarking suite did not have an option of obtaining the start position using reverse scoring and instead performs a complete traceback which is a very slow, so in the interests of fairness, we did not include Seqan in these experiments. These experiments were again performed using a single socket (16 CPU cores) and a single GPU first and then repeated for full nodes to evaluate their ability for large scale bioinformatics tools.It can again be observed in Fig. We repeated the same experiments for Protein datasets; for these experiments we do not include GASAL2 library because it does not support protein alignments. Figure To evaluate the performance for protein alignments with reverse scoring phase turned on, we compared only the SSW-Library and ADEPT because the Seqan test suite and NVBIO do not have the reverse scoring feature. For single socket (16 CPU core) vs single GPU comparison, ADEPT out-performs SSW for the Protein-1 and Protein-2 datasets Fig. . But forBut for node level analysis, ADEPT Fig.  out perfTo demonstrate ADEPT-SW’s effectiveness in preparing large-scale bioinformatics software pipelines for GPU-heavy systems, we chose two applications from different domains. The first is MetaHipMer , 42, whiMetaHipMer is a specialized version of a large scale denovo genome assembler HipMer . MetagenA simplified overview of the alignment phase of MetaHipMer can be seen in Fig. We modified the MetaHipMer pipeline so that instead of performing a SW alignment immediately after the lookup, it adds it to a batch of alignments. Once the batch size is large enough, a call is made to the ADEPT-SW kernel which then takes control. The ADEPT-SW kernel detects the number of available GPUs and the number of processes running on the CPU and then performs a mapping from CPU to GPU in a round-robin fashion. If the number of GPUs is smaller than the number of processes on the node, multiple processes may be mapped to a same GPU. During the mapping the ADEPT-SW kernel ensures that if a GPU is shared by more than one process, the global memory of the GPU is divided among the processes such that there is no overlap. The ADEPT-SW driver then launches GPU kernels as shown in Fig. For this study a UPC++ based version of MetaHipMer was used. All the experiments were performed on Cori GPU Nodes ; each GPFigure PASTIS  is a disPASTIS needs certain information regarding the aligned sequence pairs in the construction of the protein similarity graph. These information include average nucleotide identity, raw alignment score, coverage, etc. The alignments that do not meet certain criteria are eliminated and they are not included in the protein similarity graph. Note that each aligned sequence pair corresponds to a possible edge in the protein similarity graph, where the weight of the edge signifies the strength of similarity between the aligned sequence pair. For that purpose, PASTIS can make use of either average nucleotide identity or normalized raw alignment score.PASTIS originally relies on Seqan C++ library   for aligIn the evaluation of PASTIS, we follow a similar setting to that of MetaHipMer. The experiments are performed on a single node of Cori GPU Nodes . We usedFigure Aligning two biological sequences is fundamental to a majority of computational pipelines in modern bioinformatics. Applications such as homology, the study of proteins and genes, protein database search, protein clustering, de novo genome sequencing, metagenome analysis and functional genomics and proteomics, frequently make use of sequence alignment algorithms at their core. Optimal sequence alignment algorithms like Smith-Waterman are based on dynamic programming approaches which makes them computationally intensive and difficult to parallelize owing to their convoluted dependencies. With the move of supercomputing facilities towards heterogeneous architectures, a lot of effort is being made to port existing bioinformatics workflows to the new architectures. In this process, sequence alignment algorithms have emerged as a computational as well as an implementation challenge. Existing GPU libraries employ domain and application specific strategies for optimizing an algorithm’s performance on GPUs, but such strategies are not generic and have restricted use-cases. This called for a domain independent strategy that targeted the hardware for speed and not the data, similar to the methods widely available for CPUs.As a way forward through this problem, we have introduced ADEPT in this paper, which is a domain independent strategy for sequence alignment that leverages the GPU’s architecture for accelerating dynamic programming based sequence alignment algorithms. We introduced a novel data structure to tackle the inter-thread dependencies and utilized register-to-register data transfers for efficient communication between CUDA threads. We demonstrated the performance of this strategy by implementing the Smith-Waterman algorithm, an optimal local alignment algorithm, and comparing its performance with similar approaches for CPU SIMD units and the existing fastest GPU implementations available for DNA and Protein alignment. ADEPT has shown performance that either closely matches or is better than existing CPU and GPU methods. We used a variety of real world datasets, ranging from proteins and DNA sequences with varying reference and query lengths, to rigorously evaluate the performance of ADEPT. For DNA and protein use-cases, ADEPT demonstrated a peak speed up of 10x and 7x respectively in node-to-node analysis against CPU libraries, while out-performing or closely matching the performance of existing GPU libraries. We also demonstrated the usability of ADEPT by integrating it into existing bioinformatics software pipelines and demonstrate a 10% and 30% boost in performance of MetaHipmer and PASTIS softwares.ADEPT is a generic strategy and can be easily extended for other dynamic programming-based sequence alignment algorithms such as those used for global and semi-global alignments. We hope ADEPT’s capability of exploiting a GPU’s architecture to provide a unified solution to GPU sequence alignment, its ability to scale across multiple GPUs and ease of use will enable it to take a central role in future high-performance bioinformatics application development and porting.Project Name: ADEPT based implementation of Smith-WatermanProject home page: https://github.com/mgawan/GPU-BSWOperating System: LinuxProgramming Language: C++, CUDAOther requirements: CUDA 10 or laterLicense: OSSAny restrictions to use by non-academics: use the included license


In recent years, there has been an explosive increase in the amount of bioinformatics data produced, but data are not information. The purpose of bioinformatics research is to obtain information with biological significance from large amounts of data. Multiple sequence alignment is widely used in sequence homology detection, protein secondary and tertiary structure prediction, phylogenetic tree analysis, and other fields. Existing research mainly focuses on the specific steps of the algorithm or on specific problems, and there is a lack of high-level abstract domain algorithm frameworks. As a result, multiple sequence alignment algorithms are complex, redundant, and difficult to understand, and it is not easy for users to select the appropriate algorithm, which may lead to computing errors. Here, through in-depth study and analysis of the heuristic multiple sequence alignment algorithm (HMSAA) domain, a domain-feature model and an interactive model of HMSAA components have been established according to the generative programming method. With the support of the PAR (partition and recur) platform, the HMSAA algorithm component library is formalized and a specific alignment algorithm is assembled, thus improving the reliability of algorithm assembly. This work provides a valuable theoretical reference for the applications of other biological sequence analysis algorithms. Since the beginning of the 21st century, with the development of high-throughput sequencing technology, gene sequencing has become much cheaper and more efficient, enabling the development of various genome projects. Since the implementation of the Human Genome Project , the amoSequence alignment is a fundamental method to study biological sequence data in bioinformatics . The theHMSAAs include progressive alignment  and iterAt present, most research on sequence alignment algorithms focuses on the optimization of specific steps of a particular algorithm. The optimization effect on different sequences will be different, and the diversity and complexity of sequence alignment algorithms may make it difficult for users to select an algorithm appropriate to the characteristics of a given sequence, resulting in unnecessary computing errors in practice. On the other hand, it may be difficult for users to understand the structure of a sequence alignment algorithm, which may affect its correct use and to some extent affect the accuracy of the sequence analysis. The specificity and low abstraction of a sequence alignment algorithm reduce its reusability and maintainability. Therefore, it is necessary to study sequence alignment algorithms at the domain level. Concerns on algorithm families will be helpful for extracting the commonality and variability of different algorithms and for the formal development of sequence alignment algorithms.In this work, the generative programming method is used to design an abstract generic algorithm component library, after which a specific alignment algorithm for the HMSAA domain is assembled, thus improving the reliability and reusability of the algorithms. First, domain analysis of HMSAA is carried out, the common domain features and variability features are identified, and a domain feature model of HMSAA is established. Furthermore, relationships among features are analyzed and an interaction model of algorithm components is designed and constructed. Finally, using a generic abstract programming language, Apla, the domain components are formally implemented and a high abstract component library is built on top of Apla.Software reuse is considered to be one of the solutions to the “software crisis.” High-quality software reuse can improve the efficiency and quality of software development and ultimately result in the construction of an industrialization pipeline to develop software. Generative programming  is the uA domain model based on generative programming includes a problem space, a solution space, and domain-specific configuration knowledge for mapping between the two. The problem space is used to represent the requirements of the customization system, and is mainly for use by application programmers and customers. The solution space includes the implementation components required for the system family implementation and the combination, dependencies, and interactions among implementation components. Domain-specific configuration knowledge is mainly used to separate the problem space and solution space, which not only reduces the redundancy and coupling of the implementation components but also improves their composability and reusability. The composition of such a generative domain model is shown in etc., in a software system. Features are the domain knowledge accumulated by users and experts from long-term practice in a domain. Feature modeling is an activity that models the commonality and variability of features and the relationships among them. etc. This was based on service analysis activities, functional analysis activities, and behavioral characteristics analysis in combination with domain terminology analysis, commonality and variability analysis, interactive process analysis, and quality demand analysis concurrently, with continuous retrospective refinement to finally obtain the feature model. The domain modeling process is illustrated in Domain modeling requires the identification and modeling of key concepts . FeaturePAR   Apla subroutine parameterization includes procedure parameterization and function parameterization. In a subroutine, the keywords proc and func are used to declare procedure parameters and function parameters, and the procedure or function is used as a parameter list. 3) As well as the predefined ADT in Apla, users can create custom ADT to make the language more flexible and the program description more powerful. These custom operations include the definition and the implementation of ADT. The ADT definition contains the operation name, the operation type, the parameters of the operation, etc. The ADT implementation gives the specific implementation methods of these operations, and define, ADT, enddef, implement, endimp, and other keywords are used to describe the custom ADT. In addition, the PAR platform supports the transformation of Apla into an executable high-level programming language such as C++ or Java.The generic programming mechanisms supported by Apla include type parameterization, subroutine parameterization, and user-defined generic abstract data types (ADT). 1) Apla introduces the keyword seq_check), heuristic alignment mode selection (heur_mode), pairwise sequence alignment operation (psa), distance matrix (dist_matrix), result output (result_op), progressive alignment , and iterative alignment  are the main functions in the domain. Progressive alignment and iterative alignment are sub-functions of heuristic alignment mode selection. Sequence legality check, heuristic alignment mode selection, and alignment result output are mandatory, where as function, pairwise sequence alignment operation, and distance matrix are optional. For progressive alignments, the progressive alignment mode  is a behavioral feature that has the following three values: the phylogenetic tree (phy_tree), the extended library (expan_lib), and the center alignment. For pairwise sequence alignment operations, the pairwise sequence alignment mode (psa_mode) is a behavioral feature that has two values, fast alignment (k-mer) and dynamic programming alignment (dp). According to the above analysis, a feature model was constructed for the domain, as shown in In this section, the FODM method is used to construct the feature model according to the service, function, and behavior characteristics in the HMSAA domain. Heuristic multiple sequence alignment operations are core services in the domain. The sequence legality check ;2 Penalty component  procedure sometype is a keyword in the Apla language that defines the type variable. GapOpen, GapExtend, and score represent the penalty points of open vacancy, extended vacancy, and non-vacancy, respectively.We designed the penalty model as an ADT, using an affine penalty model, where penaltyMatrix(sometypeelem);  GapOpen : Integer;  GapExtend : Integer;  score:array[array[Integer]];  enddef.3 heur_mode component  define ADT Heur_mode component is defined as an ADT that selects the operation mode of multiple sequence alignment and defines the data structure and information required for alignment. The setPenaltyMatrix, setGapOpen, and setGapExtend functions are to set penalty matrix, open vacancy penalty, and extend vacancy penalty, respectively. The generic procedure tree_prog_align sets the alignment mode to one designated by the user. The useLib means to select the expan-lib component, useFullPW denotes the use of the conventional dynamic programming pairwise sequence alignment algorithm, useIter represents the iteration, and treeAlgorithm is the algorithm to generate the phylogenetic tree.The heur_mode(sometypeelem);function setPenaltyMatrix:Array[Array[Integer]];functionsetGapOpen(gapOpen: Integer): Integer;function setGapExtend(gapExtend:   Integer): Integer;procedureprog_alignenddef.4 dist_matrix componentdefine ADTdist_matrix is defined as an ADT that calculates the distance matrix element and returns it using the score of the pairwise alignment, and the pairwise sequence alignment operation is defined as the generic parameter. The function getDist is used to get the data from the distance matrix. Proc psa is described in detail in reference :Array[Array[Integer]];//.  function getDist : Integer;······enddef;5 phy_tree componentdefine ADT dist_matrix (sometypeelem);  function Phylotree data structure is defined as an ADT, facilitating subsequent operations on the tree. The parameters treeMess, left, and right, respectively, represent the information of the tree node and the left and right subtrees. The phylotree component is defined as an ADT that generates a phylogenetic tree using the data in the distance matrix. The ADT contains the generic procedure generateTree and takes selAlgorithm as its generic parameter. The generic procedure can generate phylogenetic trees through different algorithms. The function calWeight calculates the weight of each sequence when calculating the score of multiple sequences alignment, the generic procedure getStepsForMSA is used to generate the sequence of subsequent multiple sequence alignments, and the generic procedure readTree is used to read information from the generated phylogenetic tree.phyloTree (sometypeelem);  treeMess: Array[Array[Integer]];  left: Integer;  right: Integer;enddef.define ADT phy_tree (sometypeelemMatrix);  procedure generateTree;  function calWeight : Array[Integer];  procedure readTree ;  procedure getStepsForMSA ;······enddef.6 prog_align componentdefine ADT prog_align component is defined as an ADT that includes the generic procedure multiSeqAlign, which performs progressive alignment according to the alignment order obtained from the phylogenetic tree and the sequence weight.The prog_align (sometypeelem); procedure multiSeqAlign ; ······enddef.7 result_op componentdefine ADT result_op component is defined as an ADT. It is composed of two generic procedures, multiAlign_op and phyloTree_op. The multiAlign_op procedure annotates the results of multiple sequence alignments and outputs them; pathAlignOutput is the path of the output file. The phyloTree_op procedure outputs the phylogenetic tree; here, pathTreeOutput is the path of the output file.The result_op(sometypeelem) procedure multiAlign_op ; procedure phyloTree_op ;enddef.define ADT clustalW, is assembled on top of the HMSAA component library introduced in previous section. The Apla program is as follows.In this section, a phylogenetic tree-based progressive alignment algorithm, program clustalW;const/* input sequences*/varseqs, seqsName: Array[String];//Seqs is the sequence to be aligned  //seqsName is the identification name of the sequenceconst pathTreeOutput, pathAlignOutput: String;/*omit the initialization of pairwise sequence alignment*/ADT pm: new penaltyMatrix ;ADT psa: new psa (……);ADT distM: new dist_matrix (psa);ADT phyloTree: new phylotree ;ADT tree: new phy_tree ;ADT msa: new prog_align (tree);ADT mode: new heur_mode (pm);varclustalw: mode; gapOpen, gapExtend: Integer; penalty: Array[Array[Integer]];beginclustalw.setPenaltyMatrix ;clustalw.setGapOpen (gapOpen);clustalw.setGapExtend (gapExtend);end;ADT resultOp: new result_op ;//instantiate and initialize the required componentsprocedure heur_multiple_sequence_align ;//heuristic multiple sequence alignment operationsvarNJTree: String; result: Booleanbegin check (seqs);clustalw.prog_align ;if→if →distM.calDistMat (psa);tree.generateTree ;msa.multiSeqAlign ;resultOp ;end.As the Apla language cannot run directly, in this section we make use of the PAR platform to transform the Apla algorithm components into the corresponding C++ components.struct data types in C++, such as penaltyMatrix and phyloTree. The results are as follows.ADT algorithm components in Apla containing only data members are transformed into struct penaltyMatrix{  int gapOpen;  int gapExtend;  vector < vector < int>> score;};struct phyloTree{  vector < vector < int> > treeMess;  vector < double > leftBranch;  vector < double > rightBranch;};dist_matrix and phy_tree. The function body code is long, and so part of it is omitted here. The partial result of the transformation is shown in ADT components containing data members and member functions are transformed into classes in C++, such as clustalW algorithm program is run through manual assembly of the components, as shown in Generic procedures and functions defined in Apla are converted into separate class member functions in C++ to reduce coupling between components. In particular, the calling functions are converted into indicator functions in C++, and the generic parameter is converted into the pointer parameter to implement the polymorphism of the Apla program. After converting each component into C++, the Apla code for the heuristic multiple sequence alignment operation is converted into the main function executed in C++; finally, the Cyprinus carpio (common carp) alpha-globin, Homo sapiens (human) alpha globin, Mus musculus (house mouse) alpha-globin, Capra hircus (goat) alpha-globin. The alignment results between our algorithm and the other two Clustal algorithms are shown in To test the program, we used four pieces of DNA data, As a key topic in bioinformatics research, sequence alignment algorithm and its applications have attracted extensive research attention. However, there has been no work considering it as a domain for high-level abstraction to improve the reliability and the productivity of the algorithms, and to reduce the probability of suboptimal solutions, errors of the algorithm, etc.Generative programming and the composition of a generative domain model are first briefly presented in the paper, and the FODM method is described. The feature model can be obtained by taking the characteristics of the service, function, and behavior characteristics of the domain into account, and carrying out a series of feature analysis activities.By using generative programming and feature modeling, the HMSAA domain has been analyzed, resulting in the following three algorithm classes: progressive alignment algorithms based on tree and central alignment; progressive alignment algorithms based on the compatible optimization objective function to form an expansion library; and multiple sequence alignment algorithms based on a combination of iteration and increment. Through analysis of this domain, general and variable features have been extracted and mapped to components, and an HMSAA feature model has been established. Moreover, an interaction model of HMSAA domain components has been designed based on the relationships among features and formally implemented using the generic abstract programming language Apla in support of the PAR platform. An HMSAA component library has been established, the reliability of which can be guaranteed owing to the ease of verification with the Apla language.generateTree, provides a diversity of algorithm components assembly as well as a good demonstration of the connections between algorithm features, thus improving the understandability and ease-of-use of algorithms.It is expected that the formal components could be automatically or semi-automatically assembled to generate a specific problem-solving algorithm, thus reducing the errors resulting from manual algorithm selection for multiple sequence alignment, and improving the algorithm efficiency, which will enable assembly of a new, more efficient, multiple sequence alignment algorithm. Furthermore, the high-level abstraction of generic components, such as via the interface and use XML files to describe the composition and constraint relations among components, without any change to the component library. We are encouraged by the success of algorithm assembly on the PAR platform.Next, we will release our codes in GitHub. Future work also include developing a user-friendly visual interface to facilitate component assembly. Users will be able to generate different sequence alignment algorithms by selecting different components The methodology and techniques for HMSAA are not only applicable to multiple sequence alignment algorithms but also have theoretical reference significance and practical application value for other biological sequence analysis algorithms, such as the assembly algorithm based on DeBruijn graph structure used in the process of gene assembly .We are cThe datasets generated for this study are available on request to the corresponding author.HS instructed the whole research work and revised the paper. XZ did the codes work and the experiments. All authors read and approved the final manuscript and are agree to be accountable for all aspects of the work.This work was supported by the National Natural Science Foundation of China under Grant Nos.61662035, 61762049, and 61862033.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Additionally, all protocols are made available with insights into modifications that may improve accuracy at new targets.As sequencing methodologies continue to advance, the availability of protein sequences far outpaces the ability of structure determination. Homology modeling is used to bridge this gap but relies on high-identity templates for accurate model building. G-protein coupled receptors (GPCRs) represent a significant target class for pharmaceutical therapies in which homology modeling could fill the knowledge gap for structure-based drug design. To date, only about 17% of druggable GPCRs have had their structures characterized at atomic resolution. However, modeling of the remaining 83% is hindered by the low sequence identity between receptors. Here we test key inputs in the model building process using GPCRs as a focus to improve the pipeline in two critical ways: Firstly, we use a blended sequence- and structure-based alignment that accounts for structure conservation in loop regions. Secondly, by merging multiple template structures into one comparative model, the best possible template for every region of a target can be used expanding the conformational space sampled in a meaningful way. This optimization allows for accurate modeling of receptors using templates as low as 20% sequence identity, which accounts for nearly the entire druggable space of GPCRs. A model database of all non-odorant GPCRs is made available at  www.rosettagpcr.org) for use in future drug development.Structure-based drug discovery is among the new technologies driving the development of next generation therapeutics. Inherent to this process is the availability of a protein structure for virtual screening. The most heavily drugged protein family, G-protein coupled receptors (GPCRs), suffers from a lack of experimental structures that could hinder drug development. Technical challenges prevent the determination of every protein structure, so we turn to computational modeling to predict the structures of the remaining proteins. Traditional techniques fail due to low sequence identity of available templates. Here, we test improvements to model building and identify changes that can improve accuracy of modeling from templates below 40% identity. We apply our findings to model building of all non-odorant GPCRs to build the RosettaGPCR database ( G-protein coupled receptors (GPCRs) are the largest family of membrane proteins in the human body comprising nearly 800 distinct receptors . They orThe first atomic resolution structure of a GPCR was rhodopsin in 2000, in part due to its high abundance and stability from native sources . For mosGiven this knowledge gap, homology modeling is an important tool for generating models of as-of-yet undetermined protein structures. Homology modeling uses a protein template with a shared topology to map the target sequence onto its backbone coordinates in a process called threading . Early ha priori if the output models will be reliable.While other methods predefine template segments for various parts of the target model or averages template structures, Rosetta handles multiple templates simultaneously during its modeling process . Rosettawww.rosettagpcr.org). While we focus on GPCRs for this benchmark, we emphasize that these modifications can assist in the modeling of any target from templates with low sequence identity.Given the past success of Rosetta in single template homology modeling of GPCRs  and the For this study we chose a subset of 34 crystal structures of GPCRs covering the four structurally characterized classes A, B, C, and F. In total there were 29 Class A receptors, two Class B receptors, two Class C receptors, and one Class F receptor . ImportaInitial alignments for the benchmark set were obtained from the GPCRdb . This laFor all receptors, a pairwise identity matrix covering the transmembrane bundle and loops and excluding long termini was generated using ClustalOmega . The repMembrane spanning topology files were generated by submitting the sequence of the target proteins to Octopus . The outwww.rosettagpcr.org and www.github.com/benderb1/rosettagpcr.Alignment of sequences without known structure was accomplished similarly as above. First, alignments were extracted from GPCRdb. Then the highly conserved residues in each helix were aligned with the template MSA. Positioning of residue x.50 (BW numbering) often corrected helix alignments but gaps and deletions were propagated throughout families. For receptors lacking the highest conserved residue in each helix, other motifs such as DRY, NPxxY, and CxxP were used for helix positioning. The loops were aligned as such: ECL1, ICL1, and ICL2 were aligned using common sequence motifs (eg. xWxxG in ECL1). For ICL3, sequence alignments were maintained within families particularly with receptors with short loops such as the Class B and chemokine receptors. For the majority of receptors, the ICL3 sequence was split at the halfway point between TM5 and TM6 and the halves were adjoined directly to the end of TM5 or beginning of TM6, respectively. For ECL3, particular attention was given to the presence of cysteines for either internal ECL3 disulfides or disulfides between the N-terminus and ECL3. These cysteine residues were used for alignment. For receptors lacking disulfide bonds, patterns identified in template families were used to fix family alignments. Remaining receptors again had the sequence of the loop halved and adjoined the halves to their next helical sequence for peptide insertion overlap. For ECL2, targets were grouped by putative ligand binding type: i.e. aminergic, lipid, peptide, unknown. Based on this grouping, the alignments were carried out specifically for their family type . For receptors with unknown ligand type or dissimilar ligands (i.e. protons), the loop sequence was first divided at the conserved cysteine residue and this residue was aligned generating two shorter loops. These loops were then halved and adjoined to their nearest fixed structural feature, either a transmembrane helix or the conserved disulfide bond with TM 3. The full MSA of all receptors is available at With all input files in hand, target sequences were threaded onto the pre-aligned templates using Rosetta’s partial_thread application . Threadei to i+4 periodicity in every α-helix [Inherent to any homology modeling protocol is an alignment between the sequence of the target protein and the template structure. This alignment maps the target sequence onto the template structure in a process called threading . Sequenc α-helix . Inserti α-helix , they di α-helix . TherefoWe compared this new alignment to other sequence- or structure-based alignment methods. For each receptor in the benchmark, 100 models were generated for each of the six alignment methods tested. The average RMSD for a target protein was divided by the average RMSD for the same target using the new alignment resulting in a fold change and the average across the full benchmark is reported . As seenOur previous benchmark of GPCR modeling relied on single-template threading . We wantWhile peptide insertion helped improve accuracy in the TM and ECL2 regions, overall model accuracy weakened when using a single template with sequence identity less than 40% to the target model. Therefore, we expected that multiple templates could overcome the shortcomings of any single template when modeling a target with low identity templates . We geneAs reported previously for multiple template homology modeling, the use of multiple templates, while improving results over the single template approach, will weaken model accuracy if too many templates are used . Thus, wWe next compared our method with other GPCR modeling servers. Three servers with publicly available databases of GPCR models, GPCRdb , GPCR-I-In our previous work on GPCR modeling using single-template threading, it was found that templates needed to have greater than 50% sequence identity for accurate models . Subsequwww.rosettagpcr.org. This is, to the best of our knowledge, the only Rosetta-based GPCR server available, which distinguishes it from the many Modeller-based servers. While it is true that the Robetta server exists for general homology modeling, it is not designed with a membrane potential and it relies on automatic alignment methods, which we show to be insufficient for low identity templates. The advantage of using a Rosetta-based modeling server is that the models can be used directly in other Rosetta applications like ligand docking, protein-protein docking, and design. Switching models between different programs can result in problems due to intrinsic differences in energy functions. Currently all models are in the inactive state though will likely expand to active state models as more active state structures are determined by cryo-electron microscopy. Additionally, it has been shown that co-modeling ligands into homology models can improve the accuracy of a given model with respect to an experimental structure. We have preliminary data suggesting that to be true for RosettaGPCR as well  by identity for the entire set of non-olfactory human GPCRs . Out of  as well . HoweverAccurate modeling of targets from low-identity templates is a challenge for computational biology. GPCRs represent an unmet need in this area as most GPCRs remain structurally uncharacterized and possess identities on average below 40% to other receptors. Many of these proteins already have FDA approved drugs targeting them , but a dS1 TableReceptor name and the corresponding PDB ID that was used for accuracy measurements.(TIF)Click here for additional data file.S2 TableYellow highlighted templates were not used for general modeling because they have sequence identities greater than 40%. Bolded templates were used for single-template high identity modeling to compare to previous benchmark.(TIF)Click here for additional data file.S1 Figwww.rosettagpcr.org.The alignment for all 34 receptors is shown using Aline . Identic(TIF)Click here for additional data file.S2 FigFor each receptor in the human genome, the best template was identified in the PDB. The sequence identity of the best available is plotted. Most templates cross the 20% threshold identified as critical for accurate modeling. The previous threshold of 40% identity is highlighted in red, and the new 20% identity threshold is highlighted in black.(TIF)Click here for additional data file.S3 FigOne hundred models of the D3 receptor was modeled either with multiple low identity templates (red) or with a single high identity template (green) and the full model RMSDs are plotted as violin plots. As expected, the high identity template yields higher accuracy models on average. Incorporation of a ligand during the modeling process further improve the accuracy compared to the apo state for both multiple low identity templates (cyan) and a single high identity template (purple).(TIF)Click here for additional data file.S1 Protocolwww.github.com/benderb1/rosettagpcr.Step-by-step guide to build models using this pipeline. Input data and scripts available at (DOCX)Click here for additional data file.


The advancement of SMRT technology has unfolded new opportunities of genome analysis with its longer read length and low GC bias. Alignment of the reads to their appropriate positions in the respective reference genome is the first but costliest step of any analysis pipeline based on SMRT sequencing. However, the state-of-the-art aligners often fail to identify distant homologies due to lack of conserved regions, caused by frequent genetic duplication and recombination. Therefore, we developed a novel alignment-free method of sequence mapping that is fast and accurate.Spaced context based Locality Sensitive Hashing. With multiple spaced patterns, S-conLSH facilitates a gapped mapping of noisy long reads to the corresponding target locations of a reference genome. We have examined the performance of the proposed method on 5 different real and simulated datasets. S-conLSH is at least 2 times faster than the recently developed method lordFAST. It achieves a sensitivity of 99%, without using any traditional base-to-base alignment, on human simulated sequence data. By default, S-conLSH provides an alignment-free mapping in PAF format. However, it has an option of generating aligned output as SAM-file, if it is required for any downstream processing.We present a new mapper called S-conLSH that uses spaced-context is especially suitable for extracting distant similarities. The variable-length spaced-seeds or patterns add flexibility to the proposed algorithm by introducing gapped mapping of the noisy long reads. Therefore, S-conLSH may be considered as a prominent direction towards alignment-free sequence analysis.S-conLSH is one of the first alignment-free reference genome mapping tools achieving a high level of sensitivity. The  Minimap2  to n] are the sequences hashed at value x.The reference genome is sliced into overlapping windows, and these windows are hashed into hash tables using suitably designed S-conLSH functions  to generate alignment in SAM format using ksw library (https://github.com/attractivechaos/klib). Some key aspects of S-conLSH are detailed in the following subsections.The algorithm S-conLSH for mapping noisy long reads to the reference genome essentially consists of two steps, reference genome indexing and read mapping. The complete workflow of S-conLSH is provided in Fig. Locality Sensitive Hashing , 18 is ax of length d over some set x[i] denotes the ith symbol of x, and x[i..j] denotes the (contiguous) substring of x from position i to position j. If X, for any For a string The definition of Locality Sensitive Hashing as introduced in , 18 is gLocality Sensitive Hashing) . Here, context factor.) Let alphabet. Let d be integers with X be the set of all length-d words over U be the set of all length-sensitive family X to U is called sensitive, if for each  Let pattern of length match position and ‘0’ represents don’t-care position. Let x be a sequence of length d over alphabet sw over spaced-context ofxwith respect to) Let x with respect to the binary pattern z zeros are present between two successive contexts of 1’s in d be an integer with X be the set of all length-d words over U be the set of all length-sensitive hash function X to U is called sensitive, if for any (L functions corresponding to L different patterns generated using Algorithm 1. Multiple pattern based functions enable gapped-mapping of the reads as illustrated in Fig. Therefore, instead of restricting similarity over the r conLSH , S-conLSentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaentclass1pt{minimaS is the set of all spaced-contexts of length To obtain an integer hash value from the Spaced-context, an encoding function Additional file 1. Supplementary material which contains additional results and instruction manual of the software proposed in the main article.