GASAL2, a GPU library for aligning DNA and RNA sequences that outperforms existing CPU and GPU libraries.Due the computational complexity of sequence alignment algorithms, various accelerated solutions have been proposed to speedup this analysis. NVBIO is the only available GPU library that accelerates sequence alignment of high-throughput NGS data, but has limited performance. In this article we present The GASAL2 library provides specialized, accelerated kernels for local, global and all types of semi-global alignment. Pairwise sequence alignment can be performed with and without traceback. GASAL2 outperforms the fastest CPU-optimized SIMD implementations such as SeqAn and Parasail, as well as NVIDIA’s own GPU-based library known as NVBIO. GASAL2 is unique in performing sequence packing on GPU, which is up to 750x faster than NVBIO. Overall on Geforce GTX 1080 Ti GPU, GASAL2 is up to 21x faster than Parasail on a dual socket hyper-threaded Intel Xeon system with 28 cores and up to 13x faster than NVBIO with a query length of up to 300 bases and 100 bases, respectively. GASAL2 alignment functions are asynchronous/non-blocking and allow full overlap of CPU and GPU execution. The paper shows how to use GASAL2 to accelerate BWA-MEM, speeding up the local alignment by 20x, which gives an overall application speedup of 1.3x vs. CPU with up to 12 threads.The library provides high performance APIs for local, global and semi-global alignment that can be easily integrated into various bioinformatics tools. Many applications for processing NGS sequencing data depend heavily on sequence alignment algorithms to identify similarity between the DNA fragments in the datasets. Well known programs for DNA mapping such as BWA-MEM  and Bowtglobal alignment and local alignment. The former is performed using the Needleman-Wunsch algorithm [Global alignment: In global alignment, also known as end-to-end alignment, the goal is to align the sequences in their entirety while maximizing the alignment score.Semi-global alignment: Unlike global alignment, semi-global alignment finds the overlap between the two sequences, allowing to skip the ends of a sequence without penalty. In semi-global alignment the gaps at the leading or trailing edges of the sequences can be ignored, without inducing any score penalty. Different kinds of semi-global alignments are possible depending on which sequence can have its beginning or end be skipped. GASAL2 supports all kinds of semi-global alignments where any combination of beginning or end of a pair of sequences can be ignored.Local alignment: In local alignment, the goal is to align two sequences so that the alignment score is maximized. As opposed to global alignment, the final alignment may not contain the whole of the sequences. No penalty is induced by misalignments in the beginning and end of the sequences, and the score is kept positive.Sequence alignment is the process of editing two or more sequences using gaps and substitutions such that they closely match each other. It is performed using dynamic programming. There are two types of sequence alignment algorithms for biological sequences: lgorithm  (NW), whlgorithm  (SW) is lgorithm  to use aS1 are not penalized. The alignment score is 7, while the start and end positions of the alignment on S2 are 2 and 10, respectively. For local alignment, the final alignment score is 10. The end-positions of the alignment on S1 and S2 are 12 and 10, respectively. The start-position is 3 on both sequences.Figure CUDA which is an extension of C/C++. The function that executes on the GPU is known as kernel. The data to be processed by the kernel is first copied from the CPU memory into the global memory of the GPU. The CPU (known as the host) then launches the kernel. Once the kernel is finished the results are copied from the global memory back into CPU memory. This copying of data back and forth between host and GPU is quite time expensive. Therefore, data is transferred between the host and GPU in the form of large batches to keep number of transfers at minimum. Moreover, the batch should be large enough to fully utilize the GPU resources.Graphical Processing Units (GPUs) were developed for rendering graphics, but are now being used to accelerate many other applications due to their massively parallel architecture. The GPU architecture varies from one vendor to the other and even across different GPU generations from the same vendor. Here we give a general overview of state-of-the-art NVIDIA GPUs. The cores of a GPU, known as streaming processors (SPs), groups of which are organized into a number of streaming multiprocessors (SMs). Each SM has a set of SPs, a register file, one or more thread schedulers, a read only memory, L1 cache, shared memory, and some other hardware units. All SMs access the DRAM  through a shared L2 cache. The programming language for NVIDIA GPUs is known as warp. A warp is a set of 32 GPU threads that execute in lock-step . Therefore, if one or more threads execute a different instruction, different execution paths are serialized causing performance loss. This phenomenon is known as divergent execution and should be avoided as much as possible. Moreover, to achieve good memory throughput the memory accesses should be coalesced .At every clock cycle each SM executes instructions from a group of threads known as a CUDA streams, it is possible to launch one or more kernels on GPU before the results of a previously launched kernel has been copied back to the CPU. CUDA streams also allow to asynchronously perform the copying operations. Hence, one can just launch all the operations and perform other tasks on the CPU. Subsequently, the cudaStreamQuery API function can be used to test whether all the operations in a given stream have completed or not.To allow the overlapping of GPU and CPU execution, all the GPU kernel launches are asynchronous i.e. control is immediately returned to the CPU after the kernel launch. In this way, the launching thread can perform other tasks instead of waiting for the kernel to complete. Using The sequences to be aligned in NGS processing are generated specifically for each experiment. In contrast, in database searching, the database of sequences is known in advance and may be preprocessed for higher performance.query profile, is not feasible in NGS data alignment.In database search programs, one or few query sequences are aligned against all the sequences in the database (may be regarded as target sequences), whereas the processing of NGS data requires pairwise one-to-one, one-to-many or all-to-all pairwise sequence alignment. Due to this, a common performance improvement technique in database search programs, like using In programs containing GPU accelerated sequence alignment, the alignment step is tightly coupled with the rest of the program. The GPU alignment kernel is specifically tailored to meet the requirements of the program. Therefore, reusing the kernel to accelerate the sequence alignment in other programs is not easy.GPU acceleration of sequence alignment has been the topic of many research papers like –13. AparDue to these differences, GPU accelerated database search cannot be used to accelerate the alignment step in NGS data processing programs. gpu-pairAlign  and GSWAA GPU accelerated DNA/RNA sequence alignment library that can perform global, semi-global  as well as local alignment between pair of sequences. The library can compute the alignment score and the actual alignment between two sequences by performing traceback. The actual alignment is generated in CIGAR format and contains the exact position of matches, mismatches, insertion and deletion in the alignment. Optionally it can compute the alignment score with only the end, and if required, the start position of the alignment.The library uses CUDA streams so that the alignment functions can be called asynchronously and the host CPU can perform other tasks instead of waiting for the alignment to complete on the GPU.GASAL2 is the fastest sequence alignment library for high-throughput Illumina DNA sequence reads in comparison to highly optimized CPU-based libraries, and it is also much faster than NVBIO, NVIDIA’s own GPU library for sequence analysis.GASAL2 can be easily integrated in bioinformatics applications, such as accelerating the seed-extension stage of BWA-MEM read mapper.None of the previously published research efforts have developed any GPU accelerated sequence alignment library that can be easily integrated in other programs that require to perform pairwise alignments. NVBIO  is the opacked into unsigned 32-bit integers. If needed, any number of sequences can then be reverse-complemented. Finally, the alignment is performed and the results are fetched back from the GPU memory to the CPU memory. This section gives an overview of the implementation choices of GASAL2 and describes various stages in the data processing pipeline performed on the GPU.In this paper, we describe GASAL2, a GPU accelerated library for pairwise sequence alignment. The sequences are first transferred to the GPU memory, where they are The user passes the two batches of sequences to be pairwise aligned. A batch is a concatenation of the sequences. Each base is represented in a byte (8-bits). DNA and RNA sequences are made up of only 5 nucleotide bases, A, C, G, T/U (T in case of DNA and U in RNA) and N (unknown base), 3 bits are enough to represent each symbol of a sequence. However, we represent each base in 4 bits for faster packing. Due to the compute bound nature of the GASAL2 alignment kernel, using 3-bits does not result in any significant speedup over the 4-bit representation, but instead complicates the data packing process. Registers in the GPU are 32-bits wide. Therefore, a batch of sequences is packed in an array of 32-bit unsigned integers in which each base is represented by 4 bits. NVBIO also packs the sequences on CPU using 4 bits per base. As the total number of bases in a batch is quite large, packing the data on the CPU is very slow. Figure GASAL2 is able to reverse and/or complement any number of sequences from any batch. Any sequence can be flagged to be reversed, complemented, or reverse-complemented. The reverse-complementing process is performed on the GPU on already packed sequences to take advantage of the high parallelism of the task.H, E and F. The matrices are shown in Fig. O(n2) to O(n). Since, the stored column has only 8 elements it can easily reside in the GPU register file. For ease of representation, Fig. The sequence alignment kernel is launched to perform pairwise alignment of the sequences using affine-gap scoring scheme. GASAL2 employs inter-sequence parallelization and each GPU thread is assigned a pair of sequences to be aligned. All pairs of sequences are independent of the others, so there is no data dependency and all the alignments run in parallel. An alignment algorithm using affine-gap penalties compute cells in three dynamic programming (DP) matrices. These matrices are usually named as direction matrix is stored in the global memory of the GPU while computing the alignment. The direction matrix is similar to the one shown in Fig. S1|×|S2| cells. Each cell is represented by 4-bits in the memory. The lower 2 bits are used to encode whether the current cell is match, mismatch, insertion or deletion. The upper two bits are for the next cell on the alignment path. If the next cell is a gap then the upper bits of the current cell represent whether it is a gap-open or gap-extension, one bit each for insertion and deletion. The direction matrix is stored in the memory using uint4 CUDA vector data type. uint4 has 4 aligned 32-bit unsigned integers. A single store/load instruction is required to access uint4 data from the memory. A single uint4 data element can store 32 direction matrix cells, and hence half the cells in a tile. Moreover, the direction matrices of all the pairs aligned on the GPU are stored in an interleaved fashion for coalesced memory access. The actual alignment is generated using the direction matrix by starting from the end cell and tracing back to the start of the alignment to compute the exact location of matches, mismatches, deletions and insertions.For computing the traceback a The output of this stage depends on the users choice. There are three possible outputs: 1) only score and end-position of the alignment. 2) score, end-position and start-position of the alignment without performing traceback. 3) score, end-position, start-position and actual alignment in CIGAR format.GASAL2 supports various kinds of parameters for kernel launches, to tailor the results to the user’s need. For example, the traceback will only be calculated if the user requests it. In addition, GASAL2 can adapt to any kind of semi-global alignment where the initialization or the search for a maximum can vary, depending on the user requesting the beginning and/or the end of any sequence.if statement slows down the whole kernel dramatically . Duplicating the kernels is not a viable solution for code maintenance: for example, for the semi-global kernel, there are 24=16 types; and adding the possibility of asking for the start-position doubles this number.Dealing with this kind of issue is not trivial in the case of GPU programming, as creating a simple branch through an if statements that are resolved at compilation time, to prune the useless branches.The solution that we adopted allows to generate all the possible kernels at compilation time, so that they are all ready to run at full speed without branches. CUDA implementation of C++ templates (according to C++11 specifications) allows to generate all template-kernels at compile time. The programming model that we adopted allows to create a new kernel specialization by writing blocks, and blocks are grouped into kernel grid. A block is run on a SM that has several hardware resources such as cores, register file, cache, etc. Two parameters characterize the kernel launch: the block size, which is the number of threads in a block.the grid size, which is the total number of blocks.GPU threads are organized in SM occupancy. The SM occupancy is the ratio of number of active warps and the maximum number of warps allowed on a SM. Increasing the occupancy helps in memory-bound applications. Large occupancy makes sure that they are always enough number of warps that are ready to be scheduled to the streaming processors so that all cores (SP’s) in the SM are fully utilized. GASAL2 alignment kernel is not memory-bound. It can compute a 8x8 tile of cells in only 2-3 memory requests. Thus, increasing the occupancy does not help much. However, GASAL2 alignment kernels use a block size of 128 for reasonable occupancy value. GASAL2 uses the inter-sequence parallelization and each GPU thread performs only one alignment. Hence, the grid size is always the ratio of number of alignments to be performed and the block size (128).Block size affects the asynchronous or non-blocking alignment function call as opposed to synchronous or blocking call used in GASAL [GASAL2 allows the user to overlap GPU and CPU execution. This is known as in GASAL . In a blGASAL2 can generate the actual alignment between a pair of sequences by computing traceback. The traceback contains the exact position of matches, mismatches, insertion and deletion in the alignment. This facility is not provided in GASAL.GASAL2 is much faster than NVBIO.Asynchronous execution. This is a unique facility that is not available in NVBIO or GASAL.In NVBIO and GASAL, an ambiguous base (N) is treated as a ordinary base having the same match and mismatch scores as A, C, G or T. But, in most sequence analysis programs, the match/mismatch score of "N" is different. For example, in BWA-MEM the score of aligning "N" against any other base  is always -1. Extending NVBIO to adopt this new scoring scheme to handle "N" increases the execution time of GPU kernels by 30% for global and semi-global alignment, and by 38% for local alignment. In GASAL2 the score of aligning "N" against any other base is configurable. Due to this, the execution time of global, semi-global and local kernels is higher than that of GASAL by 17%, 15% and 6%, respectively.In GASAL, the GPU memory allocations are performed just before the batch of sequences are copied from CPU to GPU. The allocated memory is freed after the alignment is complete and the results are copied from GPU to CPU. If the input batch is not very large, the time spent in memory allocation and de-allocations may become significant and, thus reduces the performance. In GASAL2, we have a separate API function for memory allocation and de-allocation which is called only once at the beginning and end of the program, respectively. At the beginning of the program, user calls the memory allocation function by passing an estimated input batch size. Separate data structures are maintained to keep track of the allocated memory. If the actual input batch is larger, GASAL2 automatically handles the situation by seamlessly allocating more memory. The allocated memory is freed up at the end of the application.GASAL2 supports all types of semi-global alignments. NVBIO and GASAL supports only one type of semi-global alignment in which the gaps at the beginning and end of the query sequence are ignored.GASAL2 can also compute the second-best local alignment score. GASAL only computes the best score.GASAL2 has a reverse-complementing GPU kernel. In NVBIO and GASAL, the user has to manually reverse-complement the sequence before writing it to the input batch.The advantages of GASAL2 over GASAL and NVBIO are listed below: one-to-one pairwise alignments between two set of sequences. We considered the case of DNA read mapping. Read mappers have to perform billions of one-to-one pairwise alignments between short segments of DNA and substrings of the reference genome. In this paper, we also perform one-to-one pairwise alignments between two set of sequences for evaluation purposes. Affine-gap scoring scheme is used in which the match score, mis-match penalty, gap open penalty and gap extension penalty is 6, 4, 11 and 1, respectively. In the rest of the paper,we will refer to the substrings of the reference genome as target sequences. The length of the read sequence is fixed, while the length of the target sequence may vary. Table To evaluate the performance of GASAL2 we performed The CPU-based libraries are executed on a high end machine consisting of two 2.4 GHz Intel Xeon E5-2680 v4 (Broadwell) processors and 192 gigabytes of RAM. Each processor has 14 two-way hyper-threaded cores. Hence, there are 28 physical and 56 logical cores in total. We measured the execution time of the CPU-based libraries with 28 and 56 threads and reported the smallest execution time of the two. GASAL2 and NVBIO are executed on a NVIDIA Geforce GTX 1080 Ti GPU. Only one CPU thread is used in case of GASAL2 and NVBIO. GASAL2 is compiled with CUDA version 10.0.SeqAn contains the vectorized implementation of all types of alignments using SSE4, AVX2 and AVX512 SIMD instructions [chunked execution policy giving the fastest execution for short DNA reads. The chunked policy is also used to generate the results in [ructions . For Seqructions . AVX2 imsults in  for Illusults in  are not ksw module in klib [ in klib  contains in klib . We deveParasail [scan, striped, diagonal and blocked. We have used the scan approach implemented with AVX2 instructions as it is the fastest for our dataset. Parasail does not compute the start-position directly without computing traceback. Therefore, the original sequences are aligned to obtain score and end-position, then both sequences are reversed to calculate the start-position without traceback.Parasail  containsNVBIO contains the GPU implementations of local global and semi-global alignment with and without traceback. Only one type of semi-global alignment is supported shown in Fig. We compared GASAL2 against the fastest CPU and GPU based libraries available, which are: There are also very fast CPU-based libraries that compute the edit distance or sequence alignment with linear-gap penalty e.g. EDlib , BitPAl Table In this section, we compare the performance of GASAL2 and other libraries in terms of the total execution time. The total execution time is the total time required to perform all the one-to-one pairwise alignment between the sequences in the read and target set. Figure S1|×|S2|  and 250 bp (SRR835433) are used. The experiments are run on an NVIDIA Tesla K40c GPU. The GPU host machine has two 2.4GHz Intel Xeon E5-2620 v3 processors and 32 gigabytes of RAM. Each processor has six cores with 2-way hyper-threading. The BWA-MEM version used in this case study is 0.7.13. We also accelerated BWA-MEM using GASAL and compared it with the results obtained with GASAL2. The original GASAL published in [BWA-MEM is a well known ished in  has two sub-batches, numbered 1, 2 and 3. CPU execution is overlapped with the seed extension on GPU. This is achieved via the GASAL2 asynchrnous alignment function call facility. Empty time slots on the CPU timeline are also present in (c), but these are much smaller than (b). These empty slots in (c) will not be present if extension on GPU is faster than post-extension processing or vice-versa. We test both approaches i.e. (b) and (c), to accelerate the extension step of BWA-MEM. In practice, due to load balancing (explained below) we used a batch size that varies between 5000 to 800 reads. The number of sub-batches are either 5 or 4.Figure wasted time as the difference between the finishing times of slowest and the fastest thread. By applying our dynamic batch size approach the wasted time is reduced by 3x for 150bp reads and 2x for 250 bp reads with 12 CPU threads.In the original BWA-MEM, each thread is assigned a number of reads to process and one read is processed by a thread at a time. If a thread has finished processing all of its allocated reads, it will process the remaining reads of unfinished threads. Due to this, all of the threads remain busy until the whole data is processed resulting in maximum CPU utilization. On the other hand, in case of GPU acceleration reads are processed in the form of batches. Therefore, some threads may finish earlier than others and remain idle while waiting for all of the threads to finish. The idle time of these threads causes underutilization of the CPU. Decreasing the batch size helps to increase the CPU utilization, but at the same time may reduce the alignment function speedup due to increased data transfer overhead and poor GPU utilization. To circumvent this problem, we used dynamic batch sizes in our implementation. At the start, the batch size for each CPU thread is 5000 reads, but can be reduced to as low as 800 reads, depending upon the number of free threads which have finished processing there allocated reads. Doing so help to reduce the time wasted by a CPU thread in waiting for other threads to finish. We measured the For 150bp reads, Fig. ideal-total is total execution time for the case in which the time spent in the extension step is zero, and thus, represents the maximum achievable speedup. For 1 to 4 CPU thread, the GPU speedup is almost identical to the ideal one. For higher CPU threads, the speedup is slightly smaller than ideal. For 12 threads, the GASAL2 speedup and ideal speedup are 1.3 and 1.36, respectively. Since the time consumed by the seed extension function in BWA-MEM is 25-27%, the total execution time of GASAL is only slightly higher than GASAL2. For 12 threads, the GASAL speedup is 1.26. The main cause of the difference between ideal and actual speedup for higher number of CPU threads is imperfect load balancing between the CPU threads.Figure Same analysis is repeated for 250 bp reads. Figure Figure In summary GASAL2 gives seed-extension speedup in excess of 10x even when 12 CPU threads share a single NVIDIA Tesla K40c GPU.https://github.com/nahmedraja/GASAL2.In this paper, we presented GASAL2, a high performance and GPU accelerated library, for pairwise sequence alignment of DNA and RNA sequences. The GASAL2 library provides accelerated kernels for local, global as well as semi-global alignment, allowing the computation of the alignment with and without traceback. It can also compute the start position without traceback. In addition, one-to-one as well as all-to-all and one-to-many pairwise alignments can be performed. GASAL2 uses the novel approach of also performing the sequence packing on GPU, which is over 750x faster than the NVBIO approach. GASAL2 alignment functions are asynchronous/non-blocking which allow fully overlapping CPU and GPU execution. GASAL2 can compute all types of semi-global alignments. These represent unique capabilities not available in any earlier GPU sequence alignment library. The paper compared GASAL2’s performance with the fastest CPU-optimized SIMD implementations such as SeqAn, ksw, Parasail and NVBIO . Experimental results performed on the Geforce GTX 1080 Ti GPU show that GASAL2 is up to 5.35x faster than 56 Intel Xeon threads and up to 10x faster than NVBIO with a read length of 100bp, computing only the score and end-position. For 150bp reads, the speedup of GASAL2 over CPU implementations (56 Intel Xeon threads) and NVBIO is up to 4.75x and up to 7x, respectively. With 300bp reads, GASAL2 is up to 3.4x faster than CPU (56 Intel Xeon threads) and NVBIO. The speedup of GASAL2 over CPU implementations (56 Intel Xeon threads) computing start-position without traceback is up to 6x, 5.3x and 4x for 100, 150 and 300bp reads, respectively. With start-position computation, the speedup of GASAL2 over NVBIO is up to 13x, 8.7x and 4.4x for 100, 150 and 300bp reads, respectively. With traceback computation GASAL2 becomes even faster. GASAL2 traceback alignment is 13x and 20x faster than SeqAn and Parasail for read lengths of up to 300 bases. The GPU traceback alignment kernel of GASAL2 is 4x faster than NVBIO’s kernel, giving an overall speedup of 9x, 7x and 5x for 100, 150 and 300bp reads, respectively. GASAL2 is used to accelerate the seed extension function of BWA-MEM DNA read mapper. It is more than 20x faster than the CPU seed extension functions with 12 CPU threads. This allows us to achieve nearly ideal speedup for 150 bp reads. The library provides easy to use APIs to allow integration into various bioinformatics tools. GASAL2 is publicly available and can be downloaded from: Project name: GASAL2- GPU Accelerated Sequence Alignment Library.Project home page:https://github.com/nahmedraja/GASAL2Operating system(s): LinuxProgramming language: C++, CUDAOther requirements: CUDA toolkit version 8 or higher.License: Apache 2.0Any restrictions to use by non-academics: Not applicable


Accurate and fast aligners are required to handle the steadily increasing volume of sequencing data. Here we present an approach allowing performant alignments of short reads (Illumina) as well as long reads , while achieving high accuracy, based on a universal three-stage scheme. It is also suitable for the discovery of insertions and deletions that originate from structural variants. We comprehensively compare our approach to other state-of-the-art aligners in order to confirm its performance with respect to accuracy and runtime. As part of our algorithmic scheme, we introduce two line sweep-based techniques called “strip of consideration” and “seed harmonization”. These techniques represent a replacement for chaining and do not rely on any specially tailored data structures. Additionally, we propose a refined form of seeding on the foundation of the FMD-index. Alignment is an important stage in the analysis of sequencing data. Here, the authors present a fast and accurate alignment approach suitable for long and short reads, and introduce two line sweep-based techniques which can replace the often-used chaining approach. Smith-Waterman’s algorithm4 provides optimal sequence-to-genome alignments on the foundation of dynamic programming. However, due to its high computational costs, this algorithm is not suitable for high-throughput sequence alignment, where a large amount of reads  has to be processed in a reasonable amount of time. For this purpose, several specially tailored aligners have been proposed in recent years10. These aligners aim at getting optimal alignments, but do not guarantee actually obtaining them. They can be categorized according to the read sizes that they are intended for. Short read aligners are meant to align reads from 36 nucleotides (nt) up to a few hundred nt. Long read aligners are meant to align reads of a few hundred nt up to several thousand nt or even up to several ten thousand nt. Reads that exceed 40,000 nt are possible already11. Instances of short read aligners are SOAP212, GEM10, Bowtie 26 and BWA-MEM5. For long read aligners this list can be extended by Minimap29, BLASR7, GraphMap8 and NGMLR13. In an initial step, which is called seeding14, most of these aligners search for short matches  between read and genome. These seeds may be non-consecutive matches17. There are two popular approaches for seeding18. It can either be done using some index as e.g. the FMD-index19 or on the foundation of hash tables. The FMD-index allows the computation of supermaximal exact matches (SMEMs)19. Seeding with hash tables can be achieved in a memory efficient fashion by using minimizers20. A magnitude of techniques has been devised for seed processing that follows the initial seeding. Notable here are the popular chaining23 approach and the line sweep paradigm. Chaining is used in many state-of-the-art aligners as e.g. BWA-MEM5, Minimap29 and Bowtie 26, while the line sweep paradigm is exploited in e.g. Mashmap24 for the computation of homology maps between genomes.Within computational genomics, there are various types of alignment problems that can be classified as shown in Fig. 6, BWA-MEM5, Minimap29 and BLASR7, rely on sequence-to-sequence alignments for filling gaps or for alignment extension purposes. As for sequence-to-genome alignments, it is possible to rely on dynamic programming25 for finding optimal alignments here. But, such optimal alignments are computationally expensive.Sequence-to-genome alignment is next to sequence-to-sequence alignment in Fig. 26. Divergent homology detection tries to identify statistically significant matches to a query sequence in a database of reference sequences31.Aside of pairwise alignments, there are two additional subcategories within the family of sequence-to-sequence alignments: With multiple sequence alignments more than two sequences are aligned simultaneouslyhttps://github.com/ITBE-Lab/MA.Here we introduce an algorithmic scheme for efficiently performing sequence-to-genome alignments for read sizes ranging from 100 nt up to several ten thousand nt. The proposed scheme allows high throughput alignments and shows an excellent overall accuracy compared to other state-of-the-art aligners as reported in the results section. Particularly, for alignments comprising many insertions and deletions our approach shows superiority compared to others. As part of our scheme, we propose a refined form of seeding using the FMD-index. This refined form of seeding relies on a divide-and-conquer technique and computes seeds of higher relevance than SMEMs as mentioned in the discussion. Further, we propose two line-sweep-based techniques called strip of consideration (SoC) and seed harmonization for quickly identifying promising positions on the genome and efficiently computing sets of consistent seeds. In combined form, these two techniques represent a drop-in replacement for chaining. Further, our approach does not rely on specially tailored data structures and it can be described concisely in pseudocode. Using generated reads, we comprehensively compare several state-of-the-art aligners with each other and with our approach. An open source application called MA (Modular Aligner) implements our approach. MA has a modular design and is publically available via GitHub at 19, which represents a full-text substring index. In stage 2, the seeds obtained in stage 1 are filtered and harmonized, which results in a consistent and spatially local set of seeds. Remaining gaps between seeds are filled using DP25 in stage 3. Apart from the FMD-index, our approach does not require any specially tailored data structures.Our approach performs alignments by completing the following three stages: (1) Seeding, (2) seed processing and (3) dynamic programming (DP). Seeding finds perfectly matching substrings between a reference and a query. We do this using the FMD-index32. Our seeding follows a divide and conquer approach that starts at the query center. By applying maximal extensions, a query section  is decomposed into three intervals, where the central interval is defined by two or more maximally spanning seeds. These maximally spanning seeds are computed in a two-step process. First, we maximally extend forwards and then we maximally extend backwards. By doing so, we obtain one or more seeds that all cover the same query interval. Then, we apply the opposite order of maximal extensions for getting one or more additional seeds. The intervals remaining on the left and right are processed recursively. This approach creates less irrelevant seeds than other seeding approaches19 as reported in the discussion.Stage 1: Seeding on the foundation of an FMD-index requires an extension process33 but our approach does not rely on this technique. Instead, our approach does seed processing in two steps called strip of consideration (SoC) and seed harmonization. The SoC identifies best-scored sets of spatially local seeds. Although not algorithmically realized this way, the computation of SoCs can be imagined as follows: By sliding a fixed size window along the reference, we search for the best-scored window positions, where the score of a position is the accumulative length of all seeds within the window on that position. The overall time complexity of the SoC computational is limited by the complexity of an initial seed sorting. If the index used for seed generation is able to deliver the seeds in correct order, then the SoC can be computed in a single pass in linear time.Stage 2: Informally, two seeds are consistent, if they appear in the same order on reference and query. Further, two seeds are spatially local, if the gap between them is small enough so that an alignment comprising both seeds can have a positive score. In stage 2, sets of consistent spatially local seeds are computed in order to get alignments with optimal scores. We call this stage seed processing. Seed processing can be done by chainings and s' must fulfill one of the following two properties exclusively: Either, the query and reference start positions of s are smaller than the corresponding start positions of s′, or, the query and reference end positions of s are larger than the corresponding end positions of s′. In the case of a pair of inconsistent seeds, we purge one of them, where we rely on the following decision principle: Using a heuristic approach, we define a line, called δ-guideline that roughly approximates the expected shape of the alignment. The seed showing the larger distance to this line is purged, while the closer seed is kept.Seed harmonization computes a consistent subset of a SoC-seed set. As opposed to chaining, seed harmonization represents a purging technique. In order to be consistent, two seeds Theoretically, our purging approach could have a squared complexity behavior. However, it requires highly specialized examples to obtain such a worst-case behavior. Practically, the runtimes are driven by an initial seed sorting that is required as part of the harmonization.34. They are all controlled by some scoring scheme. Like other long read aligners7, we make use of banded DP36. We primarily use it for filling small gaps occurring between the seeds of a harmonized SoC-seed set. Due to this limited use of DP, its overall computational penalty is proportional to the dissimilarity of query and reference. So, in the case of high seed coverage DP costs become low.Stage 3: For computing optimal alignments, there are several DP algorithms9, NGMLR13, GraphMap8 and Blasr7 for long reads as well as BWA-MEM5 and Bowtie 26 for short reads. The lack of ground truth with real-world data prevents the direct inference of an alignment’s accuracy. Hence, our evaluation relies on simulated reads. For creating Illumina reads, we use DWGSIM37. Pacific Biosciences (PacBio) reads and Ultralong Oxford Nanopore (UON) reads are generated from sampled error profiles. These error profiles are computed by evaluating alignments obtained via Minimap2 for the datasets PacBio-MtSinai-NIST and Ultralong Oxford Nanopore (UON) of HG002 (AJ Son)38. Our approach resembles the technique used by SURVIVOR13 and is comprehensively described in Supplementary Note In the following, we compare MA with several other aligners for short and long reads, where we analyze Minimap2We now discuss the behavior of all inspected aligners. For long reads (PacBio and UON), all aligners deliver a similar high accuracy within the left third of the diagrams. Then, the accuracy deteriorates with different speeds, where MA is only surpassed by GraphMap. Further, there are significant differences with respect to the runtime, where Minimap2 and MA belong to a more performant group than GraphMap, Blasr and NGMLR.The read distributions with PacBio and UON (as shown by the bar diagrams) indicate the existence of notable amounts of reads within the range of two times up to three times of the mean error. Therefore, the different accuracy levels of the inspected aligners within this range are relevant to their assessment. For all generated reads, we assume that the location of the genome section used as template for the read’s generation is the read’s only correct position. If a template is chosen from an ambiguous section of the reference genome, an aligner can deliver an alignment for an alternative position of this ambiguous section. Such different positions are assessed as wrong positions, because they are unknown during read generation. This explains the inability of all aligners to reach 100% accuracy for perfect reads for all analyzed datasets.5 and visualizes the behavior of aligners depending on the mapping quality. The figure shows that MA’s mapping behavior is superior to the behavior of Bowtie 2 but inferior to that of BWA-MEM. However, MA is roughly twice as fast as BWA-MEM and three times faster than Bowtie 2. Additional information can be found in Supplementary Note An evaluation for short reads (paired Illumina reads) is given in Fig. 39. MA can perform long read alignments while still having the footprint of a short read aligner.Table 13, where our accuracy categorization is similar, but our measurement approach is different . Despite these differences, both sets of diagrams confirm a low level of precise SV (indel) detection for GraphMap and Blasr. Within the category “precise” of Sedlazeck et al.13, we distinguish a subcategory “split”, where the aligner finds accurate end-points of the SV but reports the SV as chimeric alignment . The justification for this additional category is the indirect indication of the SV in contrast to the direct indication with “precise”. MA recognizes SV via the SoC. Therefore, it produces very few “split” alignments in contrast to Minimap2 and NGMLR. This behavior is advantageous in ambiguous regions as indicated in the discussion. All aligners show a similar decreasing level in SV detection for increasing SV sizes. This is to be expected because with increasing SV size an increasing portion of the read is not related to the reference genome anymore.Now we additionally analyze MA with respect to the mapping of two types of structural variants: insertions and deletions. For this purpose, we generate reads on the foundation of the sampled UON mean error and inject exactly one SV into each generated read. According to an aligner’s CIGAR for a generated read, we distinguish among six categories of SV detection accuracy: Precise: The aligner reports the accurate SV position. Split: The SV is reported accurately using two alignments. Indicated: The positon of the SV is misplaced more than 10nt. Forced: The SV is delivered in fragmented form. Trimmed: Reads are clipped near the SV. Unaligned: There is no alignment close to the SV position. A detailed explanation of these categories is given in Supplementary Note 19 as shown by the runtimes for UON reads in Fig. One highlight of our work is the proposed technique for computing sets of consistent spatially local seeds. We compute these sets using the SoC followed by seed harmonization. The efficiency of this backend allows performant alignments of long reads using SMEMsSome aligners, e.g., GraphMap, rely on a bucketing technique instead of the SoC. Bucketing techniques have several disadvantages compared to our approach: Bucket-boundary crossing alignments require special consideration. If the size of all buckets is fix, the number of buckets is proportional to the genome size. Therefore, the costs (runtime and memory) for the discovery of the best buckets become proportional to the genome size.23. In the following, we will compare our technique to chaining.Another replacement for the SoC is chaining, which has received a lot of attention in research already9, but for the price of a squared worst-case complexity. Alternatively, it can be done on the foundation of special tailored data structures in time O (n log n log log n) (with gap-costs21) or O (n log n) (without gap-costs23). However, the latter solutions rely on complex data structures and, to our knowledge, they have not been integrated into any aligner so far. Our technique does not require sophisticated data structures and is, due to its simplicity, highly performant. The time complexity of our SoC computation is driven by the initial sorting of all seeds. For specially constructed examples, the seed harmonization can show a squared worst-case complexity. On average, however, the seed harmonization can be expected to be completed in time O (n log n), where n is the number of seeds.All chaining techniques share the property that they sequentially grow chains of seeds. This can be done without sophisticated data structures22 cannot cope well with overlapping seeds, because they cannot concatenate them. The following example shall explain the severity of this limitation: Let the string AATGG be the query and let the string AATCTGG be the reference. There are two non-enclosed  seeds (>1nt) on the reference T on either side of the C. Such overlaps occur with a probability of 7/16 for seeds that are separated by indels . Hence, such cases should be quite common on nucleotide sequences. Chaining has to drop one of these seeds, because a concatenation is impossible. With our approach, the seed harmonization keeps both seeds. The overlap among them does not cause any trouble.The chaining approaches with gap costs21 or accumulative seed lengths23). Figure δ-guideline) delivers global information about the expected overall alignment path. In our approach, we pick the seed closer to the δ-guideline. This should be advantageous, because an alignment via s allows more substitutions of gaps by matches or mismatches than an alignment via s′.In the case of pairs of contradicting seeds, one seed has to be chosen in favor of the other seed or both have to be purged. Chaining has to perform the required decision on the foundation of the local context. Several strategies have been proposed for this purpose Alignments in repetitive regions of a genome demand for long seeds because short seeds can show unmanageable amounts of repetitive occurrences in these regions . Let S be a set of seeds and let S′ ⊆ S be the set of relevant seeds in S. We define the relevance rate of S as the ratio |S′|/|S|. For simulated UON reads (mean error sampled from HG002 (AJ Son) by Zook et al.38), maximally spanning seeds deliver a relevance rate of 4.4% , while for SMEMs the rate is 1.5% merely. With simulated PacBio reads, the relevance rates are 5.0%  and 1.7% (SMEMs).Maximally spanning seeds constitute a subset of SMEMsWe inspect MA with respect to insertions and deletions that represent SVs. Apart from indels, there are other forms of structural variants as e.g. duplications, translocations and inversions. Currently, MA does not distinguish between insertions and duplications; it always reports insertions. Translocations can be discovered via SoCs, if the translocated segment forms a separated SoC of sufficient score. Such translocations are reported via supplementary alignments. In the context of the SoC, sufficiently sized inversions resemble translocations between forward and reverse strand. Therefore, MA can report sufficiently sized inversions via supplementary alignments like translocations. Our SV analysis for insertions and deletions comprises a category “split”. “Split” alignments report the SV accurately but indicate that an aligner discovers both ends separately. However, if one of both ends originates from an ambiguous region, the ambiguity can prevent the SV discovery. SoC based seed processing avoids such splitting and so it helps with the precise discovery of SVs in such cases.Overall, the proposed algorithmic approach allows a modular application architecture as shown by our implementation (see Code availability section). MA’s codebase consists of individual modules, realized in C++, which are coupled using a dependency graph for getting a functional aligner. The module coupling can be achieved via Python or C++ itself. MA comprises a graphical user interface application as well as a command line application that are both realized via a C++ based module coupling. The graph-based design allows modules to be interchanged seamlessly, which boosts flexibility and supports experimental work with alternative aligner design. Therefore, MA can be perceived as a general framework for aligner design and construction.A| shall denote the alignment length. With this plane, the reference R is on the x-axis and the query Q is on the y-axis, with |Q| < |R|. Please note: The sequences r0, r1, …, rA|| and q0, q1, …, qA|| are both monotonically increasing. Two consecutive points  and  are always associated in one of the following four ways:rn+1 = rn + 1, qn+1 = qn; visually this is equal to a horizontal path extension.Deletion: rn+1 = rn, qn+1 = qn + 1; visually this is equal to a vertical path extension.Insertion: rn+1 = rn + 1, qn+1 = qn + 1; visually both appear as diagonal path extensions.Match or Mismatch: We consider alignments as a path that follows a sequence of points For example, in Fig. The alignment process is decomposed into three separated stages: seeding, seed processing and dynamic programming (DP). In the following three sections, we will explain our approaches for all three stages in detail.q, r, l), where q and r are the starting positions on query and reference, respectively l denotes the length of the seed 17.Seeding computes a set of seeds, where a seed is a perfectly matching section between query and reference. Each seed is represented as a triple , OR,c containing the positions of all occurrences of c in R. The position of a string is the index of its first character. Given a set OR,s of all positions of the string s in R and a single character c, OR,c⊕s, where c ⊕ s denotes the string concatenation of c and s. Similarly, OR,s⊕c. We assume that all three operations are performed in time O(1). All OR,s shall be represented implicitly  so that runtime does not depend on the size |OR,s|. Further, the size |OR,s| must be obtainable in constant time. Our approach is built on the foundation of FMD-indices19, but it is possible to use other index-models  as well. Our code for the FMD-index as well as the pack management is imported from the code of BWA-MEM5 and adapted to our needs. FMD-indices are based on FM-indices, which form a space-efficient family of indices40. FMD-indices are optimized for searching on genome data, where the forward strand and its reverse complement can be expressed in terms of each other. FMD-indices allow extending on both strands simultaneously19. Further, because any sequence and its reverse complement have suffix arrays of equal size, FMD-indices support forward extensions by interval decomposition.In order to refrain from scanning the reference, our approach relies on a full-text substring indexS is fully covering if and only if there is a seed  ∈ S for each position i on the query (0 ≤ i < |Q|), so that q ≤ i < q + l. If the reference alphabet ∑R is a proper subset of the query alphabet ∑Q, it is impossible to compute a fully covering set of seeds. So, we assume A set of seeds q, r, l) is non-enclosed with respect to a set of seeds S if and only if there is no other seed  ∈ S with q′ ≤ q < q + l ≤ q′ + l′.A seed  in S we have q ≤ i < q + l and it is not possible to construct a seed  with q′ ≤ i < q′ + l′, so that q′ < S.begin or q′ + l′ > S.end.We now give two definitions that formalize the outmost query positions reached by a set of seeds Because our seeding technique relies on an implicit seed representation, it cannot deliberately discover seeds with specific positional properties on the reference. For this reason, the following three properties (1) fully covering, (2) non-enclosed and (3) maximally spanning are defined on the foundation of query positions.i, computes a set of maximally spanning non-enclosed seeds, where each seed covers i. Based on the resulting seeds, we then compute a fully covering set of seeds.We now propose an algorithm that, for a given query position Maximally spanning seeds. We call a repeated extension by i and continuing while i (0 ≤ i < |Q|). Starting from position i, we first maximally extend backwards and obtain a set of seeds S. Then, we extend S maximally forwards. S can comprise one or multiple seeds, where all seeds in S have identical query intervals. By starting from position i again, but using the opposite order of extensions , we get a second set of seeds S′. We return the union S ∪ S′ as outcome of the extension process. The pseudocode for this algorithmic approach is given by the procedure MAXIMALLY—SPANNING  in Supplementary Note MAXIMALLY—SPANNING always delivers a non-empty set of seeds. We now prove that the above extension process always finds maximally spanning seeds with respect to a given start position i.S' be the set of all seeds  with respect to a query Q and a reference R that span over a given query position i (0 ≤ q′ ≤ i < q′ + l′ ≤ |Q|). Let S be the set of seeds computed by MAXIMALLY—SPANNING. There is no seed  ∈ S′, so that q′ < S.begin or q′ + l′ > S.end.Let Proof: Assume, there is a seed  ∈ S′, so that q′ < S.begin. This poses a contradiction to the definition of the maximal backwards extension that continues while there is a non-empty set of matches returned by the extension. Likewise, there cannot be a seed with q′ + l′ > S.end. ■19. BWA-MEM then uses this algorithm in order to compute a fully covering set of seeds. These seeds are called supermaximal exact matches (SMEMs), where a seed is supermaximal extended if and only if it is non-enclosed. Our maximally spanning seeds represent a subset of SMEMs. We now explain which unrelated  seeds are non-enclosed but not maximally spanning. These unrelated seeds can be recognized by their query location as follows: They are completely covered by two or more longer non-enclosed seeds. Supplementary Fig. Similarly to our approach, BWA-MEM uses a seeding algorithm that initially finds non-enclosed seeds covering a given query positionBinary seeding. Binary seeding computes a fully covering set of seeds. In each step, binary seeding decomposes a given query interval iC comprises the query area covered by the seed-set that originates from position MAXIMALLY—SPANNING. Binary seeding continues recursively on non-empty iL and iR. The pseudocode for binary seeding can be found in Supplementary Note Q and reference index IR.Binary seeding terminates for any given query Proof: Let iQ. Let S be the set of seeds received by the call MAXIMALLY—SPANNING. All seeds in S overlap m and are of size 1 at least. Thus, we have S.begin ≤ m and S.end > m. We decompose iQ into p1 = S.begin and p2 = S.end. Since p2 − p1 > 0, we get (p1 − b) + (e − p2) < e − b. Hence, the size of iL and iR is smaller than the size of iQ. Calls for size zero return immediately. ■Seed processing performs the collection of a set of consistent seeds with the intention of acquiring an optimal alignment. It comprises two major steps: Strip of consideration (SoC) and seed harmonization. A SoC comprises a set of seeds that cover a candidate region for an optimal alignment. Seed harmonization erases inconsistent seeds within a SoC.Strip of consideration (SoC). Informally, two seed that can participate in an alignment with a positive score are called spatially local. In order to define spatial locality formally, we require the SoC width λ, which is defined as follows:sM be the score for a match and let pO, pE be the penalties for opening and extending a gap, respectively. Further, let |Q| denote the length of a given query Q. The width of a SoC, denoted by λ, is computed as follows:Let λ can be found in Supplementary Note λ with respect to DP, the parameters sM, pO and pE should be equal to the respective parameters in DP.The derivation of q, r, l) and  are considered spatially local if and only if |δ − δ′| ≤ λ, where δ: = r − q and δ′: = r′ − q′ . Measuring the distances of seeds using their δ-values is done in other aligners (as e.g. GraphMap8) as well.Two seeds  in AS and the score  are stored in a stack. This stack is finally transformed into a priority queue44, where the order is defined by the SoC scores. This priority queue is then forwarded to the seed harmonization step. The pseudocode for the SoC computation can be found in Supplementary Note We now explain our algorithmic approach for computing SoCs, which is related to techniques used in the context of convolutional filters in image processing. We assume that all seeds are stored as triples in an array Seed harmonization. Seed harmonization computes a consistent subset of SoC seeds by applying a guided purging of contradicting seeds. Two seeds are consistent if they occur in the same order on query and reference. All seed in an alignment must be mutually consistent. For determining consistency, we introduce the concept of shadows for a given seed s: Informally, each seed is characterized by its start point and its end point in the two-dimensional plane spanned by query and reference . If we imagine a centered spotlight shining towards the seed, a shadow is created, where the outer limits of the shadow are determined by the start and end-points defining the seed. A seed completely within the shadow of s cannot be part of any consistent alignment comprising s. Note, that a partially shadowed seed is still consistent. Further, the spotlight can be placed on either side of s. This results in two shadows, one in its second and one in its fourth quadrant. We denote these shadows by σII and σIV, respectively. The indexing scheme for shadows was chosen in accordance with the quadrant enumeration using Roman numerals in plane geometry. Figure σII and σIV are partially overlapping.Note, by using this definition δ-guideline on the foundation of all seeds in a SoC. The δ-guideline forms a decision tool during the purging of conflicting seeds. For the computation of the δ-guideline, we create a point cloud PSoC in the plane spanned by query and reference. PSoC comprises the start point, center point and end point of all seeds of a SoC. Using random sample consensus (RANSAC)45, the angle and position of the δ-guideline is computed on the foundation of the point cloud PSoC. For a pair of inconsistent seeds, we measure their proximity to the δ-guideline and delete the more distant one.In the context of the seed harmonization, we compute a δ-guideline based removal happens according to the following algorithmic scheme: Let σII (σIV) and s =  and s′ = , respectively. Our approach is based on the observation that s′ is within σII if and only if σII. Accordingly, s′ is within σIV if and only if σIV. σII is determined by (a) q ≤ q′ and r′ + l′ ≤ r + l; σIV is determined by (b) r ≤ r′ and q′ + l′ ≤ q + l. The seeds s and s′ are conflicting if and only if σII or σIV. As described above, we keep the seed closer to the δ-guideline and remove the other.In detail, the (a) and (b) represent instances of the interval-interval-inclusion problem. We call our problem interval-interval-inclusion problem for avoiding confusion with the notion interval-inclusion problem. The latter one is usually used in the context of point inclusion decisions. However, we will do enclosure decisions with respect to complete intervals here. The enclosure of complete intervals can be efficiently solved using the line sweep paradigm. Using an example, Fig. O(|S|2), where |S| is the number of seeds in a SoC. More detailed, the time complexity can be expressed as O(|S|⋅ξ + |S|) with 0 ≤ ξ ≤ |S|, where ξ is smaller than or equal to the number of contradictions of the seed with the most contradictions. Practically, however, we observed that the initial sorting of the harmonization is the driving runtime factor.The asymptotic worst-case complexity of the seed harmonization is 25 with affine gap penalties). This represents a standard approach used by other aligners as well9. Further, the first seed might not touch the query start and the last seed might not extend to the query end. In this case, we rely on an adaptive banded semi global dynamic programming approach. Practically, we relied on ksw2  for filling gaps and libGaba  for performing semi global extensions.There can still be gaps between seeds after harmonization. In order to fill these gaps, we rely on a bandwidth-limited variant of dynamic programming  There are two user defined thresholds minSeedSize and maxAmbiguity. All seeds with a size below minSeedSize are removed. Further, if there are more than maxAmbiguity many seeds with identical query intervals, we purge them all. (3) During SoC computation, we purge SoCs using two user defined thresholds γ and ξ. γ represents an imaginary minimum seed length and ξ (0 < ξ ≤ 1) is a scalar factor with respect to the query length. For a query Q, we purge all SoCs with a score (accumulated seed length) below γ and ξ · |Q|). Whenever we complete the harmonization of a SoC, we apply the above filter to the altered SoC. (4) The harmonization of consecutive SoCs happens according to the order given by the extraction from the priority queue. Depending on the query length, we apply two different heuristics, where the first one represents a filtering technique and the second one a stop-criteria. For long queries , we always inspect a predetermined number of SoCs and discard SoCs that get a lower score than preceding SoCs. For shorter queries, we trace equality using a look-ahead approach. If we observe an unaltered score for more than a given threshold number of times, we break and move forward to DP. The latter two heuristics represent the outcome of an observational optimization. Improvements with respect to the filtering of harmonized SoC could be a valuable topic in the context of further research.Our approach relies on several heuristic optimizations for getting an improved runtime behavior. (1) After seeding, we probe the periodic occurrence of sufficiently sized seeds and give up on failing queries. Our approach is accumulative here. Formally, we requireSupplementary InformationPeer Review File


Sequence alignment is crucial in genomics studies. However, optimal multiple sequence alignment (MSA) is NP-hard. Thus, modern MSA methods employ progressive heuristics, breaking the problem into a series of pairwise alignments guided by a phylogeny. Changes between homologous characters are typically modelled by a Markov substitution model. In contrast, the dynamics of indels are not modelled explicitly, because the computation of the marginal likelihood under such models has exponential time complexity in the number of taxa. But the failure to model indel evolution may lead to artificially short alignments due to biased indel placement, inconsistent with phylogenetic relationship.Recently, the classical indel model TKF91 was modified to describe indel evolution on a phylogeny via a Poisson process, termed PIP. PIP allows to compute the joint marginal probability of an MSA and a tree in linear time. We present a new dynamic programming algorithm to align two MSAs –represented by the underlying homology paths– by full maximum likelihood under PIP in polynomial time, and apply it progressively along a guide tree. We have corroborated the correctness of our method by simulation, and compared it with competitive methods on an illustrative real dataset.https://github.com/acg-team/ProPIP. Supplementary data are available at BMC Bioinformatics online.Our MSA method is the first polynomial time progressive aligner with a rigorous mathematical formulation of indel evolution. The new method infers phylogenetically meaningful gap patterns alternative to the popular PRANK, while producing alignments of similar length. Moreover, the inferred gap patterns agree with what was predicted qualitatively by previous studies. The algorithm is implemented in a standalone C++ program: The online version of this article (10.1186/s12859-018-2357-1) contains supplementary material, which is available to authorized users. Multiple sequence alignments (MSAs) are routinely required in the early stages of comparative and evolutionary genomics studies. Not surprisingly, accuracy of MSA inference affects subsequent analyses that rely on MSA estimates . MSA estAll state-of-the-art MSA programs nowadays use an evolutionary model to describe changes between homologous characters, providing a more realistic description of molecular data and thus more accurate inferences. However, a mathematical formulation of the insertion-deletion (indel) process still remains a critical issue. Describing the indel process in probabilistic terms is more challenging: unlike substitutions, indels often involve several sites, vary in length and may overlap obscuring the underlying mechanisms. Instead, the popular PRANK program adopts a pragmatic approach; it uses an outgroup to distinguish insertions from deletions during the progressive alignment procedure, so that insertions are not over-penalized . As a reA recent modification of TKF91 describes the evolution of indels on a phylogenetic tree as a Poisson process, thus dubbed the Poisson indel process or the PIP model . The indHere we propose a new progressive algorithm to estimate an MSA under the explicit model of substitutions and indels. We have re-framed the original PIP equations into a dynamic programming (DP) approach. It aligns two MSAs –represented by their homology paths on the two corresponding subtrees– by maximum likelihood (ML) in polynomial time. The progressive algorithm traverses a guide tree in postorder; at each internal node the DP is applied to align the two sub-alignments at the child nodes. The procedure terminates at the root of the guide tree, with the complete MSA and the corresponding likelihood, which by construction is the likelihood under the PIP model. We have implemented the progressive MSA algorithm in a prototype program and verified its correctness by simulation. To our knowledge, this is the first progressive MSA algorithm with polynomial time complexity, using a mathematical formulation of an explicit indel process. Note that an equivalent formulation under TKF91 or TKF92 –i.e. using the full marginal likelihood along the subtrees in question– would have exponential time complexity. Quadratic time complexity under the TKF models could be obtained  by repreThe remainder of this manuscript is organized as follows. We first introduce notation and the PIP model. Then, we describe our DP algorithm and provide the simulation results. We conclude the paper with an illustrative real dataset, where we contrast our method with PRANK, as well as with MAFFT, representing a state of the art similarity based progressive method.N leaves. τ is a directed, connected, labelled acyclic graph, with a finite set of branching points N observed taxa, represented by strings of characters from a finite alphabet Σ . There are N−1 internal vertices Ω is the most recent common ancestor of all leaves. Branch length b(v) associated with node v to its parent node pa (v). The total tree length ∥τ∥ is a sum of all the branch lengths.Let τ. We denote the distance from the root to a given point on the tree, by the same symbol τ. Atomic insertions are Poisson events with rate measure ν(dt)=λ(τ(dt)+μ−1δΩ(dt)), where λ is the insertion rate, μ the deletion rate, and δΩ(·) Dirac’s delta function. This formulation guarantees that the expected sequence length remains constant during the whole evolutionary process. Point substitutions and deletions are modelled by a continuous-time Markov process on Σε=Σ∪{ε}, where ε is the deletion symbol. Accordingly, the generator matrix Qε of the combined substitution and indel process extends the instantaneous substitution rate matrix Q by a row and a column to include ε, which is modelled as an absorbing state as there can be no substitutions after a deletion event. The quasi-stationary distribution of Qε is denoted by πε. Root Ω has a virtual infinite length stem, reflecting the equilibrium steady state distribution at the root.The PIP model describes a string-valued evolutionary process along the branches of v, the probability ι(v) of inserting a single character on branch pa (v)→v, is proportional to branch length b(v). For v≠Ω it is given by ι(v)=b(v)/(∥τ∥+μ−1); at the root atomic mass point probability ι(Ω)=μ−1/(∥τ∥+μ−1) so that β(v) associated with an inserted character on branch pa (v)→v is given by β(Ω)=1 and β(v)=(1− exp(−μb(v)))/(μb(v)).For an internal node pτ(m) of MSA m of length |m| is computable in O(N·|m|) and can be expressed as The marginal likelihood p(c) is the likelihood of a single column c, and p(c∅) is the likelihood of an unobservable character history, represented by a column c∅ with a gap at every leaf. The factor in  using a variant of Felsenstein’s peeling recursion  is the indicator function. In Eq. β(v) accounts for the probability that the inserted character does not survive till the first node below the insertion point. The recursive function and v, our DP algorithm proceeds to align the two sub-alignments obtained in the left and right sub-trees maximizing the likelihood  ,|m|) is non-monotonic  cannot be maximised by means of a standard two-dimensional DP approach . Similarly to TKF91 )Below we provide results from some basic evaluations of our proposed method.λ,μ)∈{,,,}. The resulting sequence lengths varied between 5 and 8 nucleotides.To test the correctness of the algorithm and implementation, we generated data under PIP using a simulator provided by the authors of PIP. We chose relatively small trees and short sequences to be able to perform analytical tests during algorithm design and program debugging. Specifically, we simulated 120 datasets in total, on trees with 4, 5, 6 and 7 leaves, and using the following parameter combinations . These Using our new algorithm we inferred an MSA for a challenging dataset, 23 envelope glycoprotein gp120 sequences from HIV/SIV, previous analysed by Löytynoja and Goldman . We compThe indel patterns reflected the underlying indel model or scoring function of the methods. Our algorithm favoured shorter indels, compared to PRANK and MAFFT, which reconstructed visually tidier gap regions. A phylogenetic interpretation of MAFFT’s indel placement implies few insertions, followed by several subsequent deletions, leading to a short MSA. PRANK infers a longer alignment, with phylogenetically meaningful and balanced number of insertions and deletions. Note that similar to MAFFT, PRANK also tends to block long indels. Our method infers a phylogenetically meaningful MSA, with multiple single amino-acid insertions, which sometimes fuse to mimic long indels . Our method infers short indels, which allows for gap regions with higher conservation in terms of the substitution rates; we observe more conserved columns. To quantify this, we estimated tree-lengths (in expected substitutions per site), by fitting the branch-lengths of the guide-tree topology based on the inferred MSAs using PhyML . ConsistO(Nl3), where N is number of taxa and l is the maximum sequence length. The cubic factor stems from the fact that the likelihood is not monotonically increasing in the MSA length, so that the length has to be incorporated as an extra dimension in the DP. The O(l2) entries in a specific matrix layer along that dimension  depend only on the layer above (and not on each other). Therefore, their computation can be parallelized, taking down the running time to O(Nl), assuming O(l2) processors. Further, our empirical findings show that the likelihood has exactly one maximum, suggesting an early stop condition to the DP. We are currently optimising our implementation with respect to this and other time-critical aspects. To date inference of MSAs under an evolutionary indel model (TKF91 or TKF92) has only been implement using a Bayesian framework. Such approaches are however computationally expensive with large datasets. Our method for MSA inference under PIP is the first step towards equivalent developments in the frequentist framework.Here for the first time in the frequentist framework, we have developed and implemented a progressive MSA algorithm with an explicit evolutionary model of substitutions, insertions and deletions. The evolution of indels was described as a Poisson process as part of a continuous-time Markov model known as PIP. At the core of our method we have designed a new DP algorithm for the alignment of two MSAs by ML, which exploits PIP’s linear time complexity for the computation of marginal likelihoods. The overall complexity of the progressive algorithm is Despite only allowing single residue indels our method appears to fare surprisingly well compared to other state-of-the-art popular alignment tools such as PRANK and MAFFT. Indeed, our example above  demonstrate that our new method allows inferring alignments with phylogenetically sensible gap patterns, similar to the phylogenetically-aware PRANK. In contrast to traditional aligners that do not utilise phylogenetic information to distinguish insertions and deletions, our method produces longer alignments, avoiding the artificial compression of MSAs and inferring more indels, again similar to PRANK. According to the underlying indel model, our method appears to infer more shorter indels , while longer indels are described by several subsequent indel events. Including longer indels is considered desirable, however it has not been studied whether modeling one residue indels at a time may also work well. For example, for simplicity models of codon substitution typically allow only one-nucleotide mutations. Despite this gross simplification codon models have been demonstrated to perform extremely well for practical analyses of protein-coding genes. As can be seen in our example of an HIV protein gp120, it is unclear what inferred indel pattern is more realistic . Considering the nature of HIV mutations, it is quite plausible that indel evolution of gp120 is dominated by short indel events . ArguablOur new methods provides not only a first step towards the explicit modeling of indels in the frequentist framework but also enables to test a different hypothesis of indel evolution. In our follow up studies we intend to further scrutinise the various properties of our new method, its further development including less greedy algorithm versions, variation of indel rates across sites and the approximations to include longer indels.Additional file 1Supplemental Materials. Qscores, MSA length and MSA magnifications. (PDF 3392 kb)


Third generation sequencing technologies generate long reads that exhibit high error rates, in particular for insertions and deletions which are usually the most difficult errors to cope with. The only exact algorithm capable of aligning sequences with insertions and deletions is a dynamic programming algorithm.In this note, for the sake of efficiency, we consider dynamic programming in a band. We show how to choose the band width in function of the long reads’ error rates, thus obtaining an We suggest that dynamic programming in a band is well suited to the problem of aligning long reads between themselves and can be used as a core component of methods for obtaining a consensus sequence from the long reads alone.https://forgemia.inra.fr/jean-francois.gibrat/BAND_DYN_PROG.gitThe function implementing the dynamic programming algorithm in a band is available, as a standalone program, at:  The longest sequenced reads have a length of about 55 kbp. These reads have an error rate of 10-15% . These figures are likely to change with the techniques’ improvements, but give a general idea of the characteristics of 3rd generation technologies so far.High throughput sequencing technologies are progressing at a very fast pace. Recently, third generation sequencing technologies have been launched on the market and are becoming available to the life science community. These novel sequencing technologies are based on single molecule techniques and are characterized by very long reads exhibiting a high error rate. At the time of writing, Pacific Biosciences  with very low error rates (< 1%). The length of 3rd generation reads and their high error rates, particularly for insertions and deletions, is likely to make these algorithms ill-adapted to efficiently process 3rd generation sequencing technology data. Algorithms better able to cope with reads exhibiting large numbers of insertions and deletions are needed.O(NM) in time and space, where N and M are the lengths of the two sequences to be aligned. To illustrate this point, if one wants to compare two long reads of length 50 kbp as described above, this requires allocating enough memory for a matrix of 2.5 109 elements. If this is a matrix of floats or integers (4 bytes), this requires 10 GB of memory.Until now, the only exact algorithm for aligning sequences with insertions and deletions remains the dynamic programming (DP) algorithm proposed by Needleman & Wunsch almost 50 years ago . This alO(N)  of the dynamic programming matrix is within the band if −w−(l1−l2)≤i−j≤+w. As shown on Fig. i and j indexes such that i−j>w (dashed path) or i−j<−w−(l1−l2) (solid path). The best score obtained for such a path is given by: Assuming that the length of the first sequence is g is the gap penalty and m is the match score. Indeed, this path generates w+1−(l1−l2) gaps in the horizontal sequence, w+1 in the vertical sequence and at most l2−(w+1) matches. Let sband be the best score found by applying the dynamic programming algorithm in a band and sopt be the best score obtained with the complete matrix. If the path does not leave the band, then sband=sopt. If sband≥bestout than sband is optimal since it is larger than the best score of any possible alignment that travels outside of the band. Notice that the above formula provides an upper bound of bestout, since it is assumed that all the aligned characters in the 2 sequences correspond to matches, excluding the possibility of mismatches.where w=6, is not sufficient to fit the path. In the right panel, the band width has been increased to w=7 and the path remains in the band. Although this change is marginal, it has a dramatic impact on the alignment provided by the algorithm.Using a semi-global alignment might often be more appropriate for the problem at hand. Figure What criterion should one use to detect instances where the path leaves the band in semi-global alignments? In this case, the above global alignment criterion is not applicable since one does not know in advance where will be the ends of the path.Nm, in alignments of sequences of length N and the corresponding variance are given by: We tested 2 criteria. The first one is whether or not the path reaches the edge of the band (which is straightforward to check when one performs the backtracking procedure to find the path). The second criterion is related to the distribution of the number of matches in the alignments. The expected number of matches, pu=1−pm−pi−pd is the probability of no modification of the nucleotides in the sequences. pu is an upper bound of the true probability. Empirically, we checked that this provides a very good approximation of the mean of the number of matches. We consider that the path has left the band if the corresponding score is less than where w. If it is false, we further check whether the number of matches is outside the 99% confidence half interval. If it is true, we restart the alignment with a larger value of w, else we consider that the banded DP algorithm has found the optimal solution.With these 2 criteria, we propose the following procedure to decide whether the banded DP alignment has provided the optimal solution: i) we check whether the path reaches the band edge. If it is true, we restart the alignment with a larger value of However, as shown in Fig. pu), a mismatch (pm), an insertion (pi) and a deletion (pd) of a nucleotide. We aligned all the pairs using the dynamic programming algorithm and we determined the maximal run of indels, in a particular direction, in the alignments, i.e., d above. For each set of 1000 aligned pairs of sequences we computed the mean and standard deviation of d. As awaited, the computed means are very close to zero. Simulation results for the standard deviations are shown in Table To check the validity of the assumption made in Eq. , we geneσth, calculated without the O term are larger than the experimental ones . However, the difference is sufficiently small to be of no consequence for our purpose.As expected, theoretical standard deviations, w=σd).To measure the pertinence of the two criteria described above, we used the results of the previous simulations. Table w although the result is correct. This wastes some time, but has no incidence on the correctness of the final alignment. On the contrary, when the path of the alignments does not reach the band and provides a suboptimal score  one will wrongly accept a suboptimal alignment.The proposed algorithm, would be optimal if the 2 off-diagonal elements contained 0% alignments. Here, 2.2% of the alignments provide the optimal solution and reach the band edge. It means that one will restart the alignment with a larger value of Applying the 2nd criterion after the 1st criterion, the percentage of alignments that reach the band edge but provide an optimal score decreases from 2.2 to 1.8% and the percentage of alignments that do not reach the band edge but provide a suboptimal score drops from 0.5 to 0.2%, improving by 60% the latter problematic cases.The procedure to determine whether the DP algorithm in a band finds the optimal score proposed in this note is thus not completely foolproof, but provides the correct answer in the vast majority of cases.The advent of third generation sequencing technologies that are characterized by long reads exhibiting high error rates, most notably for insertions and deletions, which are the most difficult errors to cope with, call for the development of new methods, better adapted to these features. Although the DP algorithm can hardly be defined as “new”, the problem at hand, aligning long reads that differ “only” by random sequencing errors, is ideally suited for using dynamic programming in a band.In this note, we showed how to choose the band width in function of the reads’ error rates, resulting in a subquadratic time and space algorithm and proposed a procedure to determine whether the alignment path stays in the band, thus providing the optimal score. Therefore, the DP algorithm in a band is a good contender to align long reads between themselves, and can constitute a key component of methods for correcting long read sequencing errors and obtaining a consensus sequence (to be published).