Increasing amounts of individual genomes sequenced per species motivate the usage of pangenomic approaches. Pangenomes may be represented as graphical structures, e.g. compacted colored de Bruijn graphs, which offer a low memory usage and facilitate reference-free sequence comparisons. While sequence-to-graph mapping to graphical pangenomes has been studied for some time, no local alignment search tool in the vein of BLAST has been proposed yet.We present a new heuristic method to find maximum scoring local alignments of a DNA query sequence to a pangenome represented as a compacted colored de Bruijn graph. Our approach additionally allows a comparison of similarity among sequences within the pangenome. We show that local alignment scores follow an exponential-tail distribution similar to BLAST scores, and we discuss how to estimate its parameters to separate local alignments representing sequence homology from spurious findings. An implementation of our method is presented, and its performance and usability are shown. Our approach scales sublinearly in running time and memory usage with respect to the number of genomes under consideration. This is an advantage over classical methods that do not make use of sequence similarity within the pangenome.https://gitlab.ub.uni-bielefeld.de/gi/plast.Source code and test data are available from Bioinformatics online.Supplementary data are available at  Works by unrolling cyclic parts. Read mapping on de Bruijn graphs can be done by a heuristic proposed by Algorithms for sequence-to-graph alignment have already been studied for some time. Early works date back to 1989 where a graph was used for approximate regular expression mapping . In 2000These solutions address a sequence mapping scenario where query sequences are aligned either globally or locally to the graph region that best matches the query. This approach is usually designed for rapidly mapping a huge number of queries gaining speed by the underlying assumption that a query either maps to exactly one position in the graph or to none.In this work, we study the problem of finding high-scoring local alignments between a query sequence and a graph that are likely to represent sequence homology. The exact notion of ‘high-scoring’ is based on statistical considerations. Hence, we are within the regime of the popular Basic Local Alignment Search Tool BLAST; . Some exquorum and a search color set to allow for customized searches in specific parts of the pangenome. A quorum here is meant as the number of individual sequences of the pangenome that have to share a graph sequence to consider it during search. A search color set enables the search to be focused on graph sequences that appear in individual sequences from the set. For instance, a user might be interested only in alignments shared by the majority of genomes showing a certain phenotype. Finally, we present first results of alignment statistics considering that all sequences in a pangenome are related and highly similar. This is in contrast to the general initial, underlying independence assumptions. To our knowledge, our approach is unique and has never been studied before.In comparison to the abovementioned alignment tools that work on collections of individual sequences, our approach stores genomic sequences in a graph to analyze them jointly. Moreover, by querying the graph, we are able to compare these sequences not only to the query but also among each other. This has an advantage over other approaches where an all-against-all post-processing of results would add a quadratic number of comparisons to their running time. To this end, we introduce the notion of a https://gitlab.ub.uni-bielefeld.de/gi/plast.The remainder of this manuscript is organized as follows. In Section 2, we define our model, formally state the problem and describe the algorithmic procedure. Section 3 contains use cases of our algorithm and comparisons of its performance. In Section 4, we discuss our results. The source code of our implementation, instructions on how to generate samples for our statistical parameter estimation and test data used in this article are available from string is a sequence of characters drawn from a finite, non-empty set, called alphabet. For a given string s, we denote its length by i by i and ending at position j by k is called a k-mer. For any decomposition s = xy, the  substrings x and y are called prefix and suffix of s, respectively.A query is a finite string q over genome is a set of strings over universal color set U, and the color is assigned to all strings of the genome to distinguish between sequences from different genomes.In this work, we assume that all strings are over the DNA nucleotide alphabet compacted colored de Bruijn graph of dimension k over an alphabet Σ and a color set U is a vertex-labeled directed graph k, each k-mer r appears in the label of at most one vertex of the graph, there exists an edge v to vertex w if and only if the (k−1)-length suffix of k−1)-length prefix of C assigns a color set k-mer r that appears in any vertex label. For convenience, we write v. In addition, v except its last k −1 characters, i.e. Let location in a compacted colored de Bruijn graph v, i) with k-mer r overlaps a location r is a substring of o such that v, interval location as v is denoted by L(v).We represent a path in G if i with truncated path in G from location 0, v1, …, vz)p =  z = 0 then We say that path location of a truncated path The L is then defined as t is quorum fulfilling if and only if The color set of a path location q and a pangenome represented as a compacted colored de Bruijn graph U. Apart from q and G, it takes as input a non-empty color set search color set and a quorum α between a substring of q and the sequence t, and a list of checkpoints t into maximal subpaths with the same color set, each containing a subset of m. Our algorithm allows to find local alignments for both q and its reverse complement. For the sake of simplicity, we forgo to mention the reverse complement in the following descriptions. Procedures may be assumed to work similarly for the reverse complementary sequence.Our algorithm finds high-scoring local alignments between a given query sequence seed detection step, maximal exact matches of a minimal length q and all sequences of G. Found seeds are extended without gaps in the seed extension step. Statistical considerations are used to extract biologically meaningful alignments. These are then recalculated in the gapped recalculation step to obtain the finally reported results.The algorithm consists of three basic algorithmic steps. In the q has to be specified as an input parameter, the graph G may be given or initially constructed from a set of genomes in a preprocessing step. Once G has been created for a specific value of k, an additional index is built according to a minimal seed length While the query sequence q and sequences stored in G. Array w-mer profile that contains, in form of a cumulative sum, for any w-mer ω the number of occurrences of ω in G, denoted iω is the i-th w-mer over w-mers in the graph. An occurrence of a w-mer ω in G is defined as a pair o represents the offset in ω starts. In w-mer are stored consecutively in some arbitrary order such that iω and i in w-mer falls into the v is stored. An example of the index and its use can be found in The index consists of two arrays that allow a fast location of exact matches between w-mers occurring in q. All matches are checked whether they fulfill the quorum requirement. Consecutive matches may be merged into a single interval location. A match is stored in a list directly linked to the vertex whose sequence it was found in. This allows a quick access to the match if needed during seed extension. Inside the list, matches are ordered by increasing starting offset in the vertex label. Match order is used to terminate a list iteration as soon as it is clear that a demanded match cannot be part of a list anymore, which leads to an additional speed gain.The index is used to look up all X-drop algorithm as used in BLAST along q and all truncated paths of G starting at an initial interval location . The paths are generated by a depth-first search traversal through G outgoing from vertex v. The exploration of a path stops as soon as either the current extension’s score drops below X or the quorum requirement is no longer fulfilled. Our experience shows that the exhaustive processing of all existing truncated paths starting from v through G is possible without a notable slowdown of the algorithm most of the time even though the number of paths may be very large. However, an iteration over all truncated paths can become infeasible if G has some dense regions consisting of vertices with short labels and high degrees, producing truncated paths with highly similar sequences. If these sequences are similar to the query q, the X-drop criterion does not suffice to terminate path explorations leading to suboptimal alignments within an acceptable time frame. Therefore, we introduced a threshold that limits the number of vertices that can be visited during the extension of a seed.Each match represents an interval location used as seed for an ungapped extension. The extension is performed according to an The result of the ungapped extension is a set of path locations consisting of a few biologically meaningful and many more spurious hits. Statistical significance criteria, described in Section 2.6, are used to rank the hits and separate both kinds. The most significant alignments are recalculated using standard gapped alignment with banded dynamic programming. The bandwidth is chosen according to the quality of the alignment. The final alignments are reported, together with statistics for gapped alignments.In classical pairwise (linear) sequence comparison, an extensive statistical theory exists, sometimes referred to as Karlin–Altschul theory , but subs, how many hits sH of a random query sequence of length n against the given reference (genome or pangenome) of size m are observed whose score reach or exceed s? The expected value E-value of an observed score s. Let S be the score of the best hit. Then we may observe the event s is called the p-value of an observed score s. If sE is small  normalization constant such that Z explicitly, we use the Metropolis–Hastings method: Given a current DNA sequence x, a new candidate sequence y is proposed from a neighborhood of x , we found it sufficient to discard the first five samples.In brief, let sR be the absolute number of times that score s was sampled, and let λ can be estimated by fitting a line to points s where the counts sR are consistently high, say C is estimated from the 10% of highest scores in the initial naive sampling step.The procedure yields uncorrelated score samples, which stem from the biased distribution https://gitlab.ub.uni-bielefeld.de/gi/plast. We present results of our statistical parameter estimation, followed by a performance analysis of our method. In particular, we show the advantage in runtime, memory usage and result aggregation when searching local alignments inside a pangenome with our method compared to a conventional search and analysis using other BLAST-like software tools. Afterwards, we present a more advanced use case and show that our method scales even to human data. Unless stated differently, all graphs have been built for a k-mer length of 31 (Bifrost default), and searches were performed using default parameters. To obtain statistical parameters, we used a combination of initial naive sampling followed by importance sampling, as described in Section 2.6.We implemented the algorithm described in Section 2.5 in C++ using Bifrost  as the uSalmonella enterica genomes of the same lineage (Para C) taken from n = 200, yielding the empirical complementary cumulative distribution function (ccdf) of the best hit’s score for each query for both the gapped and ungapped case. A least-squares fit of affine linear functions in the distribution’s tail, considering logarithmic cumulative relative frequencies between C = 17.45 and C = 14.81 and 6 simulations.We tested the hypothesis from Section 2.6 that ents see . As expeα is the average acceptance rate, which we typically find to be around 0.46 to 0.75, which amounts to approximately n = 200. However, it allows us to sample from high scores that are unobtainable by naive sampling, even if billions of samples were used, yielding much higher efficiency by several orders of magnitude , blastn (2.6.0+), BLAT (36×4) and UBLAST (11.0.667). The tools DIAMOND, SWORD and GHOSTZ do not support DNA to DNA alignment. RAPSearch2 was not possible to install due to a reported, but unresolved issue. LAST is known to run very slow on highly redundant datasets—index building was terminated after 10 days. BLAT was designed to search genomes (represented as target databases) for query sequences. It was only possible to run the tool for 750 genomes at once. Runs for larger pangenome sizes had to be split into separate program calls. Results and runtimes were aggregated. Similarly, UBLAST’s freely available 32-bit version has an upper database size limit such that pangenomes consisting of more than 100 genomes had to be distributed on multiple databases.Salmonella Typhimurium assemblies from a total of 19 237 that were annotated as serovar ‘Typhimurium’ from EnteroBase . We obtained queries with an average percentual identity of about 86.9% (SD 31.1) per query and genome where 55 953 query genome combinations without any alignment were considered with an identity of zero.For the comparison, we downloaded 5 000 randomly selected X-drop value. Additionally, we set the maximum number of reported results high enough to get all existing results. This was necessary, because all other tools do not compare their results internally and would otherwise report only the best result for each genome separately, hiding all further findings. For all remaining parameters, default values were used. Concrete program calls are documented at https://gitlab.ub.uni-bielefeld.de/gi/plast. Calculations were performed single threaded on a virtual machine with 28 cores and 256 GB of RAM.If possible, tools were run with the same scores for match, mismatch, gaps and the same matching if they overlap by at least 90% of the shorter alignment with respect to the query sequence. We call an alignment unique if it does not match any alignment of another tool. The comparison is shown in all other tools. Their score was low (mean 18.5 and maximum 25) and they were short (mean length 28 bp and maximum length 112 bp). They are a result of the different statistical parameters used by PLAST in comparison to other tools. As explained in Section 1, these tools do not consider a pangenomic use case where all sequences in the database are highly similar. Thus, they overestimate the chance for a random hit. Using the same statistical parameters as BLAST, all 178 results are filtered out by PLAST due to the E-value threshold. We observed 306 PLAST alignments (4.0%) that were unique with respect to at least one other tool but at the same time found by at least one other. 56 of them had a score higher than 25 (maximum 678) or were longer than 112 bp (maximum 999 bp). Generally, we found that PLAST was able to find alignments with even very low sequence identity values of up to 58%, which is comparable to BLAST (63%) in this setting.To compare the results, we scanned the output of each tool and identified corresponding alignments of PLAST and BLAST. In case of PLAST, this identification was possible using the color sets that are part of the program’s output for each alignment as explained in Section 2.4. Two results are considered Salmonella genome assemblies to generate pangenomes of different sizes and compared the tools’ performance. Runtime and memory usage are shown in Next, we randomly selected subsamples from the 5 000 All tools spend a large amount of memory for loading the highly redundant datasets and additional index data structures for fast alignment searches. Our graphical representation, in turn, allows a maximal exploitation of sequence similarity. Most parts of the graph represent sequences shared by many individual genomes that have to be stored only once. An alignment calculated for these parts is valid for all sharing genomes and does not have to be calculated multiple times. Thus, we observe a strongly reduced growth in run time and memory consumption for PLAST. Considering a pangenomic use case with constantly growing numbers of individual genomes, our approach will always be superior compared to conventional methods not making use of sequence similarity.When comparing PLAST’s run time for different quorum values and pangenome sizes, we can generally observe two driving forces (data not shown). On the one hand, the usage of a quorum has a beneficial influence on PLAST’s run time, because it allows to disregard parts of the graph harboring rarely appearing variations that may prune it considerably, especially if the overall diversity within the pangenome is high. At the same time, a quorum check is comparatively cheap if the graph has only a moderate total number of colors in it. On the other hand, for larger pangenomes, the usage of a high quorum becomes an increasing burden as color coverages have to be checked on every vertex using Bifrost’s API. The high degree of color compression generates a noticeable loss in speed in this case, which outstrips any gain by pruning from a certain pangenome size. In our experiments, it doubled computation time compared to a run without quorum for the largest pangenome size.To prevent this loss in speed, a first filter was implemented that, instead of only checking color presence, also considers the number of missing colors on a vertex and rejects it as soon as this number becomes too high during iteration. Other heuristics to speed up a quorum check are currently under development and discussed in Section 4.V. cholerae are known to contain the Vibrio Pathogenicity Island-1 (VPI-1) that consists of a whole set of genes whose sequence and order inside the island can be diverse among different strains , and the remaining 14 have been obtained from non-clinical samples and are labelled ‘environmental genomes’ (EG)  to generate chromosome-wise genomic sequences for chromosomes 2 and 15 of all 2 504 human individuals by inserting all reported variations into the GRCh37 reference sequence and built one graph (k = 63) for each of the two chromosomes. Instead of building a single graph from all human chromosomes at once, we had to perform this chromosome-wise approach as our system could not store all 2 504 human genome sequences including indexing structures at once for a comparative call of MMseqs2 (see below). Our method alone also scales to a whole human pangenome of this size.To test PLAST’s applicability beyond bacterial pangenomes, we built a human pangenome using data from the 1000 Genomes (1KG) Project phase 3 , we exclusively found the reference allele, confirming the results of the earlier studies.We exemplarily investigated a known polymorphism within the human pangenome. The single nucleotide polymorphism ancestry . It is lTo evaluate PLAST’s performance on the human pangenome, we used the graph of chromosome 2 .Furthermore, we would like to further explore the computational limits using our implementation. The here presented datasets were the largest we considered so far in terms of individual samples (Section 3.2) and the number of We also plan to further extend the concept of quorum and search color set. In particular, we would like to give more freedom to the user by allowing not only to focus on certain parts of the graph but also to explicitly exclude regions from analysis.This work is supported by the BMBF-funded de.NBI Cloud within the German Network for Bioinformatics Infrastructure (de.NBI) . It was funded in part by the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie agreement [872539] and the International DFG Research Training Group GRK 1906 to T.S.; and National Science and Engineering Council of Canada (NSERC) Discovery Grants (RGPIN-05952) and Michael Smith Foundation for Health Research (MSFHR) Scholar Award (SCH-2020-0370) to F.H. Conflict of Interest: none declared.


Protein sequence matching presently fails to identify many structures that are highly similar, even when they are known to have the same function. The high packing densities in globular proteins lead to interdependent substitutions, which have not previously been considered for amino acid similarities. At present, sequence matching compares sequences based only upon the similarities of single amino acids, ignoring the fact that in densely packed protein, there are additional conservative substitutions representing exchanges between two interacting amino acids, such as a small-large pair changing to a large-small pair substitutions that are not individually so conservative. Here we show that including information for such pairs of substitutions yields improved sequence matches, and that these yield significant gains in the agreements between sequence alignments and structure matches of the same protein pair. The result shows sequence segments matched where structure segments are aligned. There are gains for all 2002 collected cases where the sequence alignments that were not previously congruent with the structure matches. Our results also demonstrate a significant gain in detecting homology for “twilight zone” protein sequences. The amino acid substitution metrics derived have many other potential applications, for annotations, protein design, mutagenesis design, and empirical potential derivation. The results from sequence matching serve as fundamental data for many studies.13 When utilizing the massive data of protein sequences, the combination of Basic Local Alignment Search Tool (BLAST)14 with amino acid substitution matrices is powerfully informative.The rapid advances in genome sequencing and structural biology provide an opportunity to improve the understanding of the relationship between protein sequence and structure.15 developed in 1992 are the most commonly used still, in part because of s use in BLAST at NCBI. The BLOSUM62 matrix is the default matrix for protein sequence matching there and remains the standard substitution matrix in protein sequence database searches and sequence matching. Miyazawa and Jernigan16 introduced another type of substitution matrix based on the empirical amino acid contact frequencies. Muller et al17 developed the variable time maximum likelihood (VTML) substitution matrices based on divergent alignments for identifying distantly related protein sequences. More specific substitution matrices were developed for different families of proteins18 or different types of structures.19 Yamada and Tomii20 reported a matrix based on principal component analysis and the variabilities across existing substitution matrices. Several reports have utilized sequence neighbors and sequence triplets22 to develop a pairwise or three-way dependent matrix, but these are only for nearest neighbors along a sequence. Other substitution matrices28 include some that have focused on the details of the matching method itself. None of the above mentioned approaches account for the interdependences of substitutions at longer separations in sequence.There is a long history of the development of a wide variety of substitution matrices. The point accepted mutation (PAM) matrices were the first set of substitution matrices developed by Margaret Dayhoff are based on 1572 point mutations in the phylogenetic trees of 71 families of closely related proteins. The blocks substitution matrix (BLOSUM) family of substitution matrices29 The Johnson and Overington matrix also considered regions in the structure alignment, where the gaps occur.30 Blake and Cohen extracted substitutions from the superposition of protein structures to target the “twilight zone” (protein sequences with identity lower than 20%) in protein sequence matching.31 Teodorescu generated substitution matrices by combining the energy information from protein structures with substitution matrices derived from sequence data.32 However, only the BLOSUM substitution matrices have come into common e due to their limited improvements. Our own group recently explored the approach of developing different substitution matrices for different structure families but obtained relatively small gains.33 In a predecessor to this paper, we developed a new substitution matrix that combined BLOSUM62 matrix with the correlated pair information from multiple sequence alignments (MSAs)34 to show in a proof of principle that this can bring structure matches and sequence matches into agreement. That substitution matrix permits too many substitutions and was found to give false positives in homolog detection. The major challenge of incorporating structural information into a substitution matrix is to resolve the conflicting substitution information extracted from structures and sequences.People have also used structure information to derive substitution matrices. A straightforward way is to use protein structure alignment to extract the amino acid substitution information, based on the nearest residues in the two proteins in the match. Prlic et al derived substitution matrices from a set of protein structure pairs with high structural similarity but low sequence identity.37 From this, we can hypothesize that the important interactions are those between neighboring residues. Globular proteins are tightly packed, almost at the density the most densely packed spheres.39 The packing density is also related in detail to sequence entropies and amino acid conservation41 because of the free volume around a given amino acid. In a densely packed environment, it is possible that these interdependences extend over long distances. One mutation may trigger a chain reaction that leads to a requirement for multiple coordinated mutations at its successive neighboring positions. The simplest of these dependences should be manifested in be a pair of coevolving positions. At two physical adjacent positions in a structure, a large-small pair of amino acids might be substituted with a small-large pair without disrupting the structure, or a charged (+−) pair might be substituted with a (−+) pair. These types of compensatory mutations can lead directly to substitutions at a single position that would not be considered to be so similar, but at the same time are maintaining the protein stability with closely similar free energies between the substituted structures. Since spatially close residues are often evolutionarily correlated, this means that coevolving residues has passed through correlated mutations in the history of evolution, and this is the underlying reason why it’s possible to make a connection between sequence and structure information by using these types of correlations.Studies from our group have developed knowledge-based potentials derived by mining residue contact information in protein structures that have proven to be the most successful type of potentials for assessing the quality of modeled protein structures at critical assessment of protein structure prediction (CASP).43 Dunn et al added in an average product correction to mutual information to remove the background correlation signals from random noise and phylogenetic components.44 Direct coupling analysis (DCA)46 was used to extract the direct correlations to predict the residue pairs in immediate spatial contact. To achieve a similar goal, Jones et al introduced protein sparse inverse of covariance, a precision matrix based approach to filter indirect correlations.47The big data of protein sequences contain important information about correlated mutations. Such correlations can be detected by coevolution analysis methods. Mutual information based on the association of information entropy between a pair of residue positions is being widely and successfully used as a straightforward way to predict coevolved residues in structures from the apparent correlations in a sequence multiple alignment. The results are confounded by indirect correlations through intermediates.In the present study, we develop an approach to derive a new substitution matrix from a large set of sequence data that accounts for these interdependent substitutions and is intrinsically reflecting the structure information. The approach is shown in 2 |2.1 |48 because it is the largest protein domain family database that contains high-quality MSAs. The accuracy of Pfam MSAs is better because of their human curation. The selection of the training dataset is based on a balance among three criteria: (a). the coverage of protein families; (b). the number of MSAs that are suitable for coevolution calculation; (c) the quality of the representative structures. To test the ProtSub matrix, we use datasets from the 49 database and ASTRAL in structural classification of proteins — extended (SCOPe) database50 and follow the same testing schemes widely used in other studies. Both datasets have served as the “gold” standard for homolog detection test from previous studies. The testing dataset from the CATH database contains 4184 homolog families with a total of 8901 protein sequences. The testing dataset from the ASTRAL database contains 1454 homolog families and a total of 4066 sequences in the ASTRAL20 database. These two datasets ensure extensive coverage of homolog families and folding topologies.One of the biggest challenges is to select a training dataset that covers as many protein families as possible and at the same time contains high-quality MSAs, that also have at least one representative protein structure for each family. A reliable coevolution correlation evaluation requires a large number of well aligned, diverse sequences. We chose to use the Pfam-A database2.2 |In this study, we use 2320 Pfam-A MSAs to extract the coevolution correlations. Each family contains at least 1000 sequences and has at least one representative of an experimentally determined structure. In each MSA, only “ungapped” positions are considered in the calculation. Here an “ungapped” position is defined as a position containing fewer than 10% of gaps. In this study, we use mutual information to extract the coevolution correlations. Methods like DCA that predict residue contacts may lead to a biased sampling of interdependent substitutions. The transitive correlations resulting from mutual information also contain important information of interdependent substitutions, which should be considered in the process of sequence matching. The mutual information between two positions in a MSA is defined as:q equals to 21, is the number of amino acid types and counts gaps as a 21st type of amino acid. While the number of sequence in the MSA data is sufficiently large, we use frequencies to approximate the probabilities. f(Ai) is the frequency of a single amino acid type A observed at position i. f is the joint frequency for observing the co-occurrence of two amino acid types A and B at positions i and j. Both single and pair frequencies are weighted by the redundancy of the sequences in the MSA  have previously been used by Liang’s group2.4 |The elements in an amino acid substitution matrix are substitution scores. A substitution score describes the tendency of one type of amino acid being substituted, or not, by another type of amino acid. The result of the coevolution calculation indicates that coevolution correlation is ubiquitous throughout every protein family. Most amino acid mutations do not occur independently, but depend on their structural context. By extracting substitution frequencies from correlated positions selected from the previous step, we embed the dependence information of substitutions into a new set of substitution scores. Similar to the original BLOSUM series of matrices, we also derive substitution scores using log-odds ratio:f is the joint frequencies and f(A)f(B) are the expected single amino acid type frequencies. λ is a scalar factor, optimized to give the best homolog detection performance based on the training dataset (2320 Pfam domains). A semi-heuristic strategy is used here to optimize λ. First, a coarse set of values over a broad range of λ is investigated and then refined over a smaller interval, iteratively. The initial iteration scans through a range of values between 1.0 and 5.0 with a large interval of 0.5. The procedure stops when the stepping interval is less than 0.1. The outcome is to assign λ to be 2.4 in the final ProtSub matrix.where 3 |27 Comparisons of the new substitution matrix for ProtSub with BLOSUM62, and VTML200 are shown in λ adjusts the magnitude of the overall score, some differences are also adjusted accordingly. There will be further discussion about in the We compare ProtSub with two substitution matrices, BLOSUM and VTML200, which are reported to have the best performance in homolog detection from previous studies.3.1 |55 and use template modeling align 56 to generate the structure alignments.  As a result, ProtSub achieves better agreements between the sequence alignment and the structure alignment for these cases. An example of an improved alignment in In this study, we claim that using correlated substitutions integrates structural information into the new sequence substitution matrix. So the sequence alignments generated with this matrix should agree more closely with the results of structure alignments. There are a substantial number of known cases where two evolutionarily related proteins have almost identical folds but their sequences are not at present considered to be similar. We extract in total 2002 non-redundant pairs of sequences from the CATHS20 dataset that have this feature.  is calculated from the structure alignment for residue pairs in the aligned sequence segments as specified in the sequence alignment. To obtain a comprehensive comparison, we iterate different gap penalties with a fixed interval to generate sequence alignments. Gap opening penalty ranges from 1 to 21 in steps of 1.0, gap extensions from 0 to 10 in steps of 5.0. The mean RMSD of all 2002 cases is calculated based on sequence alignments generated for different gap opening and gap extension combinations, as shown in 3.2 |Sequence matching can successfully distinguish between pairs of protein sequences when the sequence identity is high. The results become unreliable, however, for matching “twilight zone” sequences (<20% sequence identity). To test the performance for homolog detection with the ProtSub matrix, we make the best-to-best comparison among the new ProtSub, BLOSUM62, and VTML200 matrices. Each sequence in the test dataset is queried against the whole dataset. Each query is also performed for all 11 build-in gap opening and extension combinations in BLAST: , , , , , , , , , , and . Then the best result for each matrix is selected to make the comparison. To avoid the bias from testing on only a single dataset, we perform the same all-to-all homolog testing on both the datasets of ASTRALS20 from SCOPe and CATHS20 from CATH database. These are the two datasets with human curated classification information for “twilight zone” protein sequences, both having less than 20% sequence identity. The ASTRAL20 dataset is a subset of the SCOPe database. This dataset has been commonly proposed as a “gold” standard for evaluating the performance of homology search methods. After removing sequences that contain Pfam domains used in the training dataset for developing the ProtSub matrix, the ASTRAL20 has in total 1454 homolog families and 4066 sequences with no greater than 20% identity to one another; The CATHS20 contains 4184 homolog families and 8901 protein sequences, which also have no more than 20% identity to one another. The results demonstrate that ProtSub outperforms for both BLOSUM62 and VTML200 in homolog detection on both datasets .58 The CVE curves are similar to receiver operating characteristic curves, but present the results in a way that is directly interpretable and suitable for sequence analysis. The “coverage” on the Y-axis is defined as the fraction of true positive homologs that have scores above a certain threshold; this reflects the sensitivity of the matrix. On the X-axis, “errors per query” (EPQ) acts as an indicator of selectivity, which shows the number of non-homologous sequences above the threshold divided by the number of queries. The EPQ axis in a CVE plot is on a log scale to show performance over a wide error range. These homolog searches are performed in an all-to-all fashion, which takes each one of sequences in the dataset as a query sequence to search against all others. In order to generate the CVE curves, the results of all queries are sorted in the order of decreasing statistical significance . In this case, by traversing the sorted list from top to bottom, the coverage is accumulated by counting the true positive homologs. We test all possible gap opening and gap extension parameters (as used by BLAST) and find the optimal set of gap parameters for each matrix. The CVE curves shown in To evaluate the performance, we use coverage vs errors per query (CVE), a method developed by Price, et al.4 |27 Since the BLOSUM62 and VTML200 derivations were based on a significantly smaller set of sequence alignments, there are gains from utilizing the present much larger sets of structure and sequence data. In addition, our results show gains over BLOSUM62 and VTML200 because of utilizing more diverse data: utilizing compensating substitutions for closely interacting residues and MSAs of larger domains. The major improvement in the performance of protein sequence matching derives substantially from incorporating structural information into the ProtSub matrix.The combination of BLAST with BLOSUM62 has been the most commonly used procedure for matching sequences for many years. A comparative study showed that VTML200 gave the best detection performance in detecting homologs for sequences having low sequence similarity in comparison with the commonly used BLOSUM, PAM and other series of matrices.4.1 |15 PROTOMAT is a two-step system that first extracts motifs from raw sequence data with the help of an amino acid substitution matrix and then generates block alignments using motifs. The block alignments that were used to generate BLOSUM matrices are refined iteratively, that is, a new substitution matrix is obtained by using the block alignment generated for the first time, then PROTOMAT uses this matrix to refine the block again, and finally uses the refined block to generate the final BLOSUM matrices.The calculation of log-odds ratios in ProbSub is similar to BLOSUM62 but the training data that is used to derive amino acid substitutions is different. The BLOSUM series of matrices were derived from more than 2000 ungapped local alignments called blocks. Each block alignment is considered to be a conserved region in a protein family. Blocks are generated using the PROTOMAT tool.49 SCOPe,50 and evolutionary classification of protein domains (ECOD)59 as a quality control. Using the HMM profile extracted with the seed alignment, the final full alignments are generated by searching a large-scale protein sequence database, such as Uniprot.In this study, we extract correlated mutations using Pfam-A MSAs. The Pfam-A database uses a Hidden Markov Model (HMM) based on a probabilistic modeling technique to construct the alignments for different protein domain families. For each protein domain family, a small-scale seed alignment is first generated from the raw protein sequence data. This seed alignment is curated by humans to increase the accuracy. The seed alignments are also compared with several existing databases including CATH,The ungapped block method is robust and fast for the conserved regions of the protein sequences in the database. However, gaps may occur in conserved regions. The restriction of no gap not only puts a limitation on the true positive coverage but also can lead to truncation of domains. Thus, the major difference between block alignments and Pfam-A alignments is that block alignments include short conserved regions, while Pfam-A alignments provide complete domains. As a result, the correlated mutations are underestimated due to the truncation of domains in the short ungapped block alignments. The block alignments used to generate the matrix contain approximately 27 000 sequences. The data has exploded recently. Here, we use more than 3000 MSAs of Pfam-A domain families, and each has more than 1000 sequence available. Our sequence number for training is more than 124 million (12 438 029) in total. The blocks used by the Henikoffs to develop the BLOCK matrices were around 60 residues in length, and so these necessarily are omitting most of the relatively common longer-range effects.4.2 |The test results are strongly affected by the amount of permitted substitutions permitted by a given substitution matrix, which reflects the probabilities in the training dataset. In the procedure of aligning sequences, adding log-odds scores can be thought of as multiplying the corresponding probabilities all together. As shown in 4.3 |λ in 60 However, when gaps are included, the scale of the scores does affect the final alignment. To test the scaling effect, we generated matrices that scale the original scores of the three substitution matrices  by scaling factors up to 2.0. Then we perform the same tests using those scaled matrices. As a result, we find that the larger the scale of the matrix, the more insensitive it is to gap penalties. In the global alignment test, matrices with a larger scale of scores do generate more compact alignments than the original matrices : https://pfam.xfam.org/. The list of Pfam Domains and the percentage of residue positions used for the final ProtSub matrix derivation are provided in the 63 and the secondary structure evaluated by d(DSSP) software.64The Python scripts used for calculating the mutual information and deriving conditional substitution matrices are available on GitHub: https://www.cathdb.info. The global alignments are generated by using the Needleman–Wunsch method from EMBOSS https://www.ebi.ac.uk/Tools/emboss/. The structure alignments are generated with the TM-align tool. The list of PDB IDs used for structure matching is given in The “twilight zone” sequences and PDB structures used for sequence alignment comparisons are from CATH (CATHS20) database: supinfo


Multiple sequence alignment is an initial step in many bioinformatics pipelines, including phylogeny estimation, protein structure prediction and taxonomic identification of reads produced in amplicon or metagenomic datasets, etc. Yet, alignment estimation is challenging on datasets that exhibit substantial sequence length heterogeneity, and especially when the datasets have fragmentary sequences as a result of including reads or contigs generated by next-generation sequencing technologies. Here, we examine techniques that have been developed to improve alignment estimation when datasets contain substantial numbers of fragmentary sequences. We find that MAGUS, a recently developed MSA method, is fairly robust to fragmentary sequences under many conditions, and that using a two-stage approach where MAGUS is used to align selected ‘backbone sequences’ and the remaining sequences are added into the alignment using ensembles of Hidden Markov Models further improves alignment accuracy. The combination of MAGUS with the ensemble of eHMMs (i.e. MAGUS+eHMMs) clearly improves on UPP, the previous leading method for aligning datasets with high levels of fragmentation.https://github.com/smirarab/sepp, and MAGUS is available on https://github.com/vlasmirnov/MAGUS. MAGUS+eHMMs can be performed by running MAGUS to obtain the backbone alignment, and then using the backbone alignment as an input to UPP.UPP is available on Bioinformatics online. Multiple sequence alignment (MSA) is a critical precursor for many downstream analyses, such as gene and species tree estimation , proteinOne of the challenges that arise in MSA estimation is sequence length heterogeneity, which occurs naturally in many datasets due to evolutionary processes that include large insertions or deletions , and also when the input sequence datasets include some incompletely assembled sequences (e.g. contigs or reads). Datasets of interest may contain some full-length sequences as well as fragmentary sequences, for example when performing taxonomic identification of reads in microbiome samples as in TIPP  full-length sequences have been developed, including MAFFT –addfragments  and techHere, we address the possibility of improving alignment accuracy compared with UPP. The main change we explore is the use of MAGUS , a new aWe performed three experiments to design and evaluate methods for MSA estimation on datasets that have fragmentary sequences. In Experiment 1, we explore the design space for computing the backbone alignment and adding sequences to the backbone alignment. We show that the best variant is to use MAGUS for backbone alignment and eHMMs for adding the remaining sequences, and denote it as MAGUS+eHMMs. In Experiment 2, we compare MAGUS to MAGUS+eHMMs and show that MAGUS+eHMMs improves or matches MAGUS alignment performance. In Experiment 3, we evaluate MAGUS+eHMMs in comparison to methods that have been shown to have high alignment accuracy. We again show that it is the overall best performing method.https://tandy.cs.illinois.edu/magus-ehmms-suppl.pdf.Due to space limitations, commands needed to reproduce the experiment, as well as we also additional figures and tables, are in In this study, we evaluate two-stage methods and other multiple sequence alignment tools  on datasets with and without fragmentation. Other methods such as Clustal-Omega , as well as most standard alignment methods, are not based on this type of two-stage approach and considered as one-stage methods in the context of this study.In general, we will describe UPP as a We explore alignment methods on a collection of biological and simulated datasets, some with introduced fragmentation (as described below). The biological datasets all have reference alignments based on structure, and the simulated datasets have true alignments. Nearly, all the datasets used in this study are from prior studies, and available in public databases see ; the remFragmentary versions of datasets. We explore two techniques for making fragmentary versions of our datasets. The script to generate an alignment with fragments is available at https://git.io/JOGO1.High fragmentary (HF) means that 50% of the sequences are made into fragments. The fragment size is sampled from a normal distribution with mean M  and a standard deviation of 60.Low fragmentary (LF) is similar to HF except that only 25% of the sequences are made into fragments and M corresponds to 50% of the original median sequence length.Simulated datasets. We use ROSE and RNASim datasets, each from prior studies. The ROSE datasets are simulated DNA sequence datasets generated using the ROSE simulator  datasets , computed by comparing the estimated alignment to the reference alignment. Each alignment is represented with its set of pairwise homologies, which allows for the calculation of false positive homologies  and false negative homologies . The SPFN  rate is the fraction of the true pairwise homologies that the estimated alignment fails to recover, and the SPFP  rate is the fraction of the pairwise homologies in the estimated alignment that are not in the reference alignment. We compute SPFN and SPFP using FastSP . We alsoWe performed three experiments. In Experiment 1, we explore variants on the two-stage algorithmic strategy for aligning datasets that contain both full-length and fragmentary sequences. We evaluate their performance on the RNASim 10K-HF dataset with respect to both alignment error (measured using SPFN and SPFP) and wall clock running time. MAGUS+eHMMs is the most accurate method . For the remaining experiments, we fix the backbone alignment size in MAGUS+eHMMs to only 1000 sequences. In Experiment 2, we compare MAGUS+eHMMs to MAGUS on the full range of testing datasets, and demonstrate the superiority of MAGUS+eHMMs. Finally, in Experiment 3, we compare MAGUS+eHMMs to two versions of MAFFT (default and L-INS-i), PASTA and UPP on the full range of testing datasets.Here, we provide additional details regarding Experiment 1. As described earlier, the basic strategy samples a desired number of sequences considered to be full-length, computes a ‘backbone alignment’ on these sequences using a preferred method, and then adds the remaining sequences into the backbone alignment using a selected strategy. We explore variants for each step, as follows:Backbone size: 1000 (‘1000bb’) or all  full-length sequences.Backbone alignment method: MAGUS or PASTA.Technique to add sequences into the backbone alignment: ‘eHMMs’ as used in UPP or MAFFT options .In our experiments on datasets with introduced fragmentation, we know which sequences are fragmentary and which are full-length, and we provide this information to the two-stage methods so that the backbone alignment is computed only on the full-length sequences. In the experiments on datasets without introduced fragmentation, we use the default technique from UPP to define the full-length sequences (from which we then select the backbone sequences): all sequences within 25% of the median sequence length.To use the eHMMs approach, we need to build eHMMs on the backbone alignment, which requires the construction of a phylogenetic tree on the backbone alignment. This is automatically performed within UPP, but when we run MAGUS+eHMMs, which utilizes the UPP code library to operate, we need to provide UPP with the backbone alignment and a tree on the backbone alignment. We perform this by computing a maximum likelihood tree on the backbone alignment using FastTree 2 , but this option encountered out-of-memory issues on the RNASim 10K-HF datasets see ; hence, Use ‘–addfragments’ for all missing sequencesUse ‘–add’ to add the missing full-length sequences, then ‘–addfragments’ to add fragmentsSame as 2 except using ‘–addfull’ for the missing full-length sequences.The first version is denoted by ‘MAFFT(frag)’ (i.e. only using ‘–addfragments’ to add the remaining sequences) and the other versions are denoted by ‘MAFFT’ (i.e. using ‘–add’ for full-length sequences and ‘–addfragments’ for fragmentary sequences) or ‘MAFFT’, depending on whether ‘–add’ or ‘–addfull’ is used.Given the details provided, we denote different variants of the two-stage algorithmic strategy in the form of ‘[backbone alignment method]+[strategy to add the remaining sequences]+[number of sequences in the backbone]’. For example, ‘MAGUS+eHMMs + 1000bb’ refers to using 1000 full-length sequences and MAGUS to construct the backbone alignment, and using eHMMs to align the remaining sequences to the backbone alignment . ‘PASTA+MAFFT(frag)+allbb’ refers to using all full-length sequences and PASTA to construct the backbone alignment, and using MAFFT ‘–addfragments’ option to align the remaining sequences to the backbone alignment.We show results for Experiments 1–3; however, MAFFT(L-INS-i) did not complete on some datasets. We show results on individual datasets where MAFFT(L-INS-i) completed in Designing methods that use MAGUS for the backbone.  Backbone tree using initial versus backbone alignment. Recall that MAGUS first computes an initial alignment that is not designed to provide high accuracy on the backbone sequences, then estimates an initial tree using FastTree 2, and finally produces a new (improved) alignment . Here, we evaluate the impact of changing how we compute the backbone tree: using the initial tree  instead using of FastTree 2 to compute a tree on the backbone alignment. This evaluation  ‘–addfragments’ with reduced memory requirement. Since MAFFT(L-INS-i) has high memory requirements, we explored the use of the ‘–weighti 0’ option to reduce these requirements (denoted as ‘MAGUS+MAFFT(linsi-frag)+1000bb’). This approach produces less accurate alignments than MAGUS+eHMMs  and the guide tree it produces has much higher tree error than the MAGUS backbone tree , PASTA, MAFFT(L-INS-i) and MUSCLE in terms of alignment accuracy. On some model conditions, MAFFT(L-INS-i) results are not shown due to out-of-memory issues. We explored alternative ways to run MAFFT(L-INS-i) to address these memory issues. These approaches did enable MAFFT(L-INS-i) to complete, but produced alignments that were clearly worse than alternative approaches (e.g. MAGUS+eHMMs) . Full reMost analyses were run on the University of Illinois Campus Cluster, and a few large datasets were analyzed on the Blue Waters supercomputer . We set 32 GB memory and 48-hour time limit for all analyses, with 16 CPUs for datasets with fragmentation and 8 CPUs for datasets without introduced fragmentation. All analyses that did not encounter errors completed within the time limit. Some methods had out-of-memory issues and so failed to complete on some datasets; see Given that the University of Illinois Campus Cluster and the Blue Waters supercomputer consist of heterogeneous computing nodes with different speeds and available memory, it is difficult to draw clear conclusions about runtime for the methods we explore. However, here, we provide some general observations about the wall clock time used in the different analyses; see In Experiment 1, our main observation was that using large backbones  and using MAFFT’s techniques for adding fragmentary sequences into backbone alignments increased runtime substantially, making the use of eHMMs to add sequences to 1000-sequence backbone alignments the most efficient technique . In ExpeThis study revealed differences between methods for alignment of datasets that contain some fragmentary sequences. While the range of model conditions was limited , we can note the following trends.We observed that the relative performance between methods depends on the dataset. For example, the differences between methods are reduced on datasets with low average p-distances (indicating low heterogeneity), but are large on the 1000M1 and 1000M2 datasets, which have the highest average p-distances (and so highest heterogeneity). Moreover, with sufficiently low heterogeneity, alignment estimation is generally easy, even when the dataset has fragments, a finding that has been observed in prior studies .However, when datasets are challenging to align, then the choice of method can matter substantially. Furthermore, we observed that two-stage methods  produce lower alignment error compared with one-stage methods when the level of fragmentation is high, and especially when the rate of evolution is high. Thus, two-stage methods are more robust to fragmentation than one-stage methods. Moreover, of all the methods we tested, MAGUS+eHMMs is the best performing one in terms of alignment accuracy, especially for the hardest model conditions. This is not surprising, since by design two-stage methods enable local alignment strategies (such as the eHMM technique) to be used to add fragmentary sequences into alignments of full-length sequences. More generally, the two-stage approach allows each type of method to be used on those data for which they are primarily designed.Having said this, it is noteworthy that MAGUS displays fairly good robustness to fragmentation. Thus, with the exception of high levels of fragmentation and high rates of evolution, MAGUS is nearly as accurate as PASTA+eHMMs (i.e. default UPP). MAGUS’ algorithmic design is similar to that of PASTA, and the only important difference is how disjoint alignments are merged. Obviously, this study shows that MAGUS’ technique for merging disjoint alignments (i.e. GCM) is more robust to fragmentation, which is intriguing, and indicates the potential for new method development in alignment merging that could further improve merged alignment accuracy. For example, GCM can be seen as attempting to solve a reformulation of the Maximum Weight Trace problem  to the pr = 0.994 and r = 0.987 for low- and high-fragmentation conditions, respectively) between the backbone alignment error in MAGUS+eHMMs and the final alignment error, across all model conditions . The previous leading method for this problem is UPP, which operates by using PASTA to compute an alignment on a sample of the full-length sequences and then adds the remaining sequences into the alignment using an eHMM. We found that replacing PASTA with MAGUS in the UPP pipeline improved accuracy, and some improvements were substantial. Given the noted difficulties in obtaining highly accurate alignments for datasets with fragmentation, MAGUS+eHMMs may provide a valuable addition to the alignment toolkit, with potential benefits to downstream analyses.We also found that except when the dataset evolves under a high rate of evolution, MAGUS is highly robust to the presence of fragmentation. This property is surprising and may be rare among methods that do not operate in two stages (i.e. by treating full-length sequences separately from fragmentary sequences). Why MAGUS is so robust is not clear, but understanding the algorithmic designs that provide this robustness would benefit developing new methods that are even more robust.This work was supported in part by NSF . This research is part of the Blue Waters sustained-petascale computing project, which is supported by the US National Science Foundation (awards OCI-0725070 and ACI-1238993) the State of Illinois, and as of December 2019, the National Geospatial-Intelligence Agency. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications.Conflict of Interest: none declared.https://datadryad.org/stash/dataset/doi:10.5061/dryad.95x69p8h8. The three largest CRW datasets  can be accessed from https://sites.google.com/eng.ucsd.edu/datasets/alignment/16s23s. The remaining datasets are available in the Illinois Data Bank at https://doi.org/10.13012/B2IDB-2419626_V1.Most of the datasets used in this study are from prior publications. High- and low-fragmentation versions of ROSE 1000M1, 1000M2, CRW 16S.M, 23S.M datasets can be accessed from btab788_supplementary_dataClick here for additional data file.


DNA sequence alignment is a common first step in most applications of high-throughput sequencing technologies. The accuracy of sequence alignments directly affects the accuracy of downstream analyses, such as variant calling and quantitative analysis of transcriptome; therefore, rapidly and accurately mapping reads to a reference genome is a significant topic in bioinformatics. Conventional DNA read aligners map reads to a linear reference genome (such as the GRCh38 primary assembly). However, such a linear reference genome represents the genome of only one or a few individuals and thus lacks information on variations in the population. This limitation can introduce bias and impact the sensitivity and accuracy of mapping. Recently, a number of aligners have begun to map reads to populations of genomes, which can be represented by a reference genome and a large number of genetic variants. However, compared to linear reference aligners, an aligner that can store and index all genetic variants has a high cost in memory (RAM) space and leads to extremely long run time. Aligning reads to a graph-model-based index that includes all types of variants is ultimately an NP-hard problem in theory. By contrast, considering only single nucleotide polymorphism (SNP) information will reduce the complexity of the index and improve the speed of sequence alignment.The SNP-aware alignment tool (SALT) is a fast, memory-efficient, and SNP-aware short read alignment tool. SALT uses 5.8 GB of RAM to index a human reference genome (GRCh38) and incorporates 12.8M UCSC common SNPs. Compared with a state-of-the-art aligner, SALT has a similar speed but higher accuracy.https://github.com/weiquan/SALT.Herein, we present an SNP-aware alignment tool (SALT) that aligns reads to a reference genome that incorporates an SNP database. We benchmarked SALT using simulated and real datasets. The results demonstrate that SALT can efficiently map reads to the reference genome with significantly improved accuracy. Incorporating SNP information can improve the accuracy of read alignment and can reveal novel variants. The source code is freely available at  Advancements in next-generation sequencing technologies open opportunities to various biological analyses, such as de novo assemblies of bacterial and eukaryotic genomes, and species classification based on metagenomics studies . Short rIt plays a critical role in medical and population genetics. Conventional aligners map sequencing reads to a linear reference genome, which represents one or a few individuals. However, such a linear reference genome lacks information on the variation in the population and consequently does not reflect the genetic diversity of individuals.Augmenting the reference genome with known genetic variants can reduce the genetic distance between the donor and reference genomes and avoid allelic bias .More than a decade ago, several short read alignment tools, such as BWA , 7, SOAPRecently, several variant-aware aligners have been developed. BWBBLE  builds aHISAT2 modify the hierarchical indexing scheme from HISAT to create a hierarchical graph FM-index, which is combined with GCSA  to alignCompared to conventional sequence alignment tools, genomic variant-aware aligners can reduce the differences between the reference and donor genomes, which leads to better alignment accuracy. However, the ability to store and index various types of variations requires precious RAM space, and aligning reads to a graph-based index is not as efficient as conventional aligners. Among all types of genomic variants, single nucleotide polymorphisms (SNPs) are approximately ten times as numerous as others.Accordingly, indexing only the SNP information can not only improve the accuracy of alignment but also enable alignment with low memory requirements.In this article, we present SALT, a BWT-based short read aligner that incorporates genetic SNPs to augment the reference genome. It can effectively map reads to a reference genome with low memory requirements. We have benchmarked SALT on both a simulated dataset representing known variations and a real sequencing dataset with UCSC Common SNPs set. The results show that SALT can achieve higher accuracy and sensitivity than aligners that do not incorporate variation information. Furthermore, SALT is very efficient to map short reads and has only a small memory footprint. We believe that SALT, as an SNP-aware read alignment algorithm, has enormous potential in variant calling and other downstream biological analyses.https://github.com/weiquan/SALT.We have implemented SALT to align short reads to a reference genome and SNP database. It performs alignment for both single-end and paired-end reads and allows multithreading. The default output format is SAM . SALT isThe performance of SALT has been compared with that of the most widely used alignment tool, BWA-MEM (version 0.7.17-r1188). The aligners were tested on two simulated datasets and two high-throughput sequencing (HTS) datasets to assess their speed, sensitivity, and accuracy. All benchmarks were conducted on a desktop computer with 32 GB of RAM and a 3.30 GHz Intel i9-7900X processor with a total of 10 CPU cores running Linux Ubuntu 18.04.We simulated 4 million 100 bp and 150 bp Illumina-like reads from the human genome GRCh38 using Mason2  with a 0PA) if its best location is within a distance of 4 bp to the original coordinate. Given a dataset with N reads and n out of N are mapped, the sensitivity and accuracy are defined as follows:Table Concerning accuracy, regardless of the parameters, both SALT.linear and SALT.snp are more accurate than BWA. An example where BWA-MEM fail to achieve satisfactory sequence alignment is shown in Fig. Concerning speed, SALT.linear with the setting “-r 21” and the setting “-r 10” runs faster than BWA-MEM on all simulated datasets.Concerning memory, the peak memory usage levels of SALT.snp and BWA are 5.09 GB and 5.24 GB, respectively. Sufficient memory for both is available on most desktop and laptop computers.To assess the performance on real data, we benchmarked all aligners on two real datasets. Four million 100 bp reads sequenced with the Illumina HiSeq 2000 (SRA ID: ERR037900) and four million 148 bp reads sequenced with the Illumina HiSeq 2000 (SRA ID: SRR1766443) were mapped to the human reference genome (GRCh38).We ran two versions of SALT, namely, SALT.snp and SALT.linear, which use an index incorporating 12.8M UCSC Common SNPs (build 151) and the original FM-index, respectively. UCSC Common SNPs is a subset of dbsnp that have a minor allele frequency (MAF) of at least 1% and are mapped to a single location in the reference genome assembly.SALT was run with different overlap lengths in the seeding phase, leading to differences in speed and accuracy. BWA-MEM was run with the default settings.PA) if the edit distance (including clipping) between the read and the reference was less than 10% of the read length. The results are shown in Table Because the exact locations of the reads in the reference genome were not known, we assumed only that the reads should be mapped to the most similar locations in the human reference genome. Therefore, a read is considered a perfect alignment . SALT is a BWT-based short-read aligner that incorporates SNPs. Evaluations on both simulated and real data suggest that SALT based on a human reference genome incorporating 12.8M UCSC common SNPs (build 151) offers improved accuracy. SALT does not output alignments with edit distances greater than 10% of the read length, which leads to lower sensitivity than BWA-MEM. However, such alignments would be more likely to result in wrong alignments. Although aligning reads to an SNP-augmented reference is slightly slower than alignment to a linear human genome in some cases, it is considerably more efficient when aligning reads to the major histocompatibility complex (MHC) region. Moreover, SALT is naturally scalable to the alignment of reads that span more complicated genomic variants through the extraction of sequences nearby those genomic variants, and its utility will increase once large databases of genomic variants are available. Finally, SALT can output information about the SNPs in datasets , which would simplify and improve the current post-alignment processing pipeline.Overall, SALT is a fast, memory-efficient and SNP-aware short read alignment tool. This method shows enormous potential in variant calling and complex variation detection for a large population of genomes.x nt, this process will potentially decrease the number of seed matches, which will reduce the accuracy and sensitivity of the sequence alignment. Therefore, a compressed data structure is needed to index a graph genome.A linear reference genome and a set of SNPs can be represented as a graph genome by iteratively adding edges corresponding to non-reference alleles and terminating at nodes corresponding to genomic loci on the initial edge . A graphFerragina and Manzini first introduced the FM-index data structure in 2000. This structure extends the BWT representation of a string by adding suffix array (SA) and character occurrence (OCC) data structures. An FM-index is a compressed representation of a string that requires considerable memory and supports searching for text within a rather low search time. We apply a modified BWT-based index (SALT-index) to index k-mers along all possible paths of a graph.We define the alleles in the linear reference genome as the primary alleles and alleles that are not in the linear reference genome as alternative alleles. K-mers without alleles or with primary alleles can be indexed by the FM-index constructed for a standard linear reference genome, which is called the CFM-index. in section 2.2, we show that k-mers with alternative alleles can also be indexed by a variation of the FM-index. Thus, we propose an algorithm for SNP-aware alignment based on this index.concatenate all genomic sequences around SNPs with the symbol build the FM-index for the primary reference, which is called the CFM-index, and the FM-index for the alternative reference, which is called the RFM-index;generate maximal exact match (MEM) seeds based on the CFM-index and RFM-index and choose candidate alignment locations;perform SNP-aware pairwise alignment between the reference and reads and report the possible alignments.A flowchart of index construction and read alignment is shown in Fig. SALT implements the SNP-aware alignment in four main steps, as follows: entclass1pt{minimaSALT implements a BWT-based index (SALT-index) and an alignment algorithm to achieve fast and sensitive alignment of reads with respect to a reference genome and a large collection of SNPs. In contrast to other BWT-based aligners, our algorithm employs two different types of indices: (1) a four-alphabet FM-index of the primary reference genome (PRI-REF) that represents all k-mers without alternative alleles; and (2) a five-alphabet FM-index of the alternative reference genome (ALT-REF) that represents k-mers with alternative alleles.p and concatenate all sequences with By building the FM-index for the primary reference genome sequence, all k-mers without alternative SNPs are indexed (CFM-index). To retrieve k-mers containing alternative SNPs, we enumerate all possible sequences in the range nome GRCh8, the SATo perform SNP-aware pairwise alignments, we use a four-bit Gray code to encode the alleles at each site on the enhanced genome reference as introduced in BWBBLE . Each biSALT utilises a typical seed-extension strategy, and the algorithm for the alignment is shown in Algorithm 1.s in the CFM-index and obtain all occurrences of s in the primary reference. Then, we search for the seed s in the RFM-index (see Algorithm 2) and obtain all occurrences of s in the alternative reference,. The altLocate function for the RFM-index is presented in Algorithm 3.The alignment process requires the identification of all SNP-aware seeds. If a SNP is not contained in a seed, then the candidate location of the seed can be found in the CFM-index. If a seed does contain an SNP, then the seed can be found in the RFM-index . In the seeding phase, for any seed that does not contain any sequencing errors or contains a known SNP, we need to determine its location in the primary reference genome and use it as a candidate for SNP-aware pairwise alignment. We first search for the seed At this stage, we perform pairwise alignment between the read sequence and the reference genome sequence at the candidate locations to calculate pairwise alignment scores and choose the highest score location as the best alignment result. SALT supports two pairwise alignment algorithms, namely, snpLV and snpSW. In snpLV, the edit distance is used as the measure of sequence similarity (known SNPs do not increase the edit distance).We adapt the Landau–Vishkin algorithm , which iThe recurrence formula  for Lando and e are the gap opening penalty and gap extension penalty, respectively; S is the substitution matrix, which describes the substitution penalty when one character in a sequence is changed to another character; S is the substitution penalty for target sequence T[i] and query sequence Q[j]; and a and b are the match penalty and mismatch penalty, respectively. The formulas given above can be adapted for SNP-aware local alignment by modifying the substitution matrix. The modified substitution matrix is defined as follows:We adapt the Smith–Waterman algorithm  to imple library  is modifith hit for the first read and the jth hit for the second read, SALT computes their distance Given the If the two hits have the correct orientation and The values of lignment  for the SALT outputs the alignments of the reads mapped to the SNP-enhanced reference. SALT considers identical penalties for primary alleles and alternative alleles.However, some downstream analyses are based on linear genomes, which requires realigning all hits to the primary reference. We present a refinement program (Polish) to realign all hits to the linear reference genome and compute the optimal alignment using the standard Smith–Waterman algorithm. The realignment results are stored in SAM format.


The rise of metagenomics offers a leap forward for understanding the genetic diversity of microorganisms in many different complex environments by providing a platform that can identify potentially unlimited numbers of known and novel microorganisms. As such, it is impossible to imagine new major initiatives without metagenomics. Nevertheless, it represents a relatively new discipline with various levels of complexity and demands on bioinformatics. The underlying principles and methods used in metagenomics are often seen as common knowledge and often not detailed or fragmented. Therefore, we reviewed these to guide microbiologists in taking the first steps into metagenomics. We specifically focus on a workflow aimed at reconstructing individual genomes, that is, metagenome‐assembled genomes, integrating DNA sequencing, assembly, binning, identification and annotation. Metagenomics greatly facilitates the identification and characterization of known and novel microorganisms. The plethora of available bioinformatics tools with various levels of complexity rely on (shared) fundamental principles and methods, which are often not detailed or scattered. Here, we review these principles and methods with a focus on reconstructing individual genomes from metagenomics data by integrating DNA sequencing, assembly, binning, identification and annotation. Thanks to the ability to bypass the labor‐intensive isolation and cultivation steps, this approach could theoretically be used to detect and characterize a much wider range of prokaryotes. However, metagenomics is a broad term that encompasses different types of analyses, with various levels of complexity. Furthermore, the underlying principles and techniques are often seen as common knowledge and either not detailed or fragmented. Therefore, we reviewed these to guide microbiologists in taking the first steps into metagenomics. We specifically focus on a workflow aimed at reconstructing individual genomes in a sample, integrating DNA sequencing, assembly, binning, identification and annotation Figure .2A few steps precede the actual DNA sequencing, that is, sample collection and storage, and DNA extraction and purification. Although out of the scope, it is important to highlight the importance of these steps  sequencers, including the MiSeq, HiSeq, NovaSeq and NextSeq series of sequencers, which produce short reads 100−300 bases) with low error rates (<1%). Although variants of the technique are used, including single‐end and mate‐pair sequencing, paired‐end sequencing in which reads represent the ends of DNA fragments with a size specified by the library preparation step is most common in metagenomics applications , exist but have mostly been replaced by those mentioned above. This is likely due to their issues with identifying homopolymeric repeats and lower throughput  sequencers, including the RS, RS II, Sequel and Sequel II sequencers, use adapters to circularize DNA fragments, which are then placed inside wells where a polymerase copies the fragment using labeled nucleotides, producing a detectable fluorescent signal when they are bound to their complement by the polymerase. As each well should only contain one fragment and each nucleotide is labeled with a different fluorescent molecule, the polymerization process produces a continuous signal that represents the fragment being sequenced. Because the fragments are circularized, short fragments may be read multiple times before the polymerization process terminates, whereas long fragments may sometimes fail to be sequenced fully Figure . As a reIn one approach, DNA is fragmented into shorter sequences of a length varying between 1,000 and 15,000 bases. Because these fragments are relatively short, the process described above can read the same sequence multiple times, with each pass of the polymerase producing a “subread” Figure . This prIn the other approach, fragmentation results in lengths of around 50 kb. This produces a much lower accuracy (an error rate of 10%−15%) when compared to CCS, but the reads are also longer and can cover larger portions of the target genome. Since the number of reads produced in a run is limited by the sequencer, this effectively also increases throughput.2.3The Oxford Nanopore Technologies (ONT) sequencers, like the PacBio sequencers, operate in real time and are capable of producing the longest reads out of the three technologies. These instruments reconstruct DNA sequences based on current fluctuations elicited by DNA molecules when they pass through nanopores embedded in a membrane genome analysis is assembly, in which reads are assembled into longer, contiguous sequences (contigs). Ideally, contigs should correspond to individual replicons, but this is rarely the case. It is also important to mention that in metagenomics the sequenced DNA fragments originate from different cells and include potential differences. As such, genomes assembled from metagenomes (MAGs) produced with current technology cannot be obtained directly after an assembly step, and are derived from the pan‐genome of taxonomically related groups within the target environment rather than individual genomes.There are currently two dominant approaches to performing assembly. The first, and older, approach is called overlap, layout, consensus (OLC), whereas the newer approach is de Bruijn graph‐based (DBG)  as the basis for building a graph, thereby avoiding explicit alignment. This graph can then be traversed to uncover the original sequence. Although it removed the need to explicitly align sequences, the DBG approach is more sensitive to sequencing errors and has trouble resolving repetitive regions. However, neither of these drawbacks is particularly relevant for Illumina short reads, which have high accuracy, and whose length is also insufficient to resolve long repeats.DBG functions in a different manner, as it was proposed to bypass the overlapping step, which rendered OLC unusable for the kind of data produced by second‐generation sequencers such as those in the Illumina family of sequencers. Rather than building a graph based on overlaps, DBG uses overlapping oligonucleotides of fixed size  is becoming more common, and available computational resources are increasing, OLC assemblers are once again gaining importance thanks to their ability to resolve repeats. However, because of this recent paradigm change, the state‐of‐the‐art in assembly software is evolving rapidly. New more performant tools that can handle multiple types of reads, either separately or together as “hybrid” assemblers, are being published frequently.3.1“Overlap−layout−consensus” (OLC) assemblers, which are conceptually more straightforward than the DBG assemblers, are separated into three steps Figure , each wiIn the first—overlapping—step, matching fragments are identified. There are ways to determine whether two reads have the potential to contain alignments rapidly, including for example whether or not they contain identical subsequences Pop, . This cak‐mer length, and can therefore be more flexible and accurate, especially for long reads, because they use the full length of the supplied reads rather than splittng them further. The main disadvantage of OLC assemblers is their speed. Although heuristic methods have been developed, which can drastically reduce the computational costs of alignment (see below), these remain slower than the DBG approach.Unlike DBG assemblers, OLC assemblers are not constrained by a fixed In the early 2000s, a number of OLC assemblers were developed, which have for the most part become irrelevant due to lacking the optimizations necessary to make these assemblers usable in a modern context. Notable among these was the Celera assembler  is assigned a numeric value that solely depends on its content. By doing this, searching for relatively large elements in a database becomes considerably faster. This is necessary for connecting the subgraphs generated from each read to each other, which is done by checking whether a given node has already been recorded in the hash table for the whole graph. Hash tables can also be used to reduce memory requirements if handled cleverly. Another, more drastic approach to reducing memory requirements is a Bloom filter. A Bloom filter is similar to a hash table in that it relies on hashes, but unlike hash tables, Bloom filters can only be used to indicate whether an element might be present or is absent. A Bloom filter stores its data in a predefined set of bits. When adding elements to a Bloom filter, multiple hashes of that element are produced, which correspond to the indices of the bits of the Bloom filter that should be set. Checking whether an element has been added to the filter, consists of checking whether all the associated bits have been set. Because each element sets multiple bits, it is possible for all the bits associated with an element that have not been added to the filter to have been set by accident. However, this issue can be minimized through proper parametrization and the resulting structure can still be very compact.Two techniques that handle DBGs stand out in particular: hash tables , then assembling these long reads. The second approach consists of creating a preliminary assembly graph using only the short reads, then using long reads as a guide to traverse said graph, which is the strategy used by most assemblers, including HybridSPAdes  standard was introduced to ensure quality , it is necessary to find a signal that is similar within closely related DNA sequences, but dissimilar to less‐related genomes. In bacterial genomes, such a signal is found in oligonucleotide frequencies, with each species having a specific set of over‐ and underrepresented oligonucleotides. This has already been observed three decades ago  whose total number is 107 according to Ankenbrand and Keller  genes, and others, such as d Keller . MaxBin d Keller  is an exAnother intuitive method to separate genomes within an assembly is to rely on coverage, which is computed by counting the total number of bases from the reads that can be aligned to the final assembly. Assuming that the relative abundances of each species within a metagenome are different from each other, the relative abundance of reads should follow a similar trend. This principle was first used in AbundanceBin Wu & Ye,  and lateComposition, universally conserved genes and coverage can be derived from any assembly without additional external information. However, a plethora of statistical and machine‐learning approaches can be used to augment and exploit these data. As an example, BMC3C uses automatic gene detection that enables codon usage analysis  . In the case of metagenomes, identification may also be performed on individual contigs or reads. This process, called classification, can be performed in combination with or as an alternative to binning, depending on the extent to which the target environment has been characterized.5.1Identification is in most cases rather straightforward—as it consists of aligning the contigs to known reference genomes to find the best match. Despite significantly improved alignment algorithms through the use of indexing and heuristics, this step remains computationally expensive if a rigorous approach involving local alignment is used  steps in the described metagenomics workflow rely on alignment, that is, the process of finding similar subsequences within sets of sequences and identifying the differences. Therefore, for the sake of completeness, alignment concepts were included in this overview.Conceptually, alignment can be seen as a number of string matching operations, which can be exact or approximate. Exact string matching is useful for locating short sequences and its results can be used as a starting point for approximate string matching of larger strings. Exact string matching is generally not relevant for longer sequences, either due to the presence of sequencing or assembly errors or to a biological divergence between the query and reference sequences. However, even with these changes, homologous sequences can still be found using inexact string matching. For this case, there are essentially two ways of aligning sequences to one another: either to identify all the changes that need to occur to convert one sequence into another  or to find the best matching regions of the two sequences .The distinction between global and local alignment is particularly relevant in genomes, as both types of alignment are used, but in different contexts. However, on the whole‐genome sequence scale, alignment must also handle transpositions, where a long sequence is located in one place in one genome and in another place in the other, and inversions, where sequences are replaced by their reversed complementary sequence. For this task, it is necessary to first identify orthologous sequences, a process that is severely complicated by the fact that there are usually differences between these sequences and that the abundance of these differences varies from pair to pair.6.1m, such as an oligonucleotide, within a larger set of characters of length n, such as a genome. The “naïve” approach to finding similar strings is simply to check every possible position within the larger set of characters, then verify if the subsequent characters match the query. However, this approach is extremely inefficient and more efficient approaches have been developed, culminating in suffix trees stored as suffix arrays.String matching is the process of finding a given set of ordered characters (a string) of length O(m) (Big O notation), rather than the O(m × n) of the naïve method. Construction and required memory are both O(n2)  time, they can also be constructed in O(n) time and occupy O(n) memory  Figure , which r Skiena, . By usinO(n) time to be built. While this is useful for performing multiple searches on the same genome, when searching for a single sequence within a larger sequence only once, an alternative approach exists that requires less than O(n) time in a usual case and O(n/m) in the best case, called the Boyer−Moore algorithm. In this approach, the query string is first used to construct a table that contains the relative location of the next identical character for each possible character. Once this array is built, it becomes possible to skip portions of the text whose ending does not match that of the query, since a mismatch can be used to infer the closest possible position of a potential match, and avoid comparing the query against all intermediate positions. Because this algorithm indexes the query rather than the larger sequence, suffix trees are generally more practical.However, the suffix array approach still requires Unlike the ideal case of searching exact matches, for which objective and fast methods have been presented above, alignment of genetic sequences typically involves mismatches. This can be because of inherent mismatches in sequences that occur when searching for known genes in novel organisms, or when attempting to find overlaps between reads of sequencing data that contain errors. In either case, approximate string matching algorithms become necessary.6.2O) computational time and space  are expected to be. This means that for sequences that are expected to be similar, distance can be computed at a low computational cost.Identifying differences between two sequences is a relatively straightforward process that usually relies on edit distances, the most well‐known of which is the Levenshtein distance, which assigns an error cost for each insertion, deletion and substitution between two strings of text, and reports the sum of produced mismatches between two strings as the distance between them. Under specific conditions, these edit distances can be calculated in Ukkonen, , where sBecause global alignment can be viewed as an optimization problem, the result of such an operation is intrinsically linked to the evaluation criteria that are sought to be optimized by the algorithm. Because of this, it is important to remember the context in which such alignments are performed. In the present work, two such contexts are particularly relevant. The first relevant context is assembly, since long‐read sequencers tend to produce more insertions and/or deletions than their short‐read counterparts  in the worst case, the computational cost of finding an optimal solution to an MSA problem is factorial in the number of sequences. This means that the only realistic options are to rely on heuristics with no guarantees of producing an optimal solution. MSA is necessary for establishing the consensus sequences, be it for establishing the “true” sequence of the genome underlying a set of sequencing reads, or to find the common ancestor of a set of orthologous genes. For global alignment, MUSCLE  can be computed within a reasonable time frame of E Edgar, , ClustalE Edgar, , and theE Edgar,  are all 6.3Unlike global alignment, where the focus lies in uncovering the “least expensive” set of changes necessary to convert one sequence to another, local alignment focuses on locating the position of matching (sub)sequences. The most common local alignment approach is the seed and extend approach, which operates by identifying short matching subsequences within the sequences to be compared and then attempts to extend them until a significant enough number of inconsistencies is detected or one of the two sequences ends. The way these shorter sequences are identified varies from program to program.Some programs, such as the first incarnation of BLAST  implemented in Bowtie , although the latter is not strictly unique to metagenomics.Unlike alignment of assembled sequences against a reference, nonspecialized tools have an insufficient performance to perform read alignment. BWA and Bowtie2 have long been established as standard tools for aligning reads against a known reference or an assembly, but are intended for short highly accurate reads produced by Illumina and older short‐read technologies. With the advent of long error‐prone reads, newer approaches such as Minimap2 had to be developed to perform the same operation on long reads. Moreover, for metagenomics specifically, these tools are not sufficient to extract meaningful information on their own, nor are they intended to align reads against large databases. Because of this, when aligning reads against a reference database, the tools mentioned in Section Gleb Goussarov: Conceptualization (lead); writing—original draft (lead); writing—review and editing ; Mohamed Mysara: Conceptualization (supporting); writing—review and editing (supporting); Peter Vandamme: Conceptualization (supporting); writing—review and editing (supporting); Rob Van Houdt: Conceptualization (supporting); writing—original draft (supporting); writing—review and editing .None declared.None required.