Using standard benchmarking methods on simulated data and a new “concordance test” benchmark on real ribosomal RNA alignments, we show that the extended program dnamlε improves accuracy relative to the usual approach of ignoring gaps, while retaining the computational efficiency of the Felsenstein peeling algorithm.A fundamental task in sequence analysis is to calculate the probability of a multiple alignment given a phylogenetic tree relating the sequences and an evolutionary model describing how sequences change over time. However, the most widely used phylogenetic models only account for residue substitution events. We describe a probabilistic model of a multiple sequence alignment that accounts for insertion and deletion events in addition to substitutions, given a phylogenetic tree, using a rate matrix augmented by the gap character. Starting from a continuous Markov process, we construct a non-reversible generative (birth–death) evolutionary model for insertions and deletions. The model assumes that insertion and deletion events occur one residue at a time. We apply this model to phylogenetic tree inference by extending the program  dnaml, in Felsenstein's popular phylip package).We describe a computationally efficient method to use insertion and deletion events, in addition to substitutions, in phylogenetic inference. To date, many evolutionary models in probabilistic phylogenetic inference methods have only accounted for substitution events, not for insertions and deletions. As a result, not only do tree inference methods use less sequence information than they could, but also it has remained difficult to integrate phylogenetic modeling into sequence alignment methods (such as profiles and profile-hidden Markov models) that inherently require a model of insertion and deletion events. Therefore an important goal in the field has been to develop tractable evolutionary models of insertion/deletion events over time of sufficient accuracy to increase the resolution of phylogenetic inference methods and to increase the power of profile-based sequence homology searches. Our model offers a partial answer to this problem. We show that our model generally improves inference power in both simulated and real data and that it is easily implemented in the framework of standard inference packages with little effect on computational efficiency (we extended  A fundamental task in sequence analysis is calculating the probability of a multiple alignment given a phylogenetic tree relating the sequences and an evolutionary model describing how sequences change over time. This quantity is already at the heart of phylogenetic tree inference by either maximum likelihood The general approach of modeling residue substitution as a continuous-time Markov process was introduced by the Jukes-Cantor model of nucleotide substitution in DNA ungapped multiple alignment using Felsenstein's peeling algorithm phylippaup* Given a substitution model and a tree, one can efficiently calculate the probability of an tkf91) tkf91 treats insertion and deletion events as a continuous-time process governed by explicit insertion and deletion rate parameters, allowing multiple insertions and deletions to accumulate at the same “site” over long times. tkf91 improved, for example, upon methods that parsimoniously assume no more than one change per site per branch, including the pioneering Bishop and Thompson pairwise alignment likelihood model that preceded tkf91 tkf91 have appeared tkf91 or related models have addressed the problem of simultaneously aligning and inferring the phylogeny of a group of related sequences tkf91 , the problem of inferring the optimal insertion and deletion history given a tree has been formally shown to be NP-complete tkf91-based or otherwise, seeks to make approximations that make calculations tractable.A variety of more formal approaches for treating insertions and deletions exist given multiple alignment and phylogeny. A tkf91-based approach for this problem tkf91 is not invariant under column rearrangement) a sum over all subalignments compatible with a given homology structure is still required. As a result, though time complexity exponential in the number of sequences can be avoided in the average case where not many gaps occur, the approach remains expensive in absolute terms; a Bayesian Markov chain Monte Carlo phylogenetic inference for an alignment of 10 globins was reported to require 3 CPU hours on a 1.25 GHz G4 Apple Macintosh tkf91 has many desirable and realistic properties as a model of evolution, it would be advantageous to have even more computationally efficient approaches, particularly for problems where a gain in efficiency might outweigh sacrificing some of the realism of the model.Here we are specifically concerned with the problem of calculating the probability of a An alternative is to use a continuous-time Markov process for insertions and deletions that remains fully compatible with the Felsenstein algorithm K+1)×(K+1) rate matrix for K residues (4 nucleotides or 20 amino acids) plus the gap character to describe the rate of change of a residue to a residue (substitution), a residue to a gap (deletion), and a gap to a residue (insertion) The originating idea is to use an extended  sequence to another, and show how that generative process relates uniquely to the column-by-column inference algorithm that one will apply to a given multiple alignment.It has generally been thought that models based on a gap-extended rate matrix must be conceptually flawed, because it appears necessary to assume that all ancestral and descendant sequences fit in a fixed number of columns. This fundamentally conflicts with allowing any number of insertions and deletions to occur, and it produces a so-called “memory effect” artifact Here we will develop such a generative model, by borrowing terms from a Markov process for a rate matrix extended for the gap character. The rate matrix includes the gap character, but the subsequent generative evolutionary model does not treat gaps as an extra residue. Rather, it describes evolutionary insertion and deletion (birth-death) events Similarly, one must have a consistent way of dealing with columns that are unobserved in the alignment of extant sequences – that is, places where ancestral residues have been inserted and deleted, where alignment columns would exist if all the ancestral sequences were known in addition to just the extant sequences. Therefore the likelihood we calculate for an extant multiple alignment will be its marginal likelihood, marginalized over all possible ancestral sequences including unobserved alignment columns that left no trace in the observed alignment.K+1 characters including gaps occur in the ancestor at the stationary frequencies of the Markov process). Because our model is non-reversible, we must always work with rooted phylogenies.Another conceptual problem of column-based models arises if one adopts the usual practice of making the substitution process reversible . If one assumes reversibility for a substitution process that includes gaps, one necessarily imposes a frequency of gap characters that is constant with respect to divergence time Because we map generative evolutionary model events onto an alignment column, the column is assumed to be correctly aligned phylogenetically – all aligned residues are assumed to be homologous and related only by substitution events. This means that no more than one insertion event may occur in any given column. Enforcing this assumption requires modification of the usual Felsenstein peeling algorithm to include some extra bookkeeping.Given alignments are unlikely to be phylogenetically correct, because humans and alignment programs tend to produce aesthetically pleasing alignments that compress columns containing few residues. Importantly, for any arbitrary tree topology and any arrangement of observed residues and gap characters in an alignment column, there exists at least one possible assignment of characters to ancestral nodes that makes all extant aligned residues homologous. Therefore the problem with using phylogenetically incorrect alignments is not that the algorithm will fail altogether , but rather that we can expect its inference ability to be degraded by forced inference of incorrect histories in phylogenetically incorrectly aligned columns. How much the overall inference is degraded by this and by the other assumptions described above is a matter for empirical testing, which we describe in the second half of the paper, after we describe the model itself.K residues, probabilistic substitution models are defined by a K×K rate matrix R such that the matrix of conditional probabilities tQ ≡ P is given byFirst, we start by solving the Markov process associated with a rate matrix extended to include a gap character. For an alphabet of K+1)×(K+1) matrix εR that depends on arbitrary rates of deletions μ≥0 and insertions λ≥0:p =  is the distribution of inserted residues ijδ stands for the Kronecker delta in the K×K subspace . Since the extended rate matrix εR has the property that each row adds up to zero, we can construct a model of evolution for the extended rate matrix defined asWe extend this to include the gap character by augmenting the rate matrix to a .Generally, the extended conditionals K×K rate matrix R, if we assume that the distribution of inserted residues in equation (2) is the stationary distribution associated to the reversible rate R  = jRπ) then one can derive the analytic expression for K×K conditional matrix tQ = tRe and the rates of insertion and deletion. This particular solution for the extended conditional probabilities is given bytγ and tξ are given byλ = μ = 0) both functions are defined as identically zero.In particular, for a reversible K×K conditional substitution matrix subspace tM in R and the insertion and deletion rates. Generally, any biologically relevant K×K substitution rate matrix R must have zero as a non-degenerate eigenvalue, and at least one other negative eigenvalue. To express tM in general form, let  represent the eigenvalues of R, for 1≤A≤K−1, with ae>0 for 1≤a≤A. Then the standard conditional substitution probabilities could be expressed as:aO is a K×K matrix of real numbers specific for each unique nonzero eigenvalue. aO matrices from the similarity transformation that relates R with its diagonal form.Finally, we have to describe how to obtain the tM is expressed as:λ = μ = 0, tM is defined as the standard substitution conditional matrix tQ in Equation 8.Now the extended conditional matrix in the substitution-only subspace R(i≠j) = jβπ+αΔij, where ijε is 1 if i,j are both either purines or pyrimidines, and 0 otherwise. The F84 rate depends on two non-negative parameters α and β, and on a stationary residue distribution π. The F84 rate matrix has non-zero eigenvalues e1 = β, and e2 = α+β. The matrix O1 is given by O1 = Δij−jπ, and the matrix O2 is given by O2 = ijδ−Δij. For the gap-extended F84 model, we obtainλ = μ = 0 is defined as the original F84 model for substitutions,For example, the F84 model et al.λ = β and μ ∝ β.The conditional model of McGuire R, or a non-reversible substitution rate matrix, A more detailed description of the characteristic differential equations for an arbitrary gap-augmented rate matrix and the particular solution described above is provided in iλπ/(λ+μ) for a residue, and μ/(λ+μ) for the gap character. However, one has to distinguish between the reversibility of a Markov process and the reversibility of an evolutionary process constructed using that Markov chain. A Markov chain is said to be reversible if and only if there exists a marginal distribution that satisfies the reversibility condition A note about reversibility. For a reversible substitution rate matrix, the particular Markov process including gaps solved here is also reversible, as can be seen by using the marginal frequencies Up to this point, this is essentially McGuire's model tπiξ in Equation 7, the probability of a deletion as proportional to tγ in Equation 6, and that of a substitution as proportional to tM in Equation 9, but we ignore the gap to gap transition of the Markov process. This allows us to derive a generative model that describes insertions and deletions not as mere “gap character” replacements on a fixed length alignment but as true evolutionary events (births and deaths of residues).We construct the generative model as an independent product of single-event  contributions. What we will do to construct the model is to borrow terms from the conditional Markov process introduced in the previous section. We describe the probability of an insertion as proportional to x = {x1…lx} that evolves to another sequence y = {y1…ly′} corresponds unambiguously to a pairwise alignment in which a substitution is represented by a conserved or mismatched column, a deletion by an ancestral residue aligned to a gap in the descendant sequence, and an insertion is represented by an ancestral gap aligned to a residue. Let xˆ = xˆ1‥xˆL and yˆ = yˆ1‥yˆL mean the aligned sequences x and y. Specifically, the probability that y was generated from x after time t with pairwise alignment xˆyˆ that includes s≤l substitutions, (l−s) deletions, and (l′−s) insertions {I1…Ils′−} (where the subset of residues x are substituted by y) is constructed astγ, tξ and π, is set to the stationary distribution of the substitutions rate matrix. In that way, for a reversible substitution rate matrix, this generative model is quasi-reversible (i.e. reversible in the substitutions subspace).A given sequence tξ)l+1 is borrowed from the Markov process. This extra term is responsible for having a normalized distribution, such that the sum of the contributions of all sequences y of all possible lengths and all possible alignments is one. The extra term accounts for the fact given a sequence x of length l, insertions can occur at (l+1) places. That is, at each of the (l+1) places that an insertion in x could occur, an insertion of length z occurs with probability In equation (12), everything but the term , one can calculate the expected length (in residues without gaps) of descendant sequences originated from an ancestral sequence of length tkf91 model Crucially, this model does not assume that all observed sequences are generated from a preset number of aligned columns. Thus there is no “memory effect” εrate.We have implemented this generative model in a computer program for evolving sequences named conditional probabilities of the generative model, one wants to calculate the joint probability of a multiple alignment of sequences generated with the model, given a phylogenetic tree. Before doing that, let us consider the joint probability of a pairwise ancestor/descendant alignment, as a building block for a full-fledged algorithm for phylogenetic inference on multiple alignments.Given the xˆ,yˆ) after divergence time t using the expressionεP(x) is the probability of the ancestral sequence. This distribution is a prior, not determined by the generative model. In order to have this prior factorize effectively into columns, we will assume the length of ancestral sequences follows a geometric distribution (1−p)lp for a sequence of length l with arbitrary Bernoulli frequency parameter 0<p<1.Using the same notation as in the previous section, we can calculate the joint probability of the pairwise ancestor/descendant alignment  = (1−p)(1−tξ) that cannot be associated to any observed column. Intuitively this term can be viewed as an extra (terminating) column (★). An analogous “extra column” term will appear in the multiple alignment case with the Felsenstein pruning algorithm.The joint probability of a pairwise ancestor/descendant alignment can then be factorized as a product of terms over alignment columns as:In From the joint alignment probability distribution, we can also calculate the expected frequencies of insertions ents see . They arBecause the model is non-reversible, Felsenstein's peeling algorithm is an efficient algorithm for calculating the probability of a multiple alignment given a tree and a Markov substitution model. Extending the Felsenstein algorithm to include insertion and deletion events with the model we propose here requires four modifications. Those are: extra bookkeeping in the Felsenstein recursions to enforce that no more than one insertion occurs per column, so that all aligned residues are homologous; including a term from the prior ancestral sequence length distribution in the calculation of each individual column likelihood; including in the overall alignment likelihood the extra normalization terms collected in the “extra column” (★); and finally, marginalizing the contributions of possible ancestral residues that have left no trace in extant sequences. Otherwise, the substitution model assumed by the Felsenstein peeling algorithm is simply replaced by the generative model described in the previous section by Equation 12 which includes substitutions, insertions and deletions.u in the alignment, let uP be the probability up to node k given that the character (residue or gap) at node k is i. For a residue i, uP cannot contain any insertion only substitutions and deletions. For a gap, uP has to include one and only one insertion. Thus, uP is defined to be zero if all the nodes under k are gaps for that position. These probabilities are calculated recursively starting from the leaves of the binary tree as,Using the notation of k is a leaf, for a residue i,If node k is not a leaf, for a residue i,k, k to its left and right child respectively, and where the probabilities for the daughter nodes have already been calculated by the recursion. ku stands for the subset of leaves under node k for column u, and ku = – indicates that all leaves under node k are gaps for column u. The single-event conditional probabilities are dictated by the generative model in Equation 12 as,i,j≤K, where the functions tγ, tξ and If node P, we need to use the probability of the sequence at the root. The total probability of site u is given byp is the parameter of the geometric distribution of ancestral sequences, and π are the residue frequencies at the root.The second modification is due to the existence of a length distribution for ancestral sequences for this model. In order to calculate the total probability of a given aligned column l individual columns, there is an additional term in the equation:P = (1−p)P★(rootL) is calculated by a similar peeling algorithm,Third, the factorization in columns of the unconditional in length alignment distribution leaves some normalization terms that we gather together into what we think of as an “extra column” (★) contribution. Thus, when calculating the total probability of a multiple alignment as the product of P(L0) the probability of an alignment with L0 observed columns after marginalizing the unobserved columns. The probability of an alignment with L0 observed columns and k unobserved columns is given byugap is a particular case of Equation. 25. Then, for a given alignment of L0 observed columns, we marginalize to all possible unobserved columns as,L0 columns, after marginalizing all possible unobserved ancestral residues.Lastly, one could delete an ancestral residue leaving no trace in extant sequences—a column could exist but not be present in the observed alignment—therefore we are interested in calculating λ = μ = 0 . In addition, λ = μ = 0 are the optimal parameter choices for any ungapped multiple alignment for any given tree. Notice that for the previous statement to be true, it is crucial to have the extra term (1−tξ) in the generative process of a substitution. Once we set μ = 0, the residue-residue substitution process in the presence of gaps (described by Equation 9) reaches the same asymptotic value (described by Equation 8) for any value of λ, including both the limit λ = 0 and the limit λ = ∞. It is the extra term that renders the total probability of any finite-length alignment to zero in the case λ = ∞, and makes λ = 0 optimal on ungapped alignments.This algorithm will now reproduce the results of the original Felsenstein algorithm for ungapped alignments when the parameters K and a multiple alignment of l columns and n sequences.The extended peeling algorithm has worse-case time complexity dnaml . After each new branch is added to the tree using dnaml's Newton-Raphson method, dnamlε midpoint roots the tree, and then jointly optimizes both branch lengths and gap parameters using conjugate gradient descent. Midpoint rooting is an easy but simplistic rooting method; we made no attempt in dnamlε to optimize the placement of the root, though this would be possible without a significant efficiency cost, using methods described in p could be estimated using similar techniques.For simplicity, in this paper's results, we approximate the average length of ancestral sequences . Given a gapped alignment, comparison of the two implementations allows us to ask how much performance improves when gaps are treated by our extended model as opposed to treating gaps as missing data.Given an ungapped alignment, dnamlε and dnaml in three types of benchmarking experiments. First, using simulated ungapped alignment data, we confirmed that the two programs give essentially identical results when no insertions and deletions are present. Second, we assessed the ability of the two programs to accurately infer phylogenetic tree topologies for simulated alignment data with insertions and deletions from arbitrarily generated trees. Third, we assessed the programs' ability to correctly infer phylogenetic trees on real ribosomal RNA data.We compared The problem with benchmarking phylogenetic inference methods is knowing the ground truth. Evolutionary trees are only experimentally known in a few unusual cases for short time scales and rapidly evolving organisms; all other trees have been inferred. The strength of simulated data experiments is that the tree is known, but their weakness is that the simulation must assume an evolutionary model that may be biologically unrealistic and/or too similar to the evolutionary model assumed by the inference method to be tested. On the other hand, the strength of testing on real biological alignments is that the data are realistic, and the weakness is that the true tree is unknown. To evaluate inferences on real data, we have developed a “concordance test” that does not rely on the true tree being known. Testing on both simulated and real data should help compensate for weaknesses of either approach.dnamlε and dnaml produce essentially identical inferences when no insertions or deletions are present. This is a control experiment, making sure that dnamlε correctly infers that optimal deletion and insertion rates are 0 on ungapped alignments, and that the likelihood calculation correctly reduces to the original ungapped version.We first made sure that We generated simulated 8-taxon rooted trees using the algorithm of Kuhn and Felsenstein rose to generate simulated alignments from these sampled phylogenetic trees rose because it allows us to implement reasonably realistic models of insertion and deletion, which we will describe and use in the next section; in these initial ungapped control experiments, we turn off the insertion and deletion parameters. We modified rose to use the F84 rate matrix model for residue substitution, the same model used in dnaml and dnamlε. We used a uniform stationary distribution of 25% for each residue, and a transition/transversion ratio of 2.0 (the dnaml default).We used the program dnamlε reports time in units of changes per site, where changes include substitutions, insertions, and deletions.) For each choice of average branch length, we sample 100 different trees. For each tree, we generate different alignment lengths ranging from 50 to 1000 in steps of 5. A total of 900 trees and 171900 alignments were generated . For each alignment, we infer a maximum likelihood tree topology using dnamlε and dnaml.We sampled trees for 9 different average branch lengths, ranging from 0.005 to 2.0 substitutions/site and branch in roughly 2–3× multiplicative steps (corresponding to average pairwise identities ranging from 98% down to a fully saturated 25% in the 8 sequences). , the symmetric difference distance (SDD) dnaml's optimizer, which infers overly large branch lengths , whereas dnamlε results, produced by a different implementation of optimizer, are closer to the correct value (1.55±0.05). Overall, with the exception of this minor difference in numerical optimization, these results indicate that dnamlε and dnaml do produce essentially identical results on ungapped alignments, as expected.The results are shown in dnamlε would produce better inferences on gapped alignments than dnaml. rose simulates insertions and deletions by inserting and deleting a block of residues at a random position according to a chosen length distribution, and controlled by insertion/deletion probability parameters insp and delp per unit branch length. Note that because rose generates multi-residue insertions and deletions, it simulates a different and more biologically realistic process than the column-independent insertion/deletion process our model assumes. We therefore expect our model to suffer in the simulation from its assumption that gaps are uncorrelated, much as it would in real data; the question is whether this is outweighed by having at least some model of insertion and deletion, compared to dnaml's treatment of gaps as missing data.Using the same protocol as above, we then asked whether rose sample insertion and deletion lengths l from a Poisson distribution λ = 0.5. This particular distribution allows with probability 0.3935 insertions or deletions of more than one nucleotide at the time. This is still a weak violation of the column independence assumptions, and in the next section we will test the limits of our model regarding this issue. We used a range of 14 different gap probability parameters insp = delp, ranging from insp = delp = 0 (no gaps) to insp = delp = 0.1.We had dnaml , and in the second (blue points), setting insp = delp = 0.001 and an average substitution branch length of 0.07 produced alignment datasets with substitution events that averaged 20% in pairwise comparisons, and insertion/deletion events that averaged 13% in pairwise comparisons. These average percentages of substitutions and gaps in the blue dataset are reasonably realistic, comparable to those observed in our real rRNA alignments; see dnamlε being able to use the information in gaps in addition to substitutions: for dnamlε inferences, the blue dataset (the one with gaps) is more informative, and accuracy increases.Results of an illustrative comparison of two choices of gap probability and average branch length are shown in dnamlε versus dnaml for a large range of alignments with gaps. In order to summarize with one number curves similar to those presented in auc) in analogy to ROC curves. In particular, we introduce the relative auc difference, f, the quantity auc for dnamlε versus that of dnaml relative to the larger of the two auc's. dnamlε shows consistently better performance than dnaml, and the improvement increases with the percentage of gaps in the alignment.dnamlε infers longer branch lengths than dnaml (in units of changes/site) because it counts insertions and deletions as changes, in addition to substitutions.clustalw version 1.83. The more noticeable effect of realigning is that the columns of the alignment get compressed. The number of substitutions increases and the number of gaps decreases respect to the phylogenetically correct alignments. As expected, performance of dnamlε declines in terms of predicting the phylogenetic topology correctly, . However, as seen in dnamlε still shows improvement in all cases in terms of the nBSD measure which takes into account the combined effect of both the accuracy of the topology and the accuracy of the branch lengths. These results show that the loss due to the lack of a phylogenetically correct alignment seems to be compensated by the contribution to branch lengths introduced by treating insertions and deletions as real evolutionary events.dnamlε is similar to that of dnaml. Both scale linearly with alignment length . For both methods, the scaling factor increases as the number of substitutions and the number of gaps increases  about a three-fold increase in average run time compared to dnaml, whether the alignment has gaps or not.The time performance of ases see . dnamlεεrate. The model inserts residues following a geometric distribution controlled by the time-dependent parameter tξ in Equation 7. We applied the protocol described in the previous section. Results for gapped alignments with a large range of degrees divergence (comparable to those presented in the previous section) are given in As a matter of completeness, we have also sampled from the generative model described in this work using the program rose sample insertions and deletions from a distribution that simulates a DNA alignment of a protein-coding region. To that purpose, we used a nucleotide version of the empirically-derived coding gap distribution introduced in simprotrose in order to allow insertions and deletions to occur randomly but only at multiple of three positions with respect to the start of the alignment.In order to test the limitations of considering individual gaps as independent events, we also had simprot gap length distribution depends on one parameter c/t. We selected three different values, in increasing order of divergence: c/t = 100 for which p(3 nts) = 0.7238 and p(24 nts) = 0.0001; a second distribution c/t = 6 which is a good approximation to a empirically-determined distribution trained on protein sequences with less than 100 PAM sequence divergence c/t = 1 which corresponds exactly to a distribution empirically determined from proteins sharing no more than 25% sequence identity c/t = 6 distribution, one has p(3 nts) = 0.4510 and p(99 nts) = 0.0009. For the c/t = 1 distribution, one has p(3 nts) = 0.2938 and p(99 nts) = 0.0034. For each of the three protein-coding gap distributions, we applied the same protocol as for the Poisson distribution used previously.The c/t = 100 coding gap length distribution dnamlε still shows an improvement respect to dnaml at least for the two topological measures, this is despite the fact that this distribution breaks the column independence assumption more strongly than the Poisson distribution used in the previous section (p(l≥6 nts) is 0.2762 for this distribution, compared to 0.0002 for the Poisson distribution). However, for the two more divergent coding gap length distributions, the size of the insertion/deletion blocks is so large that by taking gaps into account, we cannot even reproduce the topology of the original substitutions-only tree.As we observe in dnamlε does not suffer for some unexpected reason from significantly more systematic long-branch attraction than dnaml itself, we used 4-taxon trees depending on two parameters t3), the other parameter is the branch length of the other two opposite leaves . Using the same protocol as before, we assessed the programs performance on alignments generated according to 4-taxon trees in which the two-branch parameter is 10 times longer than the three-branch parameter, for all gap length distributions used above.To make sure that dnamlε tends to perform better than dnaml, and the improvement is larger for the case prone to long branch attraction (t3 = 10t2 or r = 10) than for the balanced case in which all five branches are equal (t3 = t2 or r = 1). For the realigned tests, both dnaml and dnamlε show a certain long-branch attraction tendency when the amount of substitutions nearly triples that of gaps. For the more extreme coding distribution, the effect seems to be more severe for dnamlε. This effect can be attributed to the column independence assumption. These results show that while there is some systematic long-branch attraction for dnamlε and dnaml that effect occurs in some extreme situations which we do not see reproduced in real alignment such as those of rRNA.Results are presented in We developed a test to evaluate a phylogenetic inference method on real data, called the “concordance test”. We split an alignment randomly into two disjoint sets of columns, infer a tree on each, and ask if the two trees are identical. The frequency that the same tree is obtained for different subsets of columns from the same alignment should be correlated with phylogenetic inference accuracy, because in general, we only expect to obtain the same tree if it is the correct tree. The concordance test should be well suited for evaluating whether a modified method extracts more information from a given alignment, which is the question at hand here. The test may be less suited for evaluating the absolute accuracy of an entirely new inference program, because some systematic errors, such as long branch attraction, could result in agreement on the wrong tree; simulations could be used to detect such systematic problems, though.We found it is important to choose columns randomly, rather than simply splitting an alignment in half, because real alignments often contain preferentially 5′- or 3′-truncated sequence fragments.dnamlε and dnaml on real alignments, we applied the concordance test to a large number of ribosomal RNA alignments. We obtained curated rRNA alignments from the Comparative RNA Web Site  dnamlε and dnaml methods.In order to evaluate the performance of dnamlε shows tree concordance of 27.9% for SSU and 46.6% for LSU, while dnaml shows tree concordance in 16.9% for SSU and 35.7% for LSU. The error estimate for all these results is about 0.5–0.6%, which indicates that the improvement obtained by dnamlε is significant. LSU alignments are longer than SSU (4205±1179 versus 1959±579), probably explaining the better performance. For alignments with few gaps, the two methods produce similar results. The improvement of dnamlε over dnaml increases with the frequency of gaps in the alignments.Results are summarized in dnamlε taking about two- to three-fold longer.With respect to computational time, We have presented a non-reversible probabilistic model of sequence evolution accounting for substitutions, insertions, and deletions that is based on a continuous-time Markov process. This model does not assume a pre-stated number of columns. Rather, it describes a generative evolutionary model of substitution, insertion, and deletion events, starting from an explicit prior distribution over ancestral sequences of any length. This avoids the conceptual flaws that easily arise in column-independent models based on a Markov process that includes the gap character. The model remains compatible with efficient post-order transversal algorithms to calculate the likelihood of a phylogenetic tree. The model can also be used to calculate the probability of ancestral sequences with an arbitrary number of residues that leave no trace in the observed alignment.et al. have already made the same observation To do this, the model assumes column independence, an assumption that is problematic because insertions and deletions typically involve multiple residues at the same time. Although we can indeed produce synthetic examples for which our model breaks down due to the column independence assumption, our results for real rRNA alignments show that the gap-extended model is able to produce better trees than the standard model, indicating that the cost of the assumption is outweighed by the gain in modeling gaps instead of ignoring them as missing data. McGuire The model introduces as free parameters the rates of insertions and deletions, and the geometric probability parameter for the distribution of ancestral sequences. In the results presented here, for simplicity, we have used the F84 substitution rate matrix, identical deletion rates for all residues, and assumed that inserted residues have the same probability distribution as the stationary distribution of the substitution process. Our model does not require these simplifications. Our results may be generalized to arbitrary substitution rate matrices, deletion rates, and inserted residue probability distributions.It would be straightforward to integrate this method into any probabilistic model of sequence alignment, including phylo-HMMs tkf91 model and McGuire's model are designed to be reversible. Our model is non-reversible with respect to insertions and deletions. Reversibility is a mathematically convenient assumption in phylogenetic inference , and for residue substitution events, usually seem reasonable. However, in the case of insertion/deletion processes, reversibility seem less easy to justify, if one expects the insertion rate λ and the deletion rate μ to behave as independent parameters.The λ and deletion rate μ; moreover, the constraints imposed by reversibility can be counterintuitive. Consider what happens in a case of zero deletion rate and a positive insertion rate. For McGuire's model, reversibility requires that the frequency of gaps is a constant (π–), and that it is related to the rates of insertions and deletions by the condition π– = μ/(λ+μ). For the tkf91 model, reversibility requires a length distribution for evolved sequences identical to that of ancestral sequences which is geometric with parameter λ/μμ = 0 automatically implies that the gap frequency is zero, effectively converting the model into a substitution-only model, regardless of the insertion rate. In tkf91, the length distribution breaks down entirely in the μ<λ regime, and gives arbitrarily large joint “probabilities”. Our model remains valid for any arbitrary (positive or zero) values for insertion and deletion rate.For a model to be reversible with respect to insertions/deletions, obviously constraints must be imposed on the insertion rate Another consequence of imposing reversibility is that the expected frequencies of insertions and deletions in a pairwise alignment must be identical. For McGuire's model, the expected frequencies of insertions and deletions in a pairwise alignment are given by,π– is the constant frequency of a gap, and λ+μ = β(1−π–), where β is one of the two parameters of the F84 substitution model tkf91 model, the expected frequencies of insertions and deletions in a pairwise alignment are given by (see where n by see :(31)tkf91 for small divergence times, the expected frequency of insertions is solely dependent on the deletion rate μ; indeed, Equation 31 does not depend on the insertion rate at all. If one desires insertion and deletion rates to be independent parameters, it is more logical to expect that the observed frequencies of deletions and insertions should be in general different, and should depend on both deletion and insertion rates. This is the case in our model, shown in Equations 16 and 17. At one extreme, for λ = 0 (zero insertion rate), λ = ∞, t.Therefore in However, this is not to say that our model is free of its own problems. Most significantly, our model's assumption of column independence is problematic. Rather, in return for this simplification, our model's advantage is that it allows computationally efficient likelihood inference while using a birth-death generative model allowing arbitrary rates of insertion and deletion.phylip 3.66 package dnamlε , the C source code for evolving sequences with the generative model (εrate ), the modified rose package (version 1.3) dnamlε uses the easel sequence analysis library  which is also provided.The C source code for the modified Dataset S1Supplemental Material(24.89 MB GZ)Click here for additional data file.Text S1Appendix 1(0.16 MB PDF)Click here for additional data file.Text S2Appendix 2(0.11 MB PDF)Click here for additional data file.Text S3Appendix 3(0.14 MB PDF)Click here for additional data file.Text S4Appendix 4(0.13 MB PDF)Click here for additional data file.


Accurate and sensitive performance evaluation is crucial for both effective development of better structure prediction methods based on sequence similarity, and for the comparative analysis of existing methods. Up to date, there has been no satisfactory comprehensive evaluation method that (i) is based on a large and statistically unbiased set of proteins with clearly defined relationships; and (ii) covers all performance aspects of sequence-based structure predictors, such as sensitivity and specificity, alignment accuracy and coverage, and structure template quality.With the aim of designing such a method, we (i) select a statistically balanced set of divergent protein domains from SCOP, and define similarity relationships for the majority of these domains by complementing the best of information available in SCOP with a rigorous SVM-based algorithm; and (ii) develop protocols for the assessment of similarity detection and alignment quality from several complementary perspectives. The evaluation of similarity detection is based on ROC-like curves and includes several complementary approaches to the definition of true/false positives. Reference-dependent approaches use the 'gold standard' of pre-defined domain relationships and structure-based alignments. Reference-independent approaches assess the quality of structural match predicted by the sequence alignment, with respect to the whole domain length  or to the aligned region only . Similarly, the evaluation of alignment quality includes several reference-dependent and -independent measures, in global and local modes. As an illustration, we use our benchmark to compare the performance of several methods for the detection of remote sequence similarities, and show that different aspects of evaluation reveal different properties of the evaluated methods, highlighting their advantages, weaknesses, and potential for further development..The presented benchmark provides a new tool for a statistically unbiased assessment of methods for remote sequence similarity detection, from various complementary perspectives. This tool should be useful both for users choosing the best method for a given purpose, and for developers designing new, more powerful methods. The benchmark set, reference alignments, and evaluation codes can be downloaded from  Despite current comprehensive efforts ,2, threeFirst, although SCOP folds group proteins with general structural similarity and SCOP superfamilies correspond to evolutionary groups, the relationships between superfamilies within the same fold remain undefined: some superfamilies within a fold are very similar and can serve as good templates for each other, others are quite different, and fold similarity between them may not be useful for modeling. Second, although many SCOP folds are structurally distinct, there are multiple examples of folds with pronounced structural similarity between them, such as Rossmann-like doubly-wound sandwich folds of α/β SCOP class, Immunoglobulin-like β-sandwiches, and β-propeller folds. Third, existing manually curated classifications do not provide reference alignments for the evaluation of sequence-based alignment accuracy. In the published surveys, the accuracy of sequence alignments is usually assessed on a different protein set, by comparing sequence alignments with the 'gold standard' pairwise structure-based alignments produced by some software, for instance DALI )Reference-dependent evaluation of method's performance can be performed with two types of reference: (i) known relationships between domains of the testing set; and (ii) known structure-based alignments of domain sequences. In fold recognition assessment, evaluation (i) has been traditionally performed using a large-scale structure classification (SCOP or CATH), whereas evaluation (ii) has been ignored. In other words, a true similarity prediction providing incorrect alignment has been treated the same as a prediction providing correct alignment. In our benchmark, this traditional approach Fig.  is complBoth types of reference-dependent criteria would penalize short alignments that are structurally reasonable but differ from the 'gold standard', for example, alignments of similar structural fragments  Therefore reference-dependent evaluation may be considered a global rather than local assessment of structure prediction.In addition, we perform evaluations that do not require any prior knowledge about protein relationships or alignments. These evaluations are based on the quality of structure superposition suggested by a sequence alignment of interest, reflected in the GDT_TS score ,20. The As an illustration, we use our benchmark to compare the performance of several methods for the detection of remote sequence similarities: PSI-BLAST version 2.2.1 , methodsFor the assessment of alignment quality, all pairs of domains defined as similar are used, and various parameters of the produced alignments are calculated for several ranges of sequence identity Fig. . For eacdeveloper  monotonically depends on the 'raw' score of the produced alignment. Therefore, for the hits produced by a single query the ranks of E-values are the same as the ranks of raw scores. Thus splitting the assessment into individual queries in effect bypasses the procedure of estimating statistical significance altogether. Comparing the results of this assessment Table  to the rThe results produced by the presented benchmark are in general accord with previously reported evaluations ,8,29 in We also introduce evaluation approaches that have not been used in similar systems before. In some cases, the results produced by these approaches differ from the results of more traditional evaluations. First, we provide GDT_TS-based reference-independent assessments of similarity detection Fig. . The twoWe have developed a comprehensive benchmark system that serves as a tool for a statistically unbiased assessment of methods for remote sequence similarity detection, from various complementary perspectives. This tool should be useful both for users choosing the best method for a given purpose, and for developers designing new, more powerful methods. The benchmark set, reference alignments, and evaluation codes are available to download from .first construct a dataset of diverse representative SCOP domains and define the similarity relationships between them. Our aim is to overcome drawbacks of SCOP as a reference for such relationships, namely to reduce the number of undefined relationships between domains that belong to different superfamilies of the same fold, and to establish similarities between related folds. To define the domain relationships, we complement the SCOP superfamily classification with SVM-optimized linear combination of several structure- and sequence-based similarity scores. Second, we develop the protocols for the evaluation of similarity detection. Each protocol processes the list of alignments sorted by their statistical significance and produces a sensitivity, ROC-like curve based on a corresponding definition of true/false positives. To define true and false positives, we implement both reference-dependent and reference-independent approaches  delineation of a large, statistically balanced testing set of protein domains; (ii) protocols for the evaluation of similarity detection; and (iii) protocols for the evaluation of alignment quality. Accordingly, we hes Fig. . Similarnts Fig. .To produce a testing set that would present a considerable challenge for sequence similarity detection, we focused on fairly low cutoffs of sequence identity between individual domains (15–25%). Astral Compendium ,38 proviNid is the number of aligned identical residues, Lali is the length of the aligned regions; (ii) percentage identity over the shorter sequence, Lshorter is the length of the shorter of the two sequences; (iii) identity within the aligned blocks combined with a background identity for the unaligned regions, Lunali is the length of the unaligned regions in the shorter sequence, and pidrandom is the background identity of random alignments. We estimated the background identity as fi is the background frequency of amino acid type i. Background residue frequencies either estimated by Dayhoff [pidrandom = 6%, consistent with our observations of unaligned regions in DALI alignments.To probe the influence of gaps and unaligned regions in the structure-based alignments, we tested three methods of calculating sequence identity: (i) percentage identity within aligned blocks only,  Dayhoff  or derivTo ensure that the dataset (i) includes representatives from each SCOP superfamily; and (ii) contains the largest possible number of domains, we use a novel method of domain selection. In each SCOP superfamily, we use a dynamic programming algorithm to select the largest number of domain representatives with sequence identity below a given cutoff.pid(1), pid(2), and pid(3)). Our assumption is that the best measure of sequence identity should most closely reflect the actual evolutionary distance between the domains, regardless of the specific differences in the alignment methods. Thus we look for the measure that provides the largest overlap between representatives produced by all three aligners, and the smallest number of unique representatives produced by only one aligner and not by others. Screening the identity cutoffs in the range of 15–25%, we find that the size of the overlapping representative set is about 3500–4500 representatives for all pid measures. Fixing this size to approximately 4000, which corresponds to the sequence identity of ~ 20%, we compared the identity measures by the number of generated unique representatives. Fig. pid(1) (A), pid(1) (B), and pid(1) (C). The simplest identity measure pid(1) is clearly the worst, producing the largest number of unique representatives. The total numbers of unique representatives for pid(2) and pid(3) are lower. As the best measure, we choose pid(3), because it (i) produces the lowest average number unique representatives (2.7%); and (ii) unlike pid(2), generates a similar number of representatives for each aligner. Thus, for the final dataset, we select the overlap between the representatives produced by all three aligners using identity measure pid(3)  and each of the identity measures used ((3) Fig. .-4 for up to 2 iterations). Secondary structures for each of the representative sequence are predicted using PSIPRED [Multiple sequence alignments for each sequence in the representative dataset are generated using PSI-BLAST with conservative settings , where TP and FN are the numbers of true positives and false negatives, respectively) at a given level of selectivity (selectivity = TP/(TP+FP), where FP is the number of false positives). These sensitivity values for the evaluated methods are compared using paired t-test and non-parametric paired Wilcoxon rank test. We find that 50% level of selectivity reveals the most significant differences between the compared methods, and that results are similar for the t-test and Wilcoxon test.To evaluate the quality of similarity detection by the method of interest, we use (i) sensitivity curves based on all-to-all comparisons in the benchmark domain set; and (ii) paired comparison between the methods by sensitivity/selectivity measures for individual queries from this set. Sensitivity curves are based on the list of alignments (hits) sorted by their statistical significance, with the most significant hits on top. Each hit is assessed as true or false positive by a corresponding protocol as described below, and the number of true positives is plotted against the number of false positives, starting from the top of the list. Different protocols for the assessment of predictions produce different assignments of true/false positives, resulting in different curves. Such evaluation of all-to-all domain comparisons might be biased if a few query domains produce many highly significant hits that dominate the top of the hit list. Hence, in addition we compare the quality of detection query by query. We construct sorted lists of hits for each query domain, and consider sensitivity   to selecpid(1) and pid(2), see above), BLOSUM score, and alignment coverage; structure scores included two modifications of GDT_TS [For each domain pair, we compute structure superpositions with three programs  and use these superpositions to derive structure- and sequence-based similarity scores. Ten types of similarity scores were computed for each pair on all three structure alignments. Sequence scores were two identity measures  and clearly dissimilar domains (different SCOP classes). As a training set, we randomly select 2000 pairs of SCOP domains from four major SCOP classes . Among them, 1000 pairs belong to difference SCOP classes and are labeled as negative for SVM training; 1000 pairs belong to the same SCOP superfamilies and are labeled positive. When SVM is trained in the initial setting, with the full set of 30 input features , the resulting classification accuracy is 94.8%. We find that classification accuracy increases when some of the features are removed from the set. In order to determine the importance of each feature for the discrimination, we transform the raw similarity scores into Z-scores, train SVM and find the weight for each feature in the resulting SVM model. According to these weights, five features dominate the classification (relative contributions indicated in brackets): DALI Z-score (52%), FAST score (32%), coverage of FAST alignment (6.5%), GDT_TS of TM alignment (5.0%), and BLOSUM score of DALI alignment (4.6%). Combination of these five features produces the best SVM prediction accuracy of 95.7%.vs. TIM barrels). Forcing such cases to either of the two categories might bias the evaluation protocol. Therefore, following others [The resulting SVM makes a binary classification of domain pairs into the categories of similar and dissimilar. However, there is a number of domain pairs that share short regions of similarity but are poor global structural templates for each other  that share some local similarity but are globally different Table  and 3. TFinally, we combine our SVM classification with the expert similarity definitions in SCOP Fig. . A domaiIn order to evaluate the quality of a sequence-based alignment between query and hit in terms of its usefulness for structural modeling, we compare the sequence-based alignment to the structure-based (DALI-generated) reference alignment of the pair Fig. . The 3D GDT_TS  and Live-4 for up to 2 iterations). Then PSI-BLAST is used to compare the resulting alignments for each domain to all individual sequences from families of homologs for other domains. The sequence with the best E-value is chosen in each family. This best E-value and the corresponding sequence alignment are assigned to the comparison of the two domains. The reference-dependent template quality evaluation is applied to these alignments and the 'false' pairs are used as negative examples. PSI-BLAST alignments of all pairs with significant E-values  are used as positive examples. Fig. All-against-all pairwise sequence alignments are generated using PSI-BLAST for 500 randomly chosen domains. PSI-BLAST is first used to find and align a family of homologs for each domain sequence . Otherwise, it is considered to be 'false' Fig. .We define that reference-independent evaluation does not use any other information except the alignment generated by a sequence method under evaluation and 3D structures of the aligned proteins. Thus we directly evaluate the usefulness of the sequence-based alignment for structure modeling. This evaluation is more in-line with model-to-structure comparison paradigm that bypasses the reference , as established by CASP experiments. However, reference-independent evaluation, despite its conceptual simplicity, is still rarely used in the field of sequence analysis .We start from a pairwise sequence alignment being tested, and superimpose 3D structures of the aligned proteins to minimize RMSD for the residue equivalencies given by the sequence alignment Fig. . If the In a global mode Fig. , we evalReference-dependent evaluation is necessarily done in a global mode, because it relies on a single structure alignment generated by DALI. However, there could be many reasonable local structure alignments for a pair of proteins, i.e. each helix can be superimposed with another helix. Local evaluation can be easily done in a reference-independent way Fig. .We define a local version of GDT_TS score as:p1, p2, p4, p8 are the fractions, normalized by the query length, of aligned Cα atoms that are within the distance of 1, 2, 4, 8Å from each other in an RMSD-minimized superposition, and Lali is the length (number of residues) of the aligned region. However, we cannot use lGDT_TS directly because it has a significant dependency on aligned domain length. If the aligned region is very short, essentially all residues in the aligned region will be very close to each other and the resulting lGDT_TS will be artificially large. In the extreme case, if there is only one residue aligned, the lGDT_TS will be a perfect score of 1.0. This length dependency effect is also present in the global GDT_TS, but it does not affect the global mode evaluation much because the length of the query, and not of an alignment) is used in normalization. When the aligned region is very short compared to the length of the query, the resulting global GDT_TS is small. Thus the global GDT_TS favors long alignments, while the local GDT_TS favors short alignments.where f(L) = cLb to fit the lGDT_TS mean and sd , and minimum RMSD structure superposition was performed based on this alignment. lGDT_TS score is calculated based on this superposition. The study of lengths from 3 to 500 reveals power-law dependency of the mean and the standard deviation of lGDT_TS on length. Fig. . Using t sd Fig. , we obtaL is transformed to a length-corrected Z-score. By using this Z-score of lGDT_TS, we are able to find alignments with lGDT_TS scores significantly higher than random for that length. Alignments with Z-scores higher than 3 are considered to be of reasonable quality  is the developer) is the ccuracy' : Qlocal Based on evaluated alignments and domain 3D structures, we calculate original GDT_TS score  and 'local' GDT_TS score normalized by the alignment length and adjusted according to the length dependency of random GDT_TS scores (see section 2.2.2).YQ carried out the methodology development and ROC-based evaluation of methods for sequence comparison, as well as drafted the manuscript. RIS participated in the methodology design, evaluation, and writing the manuscript. YW carried out the evaluation of homology detection using paired tests, and of alignment quality. BHK provided the structure-based sequence alignments and scores. NVG carried out the methodology design and participated in drafting the manuscript. All authors read and approved the final manuscript.domain_seqs.fa. Sequences of protein domains included in the benchmark set.Click here for file


Multiple protein templates are commonly used in manual protein structure prediction. However, few automated algorithms of selecting and combining multiple templates are available.Here we develop an effective multi-template combination algorithm for protein comparative modeling. The algorithm selects templates according to the similarity significance of the alignments between template and target proteins. It combines the whole template-target alignments whose similarity significance score is close to that of the top template-target alignment within a threshold, whereas it only takes alignment fragments from a less similar template-target alignment that align with a sizable uncovered region of the target.-4). Compared with the ideal approach that always uses the best template, the multi-template approach yields only slightly better performance. During the CASP7 experiment, the preliminary implementation of the multi-template combination algorithm (FOLDpro) was ranked second among 67 servers in the category of high-accuracy structure prediction in terms of GDT-TS measure.We compare the algorithm with the traditional method of using a single top template on the 45 comparative modeling targets (i.e. easy template-based modeling targets) used in the seventh edition of Critical Assessment of Techniques for Protein Structure Prediction (CASP7). The multi-template combination algorithm improves the GDT-TS scores of predicted models by 6.8% on average. The statistical analysis shows that the improvement is significant (p-value < 10We have developed a novel multi-template algorithm to improve protein comparative modeling. Protein structure prediction is one of the most important problems in structural bioinformatics -3. CompaIn general comparative modeling involves four steps ,20,21:  that have only one template found by PSI-BLAST are excluded. The single-template algorithm always uses the most significant template with the lowest e-value of PSI-BLAST.Table -5 under the hypothesis that the difference is 0. So combining multiple templates and their alignments significantly improves the quality of comparative modeling over the single-template approach.We conduct a paired t-test  on the GDT-TS scores of 45 targets. The p-value of getting an average score difference >= 4.56 is 3.5 × 10A special category of template-based modeling (i.e. high-accuracy modeling), where models have GDT-TS scores > 80 compared to experimental structures, is particularly useful for understanding protein function at the atomic level. To emphasize its importance, CASP7 dedicated a category of high-accuracy template-based modeling to evaluate methods on the targets for which there is at least one template with LGA-S score > 80 and at least one method produced a model with GDT-TS > 80. Using this criteria, CASP7 classifies 28 domains from 24 protein targets into the high-accuracy modeling category. Among them T0326 and T0328 have only one template. The templates for T0311 and T0367 cannot be found by PSI-BLAST. The structure of T0334 is not released at the time of writing the paper. So we exclude these five targets and select the remaining 23 high-accuracy domains to compare the multiple-template combination algorithm with the single-template approach.Table A very challenging problem of comparative modeling is to improve the model accuracy over the best templates ,52,53. TWe use the best templates for the comparative modeling targets provided at the CASP7 web site. We select 27 targets whose best templates can be identified by PSI-BLAST to compare the multi- and best template methods. We also use the PSI-BLAST alignment between the best templates and the targets to generate structure models.As shown in Table We conduct a pairwise t-test on the GDT-TS scores of two approaches . The p-value of the statistical analysis is 0.28. Thus, on average, the GDT-TS score increase of using multiple templates over the best possible template is not significant.We compare the accuracy of our multi-template combination algorithm against the other servers that participated in the CASP7 community-wide experiment. Two of our servers, FOLDpro and 3Dpro, along with 65 other servers were evaluated in the category of high-accuracy structure prediction in CASP7. FOLDpro and 3Dpro used the same multi-template combination algorithm, but run on different versions of non-redundant sequence databases. Table Furthermore, using the more strict measure GDT-HA specially designed for high-accuracy models, FOLDpro and 3Dpro are ranked third and fifth according to the official CASP assessment ,58. We aTable Since the main goal of this paper is to demonstrate the effectiveness of using multiple templates instead of evaluating different predictors in CASP7, readers are advised to check out the CASP7 assessment papers published in the Proteins journal for the thorough evaluations using different measures such as GDT-TS, GDT-HA and AL0.The correct usage of multiple homologous templates in general but, not always can improve comparative modeling ,61. As tFigure The other good example is a two-domain target T0324, where the improvement of using multiple templates on the second domain (4-helix bundle) is drastic. The GDT-TS score of the second domain is increased by 48 Table . A close-153. The RMSD between other three significant templates  and the experimental structure is 3.22 Å for 250-residue aligned region, 2.33 Å for 262-residue aligned region, and 3.20 Å for 250-residue aligned region respectively. These three templates are much more different from the target structure than the best template. However, because the alignments between these three templates and the experimental structure are very significant , these three templates together with the best one are combined to generate models for the target.However, multi-template combination may occasionally decrease the model quality when there is a very good template that is much closer to the target than all other templates. One such an example is T0291 (length = 310). The most significant and best template is 1JPA, whose RMSD with the experimental structure (2GSF) is 0.72 Å for 264-residue long aligned region. The sequence identity and PSI-BLAST e-value is 81% and 10The RMSD between the model and the experimental structure is 2.63 Å for 271-residue long alignment, which is better than two templates (2SRC and 1KSW), but worse than the best template (1JPA). This example shows that combining multiple templates may not help if one template is much closer to the true structure than all other templates.We consider the following factors contributing to the improvement. First, statistically, the average model generated from multiple templates is better than the single top template on average. This is due to the ability of Modeller extracting spatial restraints from multiple templates and weighting them effectively. The weighting scheme can weight the most likely distance restraints more, resulting in picking correct aligned regions from different templates in most cases . The effSecond, multiple templates contain sequence and structure conservation and variation information (e.g. conserved distance restraints), which is not available in a single template. The evolutionary information is often useful to improve both secondary and tertiary structure prediction ,61-63Third, PSI-BLAST can generate good local alignments for homologous proteins (comparative modeling), particulary for close homologs (easy comparative modeling). In fact, PSI-BLAST, a profile-sequence local alignment method, generates better alignments for the comparative modeling (or easy) targets than the profile-profile alignment methods  is set to 3; and the e-value threshold (-e option) for inclusion in the final profile is set to 0.001. Second, it uses PSI-BLAST to search the profile against a template library compiled from the proteins in the Protein Data Bank 87], and JC designed the algorithm, wrote the program, carried out the experiments, and authored the manuscript. All authors read and approved the final manuscript.The description of supplemental materials. This file describes how to use supplemental materials. It is a text file that can be viewed by any text viewer.Click here for fileThe compressed file of supplemental materials. The file can be decompressed by winzip on Windows or by tar xzf on Linux. The file includes query-template alignments of both the single-template approach and the multi-template approach that were used to generate the structure models for the CASP7 comparative modeling targets.Click here for file


The comparison requires reliable sequence alignment, which remains a challenging task for less conserved noncoding sequences. Furthermore, the existing models of DNA sequence evolution generally do not explicitly treat the special properties of CRM sequences. To address these limitations, we propose a model of CRM evolution that captures different modes of evolution of functional transcription factor binding sites (TFBSs) and the background sequences. A particularly novel aspect of our work is a probabilistic model of gains and losses of TFBSs, a process being recognized as an important part of regulatory sequence evolution. We present a computational framework that uses this model to solve the problems of CRM alignment and prediction. Our alignment method is similar to existing methods of statistical alignment but uses the conserved binding sites to improve alignment. Our CRM prediction method deals with the inherent uncertainties of binding site annotations and sequence alignment in a probabilistic framework. In simulated as well as real data, we demonstrate that our program is able to improve both alignment and prediction of CRM sequences over several state-of-the-art methods. Finally, we used alignments produced by our program to study binding site conservation in genome-wide binding data of key transcription factors in the Drosophila blastoderm, with two intriguing results: (i) the factor-bound sequences are under strong evolutionary constraints even if their neighboring genes are not expressed in the blastoderm and (ii) binding sites in distal bound sequences (relative to transcription start sites) tend to be more conserved than those in proximal regions. Our approach is implemented as software, EMMA , ready to be applied in a broad biological context.Cross-species comparison has emerged as a powerful paradigm for predicting  Drosophila genomes for regulatory annotation. However, because noncoding sequences are much less structured than coding sequences, their interspecies comparison presents technical challenges, such as ambiguity about how to align them and how to predict transcription factor binding sites, which are the fundamental units that make up regulatory sequences. This article describes how to build an integrated probabilistic framework that performs alignment and binding site prediction simultaneously, in the process improving the accuracy of both tasks. It defines a stochastic model for the evolution of entire “cis-regulatory modules,” with its highlight being a novel theoretical treatment of the commonly observed loss and gain of binding sites during evolution. This new evolutionary model forms the backbone of newly developed software for the prediction of new cis-regulatory modules, alignment of known modules to elucidate general principles of cis-regulatory evolution, or both. The new software is demonstrated to provide benefits in performance of these two crucial genomics tasks.Comparison of noncoding DNA sequences across species has the potential to significantly improve our understanding of gene regulation and our ability to annotate regulatory regions of the genome. This potential is evident from recent publications analyzing 12  A CRM contains a number of transcription factor binding sites (TFBSs), which read the expression level of the cognate transcription factors (TFs) and drive the appropriate expression pattern through the combinatorial interactions among TFs, their co-factors and the basal transcriptional machinery cis-regulatory sequences, including computational prediction of CRM sequences The spatial-temporal expression pattern of a gene is controlled by its regulatory sequences, sometimes called a The first major limitation of this two-step procedure arises from errors in alignment. It has been shown that alignment procedure may seriously affect the results of comparative genomic analysis such as reconstruction of phylogenetic trees and inference of positive selection a priori.Several approaches have been proposed to address one or more of the problems discussed above. The programs Stubb Our philosophy of doing cross-species sequence analysis is: firstly, the method should be based on an explicit model of sequence evolution, as expressed in Drosphila melanogaster. We took advantage of the recent genome-wide binding data on key TFs involved in blastoderm-stage gene regulation  model for TFBS evolution Gain and loss of TFBSs are commonly observed across a large evolutionary spectrum: e.g. fungi Drosophila early developmental CRMs The main difference between the two scenarios is: in the former, the changes of TFBSs are driven by external selection forces while in the latter, the changes are mainly dominated by the stochastic forces of mutation and random drift, with selection being weak. In our model, we adopt the second scenario as it is a more “parsimonious” explanation of TFBS gain and loss, and is more consistent with our current knowledge about the Our specific model formulates the ideas discussed above. We follow the usual definition of binding energy of a TFBS, for example We illustrate our model of TFBS loss in cis-regulatory Module Analysis).We solve the following computational problems: given two orthologous sequences  and a set of TF motifs, (1) align the two sequences and annotate the TFBSs; (2) predict if the sequence is a CRM targeted by the given motifs. We use dynamic programming to simultaneously find the optimal alignment and TFBS annotation. For the second task, we use a likelihood-ratio test of two models: the CRM evolutionary model and the background evolutionary model where no motif is used. Computation under each model is also done by dynamic programming, summing over all possible alignment paths and annotations of TFBSs. The details of the algorithms can be found in cis-regulatory evolution.Simulation of sequence evolution is a useful strategy for assessing computational methods, as the true evolutionary history is often unknown for real data. In addition, simulation is an important way to understand how various factors, such as divergence time or the presence of TFBSs, affect the performance of a computational procedure, since such questions are generally difficult to answer analytically. We developed a simulation program that can generate orthologous CRM sequences according to our evolutionary model. The binding site densities, branch lengths, indel rates, etc., are all user-specified parameters. Our simulator captures a richer biology of regulatory evolution than many other sequence simulators We first used simulation to study alignment methods, similar to what was done previously by Pollard et al. The specificities and sensitivities of all programs are similar . The only difference between EMMA-ANN and EMMA1 is that a fixed alignment is used in the former, so the comparison between the two should suggest how important it is to treat alignment uncertainty. In addition to the EMMA family programs and Morph, we tested the program Stubb The results are shown in In summary, we made several findings through simulation that are relevant to choosing and developing the computational tools for CRM prediction. Generally, one should use relatively divergent sequences, as long as they are alignable. The performance depends on the alignment of TFBSs, but less on the overall alignment quality, or the exact alignment parameters such as gap penalties. When the binding site turnover events are common, it is important to model the lineage specific binding sites. Finally, because of the inherent uncertainty of alignment, simultaneous inference and alignment taking advantage of the special properties of the sequences will help both tasks. All these observations support our efforts in building a comprehensive model of CRM evolution and using it as a basis for related inference tasks.D. melanogaster and D. pseudoobscura. Since it is not possible to know the true alignments of the real data, we follow the earlier approach To test if our model truly brings benefits in real-world applications over existing programs, we start with the task of alignment, and compare our program EMMA  with Lagan and Morph. We study the set of blastoderm CRMs from the RedFly database D. melanogaster can be moved by one position without changing the Lagan score, suggesting that arbitrary resolution of ambiguous alignments can contribute to small-scale alignment errors that may be important for binding sites. Finally, the last Bcd site in D. melanogaster is close to, but does not align to a potentially orthologous Bcd site in D. pseudoobscura. In EMMA alignment, all four sites in D. melanogaster are aligned with their functional orthologs in D. pseudoobscura.In our first experiment, we use the known TFBSs We next use predicted sites for further evaluation, since the number of known TFBSs is small. For each of the seven motifs, we constructed alignments with Lagan, EMMA and Morph, using only one motif a time. The results were evaluated by the number of predicted sites . We also note that the experimental setting in this paper is different from the one in EMMA substantially outperforms all other three programs with the AUC measure . AveragiDrosophila genome is under constraint One common procedure to enhance the performance of a program running on single-species data, such as Cluster-Buster, is to filter out the sequences that are not very conserved before running the program. We combined this conservation filtering  with Cluster-Buster. However, the new results are only slightly better than the original Cluster-Buster, and still lag far behind EMMA (data not shown). This can be probably explained by the fact that a large fraction of Drosophila melanogaster. Such analysis depends on the accurate alignment of TFBSs, a task that EMMA has been shown to perform better than general purpose sequence alignment tools. We took the sequences bound by each transcription factor .In this experiment, we used EMMA to study the evolutionary pattern of TFBSs in sequences involved in gene regulation in blastoderm-stage development of ed group , though ed group : bindingcis-regulatory sequences. At the heart of our approach is a probabilistic model covering important aspects of CRM evolution, including substitutions and indels in background sequences, and constraints and turnover of TFBSs. The dynamic programming algorithm allows us to efficiently carry out likelihood-based statistical inference. This framework solves the problems of the existing approaches discussed earlier. It aligns regulatory sequences by taking advantage of the tendency of conservation of TFBSs. The TFBS gain and loss model allows us to use information present in lineage-specific TFBSs. Most importantly, when used for predicting CRMs, our method treats alignment and annotation of TFBSs as random variables, summing over them and thus minimizing the impact of an uncertain alignment and TFBS annotation. Our previous programs Stubb and Morph have similar aims, but as shown in our experiments, EMMA significantly outperforms both, strongly suggesting that correct evolutionary modeling is essential to fully utilize the sequence information.We have proposed an integrative framework for cross-species analysis of Our model is related to existing models of regulatory sequence evolution, but different from them in several key aspects. Our idea of generation of a new binding site is similar to cis-regulatory sequences. Our method shares key features with statistical alignment: explicit modeling of indel evolution; and a probabilistic treatment of alignment uncertainty. Statistical alignment started with the pioneering work of Thorne et al. on pairwise sequence alignment cis-regulatory modules. Thus the modeling of substitution and indels, the characteristic feature of statistical alignment, has to be integrated with a model-based treatment of binding site evolution.Our model is also an extension of statistical alignment , which are based on footprinted binding sites Drosophila genomes. The distribution of nucleotides in the background sequences was 0.3, 0.2, 0.2 and 0.3 for A, C, G, T respectively Drosophila species, sechellia and simulansFor our simulation, we first sampled ancestral sequences from the CRM model described in the main text and evolved the sequences in two branches independently for specified lengths of the two branches. Only the two descendant sequences will be used for alignment input. We used PWMs of For the simulation under CisEvolver, we used the same parameters except that the indel lengths are specified by their empirical frequencies D. melanogaster blastoderm CRMs from RedFly database D. pseudoobscura by using the LiftOver tool from UCSC Genome Browser D. pseudoobscura orthologs were discarded. We used seven TFs important for early development in our analysis: Bcd, Cad, Gt, Hb, Kr, Kni and Tll and the PWMs of these TFs were taken from the same source we used for simulation. Bona fide binding sites were collected from FlyReg We took 67 http://rana.lbl.gov/˜dan/trees.html), since the noncoding sequences in general are under a high level of constraint . Cluster-Buster was run under the default setting, as we do not have extra data for training parameters of Cluster-Buster.For each of the seven TFs, we took all RedFly CRM sequences that contain at least one FlyReg site of this TF as the set of positive sequences. Each positive sequence was expanded or truncated so that the length was 1000 bp. 500 random sequences of length 1000 bp each were chosen randomly from the D. melanogaster genome. The alignment of the sequences with their orthologs in D. pseudoobscura were constructed using EMMA. The binding sites in both species were then predicted following the same procedure we used before for the experiment of evaluating alignment performance of EMMA. Similarly, we define a binding site as being conserved if both orthologous sites have scores greater than the threshold (in situ hybridization (http://www.fruitfly.org/cgi-bin/ex/insitu.pl). A sequence belongs to the expressed group, if its associated gene is classfied as being expressed in stage 4–6 according to BDGP, and similarly for the unexpressed group .The genome-wide binding data is taken from http://veda.cs.uiuc.edu/emma/.The EMMA program, an evolution simulator, and the dataset used in this paper are all available at Figure S1A pair of orthologous sites that are functional in one species, but not the other. Two possible histories that could lead to this pattern are shown. Shaded and white box represent functional TFBS and non-functional site respectively.(0.02 MB PDF)Click here for additional data file.Figure S2TFBS conservation sensitivity from various alignment programs using simulated data from CisEvolver.(0.02 MB PDF)Click here for additional data file.Figure S3Performances of different programs for predicting regulatory targets of seven blastoderm TFs using B1H PWMs. CBust: Cluster-Buster. (A) AUC of the ROC curve; (B) the average sensitivity at the specificity level above 80%.(0.02 MB PDF)Click here for additional data file.Text S1Methods and additional results(0.07 MB PDF)Click here for additional data file.


Accurate estimation of statistical significance of a pairwise alignment is an important problem in sequence comparison. Recently, a comparative study of pairwise statistical significance with database statistical significance was conducted. In this paper, we extend the earlier work on pairwise statistical significance by incorporating with it the use of multiple parameter sets.Results for a knowledge discovery application of homology detection reveal that using multiple parameter sets for pairwise statistical significance estimates gives better coverage than using a single parameter set, at least at some error levels. Further, the results of pairwise statistical significance using multiple parameter sets are shown to be significantly better than database statistical significance estimates reported by BLAST and PSI-BLAST, and comparable and at times significantly better than SSEARCH. Using non-zero parameter set change penalty values give better performance than zero penalty.The fact that the homology detection performance does not degrade when using multiple parameter sets is a strong evidence for the validity of the assumption that the alignment score distribution follows an extreme value distribution even when using multiple parameter sets. Parameter set change penalty is a useful parameter for alignment using multiple parameter sets. Pairwise statistical significance using multiple parameter sets can be effectively used to determine the relatedness of a (or a few) pair(s) of sequences without performing a time-consuming database search. Local sequence alignment plays a major role in the analysis of DNA and protein sequences -3. It ishigher alignment scores. But the definition of high depends strongly on the alignment score distribution, which gives importance to the concept of statistical significance. An alignment score is considered statistically significant if it has a low probability of occurring by chance. Since the alignment score distribution depends on various factors like alignment program, scoring scheme, sequence lengths, sequence compositions [x and y with x <y, but x more significant than y. Therefore, instead of using the alignment score for detecting homology, the statistical significance of an alignment score is more widely accepted as a metric to comment on the relatedness of the two sequences being aligned. Of course, it is important to emphasize here that although statistical significance is a good preliminary indicator of biological significance, it does not necessarily imply biological significance [The local sequence alignment programs report alignment scores for the alignments constructed, and related (homologous) sequences will have ositions , it implificance ,11.Accurate statistical theory for the ungapped alignment score distribution is available . HoweverAlthough there exists some understanding of the statistics of gapped alignment score distributions for simple scoring schemes ,23, but Recently, a study of pairwise statistical significance and its comparison with database statistical significance  was condAccurate statistical significance estimates for pairwise alignments can be very useful to comment on the relatedness of a pair of sequences aligned by an alignment program independent of any database. And thus, it can also be used to compare different combination of alignment parameters – like the alignment program itself, substitution matrices, gap costs. A comparison of different gap opening penalties for four commonly used BLOSUM matrices using pairwise statistical significance was presented in . In addim and n, the statistics of HSP  scores are characterized by two parameters, K and λ. The probability  that the optimal local alignment score S exceeds x is estimated by:Just as the distribution of the sum of a large number of independent identically distributed (i.i.d.) random variables tends to a normal distribution , the distribution of the maximum of a large number of i.i.d. random variables tends to an extreme value distribution (EVD) . This isE is the E-value and is given bywhere K and λ can be computed analytically from the substitution scores and sequence compositions. For gapped alignments, no perfect statistical theory has yet been developed, although there exist some good starting points for the problem as mentioned before [E for a particular score x, by counting the number of times a score x or higher was attained. Since this number would be different for different number of random shuffles N (or number of sequences in the database in case of database search), a normalized E-value is defined asFor E-values less than 0.01, both E-value and P-values are very close to each other. The above formulae are valid for ungapped alignments , and thed before ,23. Reced before  applied d before  and suggd before ,27,32,33N).In theory, this normalized E-value is same as the P-value  where Seq1 is the first sequence, Seq2 is the second sequence, SC is the scoring scheme , and N is the number of shuffles. The function PairwiseStatSig, therefore generates a score distribution by aligning Seq1 with N shuffled versions of Seq2, fits the distribution to an extreme value distribution using censored maximum likelihood fitting to obtain the statistical parameters K and λ, and returns the pairwise statistical significance estimate of the pairwise alignment score between Seq1 and Seq2 using the parameters K and λ in the P-value formula. More details on pairwise statistical significance can be found in [SC for estimation of pairwise statistical significance.Consider the pairwise statistical significance described in  to be obfound in . In thisA = a1, a2,..., am and B = b1, b2,... bn using p parameter sets P1, P2,..., Pp. Let Ai and Bj be the subsequences a1, a2,... ai and b1, b2,..., bj respectively. For each alignment position  and each parameter set Pk, the algorithm keeps track of the optimal alignment score of the subsequences Ai and Bj where the last component  is scored using Pk. Dynamic programming is used to get optimal alignment for progressive alignment positions, until i becomes m and j becomes n. Appropriate modification of the algorithm also allows it to calculate the optimal local alignment. More details about using multiple parameter sets for pairwise sequence alignment can be found in [Usually, pairwise sequence alignment is done with a single parameter set . But to suit the different levels of conservation between sequences, there exists an algorithm  which cafound in .. This database was selected in [To evaluate the performance of pairwise statistical significance using multiple parameter sets, we used a non-redundant subset of the CATH 2.3 database  provideected in  to evaluected in , this daected in .4 - 1 = 15 in number). All matrices were used in 1/3 bit scale. The gap opening penalties for each of these matrices was set to the values empirically determined to be the best for this database in [We used this database and query set for experimenting with pairwise statistical significance using multiple parameter sets. For each of the 86 × 2771 comparisons, we used the maximum likelihood method with censoring left of peak with 1000 shuffles to fit the score distribution from the GAP4 program with substitution matrices BLOSUM45, BLOSUM50, BLOSUM62, BLOSUM80, and their all possible combinations  , to compThe coverage vs. parameter set change penalty plot for the substitution matrix combination of BLOSUM45 and BLOSUM62 is illustrated in Figure As pointed out earlier, SSEARCH employs the original Smith-Waterman algorithm for alignment, and is considered more sensitive than its heuristic implementations like BLAST and FASTA. PSI-BLAST uses an iterative approach with query-specific substitution matrices, and its performance mainly depends on the quality of position-specific scoring matrices (PSSMs) constructed iteratively. The results show that PSI-BLAST gave poorer performance than pairwise statistical significance using multiple parameter sets, even with PSSMs constructed against the benchmark CATH database used in our experiments. However, using PSSMs derived against BLAST non-redundant protein database has been shown to give better results  as it usIt is important to note that the proposed method is not a database search method but statistical significance estimation method for pairwise local alignment, and the comparison with database search programs like BLAST, SSEARCH, and PSI-BLAST is of their statistical significance estimation strategies. The proposed method, as of now is not scalable to a database search, but can be used to refine the results from a fast database search program like BLAST.k parameter sets increases the computation time by a factor little more than k. Therefore, faster methods for significance estimation can be very helpful.Since pairwise alignment using multiple parameter sets takes more computational time than using a single parameter set, pairwise statistical significance estimation using multiple parameter sets also takes more time than pairwise statistical significance estimation using a single parameter set. In general, using This paper extends the work on pairwise statistical significance by incorporating in it the use of multiple parameter sets , and compares it with database statistical significance for the knowledge discovery application of homology detection. The results show that pairwise statistical significance using multiple parameter sets performs better than pairwise statistical significance using a single parameter set. It also performs significantly better than database statistical significance using BLAST and PSI-BLAST, and comparable and at times significantly better than database statistical significance using SSEARCH. Further, an empirical justification of the use of parameter set change penalty is provided.Since PSI-BLAST results can be improved by using better quality PSSMs derived from larger protein databases, we believe that the performance of pairwise statistical significance can also be improved using sequence-specific/position-specific substitution matrices, which is a significant part of our future work. Another important contribution can be to estimate the pairwise statistical significance accurately in less time, since using multiple parameter sets increases the significance estimation time. Faster methods for determining pairwise statistical significance would be very useful.The authors declare that they have no competing interests.AA conceived the study based on the initial idea given by XH. Both AA and XH did the programming, AA carried out the experiments and analysis, and drafted the initial manuscript. Both AA and XH read and approved the final version of the manuscript.