Multiple sequence alignment is fundamental. Exponential growth in computation time appears to be inevitable when an optimal alignment is required for many sequences. Exact costs of optimum alignments are therefore rarely computed. Consequently much effort has been invested in algorithms for alignment that are heuristic, or explore a restricted class of solutions. These give an upper bound on the alignment cost, but it is equally important to determine the quality of the solution obtained. In the absence of an optimal alignment with which to compare, lower bounds may be calculated to assess the quality of the alignment. As more effort is invested in improving upper bounds , it is therefore important to improve lower bounds as well. Although numerous cost metrics can be used to determine the quality of an alignment, many are based on sum-of-pairs (SP) measures and their generalizations.v-way alignments. However judicious selection of a subset of all 3-way alignments can yield a further improvement with minimal additional effort. On the other hand, a simple heuristic to select a random subset of 3-way alignments (a random packing) yields accuracy comparable to averaging all 3-way alignments with substantially less computational effort.Two standard and two new methods are considered for using exact 2-way and 3-way alignments to compute lower bounds on total SP alignment cost; one new method fares well with respect to accuracy, while the other reduces the computation time. The first employs exhaustive computation of exact 3-way alignments, while the second employs an efficient heuristic to compute a much smaller number of exact 3-way alignments. Calculating all 3-way alignments exactly and computing their average improves lower bounds on sum of SP cost in Calculation of lower bounds on SP cost  can be improved by employing a mixture of 3-way and 2-way alignments. For 1 ≤ i ≤ v, let ℓi denote the length of word wi, and let σij denote the jth character of wi. A string xi over Σ ∪ {-} is an extension of wi of length M if there exist indices 1 ≤ d1 <d2 < ⋯ M so that, for 1 ≤ j ≤ ℓi, the djth character of xi is σij, and whenever j ∉ {d1,..., jth character of xi is -. Let Ext be the set of all extensions of word wi to length M.Let Σ be a finite alphabet, and consider alignment of two words wi and wj consists of an extension xi ∈ Ext and an extension xj ∈ Ext. The cost of such an alignment, denoted cost, is determined position by position. In the simplest case, a position contributes a cost of 1 if the extensions do not have the same character in that position, 0 otherwise. Numerous different cost measures are of interest, but for our purposes counting matches of a symbol of Σ against – (indels) and matches of two different symbols of Σ (substitutions) suffices.An Pairwise alignment of sequences wi and wj chooses M, xi ∈ Ext and xj ∈ Ext, to obtain an alignment with SP cost cost. The alignment with lowest cost has SP costv sequences w1,..., wv consists of extensions {xi ∈ Ext : 1 ≤ i ≤ v}. The sum-of-pairs or SP cost is ∑i<j≤v 1≤cost, and the multiple alignment with lowest cost has costThe alignment of One natural objective is to find a multiple alignment that minimizes the SP cost. Exact methods have, for the most part, employed dynamic programming. In this environment, the execution time grows polynomially with the length of the sequences, but exponentially in the number of sequences. While improvements in time and space utilization are possible , dynalower bound on the SP cost, and compare the heuristic method with the lower bound as a measure of accuracy. Lower bounds do not provide alignments; instead, when they are in good agreement with an upper bound, they certify that the upper bound is indeed close to optimum. Lower bounds are also employed in branch-and-bound strategies for exact alignment [lower bounds on the SP cost of a multiple sequence alignment has been much less studied than heuristic methods, primarily because a lower bound does not provide a feasible alignment. Nevertheless, an effective lower bound provides the only real means to assess proximity of an upper bound to the optimum. In this paper, we examine a conceptually simple technique for calculating lower bounds on SP costs for alignments. It is based on the observation that v-way alignments for small values of v can be efficiently calculated by dynamic programming. The specific methods that we propose combines exact 2- and 3-way alignments selected to treat each 2-way alignment exactly once. Randomization of the selection of 3-way alignments to examine enables us to improve lower bounds from 2-way alignments without excessive computational cost.When heuristic methods are employed, how can one assess the quality of the alignment obtained? Feng and Doolittle  argue thlignment ,19, but mincost, isThe minimum SP cost, A simple lower bound is obtained by treating the alignment as By way of example, consider the eighteen sequences given next; these are generated from a sequence of 50 bases by random insertions, deletions, and substitutions. Our interest is in determining a lower bound on the sum of SP costs in an 18-way alignment of these sequences  is easy to demonstrate. When the decomposition has a single class, the lower bound given is equal to the SP alignment cost. So consider two classes in the decomposition c' and c". Merge these two classes to form a single class c by setting μc, {i, j} = μc', {i, j} + μc", {i, j} for every pair {i, j}, removing classes c' and c". We need only verify that, when M is chosen to yield the best fractional alignment for class c ,That this is a lower bound on y1,..., yv form a best fractional alignment (of length M) for c' and z1,..., zv a best for c". In applying this general framework, one must calculate fractional alignment costs, and this appears to necessitate solving many v-way alignment problems. Each can be solved by dynamic programming, but unless the decomposition is suitably chosen the resulting subproblems may be no simpler to solve than the original. If, in a given class of the decomposition, most weights are zero, the fractional alignment problem is substantially simplified. In particular, if all pairs containing a particular sequence have weights equal to zero in this class, the sequence does not participate in the fractional alignment. Then by choosing weights within each class so that "few" sequences appear in pairs with nonzero weight, the alignment task is reduced to one on the few sequences only, and can be solved efficiently using dynamic programming. Indeed when α = c has μc, {i, j} = 1 for exactly one pair {i, j}, this is the decomposition of the v-way alignment into where There is every reason to expect that the best alignment of two sequences may conflict with the best 2-way alignments of each with a third. For example, aligning the first three sequences of the eighteen in the example optimally, one obtains the alignmentATGGGTTGCGTCAGGAGTAAAGAAGCCAAGGGCCCGGCACTGAAGTACCAATGGGCTGTATTAAAAGTAAGGAAGACAAAGGACCAGCAATCAAGTACAGATGGGCTGCATTAAAAGTAAACAA---AA-GT-CCAGCCATAAAATACACBetween the first and second sequences, there are 15 substitutions and no indels. Between the second and third, there are 8 substitutions and 5 indels. Between the first and third there are 14 substitutions and 5 indels. The total three-way alignment score is 15 + 13 + 19 = 47. When exact two-way alignment of each pair among the three sequences is done, the alignment of the first two sequences agrees with that in the 3-way alignment. The optimal alignment of the second and third sequences does not; it has cost 12 (shown with 7 substitutions and 5 indels):ATGGGCTGTATTAAAAGTAAGGAAGACAAAGGACCAGCAATCAAGTACAGATGGGCTGCATTAAAAGT----AA-ACAAAAGTCCAGCCATAAAATACACMoreover, the optimal 2-way alignment for the first and third sequences has cost 18 (shown as 13 substitutions and 5 indels):ATGGGTTGCGTCAGGAGTAAAGAAGCCAAGGGCCCGGCACTGAAGTACCAATGGGCTGCATTAAAAGTAAACAA---AA-GTCCAGCCA-TAAAATACACThe sum of the three optimal 2-way alignment costs is therefore 45, while the optimal 3-way alignment has cost 47. Hence the use of exact 3-way alignment provides a better bound; thus rather than computing the lower bound by adding the SP scores for the three pairs of sequences, one can employ an exact 3-way alignment for all three sequences. As shown in the example, the difference in SP scores of simultaneous three-way alignment of the three sequences and sum of pairwise scores of the three pairs in this triple is 2. In this example, the best improvement that can be achieved by replacing any three 2-way alignments by the corresponding 3-way alignment is 3. To establish this, all 816 3-way alignments were calculated using dynamic programming, and compared to the sum of 2-way alignments within each.v-way alignment, a reasonable goal therefore is to choose many triples of sequences for which the 3-way alignment improves the lower bound, so as to obtain many small improvements that aggregate to form a large improvement in the bound. At the extreme, one could calculate best 3-way alignments for all v-way alignment into In O(v3) alignments rather than O(v2). It is therefore desirable to reduce the number of classes in the decomposition, both to reduce computation time and to potentially improve the bound.There are two problems with this approach. First and foremost, while some triples yield improved cost bounds over the sum of the three pairs of the triple, others may not. Simply averaging the contributions of each negates to a degree the improvements attributable to some triples. Secondly, in addition to a substantial increase in cost for each alignment, the calculation involves completing i, j} of sequences, let λij be the number of triples of T ∈ μc, {i, j} = i, j} ⊂ T, 0 otherwise. Further, whenever there is a pair {i, j} with λij = 0, form a class in which μc, {i, j} = 1 and all other weights are zero. This decomposition requires the determination of fractional 3-way alignments for each triple of We now employ only some of the 3-way alignments. Choose a collection fractional 3-way alignments can be avoided by choosing a collection λ of the chosen triples. Then the weights for the classes arising from triples are all λ, to avoid the solution of fractional alignment. Using all 2-way alignments comprises one extreme, when v sequences (and λ = v - 2). Other choices of The determination of V be a set of elements and V; we associate V with the sequences, and every set in V, v, k, λ)-packing when |V| = v, every set in k, and every pair of elements in V appears in at most λ sets in maximal when no set can be added to maximum when it has the most sets of any packing with these parameters. See [Let ers. See ,21 for rλ = 1, i.e. -packings, trivially have the property that every pair appears either in zero or in λ = 1 triples of the packing, and hence an SP lower bound can be calculated directly. Since every triple in the packing contains three pairs and every pair is in at most one triple, no -packing can contain more than v(v - 1)/6 triples. Using such a packing for v-way alignment, we therefore employ only O(v2) 3-way alignments, rather than the O(v3) to employ all.Packings with v, 3, 1)-packing should we employ to obtain the bound? To a certain extent, this decision is based on the amount of computational effort that can be expended, and the accuracy of the bound desired. Of course, the best outcome would be to achieve the best accuracy at least computational cost. In certain cases, one might be able to anticipate from exact 2-way alignments when a specific 3-way alignment is likely to lead to an improvement. For example, given three sequences w1, w2, w3, if the 2-way alignments have very different patterns of indels and substitutions, one might expect that the 3-way alignment yields higher cost than the three 2-way alignments. This appears to be difficult to quantify.The natural question to address is: Which -packing of maximum total weight. This can prove more computationally intensive than simply undertaking all 3-way alignments, since choosing a maximum weight packing is NP-hard [In the absence of a good guide to the selection of 3-way alignments that yield the best improvements, two alternatives can be pursued. If accuracy is of paramount concern, we again calculate  NP-hard . HoweverIn the running example, since an -packing has at most 48 triples, one might hope to obtain a relatively large improvement. The best improvement that we found is 47. Then the hillclimbing algorithm to maximize weight was executed. It yields a collection of 33 triples: four triples with an improvement of 3 each, eleven with an improvement of 2, and thirteen with an improvement of 1. While calculating the average of all 816 alignments increases the lower bound by only 23, this method is comparable in execution time and provides more than twice the improvement in the lower bound.This demonstrates that, if one is willing to invest the effort to calculate all v, 3, 1)-packing at random, and evaluate the bound. We therefore employ the hillclimbing algorithm (developed in the Methods section) to calculate a packing of maximum size, not using any weight measuring expected improvement at this stage. Then we calculate the optimal 3-way alignment for each triple of the packing, to determine the improvement in the lower bound. Our first attempt gave an SP lower bound of 1968 (an improvement of 22), our second a lower bound of 1966 (an improvement of 20), and our third a lower bound of 1974 (an improvement of 28). There is significant variation in repetitions of the method, as a consequence of the random selections made. Naturally, one cannot conclude much from three trials, so we report the distribution of improvements from 10,000 trials of the method  (where the maximum packing has each pair in a triple), the averages would be in closer agreement. Moreover, using a metaheuristic search strategy such as simulated annealing could prove beneficial, again at the expense of a substantial increase in computation time.The astute reader can observe that in our 10,000 trials, the average is slightly less than 23, the average of all 816 improvements. This can be attributed to the fact that in a maximum packing on 18 elements, there are nine pairs not appearing in any triple; in a similar experiment with a number of sequences v-way alignments that are optimal with respect to sum-of-pairs cost is challenging, and substantial effort has been invested in the calculation of upper bounds on SP costs. Exact alignment appears to be computationally out of reach when many long sequences are to be aligned, and lower bounds from sums of exact 2-way alignments are often too weak to be of assistance. In this paper, we establish a generic lower bound using decompositions into fractional alignments. With this we propose two improvements using exact 3-way alignments. The first still calculates all 3-way alignments, but uses a hillclimbing technique to choose a packing among these of maximal weight. This improves on the accuracy of the bound at a similar computational cost to the naive method. The second instead reduces the computation time, by choosing at random a packing, and only then computing the 3-way alignments for the triples of the packing. In our test cases, the improvements are competitive with calculating the overall average, but are achieved at reduced effort in computation.The calculation of It remains of interest to determine whether, from exact 2-way alignment data, one can determine which triples are most likely to provide an improvement, prior to calculating the 3-way alignments; see  for a reDynamic programming methods for exact 2-way and 3-way alignment that we use are standard [v, 3, 1)-maximum packing. Stinson [v(v - 1)/6 triples ); see also [v. The basic strategy follows. Start with an empty set x, y} and {x, z}, so that neither appears in a triple in y, z} also does not appear in a triple of x, y, z} to y, z} does appear in a triple {w, y, z} in w, y, z} from x, y, z} leaves the number of triples in does change, and hence when this process is repeated, choosing a different set of two pairs may permit the addition of a new triple. The method therefore never chooses to reduce the number of triples selected, and relies on "lateral" moves to enable (ultimately) the addition of a new triple when that is possible. Stinson [The main requirement is to select a random -packing, the number of pairs not appearing in triples is 0 when v ≡ 1, 3 (mod 6); 4 when v ≡ 5 (mod 6); v ≡ 0,2 (mod 6); and v ≡ 4 (mod 6)  (see ). This uv is odd then   if v ≡ 1, 3 (mod 6) then      if ε := 0         ε : 8;      else triplesRemaining : = (v(v - 1) - ε)/6;      triples Remaining : = floor(v(v - 2)/6);   else D : ∅;   while triplesRemaining > 0 do   begin   x appearing < (v - 2)/2 times in D;      Pick a random element x, y}, {x, z} not appearing in triples in D;      Pick random pairs {if pair {y, z} appears in a triple T of D      then begin      D : = D -{T};            triplesRemaining : = triplesRemaining + 1            end;         D : D ∪ {{x, y, z}};   triplesRemaining : = triplesRemaining - 1   endx, y} and {x, z}, neither appearing in a chosen triple. If {x, z} appears in no triple as well, we can simply add the triple {x, y, z}. Otherwise we delete the unique triple containing {y, z} before making the addition. The method is a hillclimbing technique because it never reduces the number of triples chosen.Intuitively, the method repeatedly chooses two pairs with a common element, {The while loop in this method employs as its only stopping criterion that it finds a maximum packing. Stinson  incorpor-packings. As each is generated, the relevant 3-way alignments can be calculated to determine the lower bound obtained from this packing. There is a natural tradeoff between computation time and the quality of the bound. Only after a packing is chosen does one need to compute the 3-way alignments involved, giving a factor of O(v) savings in computation time; however, what has been sacrificed, potentially, is the amount of improvement since the method does not consider (or even know) the improvement until the packing is fully chosen. Of course, one can apply the method repeatedly increasing the computational expense, but with the expectation of improving the accuracy.Repeatedly employing this method yields different Stinson's method can also be adapted to choose a packing whose improvement is "large". To do this, we require that all 3-way alignments have been calculated. Then we modify the hillclimbing method in two ways. First, we only adjoin a new triple if it makes a nonzero improvement . Secondly, we only replace one triple by another if it makes an improvement at least as large as the one being removed. One might also consider replacing two or three triples by one, if this yields an improvement. We did not obtain any overall benefit by allowing this. This variant of the Stinson method essentially enables us to find packings of large weight rather than with many triples; it does, however, require that all weights (improvements) be precomputed.Both authors contributed in all phases of the research.


Non-coding DNA sequences comprise a very large proportion of the total genomic content of mammals, most other vertebrates, many invertebrates, and most plants. Unraveling the functional significance of non-coding DNA depends on how well we are able to align non-coding DNA sequences. However, the alignment of non-coding DNA sequences is more difficult than aligning protein-coding sequences.Here we present an improved pair-hidden-Markov-Model (pair HMM) based method for performing global pairwise alignment of non-coding DNA sequences. The method uses an explicit model of indel length frequency distribution which can be specified, and allows any time reversible model of nucleotide substitution. The method uses a deterministic global optimiser to find the alignment with the highest posterior probability. We test MCALIGN2 in simulations, and compare it to a previous Monte Carlo based method (MCALIGN), to the pair HMM method of Knudsen and Miyamoto, and to a heuristic method (AVID) that performed very well in a previous simulation study. We show that the pair HMM methods have excellent performance for all combinations of parameter values we have considered. MCALIGN2 is up to ten times faster than MCALIGN. MCALIGN2 is more accurate in resolving indels given an accurate explicit model than heuristic methods, but is computationally slower.MCALIGN2 produces better quality alignments by explicitly using biological knowledge about the indel length distribution and time reversible models of nucleotide substitution. As a result, it can outperform other available sequence alignment methods for the cases we have considered to align non-coding DNA sequences. The advent of automated DNA sequencing methods has resulted in an enormous growth in the volume of sequence data deposited in public databases. The increasing availability of genome sequence data for many related organisms offers great opportunities to study gene function and genome evolution, but it also presents new challenges for DNA sequence analysis, especially for non-coding DNA sequences.Drosophila melanogaster are selectively constrained [For much of the past two decades, research in DNA sequence analysis has focused on protein-coding sequences, which account for only a very small proportion of the total genomic content in mammals, most other vertebrates, many invertebrates, and most plants . For exastrained . In rodestrained , Evidencstrained . DetermiAccurate inferences about the function of non-coding DNA from comparative methods depends critically on correct alignments of non-coding sequences. However, the alignment of non-coding DNA sequences is more difficult than aligning protein-coding sequences. Protein-coding sequences tend to be highly evolutionarily conserved, so insertions and deletions (indels) are uncommon and rarely cross codon boundaries. However, indel events are common in non-coding DNA, and can occur at most nucleotide sites. Numerous advances in sequence alignment methods for noncoding DNA have been made. Many recently proposed methods are based on heuristic alignment algorithms that can be very fast and accurate in cases where sequences are similar, but perform less well when sequence divergence is high . FurtherTrue evolutionary models of sequence evolution allow both multiple point substitutions and multiple indel events to affect any site in the sequence. The first true evolutionary model of indel evolution was introduced by Thorne, Kishino, and Felsenstein , the TKFPreviously, Keightley and Johnson  proposedHere, we describe an improved non-coding sequence alignment algorithm based on a generalisation of the evolutionary model used by Keightley and Johnson . We showDrosophila non-coding DNA sequence data. It has been shown that AVID performs very well compared to other heuristic methods [We have compared our method to the pair HMM method of Knudsen and Miyamoto (PairHMM_KM hereafter), which is quite similar to the present method in that it explicitly makes use of an evolutionary time parameter . We have methods ,17, so hIn our tests, the new DP method (MCALIGN2) is up to ten times faster than the previous MC method (MCALIGN), and is also faster than the pairHMM_KM method , althougFor cases of real non-coding sequence data, we also compared MCALIGN2 with AVID and CLUSTALW , and shoWe use a Bayesian statistical framework ,21 to mat be the total divergence time between two sequences, a be an alignment of two sequences, and S be the observed data, which is two non-coding DNA sequences. In a Bayesian framework, the behaviours of all variables are modelled by probability distributions. Joint inference about a and t is accomplished simply via Bayes' theoremLet P(S) that appears in the denominator of equation (1) may be difficult to calculate, but because in Bayesian inference the observed data S is held fixed, the unconditional probability P(S) is constant. We can therefore make our inference using only relative probabilities and P(S) need not be calculated. The other unconditional probability that appears in equation (1) is P(t), which is specified as a prior; our method will work for any prior.The probability t as a nuisance parameter. The posterior probability for an alignment is therefore marginal to the divergence time t, and is calculated using the integralTo calculate the posterior probability of an alignment, we consider the divergence time P(a|S) = ∫ P dt.    (2)We approximate this integral using Laplace's method, described in detail below.P, which is the joint probability of alignment a and sequences S given a divergence time t. This probability is specified according to a model. Here, we use the pair hidden Markov model (HMM) shown in Figure t, the pair HMM shown in Figure M has emission probability distribution mi:nj, and state Ix and Iy have distributions mi and nj against a gap, in each of the two sequences (labelled x and y respectively).The most difficult probability to specify in equation 1) is  is P , the transition probability back to the M state is wi/Ix,i+1 or Iy,i+1 is 1 - wi/x will not be followed directly by a gap in sequence y, and therefore there are no transitions from any of the states Ix to any of the states Iy, or vice versa. Our approach could be extended to accommodate an indel length distribution that is any mixture of geometric distributions  by duplM, Ix,1 and Iy,1 to be the same as those from the M state. We allow all states to make transitions to the End state, with a low transition probability ε. If ε is small enough, we can ignore it in all of our calculations [In order to make our pair HMMs describe a probability distribution over all possible alignments, we need to include a Begin state and an End state. We set the transition probability from the Begin state to states ulations .The emission probabilities, which determine the sequences given the pattern of indels, are derived from the general time-reversible (GTR) model of nucleotide substitution . The Jukmi and nj, which are equal for sequences x and y. The emission probabilities o, drawn from the equilibrium distribution of nucleotide frequencies, and evolving independently down two lineages, to mi in time t1 along one lineage and to nj in time t2 along the other lineage.  in time t1+ t2). Since the times t1 and t2 are individually nonindentifiable, we parameterise our model simply by the total divergence time t = t1+ t2. For a given total divergence time, the conditional probability of evolving to mi given starting with nj is Q(t), can be calculated from the fixed instantaneous rate matrix A by matrix exponentiation [The emission probabilities ntiation , that isQ(t) = etA.     (3)A. Here we estimate the rate matrix A from the same external data that is used to estimate the parameters for indels, as described below.which can be calculated using the eigenvalues and eigenvectors of P has been specified by the model, and that a prior P(t) for the divergence time has also been specified, we have developed an algorithm to infer the approximate maximum a-posteriori (MAP) alignment P(a|S), which is given by the integral in equation (2). To approximate this integral, we assume that P, when treated as a function of t with both a and S held fixed, is approximately Gaussian. Then, using Laplace's method [Given that s method , we can t that maximises P . The quantityHere is the modal dispersion, which is the reciprocal of the curvature at the mode P(a|S) ≈ P*C(S)     (6)C(S) is a constant that depends on S but not on a or t. This approximation can be made when we wish to maximise P(a|S) over a set of a for which |Va| is approximately constant. The goodness of this approximation is discussed below.where P(a|S) if and only if P. Since by definition , we see that P. Our algorithm to find a, and then maximise P over all a. An equivalent solution is to find t, for each t  and then maximise P over all t. The second solution is much easier in practice, because t, can be found using a standard dynamic programming algorithm for pair HMMs [P can be maximised using any standard algorithm for maximising a one dimensional function.Given that our approximations hold, Equation (6) shows that air HMMs , and thet (with ties broken arbitrarily). We find a straightforward Golden Section Search [P. This assumes that there is a single global optimum to be found. Actually we are able to trap events where local optima are detected. However, no local optima has ever been detected. We terminate the search when the values of t bracketing the maximum differ by less than 0.001. Moreover, we are able to terminate earlier when the optimal alignment is the same at all points within the bracketing area.The dynamic programming algorithm guarantees to find the global maximising n Search  to be adD. simulans and D. sechellia  = . The estimated rates of each type of substitution are obtained by dividing the entries in each column by the respective column sums, yielding:A GTR model of nucleotide substitution was fitted to A that satisfies Equation (3) when time is measured in units of expected substitutions, to obtain our estimate of the instantaneous rate matrix:Finally, find the matrix in silico, so that sequences are generated that are related by a known, "correct" alignment [For non-coding sequences, there are few externally verified alignments available to test the performance of alignment methods. As a substitute, we simulate sequence divergence lignment . We testP(a|S) ≃ P, of the best alignment found by MCALIGN2 with the best alignments found by MCALIGN.We compared the dynamic programming approach used here against the Monte-Carlo approach proposed previously  and the Drosophila intronic data as described above. In these comparisons we investigated cases in which the model assumed by MCALIGN2 differed from the simulation model, by using the simpler JC and K2P models to analyse data simulated under a GTR model.We also compared MCALIGN2 against AVID of Bray et al.  in simulqi, and the rates aij, and finding ones that most closely predict the observed net transition matrix P [t and a certain evolutionary model, we observed the mean and variance of the estimator of t calculated from both the true alignment and the alignments estimated by sequence alignment methods we considered here. We express the precision of the estimator as the estimated root mean squared error (e.r.m.s.e.), since none of the estimators examined are perfectly unbiased. For t, this isIn all comparisons, we calculated the fraction of correctly aligned sites by counting the number of base pairs or bases-to-gaps which were correctly aligned in a comparison to the true alignment. As an alternative measure of alignment quality, we considered the precision of divergence time estimated from the alignments. The estimator of divergence time we used was distance under the GTR model. It is made by estimating the base frequencies matrix P . This esN simulations.when there are Although our program allows any prior for divergence time, for all comparisons we used a relatively diffuse or uninformative prior:t seems to be a reasonable one.which has the mean 0.75. Because low divergences are more likely than high ones for two homologous sequences, this prior on D. simulans and D. sechellia, and point substitutions occur according to the Jukes-Cantor model to compare the performances of PairHMM_KM, MCALIGN2 (DP hereafter) and MCALIGN (MC hereafter). In this setting, the DP and MC methods aim to find the same most probable alignment, since they assume essentially the same model and prior, but use different algorithms.We generated non-coding sequence data using a model of non-coding DNA evolution in which gap lengths are parametrized by intronic data of t), and the proportions of correctly aligned sites for combinations of θ and t. All alignment methods perform similarly when the true divergence time is not too great, t ≤ 0.2, and the indel rate is not too great, θ ≤ 0.3. For these parameters, the fraction of correctly aligned bases is greater than 90% and is similar for all the three methods. The mean estimated divergence time calculated from estimated alignments are close to the true values, and the e.r.m.s.e. are not substantially greater than if the true alignment is known. However, when the divergence time t became larger (t > 0.2) or the indel rate becomes larger (θ = 0.4), the performance of the MC method becomes noticeably inferior, since the mean proportion of correctly aligned bases is significantly lower than for the alignments estimated by the DP and PairHMM_KM method, and the divergence time estimates are more biased and have larger e.r.m.s.e. For the largest indel ratio we considered, θ = 0.4, the MC method tends to estimate an alignment with too many gaps and the estimates of t tend to be higher than the true values. Table Table t = 0.05); we presume that both methods are able to find the globally most probable alignment. However, when divergence time and/or rate of indel events becomes larger, the DP method begins to outperform the MC method, in the sense that the alignments produced by MCALIGN2 have higher probabilities. For the highest divergence time (t = 0.30) and/or rate of indel events (θ = 0.40) we considered, the DP method outperformed the MC method for almost all of the replicate simulations. This clearly indicates that the MC algorithm of Keightley and Johnson [For the same simulated datasets, Figure  Johnson  gets stut and θ, 200 replicate simulations were performed, each simulating a pair of sequences of length 500 base pairs, evolving under an indel model and a general time reversible (GTR) model of nucleotide substitution, parameterised using real Drosophila data. This model is very different to the simple Jukes-Cantor (JC) model, and quite different to Kimura's 2 parameter (K2P) model. In addition to comparing MCALIGN2 with AVID, it is interesting to explore the effect of the nucleotide substitution model assumed by MCALIGN2. We aligned each simulated pair of sequences using MCALIGN2 under the assumptions of the correct GTR model, a simple K2P model with the ratio of transition events to transversion events equal to 2, and the JC model.For each combination of values of The results in Table t that are more biased and have larger e.r.m.s.e.. In particular, alignments produced by AVID exhibit consistent upward bias estimates of t, and lower means proportions of correctly aligned bases than alignments produced by MCAL1GN2. This remains true, for most of the cases we considered here, whether MCALIGN2 used the correct GTR model of nucleotide substitution, or the incorrect JC or K2P models. The improvement in alignment quality gained by knowing the correct model of nucleotide substitution is generally modest, but worthwhile.Here, lower accuracy is indicated by a lower proportion of correctly aligned bases, and estimates of divergence time We also compared MCALIGN2 with AVID and CLUSTALW using real intronic DNA sequences from mouse and rat. Although we do not know the true alignments for real sequence data, we can still judge the alignment performances of different methods by examining the plausibility of the alignments . Here we show three specific cases in which MCALIGN2 performed quite differently from AVID and CLUSTALW.As shown in Figure Figure However, MCALIGN2 does not always produce less gaps than other methods. As shown in Figure ® Xeon™ processor. Results are shown for divergence time t = 0.2 and 0.3, and ratio of indels to point substitutions θ = 0.225. Figure L2 [t = 0.2) takes about 0.1s using AVID, about 80s using MCALIGN2, about 160s using PairHMM_KM and about 350s using MCALIGN. Furthermore, it is also shown in Figure Figure igure L2 . For theThe problem of statistical inference of an alignment can be separated into two parts: specifying a scoring function, and finding an alignment that optimises that scoring function. The scoring function is specified on biological and/or statistical grounds, and determines the biological meaningfulness and accuracy of the inferred alignment. The choice of optimising algorithm determines the speed of the method, and may hamper accuracy if convergence to a global optimum cannot be guaranteed. A useful alignment method must produce biologically meaningful and accurate alignments, and also must do so quickly. There is a trade-off because the most biologically realistic scoring functions are difficult to optimise.probabilistic interpretation [evolutionary interpretation. This inclusion of a time parameter is crucial in allowing us to train or parameterize our model using closely related sequences, in order to improve the accuracy of alignments between more distantly related sequences. Although the idea of training a scoring function on known alignments is an old one ), in thHeuristic scoring functions are often chosen because an algorithm exists to optimise them efficiently. However, without any underlying evolutionary model, the alignments produced by such methods will be biased (at least at some evolutionary distances), in the sense that they will exhibit features that depart in a systematic direction from the true alignment.4) in complexity).The evolutionary model used in our method strikes a balance between biological realism and computational tractability. We ignore multiple hits of indel events, and assume a distribution of indel lengths that corresponds to an improved affine gap penalty scheme. Our model is therefore quite different from more realistic evolutionary models that account properly for multiple hits of indel events ,9,11,12.When comparing the present method (MCALIGN2) against a previous Monte Carlo approach (MCALIGN ) we are (i) MCALIGN2 uses a dynamic programming algorithm that is guaranteed to find the most probable alignment for a given divergence time, whereas the stochastic hill-climbing algorithm used in the Monte Carlo method can only search locally by making heuristically chosen adjustments to an alignment.(ii) MCALIGN2 stops its search when the maximising divergence time is bracketed to high precision, with the bracket length being reduced by a geometric factor at each step of the algorithm. In contrast, the Monte Carlo method must search until no improvement in alignment probability is found during a predetermined number of iterations.In comparisons of MCALIGN2 against the pair HMM method of Knudsen and Miyamoto, a method with an evolutionary time parameter and an affine gap penalty , we founDespite being substantially faster than our original Monte Carlo approach and the pair HMM method of Knudsen and Miyamoto, MCALIGN2 cannot compete with AVID in terms of execution time, because of the clever heuristics used by AVID. Its general strategy for aligning two sequences is to select anchors using a variant of the Smith-Waterman algorithm  to splitIn order to examine the robustness of the MCALIGN2 method, we also investigated cases in which the model assumed in the MCALIGN2 analysis was a simpler model (JC or K2P) than the model the data were simulated under (GTR). Generally, the MCALIGN2 method assuming an incorrect model still has good performance for small and medium divergence times, but for larger divergence times and/or higher indel rates, performance suffers slightly compared with when the correct GTR model was assumed. Therefore, when aligning sequences from distant species, it is desirable to use an evolutionary model that is as realistic as possible. However, it is in precisely this situation that it may be most difficult to estimate a model, because the assumption of that the evolutionary process is the same between closely and distantly related species is most likely to break down.P is approximately normal with constant variance for a certain set of high probability alignments. Because P is a product of multinomial probabilities, the normality approximation will be good for long sequences under most models of molecular evolution. The assumption of constant variance will be reasonable when high probability alignments differ from each other by only a few indels and substitutions, relative to the total sequence length. As a concrete check of this assumption, we used the Monte Carlo search algorithm of Keightley and Johnson [P(a|S) computed "exactly" (using quadrature) and P(a|S) exceeded 0.98.When inferring alignment in a Bayesian framework, divergence time is a nuisance parameter that must be eliminated by integration (Equation 2). The computational implementation of our method relies totally on being able to approximate this integral (Equations 4 and 6) rather than having to calculate it numerically using e.g. quadrature. The approximations we make will be good when  Johnson  and retat, summing over all possible alignments, using the EM algorithm [t. Although our method has a more direct Bayesian justification, given the approximations made it is likely that the two approaches will give similar results.It is worth mentioning that, to our knowledge, no better method has been found for eliminating divergence time as a nuisance parameter when estimating alignment. Most authors concentrate on finding the true MLE for lgorithm -10,35. TSequence alignment is a major issue for the evolutionary analysis of non-coding DNA. We developed a model-based method, MCALIGN2, as an improvement to the previous Monte Carlo method MCALIGN. MCALIGN2 uses a deterministic global optimiser to find the alignment with the highest posterior probability. It allows a rich class of evolutionary models of indel length along with any time reversible model of nucleotide substitution. As shown in the test results, MCALIGN2 outperforms other available non-coding DNA sequence alignment methods for all the cases we have considered.Project name: MCALIGN2Project home page: Operating system: Platform independentProgramming language: C++Other requirements: C++ compiler if downloading and compiling the source codeLicence: FSF GENERAL public licence.JW developed and tested the software in this study, and wrote the manuscript. PDK participated in the design of the study and wrote the manuscript. TJ initially conceived of and co-designed the software and wrote the manuscript. All authors read and approved the final manuscript.


GASH, a new, publicly accessible program for structural alignment and superposition. Alignments are scored by the Number of Equivalent Residues (NER), a quantitative measure of structural similarity that can be applied to any structural alignment method. Multiple alignments are optimized by conjugate gradient maximization of the NER score within the genetic algorithm framework. Initial alignments are generated by the program Local ASH, and can be supplemented by alignments from any other program.We introduce We compare GASH to DaliLite, CE, and to our earlier program Global ASH on a difficult test set consisting of 3,102 structure pairs, as well as a smaller set derived from the Fischer-Eisenberg set. The extent of alignment crossover, as well as the completeness of the initial set of alignments are examined. The quality of the superpositions is evaluated both by NER and by the number of aligned residues under three different RMSD cutoffs . In addition to the numerical assessment, the alignments for several biologically related structural pairs are discussed in detail..Regardless of which criteria is used to judge the superposition accuracy, GASH achieves the best overall performance, followed by DaliLite, Global ASH, and CE. In terms of CPU usage, DaliLite CE and GASH perform similarly for query proteins under 500 residues, but for larger proteins DaliLite is faster than GASH or CE. Both an http interface and a simple object application protocol (SOAP) interface to the GASH program are available at  The coordinates of over 30,761 protein structures are currently available at the Protein Data Bank , and evaluated several popular structural alignment servers based on this score . By usinAnother result was that there were occasionally significant differences between servers. This was particularly true for proteins with repeating motifs (e.g. TIM barrels), multiple domains, or in cases where the structurally equivalent residues represent only a small subset of the total. More recently, Levitt and co-workers concluded that there was "wide variation" in alignment quality among different programs and that the performance of any single method was much lower than using the best result from several methods . Our ownLocal ASH, based on the double dynamic programming algorithm, was developed. In contrast to our earlier program, Global ASH [Since the NER score requires an initial alignment or superposition, a straightforward way to add robustness to the optimization algorithm is to increase the number of initial alignments. For this purpose a new alignment program, obal ASH ,9, that crossover between alignments, as is done in genetic algorithms (hence the "G" in GASH). Since both the number of initial alignments and the number of crossovers is an adjustable parameter, the GASH program can make a very good estimate of the true maximum of the NER score for an arbitrary pair of protein structures. One can even import alignments from other programs, and we give an example of combining Local ASH, DaliLite, and CE alignments in this study.In addition to accepting multiple initial alignments, the GASH program allows The test-set presented here has been significantly expanded compared to earlier work. In addition to the Fischer-Eisenberg set of structural pairs, our new set consists 3,101 pairs representing many different folds, as defined by SCOP [The overall approach is to globally optimize the NER score in three steps:1. Produce a set of locally optimal alignments.2. Parse each alignment into geometrically-consistent sub-alignments using distance matrix comparison.3. Cross the alignments a fixed number of times and select the best unique set by NER maximization.This procedure as well as our earlier protocol are illustrated in figure S:The NER score has been described in detail previously. In brief, NER is a sum over all aligned residue pairs of a similarity function k corresponds to an aligned residue pair, dk is the intermolecular distance between Cα atoms in the aligned pair, and the similarity score is a Gaussian curve with unit amplitude at zero distance:where dcut defines the tolerance in the similarity score. Since the best value of dcut depends on the problem at hand, we make it an adjustable parameter on our public server. In all calculations a value of 4Å was used.The parameter Local ASH is a local structural alignment program that utilizes the double dynamic programming algorithm (DDP) . In DDP,alignment radius.Both Global and Local ASH use DDP, but with some important modifications. The first modification is a distance cutoff used to define the structural environment. A sphere with a given radius is located at the beta carbon of each residue. The structural environment of a residue is expressed as an ordered set of the vectors from the beta carbon of the residue to those of the residues whose beta carbons are within the sphere. The modified environment is called the local environment. The similarity between a pair of residues is calculated as the alignment score between the corresponding local environments. The similarity obtained from the comparison of the local environments is used for the local structural alignment. In order to avoid confusion with the distance cut-off used in the NER score, discussed below, we will refer to the ASH distance cutoff as the Local ASH uses the Smith-Waterman algorithm for local structural alignment. When a pair of structures share multiple structural similarities that can not be expressed in a single alignment, the trace-back procedure is repeated in order to enumerate high-scoring solutions. After each trace-back operation, the scores corresponding to the alignment path and the neighboring region in the similarity matrix for the upper level DP are cleared. This ensures that each alignment will trace out a unique path. Orengo and Taylor introduced a window surrounding the local alignment path as the region to be cleared . In contFor results described here, the alignment radius we set to 14Å, and the maximum number of alignments output was 25. The internal gap opening and extension penalties for the lower level DP calculation were set to 10 and 0.5, respectively. For terminal regions the gap penalties were all zero. For the upper DP calculation, the gap opening and extension penalties were also set to 10 and 0.5, respectively. The Local ASH source code can be freely downloaded .Local ASH can not "see" beyond the alignment radius. If two or more regions of structural similarity exist and are separated by a distance greater than the alignment radius, an alignment may be constructed that runs through both regions. If these multiple regions do not correspond to a superposition with the same rotation and translation values, there will be more than one maximum in the landscape of the NER scoring function for the alignment. The distance matrix comparison step is used to decouple such geometrically-distinct sub-alignments.A necessary condition for an alignment to be geometrically consistent is for all of the intramolecular distances between aligned residues in one structure to be approximately the same length as the corresponding distances in the other structure. Such a comparison of distance matrices forms the basis of the Dali target function . Here weiq,it) initiates a sub-alignment . Then we attempt to add a second aligned pair  to this sub-alignment. If we find that any of the intramolecular distances in the query  differ by more than a cutoff value from the corresponding distance in the template, , the residue pair  is rejected from the sub-alignment; otherwise, it is accepted. If the pair is rejected from all existing sub-alignments, a new sub-alignment is initialized. Each subsequent pair of residues is compared to each existing sub-alignment in this manner until all residue pairs in the original alignment have been accounted for.The parsing is iterative: the first aligned pair of residues  using an ordinary dynamic programming calculation . This reGenetic algorithms have been used to generate alignments  or super1. A stack to hold hybrid alignments is initialized with a fixed maximum length.2. Two alignments as well as a splice-point in one of them are chosen at random.3. The splice point in the second alignment is chosen so that the C-terminal portion is as long as possible without containing any of the residues in the N-terminal portion of the first alignment.4. The alignments are crossed with a single progeny: the N terminal portion of alignment 1 and the C-terminal portion of alignment 2.5. The CA RMSD of the new alignment is minimized, and the NER score calculated from the RMSD-minimized superposition. If the stack-size has not reached the maximum value, the new alignment is saved; otherwise, if this new NER score is greater than the lowest saved NER score in the stack, the lowest saved alignment is replaced by the new alignment. (This means we don't have to sort the stack but just keep track of the worst saved solution.)Ntry).6. Return to step 1 a fixed number of times  to higher-scoring ones. On the GASH server, the RMSD threshold, as well as an NER threshold for accepted solutions, can be adjusted in order to modify the number of retained alternate solutions.The DaliLite program was obtained from the FSSP  server . The DalThe CE program was obtained from the CE server . The NERIn order to assess the necessity and sufficiency of the crossover step, we report results using four versions of GASH: Default, No Cross-over, High Cross-over, and Meta. These variants are summarized below.Nstart, Ntry, and Ntop are 2,100, and 20, respectively. Initial alignments are generated by Local ASH.The default settings for Nstart = 0).In order to assess whether the extent of crossover specified by the default parameters is necessary, GASH was run with no crossing over (Nstart = 200).In order to assess whether the extent of crossover specified by the default parameters is sufficient, GASH was run with extensive crossing-over , resulting in a set of 3,102 structure pairs that were used for the present analysis. The set of queries, along with their fold classifications is given in Table The Fischer-Eisenberg set was taken from table VI in Shindyalov and Bourne .The results were generated by running each program on the command line on one of 14 identical personal computers . In order to make a fair comparison, the same re-alignment procedure was used to evaluate the NER scores and the numbers of aligned residues for all alignment methods. The numbers of aligned residues were computed for three RMSD cut-offs: 2,4, and 6Å, which we will refer to as N2, N4, and N6, respectively. Occasionally one or more of the programs failed to produce a result, or only aligned a few residues. Since we could not distinguish between software errors  and true algorithmic deficiencies, we eliminated any template from the list if any one of the methods failed to produce an NER score greater than 10. Based on the higher number of failures for DaliLite, CE and Global ASH  compared to the default, no crossover, and high crossover GASH  it is unlikely that eliminating templates in this way biased the results in favor of GASH.Averages of all similarity measures as well as the number of internal gaps were computed for each query set as well as for the entire set of results see . In addiThe raw data resulting from each of the 3,102 structure pairs using 7 different methods is too large to be presented in tabular form here. For this reason, we have prepared a link to the data on our server . The maiAlthough the main focus of this work is on optimization, we first consider whether the NER score is a valid target function for structural alignment by comparing it to a more familiar metric: the number of aligned residues under a given RMSD.In figure The fact that NER is a direct sum makes it much easier to optimize than the number of aligned residues under a given RMSD cutoff.  This fact coupled with the nearly linear agreement between the two measures validates the utility of NER as a target function for structural similarity.First we consider the improvement of GASH compared to Global ASH from the summaries given in Additional files Next we consider the performance compared to DaliLite and CE. GASH consistently aligned an equal or greater number of residues than either DaliLite or CE, independent of the measure used to define accuracy. The improvement relative to CE is particularly dramatic, with an average of 7–19 more residues aligned (depending on the measure used). With DaliLite, the improvement  was not as dramatic, but it was consistent across most query sets. In terms of CPU usage, CE was the slowest for proteins in the 100–273 residue range, and DaliLite was several seconds faster than GASH in this size range. In the 339–424 residue range, CPU times differ between the four programs by only a few seconds, on average. The greatest difference in CPU usage is seen for the largest structures: DaliLite uses only 25 and 28 seconds, for 1bgw (680 residues) and 1bxrA (1074 residues), but both GASH and CE run for longer times.It must be emphasized that these results represent use of the DaliLite and CE programs as is, without any changes in the source code for this difficult set of alignment problems. In the hands of the authors, CE or DaliLite might well yield higher NER scores and/or lower CPU times.We address the question of coverage in the initial set of alignments by comparing the default GASH performance with Meta GASH, where alignments from DaliLite and CE were added to the initial set. Since GASH uses only the crossover operation from the genetic algorithm, not mutation, the only new information it can generate is in the re-alignment step. DaliLite and CE both generate alignments by completely different algorithms from each other and from Local ASH, so if there is not enough information in the initial set of Local GASH alignments, we should see an improvement in the accuracy when alignments from DaliLite and CE are added.In terms of the NER score, there is a slight average improvement of 2 aligned residues upon using Meta GASH. This improvement can be observed consistently across most query sets, showing that we are not always locating the exact global optimum in the NER score when using default GASH. However, these differences are not great enough to justify using Meta GASH routinely, since the CPU usage would be approximately 3 times that of default GASH.In order to determine if the crossover is both necessary and sufficient, we compare default GASH to GASH without crossover and to GASH with high crossover. The no-crossover results are not as good as those of default GASH, but the improvement is not very dramatic, on average. In fact, no-crossover GASH is slightly better on average than DaliLite, and at a competitive CPU usage. However, if we consider particular cases, such as the alignment between 1gqeA and 1p32A figure , the difHere we examine the quality of 5 pair-wise alignments in detail. In addition to numerical measures, such as the number of equivalences, we consider functional information, where available. The examples chosen represent cases where GASH outperformed Global ASH in terms of the NER score, and include all-α, all-β, and mixed α/β folds. In all examples, "GASH" refers to the default GASH method. All sequence identities quoted are obtained by running the entire sequence for both query and template at the University of Southampton SBDS server , which uMyoglobin and phycocyanin belong to the same SCOP fold group but have different functions. Alignment of the two sequences yields an identity of 15%. Myoglobin utilizes a heme group to transport oxygen and phycocyanin binds a phycocyanobilin chromophore for light harvesting. The heme group bound by myoglobin and the chromophore bound by phycocyanin are positioned similarly . AlthougThe C-terminal domain of 1bxrA and the entire structure of 1egh both belong to the methylglyoxal synthase fold ,33-35. HAlanine Racimace and imidazole glycerol phosphate synthase share a TIM barrel domain with the active site located at the top of the barrel ,37-39. TBoth 1kyqB and 1jvnA contain a NAD(p)-binding Rossmann domain ,40, but Both 1bwwA and 1igyA are immunoglobulins, and contain the characteristic β-sandwich fold. 1bwwA is a single-domain structure, but 1igyB is an intact monoclonal antibody and contains both a variable domain and three constant domains. Thus, tIn The primary goal of this study was to design an algorithm that reliably maximizes the NER score for an arbitrary pair of protein structures. The results here indicate that the GASH program is successful in this regard, and that the extent of sampling can easily be increased, if necessary, by adding more initial alignments. Although we have not yet optimized every parameter used in GASH, the results in terms of NER and other scores using default parameters is encouraging. From looking at the dependence on crossover and on the initial alignment set, we can surmise that most of the improvement relative to Global ASH is due to the use of multiple Local ASH alignments.The Dali algorithm has recently been validated extensively against CATH classifications using receiver operating characteristics in two studies ,42, howeEven with a test set of 3,102 structure pairs, the improvement of GASH relative to Global ASH, both in terms of accuracy and CPU usage, is unambiguous. Moreover, as the specific examples illustrate, a higher NER score correlates well with "correctness" in terms of matching important residue pairs. The one case where GASH misaligns a residue pair found by CE immediately suggests an obvious improvement to the program: using side-chain atoms in addition to Cα atoms to define the equivalence. We intend to incorporate this improvement, as well as to look at a more comprehensive set of structure pairs in the near future..In addition to web access through both a CGI and java interface, we have developed a Simple Object Access Protocol (SOAP) server that allows GASH to be run remotely on the command-line. Sample java and Perl client programs are available for accessing the SOAP server. The java programs require installation of the Apache Axis library  and the Summary of SCOP-FSSP set results. For each query, the number of residues, number of structural templates (Ntmp), and results from each of the 7 alignment methods, are shown. For each alignment method, the NER4 score (NER), number of gaps in the core alignment (Gp), and number of aligned residues below three RMSD cutoffs  are reported. Each entry in the table represents an average over the number of templates specified in column 3. The averages on the last row refer to the total set of 3,102 structures. In the case of the number of templates, the sum, rather than the average, is given.Click here for fileSummary of Fischer-Eisenberg set results. For each structure pair, the PDB ID and number of residues, as well as results from each of the 7 alignment methods, are shown. For each alignment method, the NER4 score (NER), number of gaps in the core alignment (Gp), and number of aligned residues below three RMSD cutoffs  are reported. In the last row, the averages over all ten structure pairs are given.Click here for file


Protein sequence alignments have become indispensable for virtually any evolutionary, structural or functional study involving proteins. Modern sequence search and comparison methods combined with rapidly increasing sequence data often can reliably match even distantly related proteins that share little sequence similarity. However, even highly significant matches generally may have incorrectly aligned regions. Therefore when exact residue correspondence is used to transfer biological information from one aligned sequence to another, it is critical to know which alignment regions are reliable and which may contain alignment errors..PSI-BLAST-ISS is a standalone Unix-based tool designed to delineate reliable regions of sequence alignments as well as to suggest potential variants in unreliable regions. The region-specific reliability is assessed by producing multiple sequence alignments in different sequence contexts followed by the analysis of the consistency of alignment variants. The PSI-BLAST-ISS output enables the user to simultaneously analyze alignment reliability between query and multiple homologous sequences. In addition, PSI-BLAST-ISS can be used to detect distantly related homologous proteins. The software is freely available at: PSI-BLAST-ISS is an effective reliability assessment tool that can be useful in applications such as comparative modelling or analysis of individual sequence regions. It favorably compares with the existing similar software both in the performance and functional features. Protein sequence alignments are at the heart of many biological applications such as sequence database searches, annotation of new sequences, inference of functional regions, comparative protein modeling. Modern sequence comparison methods (e.g. PSI-BLAST ) often cThe importance of delineating reliable alignment regions has been recognized more than a decade ago, however, earlier studies focused on pairwise alignments -5. CurreHere, we present the PSI-BLAST Intermediate Sequence Search tool (PSI-BLAST-ISS) that is primarily designed to help identify reliable regions of the alignment as well as suggest potential alignment variants in unreliable regions. In comparative modeling PSI-BLAST-ISS can also help identify best matching structural templates. In addition, PSI-BLAST-ISS can be used to detect remote homologs that cannot be identified by a straightforward single PSI-BLAST search. However, it should be noted that the detection of remote homologs, unlike in the original and subsequent implementations of the Intermediate Sequence Search (ISS) strategy -17, is nSince PSI-BLAST-ISS might be most useful in comparative modeling we are going to refer to the sequence pair of interest as the target (query) and the template (reference) sequences throughout the article. However, it should be emphasized that the tool can be applied for any protein sequences that could be linked through common homologs, independently whether the three-dimensional structure for any of them is available or not.The main idea of PSI-BLAST-ISS is to obtain a number of alignment variants for the sequence pair of interest (target and template) and analyze their consistency. This idea has stemmed from previous manual analysis of multiple PSI-BLAST alignment variants suggesting that regions where variants do agree are likely to be aligned correctly and display close structural similarity .The whole PSI-BLAST-ISS procedure may be described as the following steps: 1) identification of multiple sequences related both to the target and template sequences, (2) formation of a representative set from these sequences by filtering out close homologs, (3) generation of sequence profiles for each sequence from this representative set by searching a sequence database with PSI-BLAST, (4) using each of the generated profiles to search a second sequence database that includes sequences of both the target and the template, (5) retention of all the instances of significant matches between the target and the template, (6) merging all significant target-template alignments by taking the target sequence as a frame of reference and (7) reducing the multiple variants of aligned template into the consensus sequence. The latter option enables contrasting of the region-specific reliability for multiple target-template alignments simultaneously. All the seven main steps are illustrated in a sketch of the data flow . By default, the target is searched against the non-redundant sequence database. Intermediate sequences are collected from the user-specified PSI-BLAST iteration in the resulting output file using the expectation value  threshold provided as a parameter. The reduced representative sequence set is constructed by filtering the initial set to a user-defined percentage of sequence similarity with CD-HIT , the sequence clusterization program (step 2). Optionally, a user may introduce a strict limit to the number of sequences to be included in the representative set or even supply independently pre-selected set of sequences. A PSI-BLAST-ISS user can also choose whether to collect intermediate sequences as complete protein sequences or just as sequence fragments matching the target sequence. In the case when the target sequence represents a domain that is also found in multidomain proteins the ability to select only homologous fragments of matching sequences may help to keep the ISS procedure from straying into the realm of unrelated sequences. Each of the intermediate sequences is used to generate a sequence profile in the form of the PSI-BLAST checkpoint file by running a user-defined number of PSI-BLAST iterations (step 3). The resulting checkpoint files are then used to restart PSI-BLAST searches in a second sequence database specified by the user (step 4). This database is expected to include sequences of both proteins of interest (target and template). In a common situation, when the template represents a structural template intended for use in comparative modeling, such a database may be derived by simply appending the target sequence to the PDB sequence database. In this case there is no need to define template(s) in advance since they are identified automatically. Searches against the second database generate corresponding multiple sequence alignments that contain a number of target-template alignment variants. The significance of the target-template alignment is then determined by counting the number of alignment variants that satisfy the expectation value threshold (step 5). Both parameters can be specified by the user. The significant target-template alignment variants are extracted and merged into a single multiple sequence alignment, where the target sequence is aligned with multiple instances of the template sequence according to different alignment variants (step 6). Such an alignment immediately reveals the regions where most  alignment variants are identical and thus might be considered reliable as well as those regions where there is little agreement between alignment variants and therefore unreliable. Often it is useful to analyze position-specific reliability for target alignments with multiple templates. However, it may be inconvenient to contrast/compare at once many multiple sequence alignments obtained by PSI-BLAST-ISS. To make this task easier we introduced a step (step 7) that reduces template alignment variants into a consensus template sequence for each of the target-template alignments. The consensus sequence is generated by analyzing each column of the alignment. A residue is considered conserved in the consensus template sequence if its repetition count in the corresponding position exceeds the user-defined conservation threshold.PSI-BLAST-ISS currently is implemented as a standalone UNIX-based tool meant to be installed and run locally. It consists of fairly independent modules linked together using Perl. Some of the sequence data processing tasks in PSI-BLAST-ISS are handled by a few modified SEALS scripts .PSI-BLAST-ISS produces several types of results. Perhaps the most informative output file is the FASTA-formatted sequence alignment between the target and automatically detected multiple template sequences, each represented as a consensus sequence derived from multiple alignment variants. The definition line for each consensus template sequence indicates the strength of the consensus in the interval from 0 to 1  and the number of significant target-template alignment variants that were used to produce the consensus. This output provides a possibility to simultaneously assess the alignment reliability between the target and multiple templates in a region-specific manner. In addition, the consensus strength and the number of significant target-template alignments may help in selecting templates that are structurally most consistent with the target. PSI-BLAST-ISS also produces individual FASTA-formatted multiple sequence alignment files for each target-template pair, where the target is aligned with multiple copies of the same template according to obtained multiple alignment variants. These alignment files provide a visual assessment of the region-specific alignment reliability as well as candidate alignment variants if further analysis of unreliably aligned stretches is needed. Finally, all the template sequences represented in the consensus alignment are collected together in a separate output file.Like for any method it is important to know how PSI-BLAST-ISS performs relative to other available methods. At the time of this study we have been aware of only two publicly available servers that estimate the position-specific reliability of sequence alignment using information from multiple sequences: the Consensus server  and SQUA. These proteins represent a variety of different structural folds and different degree of similarity to known structures. We ran PSI-BLAST-ISS for all the target sequences assessed in CASP6, but only those, for which PSI-BLAST-ISS with default parameters generated at least ten significant alignment variants with a structural template, were further analysed. The "gold standard" in evaluating sequence alignments is to compare them with the alignments derived from protein structure superposition. For most targets PSI-BLAST-ISS detected multiple templates but for evaluating its performance we only considered a single template for each target. The DaliLite structure comparison program [To compare PSI-BLAST-ISS and Consensus we chose protein sequences provided as prediction targets in the last round  of the c program  was usedThe data in Table A useful feature of PSI-BLAST-ISS is that it provides multiple alignment variants between the target and each template. Results in Table Often there is a need to choose a single or just a few best templates from a large number of distantly related target homologs. This becomes a challenge at low sequence similarity when the sequence signal is no longer a good indicator of structural relatedness . These targets do have evolutionary related structures in the PDB database but these relationships could not be detected by PSI-BLAST searches initiated with the target sequence. For this test we required at least one significant match to a PDB structure (template) from all intermediate sequence searches. To make the comparison compatible with the CASP6 setting we only considered structural templates that were available from PDB at the time of the CASP6 experiment. We also excluded from consideration those FR/H CASP6 targets, for which at least one domain could be matched to a PDB structure using a straightforward PSI-BLAST search. As a result, out of fourteen considered FR/H targets, PSI-BLAST-ISS was able to identify related structures for four of them . An interesting case is T0228. While direct PSI-BLAST search initiated with the T0228 sequence failed to find any related structure, PSI-BLAST-ISS identified several structures producing over ten significant matches each (a default parameter). The latter result stresses the fact that sometimes the space of homologous sequences might be skewed in such a manner that a single sequence search may not be very effective in identifying important relationships.We have described PSI-BLAST-ISS, a tool for delineating reliable alignment regions and suggesting possible alignment choices in unreliable yet structurally conserved regions. PSI-BLAST-ISS might be most useful in assessing target-template alignments in comparative modeling or judging whether the interpolation of biological information directly form alignments is feasible for individual sequence regions. Unlike two other recently published methods for predicting reliable alignment regions (SQUARE and the Consensus server) PSI-BLAST-ISS is not confined to reference (template) sequences with known three-dimensional structure. The performance of PSI-BLAST-ISS in alignment reliability estimation was directly compared with the Consensus server. We find that on a set of CASP6 targets PSI-BLAST-ISS on average is able to produce more extensive coverage of confident alignment or fewer errors, depending on the selected consensus stringency. The functionality of PSI-BLAST-ISS also extends into detection of non-apparent distant homologous relationships.Project name: The PSI-BLAST intermediate sequence search tool (PSI-BLAST-ISS)Project home page: Operating systems: Unix-based platformsProgramming language: PerlOther requirements: locally installed PSI-BLAST and CD-HIT License: NoneAny restriction to use by non-academics: NoneMM carried out the software development, programming work and participated in manuscript preparation. ČV conceived of the study, participated in its design and coordination and drafted the manuscript. All authors read and approved the final manuscript.


With the continued development of new computational tools for multiple sequence alignment, it is necessary today to develop benchmarks that aid the selection of the most effective tools. Simulation-based benchmarks have been proposed to meet this necessity, especially for non-coding sequences. However, it is not clear if such benchmarks truly represent real sequence data from any given group of species, in terms of the difficulty of alignment tasks.Drosophila group of species, and show that these data sets truly represent the variability observed in genomic data in terms of the difficulty of the alignment task. This allows us to make significant progress towards estimating the alignment accuracy of current tools in an absolute sense, going beyond only a relative assessment of different tools. We evaluate six widely used multiple alignment tools in the context of Drosophila non-coding sequences, and find the accuracy to be significantly different from previously reported values. Interestingly, the performance of most tools degrades more rapidly when there are more insertions than deletions in the data set, suggesting an asymmetric handling of insertions and deletions, even though none of the evaluated tools explicitly distinguishes these two types of events. We also examine the accuracy of two existing tools for annotating insertions versus deletions, and find their performance to be close to optimal in Drosophila non-coding sequences if provided with the true alignments.We find that the conventional simulation approach, which relies on empirically estimated values for various parameters such as substitution rate or insertion/deletion rates, is unable to generate synthetic sequences reflecting the broad genomic variation in conservation levels. We tackle this problem with a new method for simulating non-coding sequence evolution, by relying on genome-wide distributions of evolutionary parameters rather than their averages. We then generate synthetic data sets to mimic orthologous sequences from the Drosophila non-coding sequences, and shown it to be more realistic than traditional benchmarks. Apart from helping to select the most effective tools, these benchmarks will help practitioners of comparative genomics deal with the effects of alignment errors, by providing accurate estimates of the extent of these errors.We have developed a method to generate benchmarks for multiple alignments of  Drosophila species [The availability of genome sequences of closely related species  has pro species -6), offe species -12 in li species , and tha species .Given the availability of multiple tools to perform either of these two tasks, a researcher faces two important questions: "Which of the tools should I use for my task?" and "How accurate will the tool be on my data?" Answers to these come from studies that use data sets ("benchmarks") where the true answers are known, to evaluate and compare different tools. The design of benchmarks therefore directly affects the reliability of bioinformatics analyses that use those tools. The two most widely used benchmarking approaches for alignment tools are (i) to make use of biological sequences and their manually curated alignments from databases such as Homstrad , BAliBAS1) We present a new simulation-based benchmarking method that is based on the entire spectrum of values of its parameters as inferred from real data. This is in contrast to existing approaches that rely on the average observed values of the parameters.2) We quantify the difficulty of aligning a data set by leveraging recent developments  on estimDrosophila non-coding sequences. The specific alignment task we consider is that of global alignment of ~1-10 Kbp long sequences, and our conclusions may not apply to the task of local alignment, which was studied in [Drosophila genomes, and find these to be very different from previously reported values. We also evaluate two schemes for annotating insertions and deletions specifically, and find their accuracy to be comparable, and close to optimal.3) Using our new benchmarks, we evaluate and compare the accuracy of six multiple alignment tools  We find that data sets with an excess of deletions over insertions are more amenable to accurate alignment than those with an excess of insertions, suggesting an implicit bias  with respect to their treatment of indels, even though none of the evaluated tools explicitly makes a distinction between insertions and deletions.expected frequency of those events in the generated data sets. Variation in the frequency of these events, which underlie the difficulty of alignment tasks, results from the inherent randomness of the simulation process, i.e., the differences in random choices made from one "run" of the process to another. It is natural to ask if the resulting variability across data sets in a synthetic benchmark is comparable to the corresponding variability observed in real orthologous sequences. The question is particularly relevant due to the heterogeneity of non-coding sequences with respect to the density of functional elements and also motivated by the known variation in evolutionary rates across loci [Modeling of DNA sequence evolution has been studied extensively in the past, and state-of-the-art simulation programs -20 draw oss loci -36.Drosophila non-coding sequences (see Methods). A key difference in our implementation was that branch lengths  were estimated from non-coding sequences themselves, instead of synonymous substitution rates from coding sequences, as has been done previously. We elaborate on this important issue later in this section.We began by implementing the above-mentioned simulation paradigm, which we call the "traditional" paradigm, by incorporating the "constraint blocks" idea of Pollard et al.  into theDrosophila sequences from eight species (see Methods), computed the sum of branch lengths of the phylogenetic tree estimated from ~1 Kbp segments of alignment, and found the distribution of this statistic to have a large variance across the genome " alignment quality score as the agreement between two alignments, one generated from original sequences and the other from their reversed versions. Hall [Since substitution rates are generally correlated with indel rates, a large variance in the former implies a corresponding variance in indel frequencies, which of course lie at the root of the alignment problem. This suggests that if we could measure the "difficulty of alignment" in any region of the genome , we ought to see a large variability in this measure across the genome. Moreover, if the observed distribution of the alignment difficulty measure is comparable to that in a benchmark, we would be confident in making claims about performance of alignment tools based on that benchmark. The problem is that measuring alignment difficulty on real data requires knowledge of their true alignment, which is unavailable. Recent work by Landan and Graur  showed tns. Hall  later shns. Hall , but othK = 10 phylogenies {ϕ1, ϕ2, ..., ϕK} that are scaled versions of the original phylogeny ϕ0, i.e., every branch length in phylogeny ϕi is a constant factor τi times the corresponding branch length in ϕ0.  We modified the simulator to first sample at random one of the K phylogenies, and simulate according to this setting of branch lengths, with all other parameters being fixed as before. In other words, the distribution of alignment quality scores from the new simulation process is a mixture distribution, with components parameterized by different phylogenies and the probability of sampling any particular phylogeny being the mixture weight. We estimated an upper bound on the agreement between this mixture distribution and the observed distribution of alignment quality scores, by maximum likelihood training of mixture weights, through expectation-maximization algorithm [We hypothesized that the above observation about synthetic data sets was due to the use of a single setting of the branch lengths, and the relatively low variability resulting from the randomness of the process itself Figure . If thislgorithm . This "bThe above results, while encouraging in terms of better reproducing the genomic variability of alignment difficulty, were obtained by fitting parameters of the simulation process so as to best match real data. We next asked if we could achieve the same or better agreement between the synthetic and real data distributions without having seen the real distribution of alignment quality scores. This would then allow us to use the observed agreement as a relatively unbiased assessment of how realistic the benchmark is. Developing the mixture model idea from the previous section, we now computed for each parameter the entire distribution of values observed in real data alignments, just as the traditional approach estimates the average of these values. The simulation process was now made to sample each parameter independently from its empirical distribution, and then generate a data set based on the sampled parameter values. The benchmark thus constructed (comprising 10000 different data sets) was examined for its distribution of alignment quality scores, and as seen in Figure The above analysis was performed using the sum-of-pairs score (SPS), which is the simplest of the scores defined in the HoT approach . We repeK = 3, ..., 8. For each choice of K, 10000 sets of sequences corresponding to K different Drosophila species were simulated and the above alignment tools were run with default parameters or with the best setting recommended by their authors. We then compared the resulting alignments to the "true" alignments reported by the simulation program, using the following three commonly used evaluation measures [alignment agreement, which is the fraction of aligned base pairs  in the predicted alignment that agree with the true alignment, (ii) alignment sensitivity, which is the fraction of aligned base pairs of the true alignment that agree with the predicted alignment, and (iii) alignment specificity, which is the fraction of aligned base pairs of the predicted alignment that agree with the true alignment. Whereas the alignment agreement score considers aligned base pairs as well as bases aligned to gaps, the sensitivity and specificity scores are calculated only from aligned base pairs. The results of our evaluations are shown in Figure K. Its performance degrades more slowly (with increasing K) than the other tools, as a result of which the gap between Pecan and the other tools became larger more species were included in the tests. The average alignment agreement in five species alignments produced by Pecan (the species most divergent from D. melanogaster being D. pseudoobscura) was close to 80%, but degraded to ~67% when aligning all eight species.We used our new benchmark to evaluate and compare six leading multiple alignment tools that are publicly available and can align DNA sequences. These are ClustalW 2.0.5 , Dialignmeasures ,41: (i) K = 8, ClustalW showed an alignment agreement of 36% or 46% depending on whether there was an excess of insertions or deletions (respectively). The same trend was seen in terms of the alignment sensitivity and specificity measures. Noticably, Pecan was largely unaffected by this dichotomy of data sets.  that had an excess of insertions over deletions, and separately to those data sets with an excess of deletions , limiting ourselves to the eight-species data sets. All trends reported above were also seen in this alternative view of the results  also for alignment specificity, and for other species pairs and alignment programs as well  is added. Reasoning that phylogenies for which computed alignments are largely inaccurate would not be suitable for insertion/deletion annotation in any case, we chose to limit our assessment to the following five Drosophila species: D. melanogaster, D. simulans, D. yakuba, D. ananassae, and D. pseudoobscura  Indel Ratio Agreement, which is the agreement of the ratio of the number of insertions to the total number of indels between the two annotations, and (iii) Indel Annotation Coverage, which is the fraction of indel positions on which the two annotations agree (see Methods). Traditional alignment programs mark the predicted locations of insertions and deletions as "gaps", and do not proceed to annotate these gaps as being insertions or deletions. This latter task has received some attention recently with at least two "indel annotation schemes" being published, based on maximum-parsimony ("sbInfer" ) and proD. pseudoobscura, the species with the longest terminal branch in the phylogeny. Indel Ratio Agreement scores of both tools were close to optimal (1) in all five species. While the sensitivity scores of Indel Annotation Coverage of the two tools were above 90% across all five species, the specificity scores were above 90% only for the four species except D. pseudoobscura. The loss of accuracy on the D. pseudoobscura branch is presumably due to the fact that there is no "outgroup" species to aid disambiguation of insertions and deletions on this branch. We further discuss the implications of these observations in the next section. We also repeated our assessment for sequences with an excess of insertions or of deletions, as above, but no significant differences was observed between these two categories (data not shown).As summarized in Table Drosophila species. This new simulation method exploits the spectrum of values of evolutionary statistics  seen across a genome. We take advantage of an objective "alignment quality" measure to show that the synthetic sequences produced agree with real sequences not only in terms of evolutionary statistics, but are also as easy or hard to align as real data sets. In this sense, our evaluation results are more likely to reflect the actual accuracy values of alignment-related tools on data from Drosophila species. We note that our strategy of sampling parameters (used in evolutionary simulations) from their empirical distribution has parallels with traditional Bayesian inference where one integrates over  a prior distribution on parameters, rather than using a single point estimate.Choosing the most suitable tool for aligning orthologous sequences is essential to studies in comparative genomics and in molecular evolution, making it critical to develop accurate benchmarking methodology. In this study, we propose a novel simulation-based approach to generate realistic data sets mimicking orthologous non-coding sequences from multiple A key step in our benchmark construction was the ability to assess the quality of an alignment without access to the corresponding true alignment. This ability has been the result of several recent publications by other authors. Prakash and Tompa ,44 develcis-regulatory architecture, vis-a-vis the density and evolution of binding sites. However, it is not yet clear which, if any, of these different assumed models of regulatory sequence evolution is most suited to represent the variability in constraint patterns across different regions of the genome. Our simplistic "conserved block" model  seems tD. melanogaster - D. pseudoobscura) is widely different from the value observed in real non-coding sequence alignments (~0.4 substitutions/site). One could argue that this gap may be offset if we set an appropriate frequency of conserved positions (with very low rates), resulting in an average substitution rate that is close to the empirically observed value. However, this turned out not be the case for any realistic setting of the frequency of conserved positions (data not shown). We therefore chose to be guided by existing estimates of the frequency and length distribution of conserved blocks, with substitution rates that are some constant β (see Methods) times the "neutral" rate outside of the blocks, and set this neutral rate so that the average rate for the entire sequence matches observed values. Our choice reflects the philosophy that simulated data sets ought to match real data in terms of various evolutionary statistics and net alignment difficulty, and the discordance of the used neutral rate from synonymous substitution rates is ignored for the sake of practicality.Some clarification is in order with respect to our manner of choosing substitution rates for the simulation process, since it marks a significant departure from traditional thinking. The latter, as embodied in the work of Pollard et al. , prescriTo our knowledge, no previous benchmarking study has evaluated the effect of insertions and deletions on the performance of alignment tools. Some studies -25 have Drosophila species. While the accuracy was close to optimal on four of the five terminal branches, we observed that both tools over-estimate insertions as well as deletions on the longest branch (leading to D. pseudoobscura), while accurately predicting the ratio of insertions to deletions. We note that the D. pseudoobscura branch in the phylogenetic tree originates from the root of the tree, and we would expect to have better annotation results for this branch if an appropriate outgroup species was used. For studies that intend to use insertion to deletion ratio profiling to identify loci with unusual evolutionary patterns  it may s  than previously reported, and points to a clear asymmetry in the handling of insertions versus deletions by most alignment tools.We have presented a novel method for generating benchmarks of non-coding sequence alignments, that relies on a spectrum of parameter values reflecting the genome-wide variation of those parameters. We have shown our benchmarks to accurately match the difficulty of aligning real data, by taking advantage of recent developments in measurement of alignment quality. Benchmark evaluations on Drosophila genome sequences (release 5) with 14 insects were downloaded from UCSC Genome Browser Database [D. melanogaster, D. simulans, D. yakuba, D. ananassae, D. pseudoobscura, D. willistoni, D. mojavensis, and D. grimshawi. We extracted fragments of the genome-wide multiple alignments that have sequences for all eight species, whose minimum length is 1 Kbp, and which have less than 50% of their length masked . The extracted alignments were used to estimate simulation parameter values, as described below. The distribution of HoT alignment quality scores [Whole-genome multiple alignments of Database . In casey scores  was compy scores .Drosophila species were estimated from the multiple alignments described above, using Paml [D. melanogaster genome.Median branch lengths of a phylogenetic tree for eight ing Paml . This phing Paml , with thing Paml , substiting Paml  and inseing Paml , with paing Paml . The lening Paml . The lening Paml  and the to be the overall evolutionary rate (the estimated branch length), tn be the unconstrained evolutionary rate , α be the fraction of sequence length that falls into conserved blocks, and β be the ratio of the evolutionary rate of conserved blocks to that outside blocks. Then we have:The estimated median branch lengths mentioned above reflect an average of the rates in conserved and non-conserved regions of real non-coding sequences, whereas the phylogeny input to Dawg by definition represents the substitution rate outside of blocks. Therefore, the branch lengths of the phylogeny were adjusted based on the specified coverage of conserved blocks and their evolutionary rates. Let Drosophila non-coding sequences and marked consecutive columns as a conserved block if the following two conditions hold: (i) they span at least 10 consecutive non-gapped columns and (ii) Phastcons scores of all columns are greater than or equal to 0.9  and the root sequence was sampled from the D. melanogaster non-coding genome  to link their simulations to the pair of species. They later updated this value in a new phylogeny http://www.danielpollard.com/trees.html. We used their divergence estimates from the latter phylogeny and the benchmark they prescribed for this level of divergence, and evaluated the alignment programs ourselves on this benchmark.The benchmark generated by Pollard et al.  parameteNIt and NDt are true numbers of insertions and deletions, and NIe and NDe are predicted numbers of insertions and deletions.Indel Count Agreement is defined by the following formula, where Indel Ratio Agreement is defined by the following formula, with notation as above:Indel Annotation Coverage is the fraction of indel positions on which the two annotations agree.The time complexity of the Indelign program is exponential in the number of "conditionally dependent blocks" and this prohibits fast annotation of certain data sets with relatively large numbers of species . To reduhttp://europa.cs.uiuc.edu/RealisticAlignmentBenchmarks/.Source code for the modified Dawg and Indelign programs, phylogenetic trees, simulated sequences and their alignments, and computed alignments by six alignment tools are available from Indel: insertion and deletion; SPS: sum-of-pair score; CS: column scoreJK and SS conceived of the study, participated in its design, performed the analysis, and drafted the manuscript. JK developed the software and performed experiments. Both authors read and approved the final manuscript.Performance of multiple alignment tools compared by alignment sensitivity. The scores were calculated by using all synthetic data sets (left panel), and by using only data sets where the expected number of insertions is two times more than the number of deletions or vice versa (middle and right panels respectively).Click here for filePerformance of multiple alignment tools compared by alignment specificity. The scores were calculated by using all synthetic data sets (left panel), and by using only data sets where the expected number of insertions is two times more than the number of deletions or vice versa (middle and right panels respectively).Click here for fileAn example data set from the benchmark shown (in part) with true alignment (top panel) and alignments computed by each different programs.Click here for fileDependence of performance (sensitivity (left) and specificity (right)) of each alignment program on various descriptive statistics of the data sets.Click here for filePerformance of multiple alignment tools compared by alignment sensitivity of pairs of species. The scores were calculated by using all synthetic data sets (left panel), and by using only data sets where the expected number of insertions is two times more than the number of deletions or vice versa (middle and right panels respectively).Click here for filePerformance of multiple alignment tools compared by alignment specificity of pairs of species. The scores were calculated by using all synthetic data sets (left panel), and by using only data sets where the expected number of insertions is two times more than the number of deletions or vice versa (middle and right panels respectively).Click here for fileComparison of estimated alignment sensitivity and specificity, using Mlagan or Pecan, as obtained from the Pollard et al.  benchmarClick here for fileComparison of estimated alignment sensitivity and specificity as obtained from the Pollard et al. benchmark.Click here for filePhylogenetic trees and branch lengths in Newick format.Click here for fileDrosophila non-coding sequences obtained from UCSC Genome Browser Database.Genome-wide distribution of the fraction of conserved blocks estimated by using Phastcons conservation scores and multiple alignments of Click here for fileDescriptive statistics of traditional and new benchmarks.Click here for file